Security Testing: At What Level?

Now that we are on a subject of testing security and breach/attack simulation tools, one more interesting questions arises: if you test security, what constitute a “pass”? Or, alternatively, at what level do you test?

Think back the infamous bear analogy. In security, it is NOT a certainty that it is enough to outrun the slowest competitor [who the bear can then eat…chomp-chomp], you may actually have to outrun the bear [a Russian APT joke is here somewhere I am sure].

So, there are good arguments that you need to test at the level of your likely adversary. But how do you know that? Even if your threat assessment is not shitty, this is likely achievable by a tiny minority only… people who basically know what “threat assessment” (some actually call it “threat modelling” despite the appsec connotations) even means…

Here is what I am thinking:

Conceptual test levelProsCons
Test at maximum possible levelYou can test whether your security posture will withstand the best of the bestGap may be so huge, that no clear action plan is evident
Test at the level of your current securityCan test the freshly deployed controls; can actually ace the testNot immediately motivating for improvements
Test at the level of your current security PLUS 10%Easy to figure what to improveHow do you actually define and measure it?
Test at the level of your past adversaryHas solid data, that can be turned into test methodsNot a good prediction of the future
Test at the level of your likely adversaryPhilosophically, this is the best scenario!Hard to gather solid data on this, and figure it out

Now, hilarity ensues if “level of your likely adversary” is MUCH higher than “level of your current security” :-)

Ideas? Impressions?

Blog posts related to this project:

This is a Security Bloggers Network syndicated blog post authored by Anton Chuvakin. Read the original post at: Anton Chuvakin