(Third in the series regarding vulnerability)
Apologies in advance, for the length of this post…
In a perfect world…
… we’d know which specific threat agent was going to act against us and know the capability of that threat agent in absolute terms (e.g., pounds per square inch), as well as know (through testing) what our resistance capabilities are in those same absolute terms. If we had this information AND assuming this information was precisely correct all of the time, vulnerability becomes a clear and simple binary consideration — we will be or we won’t be.
Stating the obvious (anyway)
Losses occur when threat events take place that we’re vulnerable to. This is true whether we’re talking about weather events, human error, or malicious acts. Obviously, we don’t experience loss with every threat event, which means we’re only vulnerable sometimes — i.e., less than 100% of the time. This means there is some probability associated with whether we’ll be vulnerable to any given threat event. The process of measuring vulnerability is intended to help us understand what that probability is likely to be.
Simplest approach
Perhaps the simplest approach is to identify the threat community you’re analyzing risk against and simply estimate your ability to resist the capabilities of that threat community. For example, we might estimate that our web application is capable of resisting all but the top 2% of the cyber-criminal threat community — i.e., two out of a hundred hackers have the skill and resources to defeat the application’s security.
This works as a quick-and-dirty solution, and in many cases is good enough. Read on if you’re interested in a somewhat more involved approach.
Uncertainty
Unfortunately, in the real world we usually don’t know:
- Which threat agent is going to act next,
- What their capabilities are, or
- What our resistance capability is going to be
Making matters even more challenging:
- We don’t have an absolute measurement scale for some threat categories (e.g., human capability)
- Our measurements are imprecise (e.g., we can’t measure force or resistance perfectly)
- One or more of the values being measured may vary over time (e.g., hurricane wind speed varies throughout the lifetime of the storm, and strength can change throughout the lifetime of a control )
- One or more of the values being measured may vary across a population (e.g., not all hurricanes have the same wind speed)
When absolute scales apply
(Warning: This is an illustration and not an engineering exercise, for those who might want to argue details.)
Some types of threat categories can be measured using absolute scales (e.g., wind speed in miles per hour), which makes things a bit more straightforward. For example, thru testing we could estimate that a structure should be capable of resisting wind forces between 150 and 200 MPH.

By using a distribution to describe this measurement, we account for the fact that under some circumstances wind speeds of less than 150 MPH might compromise the structure, while in some circumstances the structure may be able to withstand speeds greater than 200 MPH.
If we wanted to measure the structure’s vulnerability to a specific type of storm (e.g., a tornado) we could plot a similar distribution for tornado wind speeds (black curve below). This distribution reflects the fact that wind speeds vary from tornado to tornado, ranging from under 100 MPH to over 300 MPH, with most falling in the 200 MPH range. (Keep in mind this is just an illustration and isn’t intended to reflect actual tornado data.)

In order to determine the probability of being vulnerable, we’d use a Monte Carlo function to:
- Take a random value from the tornado distribution and from the structural resistance distribution
- Compare the values — i.e., for this iteration, determine whether wind speed was greater than resistance
- If wind speed was greater, increment a counter that tracks the number of vulnerable instances
- Repeat a thousand iterations (or ten thousand, a million, etc.),
- After completing all of the iterations, the vulnerability counter divided by the number of iterations provides the probability of this structure being vulnerable to tornado winds
When an absolute scale doesn’t exist (the human threat community)
Human threat capability can be boiled down to skills and resources. Because skills and resources vary from individual to individual, we can characterize threat community capability as a distribution. At one end of the distribution are those threat agents who have the least capability, while at the other end are those who are the most capable. As seems to be the case for most things in nature (e.g., weather events), the distribution is probably pretty close to being bell-shaped (i.e., the majority of threat agents fall somewhere below those who are most capable and above those who are least capable).

A “100% secure” control (if such a thing existed) could be illustrated as existing outside of the threat community capability distribution. It would be 0% vulnerable.

More realistically, we can in most cases expect that some portion of the threat population would have the skill and resources to compromise a control (shown below).

Now, because of the uncertainties regarding threat capabilities and control strength, it would be more accurate to describe control strength as a distribution as well. For example, we expect the control is at least resistant to 90% of the general threat population, and may be resistant to as much as 99%+ of the population.

This is fine as far as it goes, but it doesn’t get us the answer we’re looking for in most circumstances. Most of the time it isn’t enough to know our vulnerability to the general threat population. In most analyses, we want to know what our vulnerability is to a particular threat community (e.g., cyber criminals, nation-state intel units, etc.). In that case, we’d have to plot the capability of the threat community in question (red distribution).

With that plotted, we can run our Monte Carlo function again, generating a probable vulnerability by taking random samples from the control distribution and the distribution of the specific threat community in question.
The key to measuring vulnerability in the absence of an absolute scale is to use the general threat population capability as the comparative baseline for both control strength and the capability of the threat community in question.
Considerations
Of course, because some malicious threat communities tend to share knowledge and tools, there can be an equalizing effect, which potentially narrows the width of the threat capability curve (shown below) but likely wouldn’t change its fundamental bell-shape. The good news is that this narrowing effect wouldn’t alter how we measure. The bad news is that it does affect vulnerability, which we know intuitively anyway.

Another consideration is the fact that the capability of the malicious population evolves over time — i.e., the curve shifts to the right along the continuum. For example, at one time in the past DES was considered invulnerable to brute force cracking. It isn’t any longer. In other words, we could say that the control stayed in place along the continuum, but the capability curve shifted to the right. This highlights the fact that it’s important to keep abreast of how threat capability evolves, so that you can evolve your defenses as well. Also, this is good fodder for the importance of defense-in-depth.
Concerns
An obvious concern is the inexact nature of these estimates and the potential for the analyst to estimate badly for various reasons. We’ve covered this issue previously in other postings, so I won’t go into it in depth now. Suffice it to say that yes, this is an inexact measurement fraught with all of the goblins that any measurement approach is subject to. That said, keep in mind a few things:
- The ability to estimate effectively can be significantly improved using calibration techniques
- There’s no such thing as a perfectly exact measurement, whether you’re using a laser or the width of your thumb to do the measuring. Therefore, the purpose of measurement is to reduce uncertainty, not eliminate it
- You can apply confidence levels to your estimates, both to describe the probability of actual values being outside of the estimated minimum and maximum, and to shape the peakedness/flatness of the curve
- Monte Carlo analysis is designed to help account for the uncertainty in measures
- You should never convey to management that these numbers are exact. In my experience management won’t have any problem with this, as the numbers they’re given from other business disciplines have precision challenges of their own.
Bottom line — If you’re trying to quantify risk, then you have to quantify vulnerability. This is one logical means of doing so. What’s more, it seems to accurately reflect how we subconsciously evaluate and quantify vulnerability anyway, only it brings the analysis to the surface. And by bringing it to the surface, it allows us to better understand and analyze risk scenarios.
If there’s interest, I can provide a couple of examples in a future post. Also, if there’s interest, I can include an example where the threat event is due to error rather than malicious intent.