Sep 01, 2019 MTTC: the Cybersecurity Success Metric that Matters Most
We spend a lot of time at Nemertes nailing down the metrics that define “success” for a particular initiative.
To be useful, a success metric must be:
- Quantitative. If you can’t measure it in numbers, it doesn’t serve to measure progress. And if it’s not something you can get better at, why are you assessing it in the first place?
- Objective. Although many times subjective success metrics (“How well do you think you’re doing at…”) correlate closely with objective ones, true credibility (particularly in the C suite) comes only from objective metrics.
- Aligned with business goals. Speaking of the C suite, any “success” metric should align with business goals; no matter how important otherwise, a metric that doesn’t conform to business goals is of limited relevance.
When it comes to cybersecurity, there’s an embarrassment of riches. Frameworks like NIST propose multiple cybersecurity metrics (along with the helpful guidance that “Organizations should be thoughtful, creative, and careful about the ways in which they employ measurements.” Not exactly actionable!) And there are multiple frameworks to choose from.
Unfortunately, more metrics aren’t necessarily better; having multiple metrics can often defeat the purpose of using the metric as an organizing principle against which all efforts should be measured. That is, if you’re going to spend time or money on something, there should be a clear link to a positive impact on the critical metric.
After considerable thought, and work with clients, we arrived at one that we believe captures the essence of what cybersecurity efforts: Mean total time to contain (MTTC).
We define MTTC as the time it takes to detect, understand, and contain a cybersecurity incident, averaged across all incidents. By “detect”, we mean “discover that an event has occurred that might, potentially, represent a security incident”. By “understand” we mean “determine that an event is in fact an incident”. And by “contain”, we mean, “limit the damage caused by the incident and prevent the attacker from causing further harm.” (Specifically, we don’t mean “remediate”, in the sense of a full remediation, which often requires re-evaluating the cybersecurity policy.) And the mean TTC is, of course, the time to conduct all three steps, averaged across all incidents.
It’s clear that the definition fulfills the first two goals (being objective and quantitative). With respect to alignment with business goals, our thinking goes like this: Every enterprise is, or will be, the target of a cybersecurity attack. Measuring the number of attacks is therefore merely a measure of the attractiveness of the enterprise as a target. And measuring the percentage of attacks that are foiled is meaningless if the .001% of attacks that get through manage to devastate the company.
Clearly, an effective cybersecurity organization should be focusing on quickly and effectively detecting and recovering from attacks: hence the emphasis on MTTC.
MTTC also helps focus the efforts of cybersecurity professionals on implementing the tools and processes that matter. Analytics, AI, and machine learning make it possible to detect events and understand that they represent real security incidents (rather than anomalous, but harmless, behavior by systems or users.) And automation shortens the time to contain after an incident has been detected. Tools that incorporate analytics, AI, and automation are in fact among the most effective in enabling cybersecurity.
In our 2019-2020 Cloud and Cybersecurity Research Study, we uncovered several fascinating facts. First is that median MTTC has dropped dramatically across all organizations since our last study (2017-2018). The top 2% of cybersecurity organizations now have a median MTTC of 2 minutes, versus 8 minutes in the previous study.
Second is that within the spectrum of tools, technologies, and strategic vendors that are intended to improve cybersecurity, there’s a range of effectiveness. Truly effective initiatives (whether tools or vendor partnerships) correlate with improvements in median MTTC by 100% or more. But there are initiatives that correlate with a significant increase (worsening) in median MTTC. And finally, as you might expect, there’s variation by size and type of cybersecurity organizations. Smaller, less sophisticated organizations may benefit from some practices (such as outsourcing the SOC) while larger, more sophisticated organizations find that these practices correlate with worse median MTTCs.
We looked at a range of initiatives, from deep network segmentation to zero trust security to microservices authentication. We addressed technologies from advanced endpoint security (AES) to behavioral threat analytics (BTA). And finally, we looked at organizational structures, including cybersecurity roles and responsibilities as well as funding and architectures. To get an overview of the results, check out our Webinar on the topic, or contact us.