Peer-Benchmarked Threat Resilience Metrics

It would be nice to have a clear answer for senior management and regulators about threat resiliency. That is, how well the organization is defending against the techniques that threat actors are using. While we’re at it, a single quantitative score to describe threat resilience would be fantastic. And could we get that benchmarked against our peers?
Introducing the “FS Index”

The Financial Services Threat Simulation Index (the “FS Index”) is a free, shared test plan for measuring threat resiliency and trending over time. The Index is facilitated by long-time FS-ISAC sponsor Security Risk Advisors, but it is developed by a wide group of intel, red, and fusion center leads from retail banks, asset managers, fintechs and insurers. We call it the Index because it is a prioritized cross section of MITRE ATT&CK. The same way that the S&P and the Dow are indexes representing prioritized market performance, the FS Index represents the most important threat actors for the financial service industry, and their attack techniques. 

• 2023 is the third year of the Index – it’s updated once per year to account for threat changes.
  
• The current test plan is 60 test cases mapped to 7 nation state threat actor groups most commonly targeting the financial services industry.
  
• The Index results are represented in a Threat Resilience Metric, a single percentage value that is also represented by a dynamic MITRE ATT&CK heatmap.

Since it is a shared test plan, all Index consumers consistently use the same test procedures to arrive at their Threat Resilience Metric – and that is how it is benchmarkable. Organizations that adopt it can trend their progress over time vs the financial services industry average benchmark value that SRA publishes quarterly. 

How to Use the Index 

Organizations can use the Index test plan by inserting it into their purple team process. You can use a spreadsheet, but it’s not recommended. Instead, Security Risk Advisors publishes the free VECTR.io platform, a purple teams management platform already used by many FS-ISAC members. VECTR is built for documenting ATT&CK-based test cases, outcomes and metrics. VECTR creates powerful visual insights into FS Index test results, including trending Threat Resilience Metrics, ATT&CK heatmaps which change color over time with maturity, and more detailed reporting about the performance of EDR, SIEM and other tools. These outputs provide structured feedback to intel, hunt, red, content and engineering teams.

Index-Driven Defense

Some CISO offices have adopted an Index-driven Defense. This means that one of the core objectives of the adversary management program is to keep pace with the Index year over year as a key performance indicator of threat resilience. These organizations have developed dashboards comparing their business units or geographies’ threat resilience metrics. When they evaluate platform replacements, candidate tools are subject to demonstrating test case outcomes improvement. The Index is not ALL that matters, but it’s a cornerstone and compass in those programs.

Index Challenges

Two concepts of the Index require getting used to:

1. It is a difficult self-exam and scoring above 80% is rare. The latest Q1 2023 benchmark is 63%.
2. It changes annually (not wholesale but there can be substantial changed content as threats evolve).  

The Index is a difficult exam because adversaries are sophisticated. It has to change annually because adversaries evolve their techniques. The Threat Resilience Metric is a threat-driven metric unlike any previous public industry effort. Adopters of the Index need to learn its place in their metrics and program maturity assessments at the onset.  Because we have established has good coverage across the sector and early adopters, we feel it is an industry leading benchmark. The Index is not intended to take the place of NIST CSF, FFIEC CAT, ISO27k, or any of the static frameworks that we’ve traditionally used to assess program maturity. Financial institutions are used to being assessed relatively high on a CMMI-like scale across NIST CSF functions and they can still do that. Adopting the Index complements those program-wide frameworks by validating maturity ratings through evidence-based testing rather than relying only on interviews and documentation.     

What to Expect When You’re Indexing

The FS Index isn’t a cake walk and most organizations score below the benchmark their first time. An initial Threat Resilience Metric of 45%-55% is not unusual. That’s the bad news. The good news is that a low initial score drives a focused action plan for improvement that can gain CISO visibility and support for significant improvement.  The Index, especially when paired with VECTR’s tracking and visualizations, helps the organization run down its weaknesses. For organizations who would be sensitive to discuss an initial low score, Security Risk Advisors recommends doing two Index exercises before widely sharing the results, because a compelling improvement story will already be on display and help get support for further adoption.

Another caution for the Index: be brutally honest. It is best used in an open book, broad stakeholder setting. If a fusion center or blue team runs it alone without the red team, or vice-versa, there is a risk that a partially successful outcome will be reported as successful, and a false sense of security. For example, a credential access test on a server that is “blocked” but not “alerted”. This should not be considered successful – not yet – until a high-fidelity alert is also sent to SIEM/SOAR.  The attacker would otherwise just move on to another technique and remain undetected. The Index should be used in a collaborative workshop settings with red, blue, intel, hunt and GRC at the table.

Flipping the Burden of Perfection  

Fusion Centers and SOCs can be crushed by the feeling they need to be perfect to thwart advanced adversaries. The Index is a carefully curated set of 60 test cases mapped to top threat actors. An organization that practices those 60 and succeeds at more than half is well on their way to flip the burden of perfection to the adversary. The adversary can no longer make mistakes and remain undetected. An organization using the Index has done its homework and practiced for the real test. It has many reliable traps and has increases the likelihood of, and time to detect. With the Threat Resilience Metric, the CISO office has a clear story to tell.  

By adopting the Index and Threat Resilience Metrics, the CISO office can learn and elevate a clear story with the following concepts to the Board. Early adopters already have.