Metrics - The Puzzle

Security metrics is something I'm having a growing interest in, for several reasons. First, its important to be able to justify the return on investment. If you can't demonstrate value of security, why should anyone implement it? Second, being a security professional I would like to know if what I'm doing is having an effect. Is there anything I can be doing different to improve my approach? Continuing my slug fest of attempting to read (and understand) NIST's SP800 series documentation, I was quite surprised to see alot of value in the SP800-80 "Guide for Developing Performance Metrics for Information Security". As with most of the SP800 papers, you have to fight your way through the government speak and academic noise. I really think you can easily cut many of the SP800 papers by at least 50%. For example, in SP800-80 you have to go through 20 pages before you get to the good stuff. However, there are two key things I REALLY liked about NIST's metrics paper.

1. First, they break metrics down into 3 different categories. This is something I never thought of, there are big differences between areas that can be measured. These are identified on page 15 as follows:

• IMPLEMENTATION: This is the one that I see most commonly used, but I feel has the least value. It tracks how much as been done. For example, how many accounts have a unique identity tied to it, how many desktop computers have been configured to a security standard, or how many end users have received security awareness training.
  
• EFFECTIVENESS: Now this is where the rubber meets the road and things get harder. How do you demonstrate the effectiveness of your security? This is what interests me the most also, what can I do to improve?
  
• IMPACT: For a NIST document, I was impressed to see this metric definition. It is the attempt to measure the impact to your business, your ROI. As discussed on various other blogs and publications, this is easily the hardest thing to measure. In fact, it is often argued if security can even be considered ROI, or is it simply loss prevention.
  

2. The second thing I liked were the examples in the appendix. If SP800-80 only broke down metrics into three categories, I would not have been overly impressed. Where the magic happens is they have examples of these metrics in the 18 different control domains identified in SP800-55 (IS0 27001/AnnexA has 11 such control domains). In addition, what I found very helpful was each security domain not only has metrics, but the overall strategic goal and information security goal of the controls.

All in all, I found this to be one of the most helpful SP800 documents I have read so far for ISMS approach. Even though its FISMA specific (as all SP800 documents are) it can apply to other models as well. However, if you are short on time I suggest skipping all the noise and going straight to the appendix on page 22 where the examples are. That is the true value of this document.