Security Metrics: Replacing Fear, Uncertainty, and Doubt and over one million other books are available for Amazon Kindle. Learn more

Vous voulez voir cette page en français ? Cliquez ici.


or
Sign in to turn on 1-Click ordering.
or
Amazon Prime Free Trial required. Sign up when you check out. Learn More
More Buying Choices
Have one to sell? Sell yours here
Start reading Security Metrics: Replacing Fear, Uncertainty, and Doubt on your Kindle in under a minute.

Don't have a Kindle? Get your Kindle here, or download a FREE Kindle Reading App.

Security Metrics: Replacing Fear, Uncertainty, and Doubt [Paperback]

Andrew Jaquith
4.0 out of 5 stars  See all reviews (2 customer reviews)
List Price: CDN$ 67.99
Price: CDN$ 42.83 & FREE Shipping. Details
You Save: CDN$ 25.16 (37%)
o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o
Only 1 left in stock (more on the way).
Ships from and sold by Amazon.ca. Gift-wrap available.
Want it delivered Tuesday, August 5? Choose One-Day Shipping at checkout.

Formats

Amazon Price New from Used from
Kindle Edition CDN $29.32  
Paperback CDN $42.83  
Join Amazon Student in Canada


Book Description

March 26 2007 9780321349989 978-0321349989 1
<>The Definitive Guide to Quantifying, Classifying, and Measuring Enterprise IT Security Operations

 

Security Metrics is the first comprehensive best-practice guide to defining, creating, and utilizing security metrics in the enterprise.

 

Using sample charts, graphics, case studies, and war stories, Yankee Group Security Expert Andrew Jaquith demonstrates exactly how to establish effective metrics based on your organization’s unique requirements. You’ll discover how to quantify hard-to-measure security activities, compile and analyze all relevant data, identify strengths and weaknesses, set cost-effective priorities for improvement, and craft compelling messages for senior management.

 

Security Metrics successfully bridges management’s quantitative viewpoint with the nuts-and-bolts approach typically taken by security professionals. It brings together expert solutions drawn from Jaquith’s extensive consulting work in the software, aerospace, and financial services industries, including new metrics presented nowhere else. You’ll learn how to:

 

• Replace nonstop crisis response with a systematic approach to security improvement

• Understand the differences between “good” and “bad” metrics

• Measure coverage and control, vulnerability management, password quality, patch latency, benchmark scoring, and business-adjusted risk

• Quantify the effectiveness of security acquisition, implementation, and other program activities

• Organize, aggregate, and analyze your data to bring out key insights

• Use visualization to understand and communicate security issues more clearly

• Capture valuable data from firewalls and antivirus logs, third-party auditor reports, and other resources

• Implement balanced scorecards that present compact, holistic views of organizational security effectiveness

 

Whether you’re an engineer or consultant responsible for security and reporting to management–or an executive who needs better information for decision-making–Security Metrics is the resource you have been searching for.

 

Andrew Jaquith, program manager for Yankee Group’s Security Solutions and Services Decision Service, advises enterprise clients on prioritizing and managing security resources. He also helps security vendors develop product, service, and go-to-market strategies for reaching enterprise customers. He co-founded @stake, Inc., a security consulting pioneer acquired by Symantec Corporation in 2004. His application security and metrics research has been featured in CIO, CSO, InformationWeek, IEEE Security and Privacy, and The Economist.

 

Foreword         

Preface            

Acknowledgments         

About the Author           

Chapter 1          Introduction: Escaping the Hamster Wheel of Pain          

Chapter 2          Defining Security Metrics           

Chapter 3          Diagnosing Problems and Measuring Technical Security  

Chapter 4          Measuring Program Effectiveness           

Chapter 5          Analysis Techniques     

Chapter 6          Visualization     

Chapter 7          Automating Metrics Calculations

Chapter 8          Designing Security Scorecards  

Index   

 

 


Customers Who Bought This Item Also Bought


Product Details


Product Description

About the Author

Andrew Jaquith is the program manager for Yankee Group’s Enabling Technologies Enterprise group, with expertise in compliance, security, and risk management. Jaquith advises enterprise clients on how to manage security resources in their environments. He also helps security vendors develop strategies for reaching enterprise customers. Jaquith’s research focuses on topics such as security management, risk management, and packaged and custom web-based applications.

 

Jaquith has 15 years of IT experience. Before joining Yankee Group, he cofounded and served as program director at @stake, Inc., a security consulting pioneer, which Symantec Corporation acquired in 2004. Before @stake, Jaquith held project manager and business analyst positions at Cambridge Technology Partners and FedEx Corporation.

 

His application security and metrics research has been featured in CIO, CSO, InformationWeek, IEEE Security and Privacy, and The Economist. In addition, Jaquith contributes to several security-related open-source projects.

 

Jaquith holds a B.A. degree in economics and political science from Yale University.

 

Excerpt. © Reprinted by permission. All rights reserved.

Preface

Preface

What This Book Is About

This book is about security metrics: how to quantify, classify, and measure information security operations in modern enterprise environments.

How This Book Came to Be

Every consultant worth his or her weight in receipts accumulates a small trove of metaphors, analogies, and witty expressions. These help explain or clarify those rarified things that consultants do and tend to lubricate the consulting process. Oh, and they also tend to be funny. One of my favorite bits—particularly relevant to the topic at hand—is this one:

No good deed goes unpunished.

This simply means that with any worthwhile endeavor comes many unwitting (and often unwanted) consequences. So it is with the world of "security metrics." As you will see in the story I am about to tell you, my steadfast belief that security metrics ought to be a very! serious! field of study! has brought with it its own punishment.

Several years ago, several colleagues and I undertook a series of elaborate empirical studies on the subject of application security. We rigorously gathered and cleansed far-flung source material, aggregated and analyzed the resulting data, built an exotic mathematical model, and wrote a short research paper on the subject, complete with eye-catching charts and graphs. It was well received by customers and media alike. Some time later I was asked to present a condensed version of our findings on an Internet webcast run by an industry trade publication. In this case "webcast" meant a PowerPoint presentation accompanied by previously taped narration. The audience, as pitched to me by the sponsor, was to include "CSOs, technologists, and decision-makers."

That sounded great; I relished the opportunity to impress the bejeezus out of the vast numbers of grand globetrotters promised by the publication. In addition, my Inner Academic had high hopes that many in the audience would send me e-mails and letters marveling at the analytical techniques we used, the breadth of the data, and the many keen insights contained in the narrative and text. How wrong I was. Instead of measured praise from academe, I received several e-mails that went something like this:

"Great presentation, but I was hoping to see more 'return on investment' numbers. You see, I really need to convince my boss to help me buy widget ______ (fill in the blank)."

And then there were the slightly more disturbing comments, like this one:

"We have no money for our security program! Oh, woe is me! What I really need is more ROI! Help me!"

I confess to embroidering the truth a tiny bit here; the second e-mail I received was not nearly so plaintive. But the theme was clear: viewers assumed that because the webcast was about "security metrics," it must be about ROI. Our marvelous metrics were the good deed; their unfulfilled expectations were the punishment.

Goals of This Book

Mercifully, the "security ROI" fad has gone the way of the Macarena. But to be absolutely sure that your expectations are managed (more consultantspeak for you), here is what this book is about, and what it is not about.

The primary objective of this book is to quantitatively analyze digital security activities. The chapters suggest ways of using numbers to illuminate an organization's security activities:

  • Measuring security: Putting numbers around activities that have traditionally been considered difficult to measure
  • Analyzing data: What kinds of sources of security data exist, and how you can put them to work for you
  • Telling a story: Techniques you can use to marshal empirical evidence into a coherent set of messages

The need for a book like this seems plain to me. Security is one of the few areas of management that does not possess a well-understood canon of techniques for measurement. In logistics, for example, metrics such as "freight cost per mile" and "inventory warehouse turns" help operators understand how efficiently trucking fleets and warehouses run. In finance, "value at risk" techniques calculate the amount of money a firm could lose on a given day based on historical pricing volatilities. By contrast, security has . . . exactly nothing. No consensus on key indicators for security exists.

The lack of consensus on security metrics is, in part, due to the fact that the culture surrounding security is largely one of shame. Firms that get hacked tend not to talk about security incidents in public. Likewise, firms that are doing the right things tend not to talk either, lest giant red bull's-eyes appear on their firewalls' flanks. When they do talk, it is typically under NDA, or at small gatherings of like-minded people. Therefore, this book, as a secondary objective, documents effective practices of firms that take the responsibility of measuring their security activities seriously.

Non-goals of This Book

This book is first and foremost about quantifying security activities. It identifies ways to measure security processes that many enterprises consider important. The metrics and analysis techniques I document here are partly of my own devising but are drawn primarily from examples collected over the course of consulting in the software, aerospace, and financial services industries. I have met and exchanged notes with many people who have started their own metrics programs and are passionate about security metrics. At a minimum, I hope you will regard this book as a useful synthesis of current security measurement practices.

The word "practices" in that last sentence is important. I chose it carefully because of the implicit contrast with an opposing word: theory. In this book you will find plenty of anecdotes, lists of metrics, and ways of measuring security activities. But I have devoted only a small part of the text to modeling security risks—that is, figuring out which threats and risks are the right ones to worry about. Risk assessment is a broad field with many schools of thought. Smart people have spent many megawatts of brainpower modeling threats, modeling the effectiveness of security countermeasures, and simulating perimeter defenses.

The first non-goal of this book, therefore, is enterprise risk modeling and assessment. This is an important endeavor that every enterprise must undertake, but specific techniques are beyond the scope of this book. Risk assessment is an organization-specific activity, and I did not want to spend half of my pages disclaiming things because "it depends on what risks your organization feels are the most important." Moreover, I did not wish to add to what is already an exceptionally rich canon of works devoted to the subject of risk modeling and assessment.

To this rather significant and somber-sounding non-goal I would like to add three more. The dearth of generally accepted security metrics often means that unscrupulous vendors manufacture blood-curdling statistics in a vacuum, devoid of context and designed to scare. Middle managers with agendas promptly recycle these metrics for their own purposes. Therefore, this book also is not about the following:

  • Budget justification: How to convince your boss to spend money on security. If your company has not yet figured out that it needs to spend money on security, it likely has deeper problems than just a lack of statistics.
  • Fear, uncertainty, and doubt (FUD): How to abuse or misrepresent data for the purpose of manufacturing security scare stories. I derive no pleasure from this, and it makes me feel cheap and dirty.
  • Funny money: Any and all topics relating to "return on security investment." In addition to its dubious merit as a measure of security effectiveness, ROSI (as it is sometimes called) is a needless distraction from empirical security measurement.

Of course, because no good deed goes unpunished, it is entirely likely that this book will be used for those purposes regardless. But that, as a student of security analysis might say, is a risk worth taking.

Audience

I wrote this book for two distinct audiences: security practitioners and the bosses they report to. Practitioners need to know how, what, and when to measure. Their bosses need to know what to expect. Not for nothing has the security domain resisted measurement. As the bedraggled security manager of a household-name financial services firm recently told me, "My boss doesn't understand what I do every day. All he understands are numbers." Bridging the yawning gap between practitioners and management is what this book aims to achieve.

Overview of Contents

This book is divided into eight chapters:

  • Chapter 1, "Introduction: Escaping the Hamster Wheel of Pain": The state of security metrics today eerily resembles a hamster wheel that spins continuously around an axis of vulnerability discovery and elimination. Thinking about security as a circular, zero-sum game cripples our ability to think clearly. This introductory chapter advocates replacing the hamster wheel with key indicators—metrics—that measure the efficiency of key security activities.
  • Chapter 2, "Defining Security Metrics": This chapter describes the philosophy behind metrics, describes business pressures driving their adoption, suggests criteria for evaluating "good metrics," and warns against red herrings and other "bad metrics."
  • Chapter 3, "Diagnosing Problems and Measuring Technical Security": Leading firms measure security activities differently, depending on need and context. This chapter catalogs the types of measurements that firms use to diagnose security problems. These include practical metrics for such topics as coverage and control, vulnerability management password quality, patch latency, benchmark scoring, and business-adjusted risk.
  • Chapter 4, "Measuring Program Effectiveness": Beyond purely technical security measures, organizations need methods for measuring strategic security activities, for tracking security acquisition and implementation efforts, and for measuring the ongoing effectiveness of security organizations. This chapter catalogs dozens of program-level metrics, using the COBIT framework as an organizing principle.
  • Chapter 5, "Analysis Techniques": To create metrics, analysts must transform raw security data into numbers that provide richer insights. This chapter describes essential techniques for arranging, aggregating, and analyzing data to bring out the "headlines." It also describes advanced analytical techniques such as cross-sectional and quartile analyses.
  • Chapter 6, "Visualization": Even the most compelling data is worthless without an effective way of presenting it. This chapter presents a myriad of visualization techniques, ranging from simple tables to two-by-two grids and intricate "small multiple" charts.
  • Chapter 7, "Automating Metrics Calculations": Most organizations have plenty of security data available to them, although it is often trapped inside proprietary tools and information islands. This chapter suggests likely sources for finding appropriate data, including firewall logs, antivirus logs, and third-party auditor reports. It also describes techniques for transforming acquired data into formats that lend themselves to aggregation and reporting.
  • Chapter 8, "Designing Security Scorecards": After an organization collects and analyzes its security metrics, only one step remains: creating a scorecard that pulls everything together. This chapter presents several alternative approaches for designing security "balanced scorecards" that present compact, holistic views of organizational security effectiveness.

In addition to these topics, this book contains a generous sprinkling of anecdotes and war stories from my personal experiences, as well as those of my interview subjects.

Thank you for purchasing this book. I hope you enjoy reading it as much as I have enjoyed writing it.


© Copyright Pearson Education. All rights reserved.


Inside This Book (Learn More)
Browse Sample Pages
Front Cover | Copyright | Table of Contents | Excerpt | Index
Search inside this book:

Customer Reviews

4 star
0
2 star
0
1 star
0
4.0 out of 5 stars
4.0 out of 5 stars
Most helpful customer reviews
3 of 3 people found the following review helpful
3.0 out of 5 stars Some gaps, but useful nonetheless July 16 2008
Format:Paperback
Andrew Jaquith's book on security metrics is refreshing in its approach. Instead of a neverending cycle of risk assessments and vulnerability patching (a process which the author humorously calls the "hamster wheel of pain"), we are told to focus on core operational security processes and measurement of key indicators.

The central premise of the book is that a "risk management" approach, as promoted by many security vendors, doesn't work. The reason it doesn't work is that it is extremely difficult to get a good handle on the true value of assets, and an accurate estimate of risk. As the author puts it, "identifying problems is easy ... quantifying and valuing risk is much harder."

The thorough discussion of information security metrics makes this book worthwhile reading. However, there is a hint of sloppy thinking sprinkled throughout, which tends to undermine one's trust in the author's intellectual honesty. For example, when discussing the importance of tracking not only inbound viruses, but outbound as well, the author makes the following statement:

BEGIN QUOTE -
Another twist I have added to the traditional antivirus statistics is a simple metric documenting the number of outbound viruses or spyware samples caught by the perimeter mail gateway's content filtering software. Why it matters is simple--it is an excellent indicator of how "clean" the internal network is. Organizations that practice good hygiene don't infect their neighbors and business partners. My friend Dan Geer relates this quote from the CSO of a Wall Street investment bank:

"Last year we stopped 70,000 inbound viruses, but I am prouder of having stopped 500 outbound.
Read more ›
Was this review helpful to you?
1 of 2 people found the following review helpful
5.0 out of 5 stars A good resource for security practitioners June 13 2007
Format:Paperback
This book had an immediate and positive impact on my work. It provided a good statement of the current business issues surrounding security metrics and provided tangible, workable implementation suggestions.
Was this review helpful to you?
Most Helpful Customer Reviews on Amazon.com (beta)
Amazon.com: 4.4 out of 5 stars  24 reviews
38 of 39 people found the following review helpful
5.0 out of 5 stars A ground-breaking book that all security managers should read Aug. 9 2007
By Richard Bejtlich - Published on Amazon.com
Format:Paperback
I read Security Metrics right after finishing Managing Cybersecurity Resources, a book by economists arguing that security decisions should be made using cost-benefit analysis. On the face of it, cost-benefit analysis makes perfect sense, especially given the authors' analysis. However, Security Metrics author Andy Jaquith quickly demolishes that approach (confirming the problem I had with the MCR plan). While attacking the implementation (but not the idea) of Annual Loss Expectancy for security events, Jaquith writes on p 33 "[P]ractitioners of ALE suffer from a near-complete inability to reliably estimate probabilities [of occurrence] or losses." Bingo, game over for ALE and cost-benefit analysis. It turns out the reason security managers "herd" (as mentioned in MCR) is that they have no clue what else to do; they seek safety in numbers by emulating peers and then claim that as a defense when they are breached.

Fortunately, Security Metrics offers another solution. The book gives readers three sets of information: theory, metrics, and tools (concepts, not programs). The theory chapters (1 and 2) were so concise yet insightful I was tempted to underline every sentence. (I am not kidding.) Even the Preface made me glad to be reading the book when it associated "security ROI" with "the Macarena" and called it a "needless distraction." I laughed in agreement when I saw Andy call "security enablement" the "Abominable Snowman: it is rarely spotted, but legions of people swear it exists. After all, as my friend Dan geer puts it, 'You don't usually see airlines advertising how their planes fall out of the sky less often than their competitors.'" Why is that? My answer is simple: security is assumed and expected. Advertising anything else has no effect or makes people suspicious. I knew this book would be good.

The metrics chapters probably list hundreds of metrics you can extract verbatim and apply to your own environment. To the reviewer who wanted to reprint them in an appendix: they're called chapters 3 and 4. My main concern with the metrics was the focus on input-centric measurements instead of results. I would have liked to read more metrics on measuring whether security programs are working, rather than what techniques and tools are applied up front.

The tools chapters were helpful to anyone needing a statistics refresher. The visualization sections were especially helpful. (Feel free to dismiss yet another ignorant review from WB, who thinks a "review" means writing a few paragraphs after flipping through the pages of five books a day.) Andy's examples of turning lousy graphs and charts into information visualization vehicles should be followed by all managers.

Security Metrics is strengthened by the many stories from the author's consulting experience. I sensed that his techniques work and are not the product of the thought laboratory alone. I found his "Balanced Scorecard" approach to be interesting, especially to the degree it ties real metrics to business operations.

I had a few issues with terminology, such as using the term "threats" on p 231 when "attacks" is more accurate. (The football analogy is correct, however.) I semi-agreed with the author's suggestion to abandon "risk management" in favor of metrics-based approaches, but I didn't think two pages (4-5) were really enough to make the case. On p 264, threats are not risks, but they help instantiate risks. On pp 78-7, "risk of exploit" should be "ease of exploitation."

These are minor concerns, given the overwhelming concentration of practical and implementation-worthy pieces of information in Security Metrics. You must read this book if you care to measure security progress. Now we need Dan Geer to extend beyond writing wise forewords and articles into the world of his own book!
18 of 19 people found the following review helpful
5.0 out of 5 stars Arithmophobes, unite! You have nothing to lose but your Threat Level color wheels. April 21 2007
By Wendy Nather - Published on Amazon.com
Format:Paperback
It's difficult to imbue a book on metrics with something other than academic theories, but Jaquith offers the working security professional a tangible lifeline. Nearly all of his suggested metrics are within easy reach, thanks to a commonsense approach and a tie-in to the instrumentation you're most likely to have in your data center.

Don't be scared off by the term "metrics," either; it's an easy read, chock full of amusing stories and turns of phrase (I thought my 80-year-old father was the only one who said "'pert near"). Jaquith focuses on the practical, from What Not to Draw (a graphics primer for charts and tables) to a Balanced Scorecard Makeover that actually looks achievable from outside the C-suite.

If your boss likes metrics, and your budget request is in jeopardy, you can't do better than this guide to making your case. Now, if only we had a practical, lightweight risk analysis methodology to go along with it ...
18 of 20 people found the following review helpful
5.0 out of 5 stars Should be in the hands of every security professional. May 16 2007
By Ben Rothke - Published on Amazon.com
Format:Paperback
The goal of security metrics is to replace fear, uncertainty, and doubt (FUD) with a more formalized and meaningful system of measurement. The FUD factor is the very foundation upon which much of information security is built, and the outcome is decades of meaningless statistics and racks of snake oil products. Let's hope that Andrew Jaquith succeeds, but in doing so, he is getting in the way of many security hardware and software vendors whose revenue streams are built on FUD.

One could write a book on how FUD sells security products. One of the most memorable incidents was in 1992 when John McAfee created widespread panic about the impending Michelangelo virus. The media was all over him as he was selling solutions for the five million PCs worldwide he said would be affected. The end result is that the Michelangelo virus was a non-event. Nonetheless, it was far from the last time that FUD was used to sell security.

The allure of FUD is that companies can spend huge amounts of money fighting nebulous digital adversaries and feel good about it. They can then put all of that fancy hardware in dedicated racks in their data center, impressing the auditors with the flashing lights giving off an aroma of security and compliance.

And that is the chaos that security metrics comes to solve. Security metrics, if done right, can help transform a company from a nebulous perspective on security to an effective one based on formal security risk metrics.

Security Metrics is a fabulous book that should be in the hands of every security professional. The book demonstrates that companies must establish metrics based on their unique requirements, as opposed to simply basing their requirements on imprecise industry polls, best-practices and other ill-defined methods.

So why don't companies do that in the first place? If security metrics can provide even a quarter of the benefits that Jaquith states, companies should run to implement them. Real security metrics require an organization to open up their security hood and dig deep into the engine that runs their security infrastructure. It necessitates understanding the internal requirements, unique organizational risks, myriad strengths and weaknesses, and much more. Very few companies are willing to dedicate the time and resources for that, and would rather build their security infrastructure on thick layers of FUD. History has shown that the security appliance of the month almost always beats a formal risk and needs assessment.

Chapter 1 lays out the problem with approaches that most companies take to risk management. The main problem is that traditional risk management is far too dependant on identification and fixing, as opposed to quantification and triage based on value. Quantifying and valuing risk is much more difficult than simply identifying, since the software tools used do not have an organization context or knowledge of the specific business domain.

Chapter 2 sets out the foundation of security metrics. The goal of these metrics are to provide a framework in which organizations can quantify the likelihood of danger, estimate the extent of possible damage, understand the performance of their security organizations and weigh the costs of security safeguards against their expected effectiveness.

The time has come for security metrics since information security is one of the few management disciplines that have yet to submit itself to serious analytical scrutiny. The various chapters provide many different metrics that can be immediately used in most organizations to address that.

The author defines various criteria for what makes a good metric. One of his pet peeves is the use of the traffic light as a metaphor for compliance. Jaquith feels that traffic lights are not metrics at all, since they don't contain a unit of measure or are a numerical scale. He suggests using traffic lights colors sparingly, and only to supplement numerical data or draw attention to outliers. He astutely notes that if your data contains more precision than three simple gradations, why dilute their value by obscuring them with a traffic light.

The chapter concludes on what makes a bad metric, defined as any metric that relies too much on the judgment of a person. These metrics can't be relied on since the results can't be guaranteed to be the same from person to person. Also, security frameworks such as ISO-17799 should not be used for metrics. The book also tackles the sacred cow of risk management, namely ALE (annualized loss expectancy), and how it is significantly misused and misunderstood in the industry.

The book states that in developing metrics, there must be formal collaboration between the business units and the security staff. This collaboration serves to increase awareness and acceptance of security. In addition, it ensures that security requirements are incorporated into the lifecycle early on. This is needed as business units generally have no clue as to what the needed security requirements are.

Chapter 5 is a short course on analysis techniques and statistics. The author quotes George Colony who stated that "any idiot can tell you what something is. It is much harder to say what that thing means". With that, the book details a number of techniques for analyzing security data (average, median, time series, etc.) and how each one should be used.

Chapter 6 is about visualization and notes that most information security professionals have no real idea how to show security, both literally and figuratively. Part of the problem is that security is proliferated with esoteric terminology and concepts, and the lack of understanding risk management amongst the masses. Part of the reason for this difficulty in sharing the security message with management is that many security practitioners lack simple metaphors for communicating priorities. This is compounded by the fact that the message is often focused exclusively on technical security issues, as opposed to the underlying business issues, which is was management is concerned with. The chapter is invaluable as it weans one off the malevolent pie chart and traffic light PowerPoint presentation.

Marcus Ranum notes that people seem to want to treat computer security like its rocket science or black magic. In fact, computer security is nothing but attention to detail and good design. FUD is all about emphasizing the black magic aspect of hackers and other rogue threats. Metrics are all about the attention to detail that FUD lives to obfuscate.

Security Metrics: Replacing Fear, Uncertainty, and Doubt is one of the more important security books of the last few years. Jaquith turns much of the common security wisdom on its head, and the world will be a better place for it. Security metrics are a necessity whose time has come and this invaluable book shows how it can be done.
10 of 10 people found the following review helpful
4.0 out of 5 stars A dispatch from the May 21 2007
By Russell C. Thomas - Published on Amazon.com
Format:Paperback
I'd really like to give this "3 " stars, but I rounded up to 4 stars. There is, currently, no book that is the "last word" on security metrics. The field is just not mature enough for that. However, this is certainly a very good and useful book for most people.

This book is for you if you are a practicing information security professional and you want to know the latest ideas about how to define, deploy, and use security metrics to improve security management. Written in an informal, personal style, Andrew's book reads like "letters from the front lines" (by analogy) than a treatise on military strategy.
The informal style makes the reading, at times, both fun and funny.

He's up front about his preferences and biases, so you know where he's coming from. But he's not bombastic. If you disagree with him on some points (as I do), so be it. His writing invites open debate on the important issues. He's also generous in quoting and crediting various members of the security metrics communities that he participates in.

Andrew falls into the "bag-o-metrics" school of thought, as contrasted from the "risk modeling" school. (This is currently a raging debate within the community.) Basically, Andrew is pessimistic about the possibility of defining any models that integrate security metrics into an overall assessment of business risk. He's especially caustic in his comments about "asset valuation" and other related approaches. Given their current state of development, I don't blame him.

Given this philosophy, Andrew proposes a long list of operational security metrics, each of which measure something very specific (and quantitative), but don't necessarily aggregate. With enough of these "point metrics", some correlations may emerge, he reasons. To help give structure to the "bag", he offers some material relating metrics to various control frameworks (eg. COBIT) and also the Balanced Scorecard. The latter was a noble attempt to fit security metrics into enterprise performance management, but I don't think he really succeeded. But he chews on the right issues and questions, so it help people who are willing to do your own research into corporate scorecards.

Two of most notable parts of the book are the "Introduction: Escaping the Hamster Wheel of Pain", and "Chapter 7 Automating Metrics Calculations". The first is good because he talks in plain, blunt language about the current dismal state of security management in most organizations (who don't effectively use metrics to drive decisions). Ch. 7 is good because it gives a good framework and snapshot in time for a fast-emerging field.

I do have some criticisms of the book, even accepting Andrew's philosophies and premises.

First, there should have been an Appendix that compiled all the suggested metrics into one place. [...]

Second, there is inadequate coverage of how security metrics can mesh with other security and risk management needs - privacy, digital rights, IP protection, forensics, fraud prevention, physical security, and business continuity, to name a few. InfoSec should not be an island. Therefore the metrics system needs to "play well with others", so to speak.

Third, there is inadequate attention to measuring knowledge of attackers and attack strategies. Basically, over the time scale of months or years, InfoSec is an evolutionary strategic "game" between attackers and defenders. This makes information security an arms race, so you need to know if you are falling behind, fighting the last war, etc. Every organization needs to constantly learn about potential attackers and new attack strategies. At least a few metrics in these areas would add big value.

Finally, there is a 49 page chapter devoted "Visualization", which I don't think is the best use of that space. While I think visualization and reporting are *critical*, I think there are plenty of other books and guides that provide guidance. I didn't see anything in this chapter that was specific to InfoSec. That said, the material is useful and valuable if you aren't skilled at visual design.

In conclusion -- a good book that has plenty of useful material, as reported from the front lines.
7 of 7 people found the following review helpful
3.0 out of 5 stars Some gaps, but useful nonetheless July 16 2008
By Jacob Gajek - Published on Amazon.com
Format:Paperback
Andrew Jaquith's book on security metrics is refreshing in its approach. Instead of a neverending cycle of risk assessments and vulnerability patching (a process which the author humorously calls the "hamster wheel of pain"), we are told to focus on core operational security processes and measurement of key indicators.

The central premise of the book is that a "risk management" approach, as promoted by many security vendors, doesn't work. The reason it doesn't work is that it is extremely difficult to get a good handle on the true value of assets, and an accurate estimate of risk. As the author puts it, "identifying problems is easy ... quantifying and valuing risk is much harder."

The thorough discussion of information security metrics makes this book worthwhile reading. However, there is a hint of sloppy thinking sprinkled throughout, which tends to undermine one's trust in the author's intellectual honesty. For example, when discussing the importance of tracking not only inbound viruses, but outbound as well, the author makes the following statement:

BEGIN QUOTE -
Another twist I have added to the traditional antivirus statistics is a simple metric documenting the number of outbound viruses or spyware samples caught by the perimeter mail gateway's content filtering software. Why it matters is simple--it is an excellent indicator of how "clean" the internal network is. Organizations that practice good hygiene don't infect their neighbors and business partners. My friend Dan Geer relates this quote from the CSO of a Wall Street investment bank:

"Last year we stopped 70,000 inbound viruses, but I am prouder of having stopped 500 outbound."

In other words, the bank's internal network is cleaner than the outside environment by a factor of 140 to 1.
- END QUOTE

Certainly, the conclusion in the last sentence cannot be supported without additional information. The volume of inbound email is likely to be drastically higher, which may account for the difference. The bank's outbound detection/prevention mechanism also may not be as efficient as the inbound.

Moreover, the metrics analysis chapter is very rudimentary and incomplete. Basic concepts like mean, median, and standard deviation are briefly discussed, but there is no mention of statistical random sampling techniques and confidence levels, which would surely be of significant importance when measuring key indicators across large populations, where a complete enumeration is either impossible, or too expensive and time-consuming. Sometimes, metrics which are "meaningful", are not the ones that are "tangible" and "easy to measure". A certain degree of statistical sophistication can be helpful in such situations.

In summary, the book offers some useful insight and practical advice for those who are charged with running an information security management program, but a healthy skepticism of the assumptions underlying the author's conclusions is warranted. In order to develop truly meaningful information security metrics, a much more sophisticated approach than what is described in this book will likely be needed.
Search Customer Reviews
Only search this product's reviews

Look for similar items by category


Feedback