The Tao of Network Security Monitoring: Beyond Intrusion Detection (Paperback)

"The book you are about to read will arm you with the knowledge you need to defend your network from attackers--both the obvious and the not so obvious...If you are new to network security, don't put this book back on the shelf! This is a great book for beginners and I wish I had access to it many years ago. If you've learned the basics of TCP/IP protocols and run an open source or commercial IDS, you may be asking 'What's next?' If so, this book is for you." --Ron Gula, founder and CTO, Tenable Network Security, from the Foreword "Richard Bejtlich has a good perspective on Internet security--one that is orderly and practical at the same time. He keeps readers grounded and addresses the fundamentals in an accessible way." --Marcus Ranum, TruSecure "This book is not about security or network monitoring: It's about both, and in reality these are two aspects of the same problem. You can easily find people who are security experts or network monitors, but this book explains how to master both topics." --Luca Deri, ntop.org "This book will enable security professionals of all skill sets to improve their understanding of what it takes to set up, maintain, and utilize a successful network intrusion detection strategy. " --Kirby Kuehl, Cisco Systems Every network can be compromised. There are too many systems, offering too many services, running too many flawed applications. No amount of careful coding, patch management, or access control can keep out every attacker. If prevention eventually fails, how do you prepare for the intrusions that will eventually happen? Network security monitoring (NSM) equips security staff to deal with the inevitable consequences of too few resources and too many responsibilities. NSM collects the data needed to generate better assessment, detection, and response processes--resulting in decreased impact from unauthorized activities. In The Tao of Network Security Monitoring, Richard Bejtlich explores the products, people, and processes that implement the NSM model. By focusing on case studies and the application of open source tools, he helps you gain hands-on knowledge of how to better defend networks and how to mitigate damage from security incidents. Inside, you will find in-depth information on the following areas. *The NSM operational framework and deployment considerations. *How to use a variety of open-source tools--including Sguil, Argus, and Ethereal--to mine network traffic for full content, session, statistical, and alert data. *Best practices for conducting emergency NSM in an incident response scenario, evaluating monitoring vendors, and deploying an NSM architecture. *Developing and applying knowledge of weapons, tactics, telecommunications, system administration, scripting, and programming for NSM. *The best tools for generating arbitrary packets, exploiting flaws, manipulating traffic, and conducting reconnaissance. Whether you are new to network intrusion detection and incident response, or a computer-security veteran, this book will enable you to quickly develop and apply the skills needed to detect, prevent, and respond to new and emerging threats.

Welcome to The Tao of Network Security Monitoring: Beyond Intrusion Detection. The goal of this book is to help you better prepare your enterprise for the intrusions it will suffer. Notice the term “will.” Once you accept that your organization will be compromised, you begin to look at your situation differently. If you’ve actually worked through an intrusion—a real compromise, not a simple Web page defacement—you’ll realize the security principles and systems outlined here are both necessary and relevant.

This book is about preparation for compromise, but it’s not a book about preventing compromise. Three words sum up my attitude toward stopping intruders: prevention eventually fails. Every single network can be compromised, either by an external attacker or by a rogue insider. Intruders exploit flawed software, misconfigured applications, and exposed services. For every corporate defender, there are thousands of attackers, enumerating millions of potential targets. While you might be able to prevent some intrusions by applying patches, managing configurations, and controlling access, you can’t prevail forever. Believing only in prevention is like thinking you’ll never experience an automobile accident. Of course you should drive defensively, but it makes sense to buy insurance and know how to deal with the consequences of a collision.

Once your security is breached, everyone will ask the same question: now what? Answering this question has cost companies hundreds of thousands of dollars in incident response and computer forensics fees. I hope this book will reduce the investigative workload of your computer security incident response team (CSIRT) by posturing your organization for incident response success. If you deploy the monitoring infrastructure advocated here, your CSIRT will be better equipped to scope the extent of an intrusion, assess its impact, and propose efficient, effective remediation steps. The intruder will spend less time stealing your secrets, damaging your reputation, and abusing your resources. If you’re fortunate and collect the right information in a forensically sound manner, you might provide the evidence needed to put an intruder in jail.

Audience

This book is for security professionals of all skill levels and inclinations. The primary audience includes network security architects looking for ways to improve their understanding of their network security posture. My goal is to provide tools and techniques to increase visibility and comprehension of network traffic. If you feel let down by your network-based intrusion detection system (NIDS), this book is definitely for you. I explain why most NIDS deployments fail and how you can augment existing NIDS with open source tools.

Because this book focuses on open source tools, it is more likely to be accepted in smaller, less bureaucratic organizations that don’t mandate the use of commercial software. Furthermore, large organizations with immense bandwidth usage might find some open source tools aren’t built to handle outrageous traffic loads. I’m not convinced the majority of Internet-enabled organizations are using connections larger than T-3 lines, however. While every tool and technique hasn’t been stress-tested on high-bandwidth links, I’m confident the material in this book applies to a great majority of users and networks.

If you’re a network security analyst, this book is also for you. I wrote this book as an analyst, for other analysts. This means I concentrate on interpreting traffic, not explaining how to install and configure every single tool from source code. For example, many books on “intrusion detection” describe the Transmission Control Protocol/Internet Protocol (TCP/IP) suite and how to set up the Snort open source IDS engine with the Analysis Console for Intrusion Databases (ACID) interface. These books seldom go further because they soon encounter inherent investigative limitations that restrict the usefulness of their tools. Since my analytical techniques do not rely on a single product, I can take network-based analysis to the next level. I also limit discussion of odd packet header features, since real intrusions do not hinge on the presence of a weird TCP flag being set. The tools and techniques in this book concentrate on giving analysts the information they need to assess intrusions and make decisions, not just identify mildly entertaining reconnaissance patterns.

This book strives to not repeat material found elsewhere. You will not read how to install Snort or run Nmap. I suggest you refer to the recommended reading list in the next section if you hunger for that knowledge. I introduce tools and techniques overlooked by most authors, like the material on protocol anomaly detection by Brian Hernacki, and explain how you can use them to your advantage.

Technical managers will appreciate sections on best practices, training, and personnel issues. All the technology in the world is worthless if the staff manning it doesn’t understand their roles, responsibilities, and escalation procedures. Managers will also develop an intuition for the sorts of information a monitoring process or product should provide. Many vendors sell services and products named with combinations of the terms “network,” “security,” and “monitoring.” This book creates a specific definition for network security monitoring (NSM), built on a historical and operational foundation.

Prerequisites

I’ve tried to avoid duplicating material presented elsewhere, so I hope readers lacking prerequisite knowledge take to heart the following reading suggestions. I highly recommend reading the following three books prior to this one. If you’ve got the necessary background, consider these titles as references.

• Internet Site Security, by Erik Schetina, Ken Green, and Jacob Carlson (Boston, MA: Addison-Wesley, 2002). This is an excellent “security 101” book. If you need to start from the ground floor, this book is a great beginning.


• Counter Hack: A Step-by-Step Guide to Computer Attacks and Effective Defenses, by Ed Skoudis (Upper Saddle River, NJ: Prentice Hall PTR, 2001). Counter Hack offers the best single-chapter introductions to TCP/IP, Microsoft Windows, UNIX, and security issues available.


• Hacking Exposed: Network Security Secrets and Solutions, 4th ed., by Stuart McClure, Joel Scambray, and George Kurtz (New York: McGraw-Hill, 2003). Hacking Exposed explores the capabilities and intentions of digital threats. By knowing how to compromise computers, you’ll understand the sorts of attacks network security monitoring practitioners will encounter.

If you need an introduction to intrusion detection theory, I recommend the following book:

• Intrusion Detection, by Rebecca Gurley Bace (Indianapolis, IN: New Riders, 2000). While not strictly needed to understand the concepts in this book, Intrusion Detection provides the history and mental lineage of IDS technology. As The Tao of Network Security Monitoring focuses on network-based tactics, you can turn to Intrusion Detection for insight on host-based detection or the merits of signature- or anomaly-based IDS.

It helps to have a good understanding of TCP/IP beyond that presented in the aforementioned titles. The following are a few of my favorite books on TCP/IP.

• Internet Core Protocols: The Definitive Guide, by Eric A. Hall (Cambridge, MA: O’Reilly, 2000). Many people consider Richard Stevens’ TCP/IP Illustrated Volume 1: The Protocols (Reading, MA: Addison-Wesley, 1994) to be the best explanation of TCP/IP. I think Eric Hall’s more recent book is better suited for modern network traffic analysts.


• Network Analysis and Troubleshooting, by J. Scott Haugdahl (Boston, MA: Addison-Wesley, 2000). Troubleshooting books tend to offer the more interesting explanations of protocols in action. Scott Haugdahl works his way up the seven layers of the Open Systems Interconnect (OSI) model, using packet traces and case studies.


• Troubleshooting Campus Networks: Practical Analysis of Cisco and LAN Protocols, by Priscilla Oppenheimer and Joseph Bardwell (Indianapolis, IN: Wiley, 2002). This title is considerably broader in scope than Scott Haugdahl’s work, with coverage of virtual local area networks (VLANs), routing protocols, and wide area network (WAN) protocols like Asynchronous Transfer Mode (ATM).

One other book deserves mention, but I request you forgive a small amount of self-promotion. The Tao of Network Security Monitoring is primarily about detecting incidents through network-based means. In some senses it is also an incident response book. Effective incident response, however, reaches far beyond network-based evidence. To learn more about host-based data, such as file systems and memory dumps, I recommend Real Digital Forensics (Boston, MA: Addison-Wesley, 2005). I wrote the network monitoring sections of the book, and coauthors Keith Jones and Curtis Rose did the host- and memory-level forensics. If you’d like to see the big picture for incident response, read Real Digital Forensics.FreeBSD is a UNIX-like, open source environment well suited for building network security monitoring platforms. If you’re familiar with Linux or any other Berkeley Software Distribution (OpenBSD or NetBSD), you’ll have no trouble with FreeBSD. I strongly recommend running NSM tools on UNIX-like platforms like the BSDs and Linux.

You might consider trying a live CD-ROM FreeBSD distribution prior to committing a hard drive to installation.FreeSBIE recently shipped version 1.0, based on the FreeBSD 5.2.1 RELEASE edition.

Live distributions boot from the CD-ROM and run all programs within memory. They can be configured to write to removable media like USB thumb drives or the hard drive of the host computer. Live distributions are a good way to test hardware compatibility before going through the time and effort to install a new operating system on a system’s hard drive. For example, before upgrading a FreeBSD 4.9–based system to version 5.2.1, I booted a FreeBSD 5.2.1–based live distribution and checked whether it saw all of the hardware properly.

Many security tools are included in the distribution, including Nessus, Nmap and NmapFE, Snort, and Ethereal. I am investigating building an NSM-minded FreeBSD-based live distribution to run the tools discussed in this book.

If you want to learn about FreeBSD, I suggest these books.

• FreeBSD: An Open-Source Operating System for Your Personal Computer, 2nd ed., by Annelise Anderson (Portola Valley, CA: Bit Tree Press, 2001). Absolute UNIX newbies will find Annelise Anderson’s book the gentlest introduction to FreeBSD.


• Absolute BSD: The Ultimate Guide to FreeBSD, by Michael Lucas (San Francisco, CA: No Starch Press, 2002). Michael Lucas has an uncanny ability to answer the questions his readers are bound to ask. Keep in mind that Annelise Anderson’s book and Absolute BSD focus on FreeBSD 4.x, so certain details might change with FreeBSD 5.x.


• The Complete Guide to FreeBSD, 4th ed., by Greg Lehey (Cambridge, MA: O’Reilly, 2003). Greg Lehey covers more than just FreeBSD; he addresses system and network administration issues as well. This is the first book explicitly written with FreeBSD 5.x in mind.

I’m often asked why I use FreeBSD and not OpenBSD. I use FreeBSD because I believe it is the best general-purpose operating system available. It has more applications in its ports tree, a larger development community, and better network and multiprocessor performance. I develop and test all of my applications and techniques on FreeBSD.

OpenBSD is more innovative in terms of security, with integrated defensive features like Systrace, the Pf firewall, increased use of privilege separation, and relentless removal of coding flaws. I believe OpenBSD may be a superior platform for building dedicated “security appliances.” Once the application is tested under a general-purpose operating system like FreeBSD, it can be deployed on a security-minded platform like OpenBSD.FreeBSD is beginning to adopt security systems like mandatory access control that are found in commercial operating systems like Trusted Solaris. In reality all three major BSD projects feed security ideas into each other, so competition among the projects is not a huge concern.

Linux and Windows users might wonder where I stand on their operating systems. I believe Linux benefits from having a very large development community. Because so many coders run Linux, users are more likely to see patches introduced to improve Tcpdump’s performance or implement other features useful to security professionals. I still prefer the BSDs to Linux because Linux is a kernel supplemented by tools selected by various distribution aggregators. There is also doubt about which Linux distribution is most likely to be used by the community. Prior to the arrival of Fedora Core, Red Hat Linux was more or less the de facto standard. Debian may be the heir to Red Hat’s throne, but that situation remains in flux. This is not the best environment for developing security applications and standards.

Windows is an operating system for consumers. It was designed to “make life easy” at the expense of security and operational transparency. The underlying Windows design model has not withstood connectivity to the Internet very well. The operating system provides far too many services on single ports. How can one disable port 135 or 139 TCP, for example, without breaking a dozen built-in applications?

I believe the supposed ease of use of a Windows system, even if one accepted this feature to be true, is far outweighed by the risk of introducing the operating system in a critical security role. Those adding a security platform to a network should not violate the first rule of the Hippocratic Oath: do no harm. I have far more confidence in the reliability and resiliency of a FreeBSD or other UNIX system compared to a Windows system.

Scope

The book is broken into five major parts, followed by an epilogue and appendices. You can focus on the areas that interest you, as the sections were written in a modular manner. You may wonder why greater attention is not paid to popular tools like Nmap or Snort. With The Tao of Network Security Monitoring, I hope to break new ground by highlighting ideas and tools seldom seen elsewhere. If I don’t address a widely popular product, it’s because it has received plenty of coverage in another book.

Part I offers an introduction to NSM, an operational framework for the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. Part I begins with an analysis of the terms and theory held by NSM practitioners. Chapter 1 discusses the security process and defines words like security, risk, and threat. It also makes assumptions about intruders and their prey that set the stage for NSM operations. Chapter 2 addresses NSM directly, explaining why NSM is not implemented by modern NIDSs alone. Chapter 3 focuses on deployment considerations, such as how to access traffic using hubs, taps, SPAN ports, and inline devices.

Part II begins an exploration of the NSM “product, process, and people” triad. Chapter 4 is a case study called the “reference intrusion model.” This is an incident explained from the point of view of an omniscient observer. During this intrusion, the victim collected full content data in two locations. We will use those two trace files while explaining the tools discussed in Part II. Following the reference intrusion model, I devote chapters to each of the four types of data that must be collected to perform NSM—full content, session, statistical, and alert data. Chapters 5 through 10 describe open source tools tested on the FreeBSD operating system and available on other UNIX derivatives. Part II also includes a look at tools to manipulate and modify traffic. Featured in Part II are little-discussed NIDSs like Bro and Prelude, and the first true open source NSM suite, Sguil.

Part III continues the NSM triad by discussing processes. If analysts don’t know how to handle events, they’re likely to ignore them. I provide best practices in Chapter 11 and follow with Chapter 12, written explicitly for technical managers. That material explains how to conduct emergency NSM in an incident response scenario, how to evaluate monitoring vendors, and how to deploy an NSM architecture.

Part IV, intended for analysts and their supervisors, completes the NSM triad. Entry-level and intermediate analysts frequently wonder how to move to the next level of their profession. In Chapter 13, I offer some guidance for the five topics with which a security professional should be proficient: weapons and tactics, telecommunications, system administration, scripting and programming, and management and policy. Chapters 14 through 16 offer case studies, showing analysts how to apply NSM principles to intrusions and related scenarios.

Part V is the offensive counterpart to the defensive aspects of Parts II, III, and IV. I discuss how to attack products, processes, and people. Chapter 17 examines tools to generate arbitrary packets, manipulate traffic, conduct reconnaissance, and exploit flaws in Cisco, Solaris, and Microsoft targets. In Chapter 18 I rely on my experience performing detection and response to show how intruders attack the mind-set and procedures on which analysts rely.

An epilogue on the future of NSM follows Part V. The appendices feature several TCP/IP protocol header charts and explanations. I also wrote an intellectual history of network security, with excerpts and commentary on the most important papers written during the last 25 years. Please take the time to at least skim that appendix; you’ll see that many of the “revolutionary ideas” often heralded in the press were in some cases proposed decades ago.

Neither Part V nor other parts are designed as “hacking” references. You will not find “elite” tools to compromise servers; if so inclined, refer to the suggested reading list. The tools I profile were selected for the traffic they generate. By looking at packets created by readily available offensive tools, analysts learn to identify normal, suspicious, and malicious traffic.

Welcome Aboard

I hope you find this book useful and enjoyable. I welcome feedback on its contents, especially tips on better uses of tools and tactics. While doing research I was amazed at the amount of work done in the field of intrusion detection over the last 25 years. Intrusion detection is only one component of NSM, but it is the general community in which NSM practitioners feel most at home.

Much of what I present is the result of standing on the shoulders of giants. Our community is blessed by many dedicated and talented people who contribute code, ideas, and resources to Internet security issues. I hope my contribution is worthy of the time you dedicate to reading it.