The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems [Hardcover]

Complex event processing (CEP) is a set of techniques and tools to help us understand and control event-driven information systems. And today, any kind of information system, from the Internet to a cell phone, is driven by events. What is a complex event? It is an event that could only happen if lots of other events happened.

For example, suppose you see a car you like at your favorite car dealership. That car is on the showroom floor only because a number of other events took place—events in the inventory control systems of the dealership and the manufacturer, shipping events, customs events at the port of entry, and so on. Of course, when you see exactly what you want in the showroom, you don't ask how or why. But if you don't see the model, make, or color you want and ask why not, you'll get an explanation about allocation quotas, backlogs at the factory, or some other factors that affect events in the causal history leading up to the event you wanted.

This illustrates one of the ideas behind CEP. Events are related in various ways, by cause, by timing, and by membership. CEP applies to electronic information systems. It makes use of relationships between events to answer questions like, "Is our system providing the correct level of service to our customers," "Will our shipment arrive on time," and "Is someone trying to steal our information?" CEP adds a new dimension of event processing to what our event-driven information systems already do.

Why is there a need for CEP? Let's look at the situation briefly.

Today's information society is founded upon gathering and sharing information. All our organizations—commercial, government, and military—are dependent upon electronic information processing. Their foundational backbone is the kind of distributed computing system based on computer networks that is nowadays called the "information technology layer" (or IT layer) of the organization. The use of these systems has expanded rapidly over the past ten years to meet the increasing demands of automation, electronic commerce, and the Internet explosion. In vestment in technology has focused on making IT systems faster, capable of handling larger and larger amounts of information, and able to collaborate with one another. We now live in the world of the open enterprise, where commerce and information move across the boundaries of organizations and nations. Our society has become dependent upon IT systems.

Less investment has been devoted to develop technology to solve the increasing problem of understanding what is happening in our IT systems. Whenever there is a crisis—a denial-of-service attack or a system failure—at first we don't understand what is going on or how to fix it, and then in the aftermath, we scramble for weeks to find out what caused it. We need to understand and control our critical information infrastructures better than that!

A lot of the information in IT systems is never recognized. Messages—or events—pass silently back and forth across our information systems as unrelated pieces of communication. They are a source of great power, for when they are aggregated together, and correlated, and their relationships understood, they yield a wealth of information. A new technology is needed to harness the power of events in global information systems. This book is about such a technology.

A few words about CEP—what it is, and where it applies.

CEP consists of very simple techniques, a mix of old and new. Some of them are well known in other kinds of computer applications, such as rulebased systems in intelligent programs. Some of them are new techniques, such as tracking causal histories of events in large distributed computer systems. Or using patterns of events and event relationships, to recognize the presence of complex events that are signified by hundreds or thousands of simpler events in our IT systems. In CEP, new techniques are combined with well-known techniques in a unified framework.

An example of the kind of electronic complex event we are talking about is the completion of a financial transaction involving a bundle of financial contracts. Several merchant banks and brokerage houses may participate in the transaction. They use a global trading network. The event itself, the completion of the transaction, might be the result of hundreds of electronic messages and entries into several different databases around the world over a span of two or three days. These events don't necessarily happen in a nice linear order, one after the other. Some of them might happen simultaneously and independently of others, mixed in with events from other transactions. We can apply CEP to the trading network to recognize not only when that complex event happens, but, more importantly, whether it is going to happen, or if it is getting off track and may not happen, and why.

CEP applies to a very broad spectrum of challenges in information systems. A short list includes

• Business process automation utilizing the Internet and electronic marketplaces
• Computer systems to automate the scheduling and control of anything from fabrication lines to air traffic
• Network monitoring and performance prediction
• Detecting attempts to intrude into computer systems or attack them

There is a fundamental reason for this broad applicability. It is simply because information systems are all driven by events. To be sure, each system, or application running on top of a system, depends upon different kinds of events. Network events are different from database events, which are different from financial trading events. But one of the major themes of CEP is that different kinds of events are related. CEP provides techniques for defining and utilizing relationships between events. CEP applies to any type of event that happens in a computer application or a network or an information system. In fact, one of its techniques lets you define your own events as patterns of the events in your computer system. CEP lets you see when your events happen. This is one way to understand what is going on in your system.

That brings us to another point—flexibility. CEP allows users to specify the events that are of interest to them at any moment. Events of interest can be low-level network monitoring alerts or high-level enterprise management intelligence, depending upon the role and viewpoint of individual users. Different kinds of events can be specified and monitored simultaneously. And the specification of the events of interest, how they should be viewed and acted upon, can be changed on the fly, while the system is running.

The users of CEP can be human, or they can be autonomous processes. The processes that manage our enterprises are becoming more complex. Linear workflow processes that epitomize document processing in commercial transactions are not capable of managing the open electronic enterprise. In the future, enterprise management processes will be designed to incorporate complex event processing in order to get the kind of events they need to operate.

Now, a few words about the book itself and what the reader should expect. First, there are two parts to this book.

Part I is for a broad audience of people with an interest in various aspects of the information society, such as electronic commerce, the Internet, B2B collaboration, or, generally, electronic information processing. Part I deals with two questions about CEP: what it is for—that is, the kinds of problems in the information society that CEP can be applied to; and what it is—a simplified view of CEP, the basic concepts and easy examples of applications. Part I includes Chapters 1 through 7.The first four chapters describe the problems and issues in IT systems that CEP applies to. The next three chapters describe basic concepts of CEP, such as what an "event" is, causal and timing relationships between events, patterns of events and event hierarchies, and how to apply them to solve the problems described earlier.

Part II consists of Chapter 8 onward. It is intended for information systems specialists with some background in software. Part II presents how-to-build-it details and case studies of CEP applications. The goal of Part II is to describe what is needed to build applications of CEP that are capable of solving real-world problems. It includes first a detailed description of a complex event pattern language, reactive event pattern rules, and event pattern constraints. Second, Part II shows how to build solutions by using the event pattern rules and constraints to build event processing agents and architectures of communicating agents. Part II also includes case studies, as large and as detailed as we can fit in a chapter of a book.

The final chapter of this book deals with the question of how to develop an infrastructure for CEP. We can look around the event-driven applications being developed in the commercial world today, utilizing the power of distributed computing, the Internet, and private networks. An almighty commercial struggle is brewing for market share in the world of eMarketplaces and electronic commerce. It is quite predictable, considering the trends in middleware, the Java world, the .NET world, the security world, and so on, that CEP will be developed as a competitive advantage. This chapter deals with leveraging these developments to build an infrastructure for CEP—now and quickly!

A word about references. This area of Internet technology is changing so quickly that any attempt to give comprehensive references would be outdated in six months. Not only that, but any less than complete set of references would be unfair to some. I assume that any reader has access to the Internet and can search for current references to, for example, "middleware" or "application server." So I have tended to include only a few references, either general references to Web sites or citations to seminal research papers that are not easily found.

At this time in our society, any technology that attempts to view and control IT systems may be seen by some as conflicting with issues concerning privacy. In fact, CEP may provide a foundation for resolving some possible conflicts. However, I cannot deal with this topic here, and I do not.

Just a little history. CEP has grown out of a research project at Stanford on event-based simulation called the Rapide project. This research took place between 1990 and 2000.Out of Rapide came some early experiments in CEP applied to viewing small communicating systems built on commercial middleware, or applied to recognizing security threats in progress on the IT layer of a large university, where hackers love to play.