What Is Complex Event Processing?

Complex event processing (CEP) is a set of techniques for capturing and analyzing streams of data as they arrive to identify opportunities or threats in real time. CEP enables systems and applications to respond to events, trends, and patterns in the data as they happen. CEP is considered largely synonymous with event stream processing, though CEP is typically more associated with searching for complex patterns and dependencies in the incoming data. The latter term tends to be more about aggregations in time windows and is commonly associated with newer streaming technologies like Apache Kafka and its ecosystem of stream processing engines.

Complex Event Processing Use Cases

CEP is typically used for scenarios in which there is a large volume of events occurring and latency requirements are very low, typically in the range of milliseconds. CEP use cases include:

• Stock market trading. A CEP application can recognize incoming stock prices, match it against a pattern, and decide whether to trigger a buy/sell decision.
• Predictive maintenance. Sensors in manufacturing facilities and large objects such as windmills and airplanes continuously collect data to determine whether there are any patterns indicating it is necessary to shut down equipment for predictive maintenance.
• Real-time marketing. Marketers rely on CEP systems to recognize meaningful patterns in customer behavior so they can present relevant offers in real time.

One use case for CEP that has seen growing interest in the last few years is for autonomous vehicles. Sensors in a self-driving car may deliver data that lets a CEP system recognize a stop sign as a red and white object, in the shape of a hexagon, at an intersection ahead. The system can then measure the distance to that stop sign along with moisture on the road, to calculate and adjust to an appropriate deceleration rate.

Complex Event Processing and Stream Processing

CEP has been around since the 1990s, both conceptually and in terms of the earliest technologies. In recent years though, the term has been used almost interchangeably with “stream processing.”

However, purists may make some distinctions. CEP usually looks at patterns of events to identify a specific event. Both the event detection and the underlying processes to get there could be quite complex. For example, a CEP application may recognize the sound of drums, the cheers and clapping of a crowd, the sounds of hounds, and the change in frequency of the instruments to put together a pattern that lets it determine that a specific type of event was occurring—a marching band was playing. A stream processing application, on the other hand, often collects a single event at a time, such as a transaction or a click on a website, analyzes and processes it, then responds accordingly.

In common vernacular, “complex event processing” and “stream processing” are essentially synonymous. In fact, the same underlying technologies may support both use cases with adequate development.