Open Source Projects:

Use Case

In-Memory Stream Processing

When recency and speed drive the value of your data, In-Memory Stream Processing solutions from Hazelcast can elevate your business to new levels of performance.

Pricing
Chat
Contact
Back to top
Overview

Let Hazelcast In-Memory Stream Processing add value to your applications.

Stream processing is a technique to process the data on-the-fly, prior to its storage. This is in contrast with a traditional batch approach, where the data set has to be completely available and stored in the database or file before processing can begin.

A stream processing framework is vital when the value of information contained in the data decreases rapidly as the data ages. The faster the information is extracted from the data and provided to consumers the better.

Hazelcast In-Memory Stream Processing

Typical streaming applications include:

  • Log analysis and monitoring
  • Fraud detection
  • Anomaly detection (IoT systems, sensors)
  • Fast business insights
  • Cleaning data for downstream processing (filtering, modifying, normalising, enriching)
  • Real-time advertising placement and reporting
  • Real-time recommendations
  • Online gaming stats
  • Payment processing

In these use cases, processing data fast is of the same importance as processing vast volumes of data.

Data streams are potentially unbounded and infinite sequences of records, and records usually represent events or changes that happen in time. Stream processing applications are observing flowing records and literally query the stream for relevant data in near real-time.

Typical stream processing tasks:

  • Algorithmic analysis of the stream data
  • Joining multiple streams
  • Enriching stream with other information
  • Implementing Event Sourcing and Command Query Responsibility Segregation Architectures
  • Moving batch tasks to near real-time
Hazelcast Jet and Stream Processing

Hazelcast Jet and Stream Processing

Stream Processing Engine

Hazelcast Jet provides the tooling necessary to build a streaming data application. It gives you a powerful processing framework to query the data stream and elastic in-memory storage to store the results of the computation.

Large Data Volumes

Hazelcast Jet processing tasks, called jobs, are distributed across the Jet cluster to parallelize the computation. Jet is able to scale out this way to process large data volumes.

Integration with IMDG

Hazelcast Jet has very high-speed integration with Hazelcast IMDG, storing large amounts of data that is joined to the Jet stream in microseconds. Latency can also be lowered by using IMDG for stream ingestion or for publishing the results.

Windowing

Hazelcast Jet works with streaming data in terms of “windows," where a window represents a slice of the data stream, usually constrained for a period of time.

Jet supports Tumbling, Sliding and Sessions Windows.

Event Time and Late Events

Hazelcast Jet supports the notion of “event time,” where events can have their own timestamp and can arrive out of order. This is achieved by inserting watermarks into the stream of events that drive the passage of time forward.

Fault Tolerance

Hazelcast Jet provides simple fault-tolerant streaming computation without external system or storage using distributed in-memory. Jobs restart automatically using in-memory snapshots, and processing can be resumed where it left off.

Processing Guarantees

The need to trade-off performance and correctness in event processing systems may not allow firm guarantees. Hazelcast Jet allows you to choose the processing guarantee at start time, choosing between No guarantee, at-least-once, or exactly-once.

Free Hazelcast Online Training Center

Whether you're interested in learning the basics of in-memory systems, or you're looking for advanced, real-world production examples and best practices, we've got you covered.