Open Source Projects:

Use Case

In-Memory Stream Processing

When recency and speed drive the value of your data, in-memory stream processing solutions from Hazelcast can elevate your business to new levels of performance.

Pricing
Chat
Contact
Back to top
Overview

Accelerate Applications With In-Memory Stream Processing from Hazelcast

Accelerate Applications With In-Memory Stream Processing from Hazelcast

Data is coming at you fast from every direction. To meet customer expectations, prevent fraud, and ensure smooth operations, batch processing simply won’t cut it.

Traditional batch processing requires data sets to be completely available and stored in a database or file before processing can begin. With in-memory stream processing platforms, you can respond to data on-the-fly, prior to its storage, enabling ultra-fast applications that process new data at the speed with which it is generated.

Use Cases

Use Cases for Stream Processing

Your business is a series of continually occurring events. You launch products, run campaigns, send emails, roll out new apps, interact with customers via your website, mobile applications, and payment processing systems, and close deals, for example – and the work goes on and on. Events happen in real time, and your environment is always changing. To compete, you need to be able to quickly adjust to those changes.

 

Your applications require the real-time capabilities and insights that only stream processing enables. You can’t rely on knowing what happened with the business yesterday or last month. You need to know, and respond to, what is happening now. Event-driven businesses depend on modern in-memory streaming applications for:


  • Real-time analytics
  • Alerting
  • Fraud detection
  • Payment processing
  • IoT data capture
  • Microservices
  • Log monitoring and analysis
  • Real-time advertising


 

Stream processing must be both fast and scalable to handle billions of records every second. Event streams are potentially unbounded and infinite sequences of records that represent events or changes in real-time.

Typical stream processing tasks include:


  • Cleaning data for downstream processing
  • Algorithmic analysis of streaming data
  • Joining multiple streams
  • Enriching streams with other information
  • Publishing notifications to subscribers

Streaming Architecture

Where Does Stream Processing Fit Into My Application Architecture?

The Hazelcast Jet stream processing platform–built on in-memory computing technology to leverage the speed of random access memory compared with disk–sits between event sources such applications and sensors, and destinations such as an alerting system, database or data warehouse, whether in the cloud or on-premises.

Developers build stream processing capabilities into applications with Hazelcast Jet to capture and process data within microseconds to identify anomalies, respond to events, or publish the events to a data repository for longer-term storage and historical analyses.

With the Hazelcast Jet stream processing platform, your applications can handle low latency, high throughput transactional processing at scale, while supporting streaming analytics at scale.

You can analyze streaming events in real-time, augment events with additional data before loading the data into a system of record, or power real-time monitoring and alerts.

Stream processing does not always eliminate the need for batch processing. Traditional batch processing may be necessary to provide a comprehensive view of historical data – think of BI reports, which may access data from a system of record that is much older than the data that lives in your stream processing platform. And batch processing enables organizations to leverage existing investments for use cases where the urgency of reacting to data is less important. In some architectures, the stream processing platform and batch processing system may sit side-by-side, or stream processing may occur prior to batch processing.

Hazelcast Jet

Hazelcast Jet and Stream Processing

Stream processing Eengine

Hazelcast Jet provides the tooling necessary to build streaming data applications. It gives you a powerful processing framework to query the data stream and elastic in-memory storage to store the results of the computation.

Large data volumes

Hazelcast Jet processing tasks, called jobs, are distributed across the Jet cluster to parallelize the computation. Jet is able to scale out to process large data volumes.

Integration with IMDG

Hazelcast Jet has very high-speed integration with Hazelcast IMDG, which can store large amounts of data to join it to the Jet stream in microseconds. Latency can also be reduced by using IMDG for stream ingestion or publishing results.

Windowing

Hazelcast Jet works with streaming data in terms of “windows," where a window represents a slice of the data stream, usually constrained for a period of time.

Jet supports Tumbling, Sliding and Sessions Windows.

Event time and late events

Hazelcast Jet supports the notion of “event time,” where events can have their own timestamp and arrive out of order. This is achieved by inserting watermarks into the stream of events that drive the passage of time forward.

Fault tolerance

Hazelcast Jet provides simple fault-tolerant streaming computation with snapshots saved in distributed in-memory storage. Jobs restart automatically using the snapshots, and processing resumes where it left off.

Processing guarantees

The need to trade-off performance and correctness in event processing systems may not allow firm guarantees. Hazelcast Jet allows you to choose a processing guarantee at start time, choosing between no guarantee, at-least-once, or exactly-once.

10s of 1000s

Transactions per second

High-Speed streaming data from multiple sources, devices, and networks

1.2 microseconds

Maximum application latency

Leverage high-speed stream processing with in-memory performance

A high-speed solution for a high-speed world
In-memory streaming is designed for today’s digital ecosystem, with billions of entry points streaming data continuously, with no noticeable delays in service. Ever.

The speed of in-memory, optimized for streaming data
The speed delivered by in-memory systems can accelerate data performance by a factor of 1000X. Adding stream processing accelerates this further, through pre-processing of data prior to ingestion.

Applicable to any process that would benefit from higher performance
Is there a single application in your business that would work better at a slower rate? Stream processing is not just faster, it’s significantly faster, which opens up new opportunities for innovation.

Free Hazelcast Online Training Center

Whether you're interested in learning the basics of in-memory systems, or you're looking for advanced, real-world production examples and best practices, we've got you covered.