Solution: The Hazelcast Jet Approach to ETL and Data Ingestion
Hazelcast Jet provides all the necessary infrastructure to build and run real-time ETL applications, so you can focus on the business logic of your data pipelines. Key components of Hazelcast Jet include:
- Pipeline API to declaratively define your data pipelines
- Connectors for extracting data from sources and loading it into sinks
- Runtime for executing data pipelines with fault-tolerance and parallel execution at scale
Hazelcast Jet can move data between a variety of systems, including Hazelcast IMDG, which is often used for operational storage or as a distributed cache. Hazelcast Jet is a very convenient tool for keeping in-memory caches hot through real-time ETL.
One popular data ingestion use case is loading event streams from Kafka into Hazelcast IMDG, essentially creating a materialized view on top of the stream for real-time querying. Learn more about loading data into Hazelcast IMDG using Jet.
How It Works
Hazelcast Jet was built for developers by developers. Therefore, its primary programming interface is a Java-based DSL called the Pipeline API, which allows you to declaratively define the data processing pipeline by composing operations against a stream of records. Common operations include filtering, transforming, aggregating, joining, and data enrichment. The Pipeline API is similar to java.util.stream. However, it has been designed to support distributed stream processing as a first-class citizen.
Connectors for Extracting and Loading Data
Hazelcast Jet provides a variety of connectors for streaming data into Hazelcast Jet pipelines and storing the results to sinks such as Hazelcast IMDG, Java Message Service, JDBC systems, Apache Kafka®, Hadoop Distributed File System, and TCP Sockets. Also, Hazelcast provides a convenience API so you can easily build custom connectors.
Running Data Pipelines
The heart of Hazelcast Jet is a high-performance execution engine. Once deployed, Hazelcast Jet performs the steps of the data pipeline concurrently, making use of all available CPU cores. Hazelcast Jet processes partitioned data in parallel. Hazelcast Jet processes data continuously, performing with millisecond latencies. The Hazelcast Jet architecture enables you to process hundreds of thousands of records per second with millisecond latencies using a single Jet node.
Fault-Tolerant, Scalable Operations
ETL jobs have to meet strict SLAs. If there is a failure in the system, the jobs cannot simply restart and still meet the business deadlines.
Hazelcast Jet uses checkpointing to enable continuity. Checkpoints are regularly taken and saved in multiple replicas for resilience. In the event of a failure, an ETL job is rewound back to the most recent checkpoint, delaying the job for only a few seconds rather than starting from scratch.
Hazelcast Jet clusters are elastic, allowing dynamic scaling to handle load spikes. You can add new nodes to the cluster with zero downtime to linearly increase the processing throughput. Learn more about how Jet makes your computation elastic.
Hazelcast Jet Briefing for the Financial Services Community
Performant, elastically scalable, and resilient, Hazelcast Jet has been used extensively in financial services for a wide range of real-time use cases. In this webinar, Hazelcast Jet product manager Vladimir Schreiner will provide an overview of Hazelcast Jet, and how it is used at some of the world’s leading banks and credit card providers.
Getting Started with Stream Processing
Java Champion, Ben Evans, will provide an introduction to stream processing and teach more about core techniques and how to get started building a stream processing application using real world use cases and live demos.
Hazelcast Jet for Hazelcast IMDG Users
Hazelcast Jet® is a 3rd generation stream processing engine that adds advanced data processing capabilities to Hazelcast IMDG®. Jet makes it simple to build distributed, fault-tolerant data processing pipelines on top of Hazelcast IMDG and provides 500% performance increase over similar processing done with Apache Spark. Just like Hazelcast®, it can be embedded into your application or run as standalone.