Use Cases

ETL and Data Ingestion

ETL is an acronym for “extract, transform, load.” Extract refers to collecting data from some source. Transform refers to any processes performed on that data. Load refers to sending the processed data to a destination, such as a database. ETL is a data processing concept dating back to the 1970s, but it remains important today because it is one of the most dominant frameworks for providing people and applications with data. Engineering and product teams load and preprocess data from a variety of sources to a number of destinations with ETL techniques and software.

Solution: The Hazelcast Jet Approach to ETL and Data Ingestion

Hazelcast Jet provides all the necessary infrastructure to build and run real-time ETL applications, so you can focus on the business logic of your data pipelines. Key components of Hazelcast Jet include:

  • Pipeline API to declaratively define your data pipelines
  • Connectors for extracting data from sources and loading it into sinks
  • Runtime for executing data pipelines with fault-tolerance and parallel execution at scale

Hazelcast Jet can move data between a variety of systems, including Hazelcast IMDG, which is often used for operational storage or as a distributed cache. Hazelcast Jet is a very convenient tool for keeping in-memory caches hot through real-time ETL.

One popular data ingestion use case is loading event streams from Kafka into Hazelcast IMDG, essentially creating a materialized view on top of the stream for real-time querying. Learn more about loading data into Hazelcast IMDG using Jet.

How It Works

Hazelcast Jet was built for developers by developers. Therefore, its primary programming interface is a Java-based DSL called the Pipeline API, which allows you to declaratively define the data processing pipeline by composing operations against a stream of records. Common operations include filtering, transforming, aggregating, joining, and data enrichment. The Pipeline API is similar to java.util.stream. However, it has been designed to support distributed stream processing as a first-class citizen.

Connectors for Extracting and Loading Data

Hazelcast Jet provides a variety of connectors for streaming data into Hazelcast Jet pipelines and storing the results to sinks such as Hazelcast IMDG, Java Message Service, JDBC systems, Apache Kafka®, Hadoop Distributed File System, and TCP Sockets. Also, Hazelcast provides a convenience API so you can easily build custom connectors.

Running Data Pipelines

The heart of Hazelcast Jet is a high-performance execution engine. Once deployed, Hazelcast Jet performs the steps of the data pipeline concurrently, making use of all available CPU cores. Hazelcast Jet processes partitioned data in parallel. Hazelcast Jet processes data continuously, performing with millisecond latencies. The Hazelcast Jet architecture enables you to process hundreds of thousands of records per second with millisecond latencies using a single Jet node.

Learn more about Hazelcast Jet performance

Fault-Tolerant, Scalable Operations

ETL jobs have to meet strict SLAs. If there is a failure in the system, the jobs cannot simply restart and still meet the business deadlines.

Hazelcast Jet uses checkpointing to enable continuity. Checkpoints are regularly taken and saved in multiple replicas for resilience. In the event of a failure, an ETL job is rewound back to the most recent checkpoint, delaying the job for only a few seconds rather than starting from scratch.

Hazelcast Jet clusters are elastic, allowing dynamic scaling to handle load spikes. You can add new nodes to the cluster with zero downtime to linearly increase the processing throughput. Learn more about how Jet makes your computation elastic.

Related Resources