Hazelcast Jet

Why are DAGs relevant to Jet processing? Jet uses DAGs to represent the execution plan. When running jobs, Jet identifies the available resources (i.e., CPU cores) and deploys tasks across the cluster to achieve the highest levels of performance. Jet simplifies this process for you by determining the optimal configuration of tasks and deploying the tasks for you, freeing you from the otherwise complicated effort of identifying where to run tasks and at what levels.

To accomplish this, Jet models its stream processing pipelines as DAGs, which lay out the tasks of a job and how they interact. It then optimizes the DAG to leverage parallelism for performance and efficiency of jobs. In the DAG representation, vertices represent the computational steps, and edges represent the data flows. Each vertex receives data from its inbound edges, performs a step in the computation, then emits data to its outbound edges. The breakdown of a job into separate vertices is possible thanks to data partitioning, so that subtasks in the overall job can be processed in parallel, independently of each other.

The key point here is that you do not have to worry about DAG planning. Jet does it all for you, so simply write your processing code using the Jet Pipeline API, and the Jet engine will take care of the runtime.

In the diagram, you can see how tasks can be distributed across cores and nodes to run in parallel. The use of cooperative threads (i.e., application-level threads) lets Jet more efficiently run that parallelism without incurring the overhead of context switching in OS-level threads. Jet coordinates the threads to deliver superior performance with no planning required on the application developer.

This visual representation of a DAG for a Jet job, as displayed in Management Center. You can see how a job is broken down into distinct tasks, and the processed data is delivered to separate sinks (i.e., destination repositories).