Hazelcast Jet 3.0 is Released

We are happy to announce the first General Availability release of Hazelcast Jet, two years after our first public release and three years after the project first started. We named this version 3.0 to match the current major version of Hazelcast IMDG, which is the underlying technology used in Jet. This version also includes several new exciting features.

Pipeline API Backward Compatibility

Over the last few versions of Jet, we have been evolving its pipeline API with new features in each version and some breaking changes. Going forward from 3.0, we are going to be using semantic versioning between releases. This means that the APIs will be backward compatible between minor versions. For a full list of compatibility guarantees, please see the reference manual.

Async Mapping and Filtering

It is now possible to do async mapping and filtering when using a context, for example when querying a remote service. This brings increased performance as it enabled several concurrent remote calls without having to wait for each call to complete. It’s possible to see a full working example of async mapping used for enrichment which calls a gRPC service asynchronously in the code samples repository:

 trades
  .mapUsingContextAsync(contextFactory, (stub, t) -> {
    ProductInfoRequest request = ProductInfoRequest.newBuilder().setId(t.productId()).build();
    return toCompletableFuture(stub.productInfo(request))
           .thenApply(productReply -> tuple2(t, productReply.getProductName()));
  })

Early Window Results

When doing sliding or tumbling windows aggregations, it is now possible to emit early results, before the window has closed. This enables the user to get a preview of a window’s findings and is particularly useful for long-running windows.

p.drawFrom(tradeSource(TRADES_PER_SEC, DURATION_SECONDS))
.withNativeTimestamps(MAX_LAG)
.window(tumbling(SECONDS.toMillis(2))
.setEarlyResultsPeriod(20)
)
.aggregate(summingLong(trade -> trade.getQuantity() * trade.getPrice()))
.drainTo(Sinks.list(VOLUME_LIST_NAME));

The full code sample is available at https://github.com/hazelcast/hazelcast-jet-code-samples/tree/3.0-maintenance/early-window-results

Improvements to Job Submission

In previous Jet versions, it was possible to have two jobs with the same name running simultaneously. Starting with 3.0, this is no longer allowed – only one job with a given name can be running at a time. Furthermore, a new method, JetInstance.newJobIfAbsent(), allows the creation of a job atomically only if it’s not previously running.

This can be useful if multiple clients are in a race to submit the same job, for example in a cloud environment. Using this method guarantees that only a single instance of the job will be submitted.

Live Job Updates and Named Snapshots (Jet Enterprise Only)

It is now possible to update the pipeline for a job by taking a named snapshot and then submitting the updated job to start from the named snapshot. This allows for updating a job without losing its running state. For the full documentation, please refer to the reference manual.

Lossless Restart (Jet Enterprise Only)

Jet can now make use of Hazelcast IMDG Hot Restart to persist the job state to disk automatically. This allows the whole cluster to be shut down and restarted, automatically restarting the job from the latest taken snapshot. For more information about this feature and how to use it, please see the reference manual.

Jet Works on JDK 11

We have now made sure that all Jet code builds and runs correctly on JDK 11 and integrated it into our nightly builds.

New Jet Command Line Interface

Jet now comes with a command line tool which can be used to submit, list and cancel jobs among other things. It replaces the previous `jet-submit` script. For a full list of commands, please see the section in the reference manual.

Improved diagnostics in Jet Management Center

Jet Management Center also brings new features in this release:

1. Improved streaming diagnostics by being able to drill down into each vertex and see the current latencies and last emitted watermarks.
2. Support for displaying map and event journal stats

Hazelcast IMDG version updated to 3.12

As previously mentioned, Hazelcast IMDG is the underlying technology for Jet and past versions used IMDG 3.10. With the recent release of IMDG 3.12, we have updated Jet to include the latest version. For a full list of new features in Hazelcast IMDG 3.11 and 3.12 please see the following blog posts:

• Hazelcast IMDG 3.11 is released
• Hazelcast IMDG 3.12 is released

TensorFlow Integration Sample

We have also added a new code sample showing how Jet could be used to score a TensorFlow model. See the full example at Hazelcast Jet Code Samples repository.

For a full list of changes, please see the release notes.