Streaming and IMDG Coming Together: Hazelcast Platform 5.0 is Released!

We are proud to announce the first beta release of Hazelcast Platform 5.0. Attentive readers might notice the name of the release is just “Hazelcast Platform”, and not “Hazelcast IMDG” or “Hazelcast Jet”.

In this blog post, we’ll explain the changes in the product name and how it relates to the IMDG and Jet products. Additionally, we’ll deep dive into the technical features coming in this major release. Make sure to review the new features because Hazelcast Platform 5.0 is packed with quite a few new shiny features.

What is Hazelcast Platform 5.0?

In the recent past, we observed emerging use cases that were a great fit for Hazelcast Jet such as digital integration hub or enrichment of streams of data with a static one. On the other hand, our world-class user base of Hazelcast IMDG continued to run the more traditional OLTP use cases.

Although Hazelcast Jet is built on top of Hazelcast IMDG, the split created massive confusion among our users about whether they should use IMDG or Jet in their particular case. In order to sort out this confusion, we decided to merge the Hazelcast IMDG and Hazelcast Jet products into a single one, simply named Hazelcast Platform.

In short, Hazelcast 5.0 can be thought of as a successor of Hazelcast IMDG 4.2 with added all the streaming and pipeline capabilities of Hazelcast Jet 4.5.

As a side note, you might come across the term “Hazelcast Platform” on our website and marketing materials. It’s a synonym for the newly merged Hazelcast 5.0+ product and is used in places where we need to distinguish between Hazelcast “the product” and Hazelcast “the company”.

Compatibility with IMDG and Jet

We focused on keeping the compatibility with the existing IMDG users. Therefore, Hazelcast Platform 5.0 is fully compatible with Hazelcast IMDG 4.x:

• There are no incompatible API changes. You don’t have to change your application code when upgrading from Hazelcast IMDG 4.x.
• The 4.x clients can connect to Hazelcast 5.0, 
• You can perform a zero-downtime rolling upgrade from 4.x to 5.0.

Additionally, we also provide a zero-downtime upgrade path for the Hazelcast IMDG 3.x users. However, in this case, some application code changes are required. To learn more about the upgrade path from Hazelcast IMDG to Hazelcast 5.0, refer to the Migrating Between IMDG and Platform 5.0 documentation section.

For the existing Jet users, the upgrade will require stopping and resuming the jobs just like it was the case while upgrading between Jet releases. For details, see Migrating Between Jet and Platform 5.0 documentation section. 

New Features

Hazelcast 5.0 is not just about that but also about major feature updates. In this section, we highlight the most notable ones but make sure to visit the release notes.

Expanded SQL Support and Production Readiness

As stated previously during several IMDG and Jet releases, our investment into SQL support continues. Hazelcast 5.0 takes a big leap forward and brings a lot of new expressions:

• INSERT, UPDATE and DELETE with IMap
• Sorting (ORDER BY) and aggregations (GROUP BY) without any limitations
• Basic JOIN support
• New operators such as IN, BETWEEN or CASE
• New functions, e.g. REPLACE, POSITION, EXTRACT, POWER, SQRT and more.

To see the full list of features, see the SQL reference. Additionally, with Hazelcast 5.0 GA which is to be released, we are lifting the beta status of the whole SQL engine and are ready to present it as production-ready. For the list of all SQL capabilities refer to the SQL reference.

 

Persistence (Enterprise Edition)

For years, we provided persistence capabilities through MapLoader/Store SPI. However, the user had to provide the actual implementation by herself. Another option was to leverage Hot Restart capabilities but in the practical application it turned out to be quite limited in certain scenarios.

We introduce new persistence functionality, simply referred to as Persistence. This feature completely supersedes the Hot Restart subsystem which in turn will be deprecated. With Persistence, the content of your IMap and ICache gets written to disk making it even more resilient towards failures. At this point, Persistence is a mirror of the in-memory content, it’s not a tiered storage for going beyond memory limits.

Using Persistence, you can newly achieve the following scenarios just with simple configuration change:

• Safely restore the member data after an unexpected crash.
• You can control whether the cluster should start rebalancing the data. For example, if you’re willingly shutting down one member just for maintenance with the intention of bringing it up soon, you don’t need Hazelcast to do the whole migration and data shuffling. Newly, you can control such behavior and thus, shorten your maintenance window to an absolute minimum.
• Transfer a member to another hardware with reduced migration times. You can take the persistence files, put them on a new machine and start the member there. The member will load the data and do only the necessary (delta) migrations drastically reducing the maintenance window needed.

Lastly, an example of the configuration follows below. For the full documentation of the Persistence feature, visit the respective Persistence documentation section.

Streaming

Although the current Hazelcast Jet users might not get that excited, it’s worth explicitly mentioning that Hazelcast 5.0 effectively brings all the battle-proven streaming capabilities to Hazelcast IMDG. 

Former IMDG users can upgrade to Hazelcast 5.0 with no risk and zero downtime enabling use cases like:

• analyze data using querying or streaming,
• add a context to the real-time events,
• combine Hazelcast data with data from remote systems, and
• ingest data using SQL or Java through a rich suite of connectors for out-of-the-box connectivity to databases and message brokers, both could and self-managed.

Not sure where to start? Get started with stream processing in the client/server mode using Java or via SQL.

Increased HD Capacity (Enterprise Edition)

Up until now, the maximum supported amount of high-density memory configured per member was 100 GBs. We’re lifting up this limit to 200 GBs per member effectively doubling the maximum storage capacity on the same hardware. For the end-users, this means reducing the hardware cost tremendously and making the life of your DevOps team easier.

Compact Serialization Format (Preview Feature)

In addition to the current serialization formats, we’re releasing a preview of the new serialization format named Compact Format. Compact Format aims not only to improve the non-functional aspect of the serialization but also the user experience.

In particular, Compact Format’s goals are:

• Require much less storage space
• Provide better latency/throughput
• Support much more extensive typeset
• Require much less configuration than Portable (zero configuration on common use-cases for Java only)
  • This format supports the schema evolution of a class and does not require any version to be specified explicitly.
  • Java DTO classes are automatically serialized without requiring a Serializer to be implemented on common use cases where the field types of an object are natively supported via Compact Format and class has a default constructor.
  • User classes do not need to implement an interface like Portable. Once compact serialization is enabled, the classes which do not match any other serialization methods will be serialized in the new format. 

To learn about the format, see Compact Serialization documentation .

Management Center UI redesign

In addition to many new features, our web-based monitoring tool Management Center has gone through a complete UI redesign. We’re bringing a consistent, modern look focused on improved user experience with the beta release of Management Center 5.0. 

Find out more about the Management Center in the Management Center Getting started guide.

Future of IMDG and Jet

We do not plan to release any new minor versions of either Hazelcast IMDG or Hazelcast Jet. All the new feature development will continue in the new Hazelcast product. However, we’ll continue to support IMDG and Jet for the standard version support windows, as well as provide the patch releases of the supported versions. 

Therefore, if you’re an existing user, you don’t have to be worried about your favorite in-memory data grid and streaming engine disappearing. It’s more of a rebranding and setting up a new direction.

Closing Words

We would love to hear from you what you think. You can reach us directly via Hazelcast Community Slack or Google Groups for discussions, or report issues on the GitHub repository.

Happy Hazelcasting!

Useful Links:

• Download Open source or Enterprise Edition
• Documentation
• Release notes