Hazelcast IMDG 4.1 BETA is Released!

We are proud to announce the release of Hazelcast IMDG 4.1-BETA-1. It is a step toward the first minor release in the Hazelcast IMDG 4.x product line. Being a beta release, it provides a preview of the features coming in future releases and is not intended for production usage.

Below, we highlight the most notable features and enhancements coming in this release. You can find the whole list of changes in the release notes. If you just can’t wait, download it immediately and get started. We’re very interested in your feedback, so don’t hesitate to reach out to us either via Slack, Google Groups or even Twitter.

Initial SQL Support

In previous releases, Hazelcast IMDG supported querying and aggregating over maps using an SQL-like syntax through the existing query engine. Still, the existing implementation has its limitations and there was a strong voice from the community that it should have additional features. 

By far the biggest feature that we’re delivering in this release is a preview of a brand new, state-of-the-art SQL engine, code named Mustang. In this release, the feature set is limited, but it already includes:

• Support for many new expressions
• The ability to efficiently query large amounts of data
• The option to make use of a new, high-performance concurrent off-heap B+ tree
• More advanced query optimization

We currently do not support features like joins, sorting and aggregations. However, we are committed to continuing the SQL support effort in the nearest and even long-term future to incrementally deliver the best distributed SQL engine. 

Additionally, we’re also expanding SQL support to Hazelcast Jet. In the upcoming release, you will be able to create jobs and use SQL to manipulate streams of data (for example, windowing on streaming data).

Wait no more to try out the new gem of Hazelcast IMDG. See the example in SQL Reference Manual chapter.

Deployment of Portable Domain Objects Without Restarts

One of the major pain points in previous releases was if users wanted to use EntryProcessors, Runnables or Callables with their own custom domain object, the domain object class had to be available on the classpath before the member starts. This in turn means that if you wanted to introduce a new domain object, you had to essentially restart the cluster. This posed a significant gap in usability and user convenience.

In this release, we are introducing a GenericRecord API that allows you to build the classes on the fly and eliminate the need for a restart. You can plug in your new object and freely use it right away in your EntryProcessors and Runnables/Callables without further hassle.

Configuration Entries Overwrites Without Configuration File Changes

Imagine you want to deploy Hazelcast IMDG members into Kubernetes. In order for the members to find each other, you need to alter the configuration, even though in a minor way.

hazelcast:
  network:
    kubernetes:
      enabled: true
      service-name: hazelcast

Unfortunately, it’s not that simple since this comes with a few dilemmas like where to put the configuration file and how to ensure its availability all the time and to all pods.

Focusing on improving the experience for flexible setups on-premises or in the cloud, we came up with a mechanism that allows you to overwrite any configuration entries using either an environmental variable or a system property. Using this option, you can reduce the whole process of deploying a configuration file, making all members able to access that resource and worrying about high availability of it down to this:

$ java -Dhz.network.kubernetes.enabled=true -Dhz.network.kubernetes.service-name=hazelcast -jar hazelcast-all-4.1-BETA-1.jar

Discover the full power of the feature in the Overriding Configuration documentation chapter.

Discovery Strategy Auto Detection

Concentrating on the user experience didn’t end with the improvement above but rather enabled us to take usability to the next level. Let’s reuse the Kubernetes configuration example above.

Even though it looks pretty easy already, what happens if you want to try to set up a cluster in AWS or Azure? You have to change the configuration again.

In order to make member discovery even simpler, Hazelcast IMDG introduces a discovery auto-detection mechanism. By specifying one simple configuration line, the member automatically detects in which environment it’s running and automatically chooses the discovery strategy. Therefore, forming a cluster in any of the major cloud vendors can’t be made any simpler than a single configuration entry change. Or can it?

Combined with the configuration override enhancement, we’re talking about further simplification of a one-liner:

$ java -Dhz.network.join.auto-detection.enabled=true -jar hazelcast-all-4.1-BETA-1.jar

While it might seem that this is the end of the simplicity journey, we thought differently. That’s why we made the auto-detection enabled by default and therefore, you are now able to form a cluster in AWS, Azure, GCP, Kubernetes and others via:

$ java -jar hazelcast-all-4.1-BETA-1.jar

or even

$ hz start

First Release of the Command Line Interface

Wait! What is that ‘hz’ command in the example above? Starting with Hazelcast IMDG 4.1-BETA-1, we’ve introduced the first version of the command line interface (CLI). With the first release of the tool, installing Hazelcast IMDG and starting a member has never been easier and is done in 40 seconds.

No more ‘java’ command to ease up the life of our non-Java users.

Currently, the CLI is not intended for production usage but rather allowing developers to get hands-on and familiarize themselves with Hazelcast IMDG as soon as possible. More package managers, functionality and production usage readiness are on the roadmap.

Parallel Migrations and Improved Failure Detection Mechanisms

In distributed systems, network failures or environmental instability is inevitable. One of the key strengths of Hazelcast IMDG has always been reliability and resilience against those scenarios. Hazelcast IMDG cluster handles them automatically with no user interaction required.

In the latest release, we invested in improving those mechanisms even more with two major enhancements: parallel partition migrations and partial member disconnection resolution.

Newly, Hazelcast IMDG members do the migrations in parallel. That results in a drastic reduction of time needed for rebalancing to complete if a failure within the cluster happens. Thus, shortening the time the cluster is in a suboptimal state. As an example, according to our tests, should a network disconnect one node in a 10-node cluster responsible for storing four terabytes of data, Hazelcast IMDG will be able to complete partition rebalancing in approximately 2 minutes when it would have previously required at least 33 minutes. 

In addition to improvements on “after something bad happened,” we made the “if something bad happened” part smarter as well. With the help of the Bron-Kerbosch algorithm, the cluster is now able to detect more complicated failures, such as a member being able to connect only to some members of the cluster but not all of them. In that case, the cluster will “kick out” the misbehaving member and ensure stability.

Both of the improvements above come out-of-the-box, so you start benefiting from them as soon as you upgrade to the GA version of IMDG 4.1 once it’s available.

Better CP Management Capabilities

Back in the 3.12 release, Hazelcast IMDG introduced the CP subsystem offering strong consistency guarantees for use cases where it matters the most. Since then, we continued improving the CP subsystem which resulted in it becoming the most popular Java implementation of the Raft algorithm.

This release follows up on that trend and brings a new CP Subsystem listener API. Using it, you can now watch, and therefore immediately react to, events such as a Raft member added or removed, a decrease in the level of availability or Raft majority group completely lost. Leveraging those APIs can ensure that your critical, strongly consistent infrastructure’s failures never go unnoticed and unhandled properly.

Security Improvements (Enterprise Only)

Enterprise edition now benefits from out-of-the-box support for member and client authentication based on Kerberos. This feature uses the JAAS login module from Hazelcast 4.0 and allows integration with LDAP for role mapping. In summary, plugin Hazelcast IMDG into your Kerberos authentication standard has never been easier.

We’re also providing support for audit logging, so integrating Hazelcast into environments with strict standards and legal requirements becomes more straightforward.

Optane Improvements (Enterprise Only)

Continuing to build on our partnership with Intel®, we focused on improving our support with Intel® Optane™ Persistence Memory Modules. Not only have we fixed an issue with exploiting the full capacity of all installed modules on the system, Hazelcast IMDG now provides performance tuning options to get the speed of Optane™ up by even 50% for some use cases, providing near-RAM like speeds.

To be more specific, we found out that Optane™’s performance can be quite hit by NUMA remote access. In order to overcome this, we introduced thread affinity to be able to specify the mapping between threads and NUMA nodes. While thread affinity is targeted at very experienced users, the benefits for Optane™ are promising and should help to squeeze out every last drop of performance.

What’s Next?

Now that beta release is out, we’re switching into the stabilization phase to fix all the bugs to ensure the world-class quality and reliability that Hazelcast IMDG always provided. We expect to release the GA version in late October/early November. Stay tuned!

Closing Words

We would be tremendously grateful for your feedback. Once again, don’t hesitate to reach out via Slack, Google Groups or Twitter. If you’re interested in the feature design details, we have also published new design documents on GitHub. Check them out!

We must now thank all of the community contributions. You can find the list of them in the release notes. Every engagement is highly appreciated, whether it’s a pull request to the production code, reporting issues, spotting typos in the documentation or just getting back to us with feedback.

Happy Hazelcasting!