Hazelcast IMDG 4.1 GA is Released!

We are excited to announce the production-ready release of Hazelcast IMDG 4.1.

Below, we highlight the most notable features and enhancements implemented in this release. You can find the list of changes in the release notes. Before we dive into the details, we want to share a few numbers:

• It took exactly 9 months since the last minor, in this case also major, release of 4.0.
• We closed more than 250 GitHub issues.
• Almost 700 pull requests got merged.
• 30+ external community contributors to various modules and integrations.

Shoutout to Community

The development period of this release was highly influenced by increased interest by the community, including successful projects as a part of Google Summer of Code and new contributors coming through Hacktoberfest. Simultaneously, Hazelcast IMDG passed another milestone by reaching more that 4,000 stars on GitHub. We would like to thank the community for its continued interest and growing involvement.

Let’s go through the new features and enhancements in IMDG 4.1.

Initial SQL Support

In previous releases, Hazelcast IMDG supported querying and aggregating over maps using an SQL-like syntax through the existing query engine. Still, the existing implementation has its limitations and there was a strong voice from the community that it should have additional features. By far the biggest feature that we delivered this release is a preview of a brand new, state-of-the-art SQL engine. In this release, the feature set is limited, but it already includes:

• Support for many new expressions
• The ability to efficiently query large amounts of data
• The option to make use of a new, high-performance concurrent off-heap B+ tree
• More advanced query optimization

We currently do not support features like joins, sorting and aggregations. However, we are committed to continuing the SQL support effort in the nearest and even long-term future to incrementally deliver the best-distributed SQL engine. Additionally, we’re also expanding SQL support to Hazelcast Jet – in an upcoming release, you will be able to create jobs and use SQL to manipulate streams of data (for example, windowing on streaming data).

Wait no more to try out the new gem of Hazelcast IMDG in the SQL Reference Manual chapter.

Deployment of Portable Domain Objects Without Restarts

One of the major pain points in previous releases was if users wanted to use EntryProcessors, Runnables or Callables with their own custom domain object, the domain object class had to be available on the classpath before the member starts. This in turn means that if you wanted to introduce a new domain object, you had to essentially restart the cluster. This posed a significant gap in usability and user convenience.

In this release, we introduced a GenericRecord API that allows you to build the classes on the fly and eliminate the need for a restart. You can plug in your new object and freely use it right away in your EntryProcessors and Runnables/Callables without further hassle.

Configuration Entries Overwrites Without Configuration File Changes

Imagine you want to deploy Hazelcast IMDG members into Kubernetes. In order for the members to find each other, you need to alter the configuration, even though in a minor way.

hazelcast:
  network:
    kubernetes:
      enabled: true
      service-name: hazelcast

Unfortunately, it’s not that simple since this approach creates a few dilemmas like where to put the configuration file and how to ensure it’s available all the time and to all pods.

Focusing on improving the experience for flexible setups on-premises or in the cloud, we came up with a mechanism that allows you to overwrite any configuration entries using either an environmental variable or a system property. Using this option, you can reduce the whole process of deploying a configuration file, making all members able to access that resource and worrying about high availability of it down to this:

$ java -Dhz.network.kubernetes.enabled=true -Dhz.network.kubernetes.service-name=hazelcast -jar hazelcast-all-4.1.jar

Discover the full power of the feature in the Overriding Configuration documentation chapter.

Discovery Strategy Auto Detection

Concentrating on the user experience didn’t end with the improvement above but rather enabled us to take usability to the next level. Let’s reuse the Kubernetes configuration example above. Even though it looks pretty easy already, what happens if you want to try to set up a cluster in AWS or Azure? You have to change the configuration again.

In order to make member discovery even simpler, Hazelcast IMDG introduces a discovery auto-detection mechanism. By specifying one simple configuration line, the member automatically detects in which environment it’s running and automatically chooses the discovery strategy. Therefore, forming a cluster in any of the major cloud vendors can’t be made any simpler than a single configuration entry change. Or can it?

Combined with the configuration override enhancement, we’re talking about further simplification of a one-liner:

$ java -Dhz.network.join.auto-detection.enabled=true -jar hazelcast-all-4.1.jar

While it might seem that this is the end of the simplicity journey, we thought differently. That’s why we made the auto-detection enabled by default and therefore, you are now able to form a cluster in AWS, Azure, GCP, Kubernetes and others via:

$ java -jar hazelcast-all-4.1.jar

…or even…

$ hz start

Installation Packages Through Brew, RPM and DEB

Wait! What is that ‘hz’ command in the example above? Starting with Hazelcast IMDG 4.1, we’ve introduced the first version of the command line interface (CLI). With the first release of the tool, installing Hazelcast IMDG and starting a member has never been easier and can be completed in 40 seconds.

We also support RPM and DEB packages:

# DEB packages
$ wget -qO - https://bintray.com/user/downloadSubjectPublicKey?username=hazelcast | apt-key add -
$ echo "deb http://dl.bintray.com/hazelcast/deb stable main" | tee -a /etc/apt/sources.list
$ apt update && apt install hazelcast

# RPM packages
$ wget https://bintray.com/hazelcast/rpm/rpm -O bintray-hazelcast-rpm.repo
$ mv bintray-hazelcast-rpm.repo /etc/yum.repos.d/
$ dnf install hazelcast     # Fedora 22+ and Red Hat Enterprise Linux 8+
$ yum install hazelcast     # Older Fedora and Red Hat Enterprise Releases

No more ‘java’ command to ease up the life of our non-Java users.

Currently, the CLI is not intended for production usage but rather allowing developers to get hands-on and familiarize themselves with Hazelcast IMDG as soon as possible. More functionality and production usage readiness are on the roadmap.

Parallel Migrations and Improved Failure Detection Mechanisms

In distributed systems, network failures or environmental instability is inevitable. One of the key strengths of Hazelcast IMDG has always been reliability and resilience against those scenarios. Hazelcast IMDG cluster handles them automatically with no user interaction required. In the latest release, we invested in improving those mechanisms even more with two major enhancements: parallel partition migrations and partial member disconnection resolution.

Newly, Hazelcast IMDG members do the migrations in parallel. That results in a drastic reduction of time needed for rebalancing to complete if a failure within the cluster happens. Thus, shortening the time the cluster is in a suboptimal state. As an example, according to our tests, should a network disconnect one node in a 10-node cluster responsible for storing four terabytes of data, Hazelcast IMDG will be able to complete partition rebalancing in approximately 2 minutes when it would have previously required at least 33 minutes.

In addition to improvements on “after something bad happened,” we made the “if something bad happened” part smarter as well. With the help of the Bron-Kerbosch algorithm, the cluster is now able to detect more complicated failures, such as a member being able to connect only to some members of the cluster but not all of them. In that case, the cluster will “kick out” the misbehaving member and ensure stability.

Both of the improvements above are available out-of-the-box, so you can start benefiting from them as soon as you upgrade to the IMDG 4.1.

Better CP Management Capabilities

Back in the 3.12 release, Hazelcast IMDG introduced the CP subsystem offering strong consistency guarantees for use cases where it matters the most. Since then, we continued improving the CP subsystem which resulted in it becoming the most popular Java implementation of the Raft algorithm.

This release follows up on that trend and brings a new CP Subsystem listener API. Using it, you can now watch, and therefore immediately react to, events such as a Raft member added or removed, a decrease in the level of availability or Raft majority group completely lost. Leveraging those APIs can ensure that your critical, strongly consistent infrastructure’s failures never go unnoticed and unhandled properly.

Security Improvements (Enterprise Only)

Enterprise edition now benefits from out-of-the-box support for member and client authentication based on Kerberos. This feature uses the JAAS login module from Hazelcast 4.0 and allows integration with LDAP for role mapping. In summary, plugin Hazelcast IMDG into your Kerberos authentication standard has never been easier.

We’re also providing support for audit logging, so integrating Hazelcast into environments with strict standards and legal requirements becomes more straightforward.

Intel Optane Improvements (Enterprise Only)

Continuing to build on our partnership with Intel®, we focused on improving our support with Intel® Optane™ Persistence Memory Modules. Not only have we fixed an issue with exploiting the full capacity of all installed modules on the system, Hazelcast IMDG now provides performance tuning options to get the speed of Optane up by even 50% for some use cases, providing near-RAM like speeds.

To be more specific, we found out that Optane’s performance can be quite hit by NUMA remote access. In order to overcome this, we introduced thread affinity to be able to specify the mapping between threads and NUMA nodes. While thread affinity is targeted at very experienced users, the benefits for Optane are promising and should help to squeeze out every last drop of performance.

Closing Words

We would be tremendously grateful for your feedback. Once again, don’t hesitate to reach out via Slack, Google Groups or Twitter. If you’re interested in the feature design details, we have also published new design documents on GitHub. Check them out!

Once again, thank all the community contributions. You can find the list of them in the release notes. Every engagement is highly appreciated, whether it’s a pull request to the production code, reporting issues, spotting typos in the documentation or just getting back to us with feedback.

Happy Hazelcasting!