Hazelcast IMDG 4.0 is Released

I joined Hazelcast in January 2014 as employee number 12. At that time Hazelcast 3.1 was the current version. Since then, about every 7-8 months we have done a “minor” release, the latest of which is 3.12. Minor in the sense that they maintained binary compatibility. We added many features and improved many aspects of Hazelcast, often driven by the needs of our users.

We also grew the engineering team from 6 engineers to almost 50. At this point, Hazelcast has several hundred person-years of effort put into the code.

We believe strongly in backward binary compatibility which is why we have kept to the 3.x major version. But the gradual accumulation of deprecated APIs clutters things up. Also, with the new CP subsystem introduced in 3.12, we had now deprecated the older AP factory methods.

So, today we’re happy to announce the general availability of Hazelcast IMDG 4.0. Consistent with semantic versioning, by changing the major version number we are taking the opportunity to make some fundamental changes to the core engine of Hazelcast, modernize the APIs and move the minimum Java version up to Java 8 at the price of changing binary compatibility.

Removal of deprecated APIs

This release sees a clean-up of deprecated APIs such as the MapReduce functionality which has been superseded by Hazelcast Jet and IMDG Aggregations.

Also, we’ve refactored a few other interfaces, such as the Migration Listener. Now it’s easier than ever to observe migration events in the cluster.

In general, any Hazelcast IMDG 3.x API that was deprecated has been removed in 4.0.

Migrating to IMDG 4.0

If you were using one of these APIs you will get a compile-time error when you migrate to IMDG 4.0. We anticipate that most users should be able to migrate to the new APIs within a few hours.

A key thing to note is that the client protocol has also been updated in 4.0. This means that you cannot use a 3.x client with 4.0 members. This is something that only changes on a major release. Once you upgrade your clients to 4.x then you can continue to use them with various 4.x Member versions.

Hazelcast has rolling upgrades, which allow you to move up a minor version without shutting down the cluster. As this is a major version change, you will need to schedule a maintenance window for the upgrade.

See the Migration Guide for details.

Concurrent primitives moved to CP subsystem

In Hazelcast IMDG 3.12 we introduced the CP subsystem, strong consistent implementation of our concurrency primitives such as FencedLock, AtomicLong & Semaphore, all based on the Raft consensus protocol. We’ve listened to the community and have now introduced an unsafe mode of cluster operation that allows the CP subsystem to function without a minimum of 3 members. This mode helps with development situations where running 3 member clusters is overkill, but also where systems wish to operate in AP mode. In Hazelcast IMDG 4.0 the only way you can access those primitives is via the CP subsystem – we removed the older factory methods which run of the AP system.

For the Hazelcast IMDG 4.0 Pro version, we’ve added the ability to persist all of the states in the CP subsystem. This capability significantly improves the overall reliability of the CP subsystem by recovering from member and cluster crash scenarios, without sacrificing safety guarantees. The Reference Manual provides further information on how to set this up.

Our entire CP subsystem implementation, including the CP subsystem persistence feature, is fully covered by the Jepsen testing framework.

Other features in IMDG 4.0

We also managed to add some new features and improvements.

Improved Performance

Standardised tests run in our labs show an overall throughput improvement of 35% with more predictable latency. Additionally, we’ve been able to make changes to our network protocols that have yielded an impressive improvement for writes. This builds on other significant performance improvements made over the past few years.

Intel Optane DC Support

Hazelcast has offered Intel Optane DC Persistent Memory support for our Hot Restart persistence layer for some time now. In IMDG 4.0 we’ve added the ability to use Intel Optane DC with our IMap, ICache and NearCache data structures, in conjunction with our off-heap HD memory engine. No major code changes are required to make use of this exciting new feature; simply amend the data structure configuration. Performance for typical values of up to 1KB is the same, but the cost is half.

Security

We’ve added Encryption at Rest to our Hot Restart persistence layer. This feature allows for symmetric encryption and key rotation without system downtime. The encryption keys can be sourced from external systems such as Hashicorp Vault or from a standard Java Keystore file.

Hazelcast has always had the ability to connect to LDAP for Authentication and Authorisation, but now we’ve made this even easier by providing an out-of-the-box connector that will link a Hazelcast IMDG cluster to an LDAP or Active Directory back-end using standard configuration.

Whilst we’re on the subject of Authentication, in IMDG 4.0 we’ve also added the ability for clients to provide Identity to RBAC via X509 Client Certificates.

More technical details on LDAP and X509 certificates can be found in the Reference Manual.

A shout out to our community

Finally, a shout out to the community. We’ve had some great contributions and I’d like to give a special thank you to Wojciech Gdela, Alex Dukhno and Sergey Bespalov for providing contributions that held up to the highest standards at Hazelcast. Thank you, you’re awesome!

Looking forward – what else is coming in 2020

The 3.x line of software cemented Hazelcast IMDG as the most performant and reliable in-memory data grid available; its popularity with developers surpassing anything else. With IMDG 4.0 we now have a basis for some exciting new developments that will drive Hazelcast into new use cases.

Some new features we plan to introduce in 4.1:

  • ANSI standard SQL Select
  • JDBC
  • Re-engineered query system
  • Advanced persistence for AP data structures such as JCache and IMap to complement CP persistence. The idea is that you can dispense with a system of record backing Hazelcast if you wish
  • Larger memory per member and per cluster to better utilize the lower cost of storing data in Intel Optane DC Persistent Memory

The other big one coming even sooner is an enterprise version of our managed cloud service, starting with AWS and followed by Azure and GCP. This will bring new convenience and simplicity to using Hazelcast.

Hazelcast IMDG 4.0 open source and enterprise are available for download today. You can also get started with Hazelcast Cloud for free.