Hazelcast IMDG 4.0 BETA is Released Matko Medenjak November 05, 2019 Share It is my pleasure to announce that after 6 years a new major version of Hazelcast has been released! This new release brings a breath of fresh air into Hazelcast while also being more robust than ever. We did try to keep enough familiarity to not surprise our users too much, but we also invested a lot of time into bringing more intuitive and simpler API and SPI, as well as added some cool features to boot! This release also introduces lots of preparations and freedom for extending Hazelcast in the future. So, I’d definitely recommend keeping an eye on Hazelcast for things to come! The major benefit of this freedom to change is that we’ve had an opportunity to make some major improvements in terms of the general performance of Hazelcast. As you can see in the graph below, our standardized lab tests show that Hazelcast IMDG 4.0 provides 35% more throughput at a more predictable rate. Also, thanks to some changes in the way our Clients communicate with the Cluster we’ve also managed to make some impressive improvements to Client-Cluster latency, especially for Writes. Now the fastest in-memory data grid is even faster. Lastly, I’d like to give a shout out to the community. We’ve had some great contributions and I’d like to give a special thank you to Wojciech Gdela, Alex Dukhno and Sergey Bespalov for providing contributions that held up to the highest standards at Hazelcast. Thank you, you’re awesome! Now, onto the cool stuff! CP Subsystem Unsafe Mode Currently to use the CP subsystem, at least 3 nodes are required. But this requirement makes it difficult to test and develop applications. Also, some users do not need the safety guarantees of a CP system but just need the functionality of data structures exposed by CP subsystem with one or two nodes. With Hazelcast 4.0, we allow access to CP data structures even with a single node without providing strong safety guarantees. This mode is called the “unsafe” mode and it replaces concurrent data structures previously deprecated in Hazelcast 3.12 (ILock, IdGenerator, IAtomicLong, IAtomicReference, ICountDownLatch, ISemaphore). For more information, please take a look at the reference manual: CP Subsystem Unsafe Mode Removal of Deprecated Concurrency API Implementations MapLoader with Custom TTL We’ve added the capability to load and store expiration times when loading and storing entries using MapLoader and MapStore. Existing MapLoader and MapStore implementations should continue to work normally but if you need to handle expiration times, you can implement the new EntryLoader and EntryStore interfaces. For more information, please take a look at the reference manual: Loading and Storing Persistent Data. JSON support for REST You can now more naturally access HazelcastJsonValue instances stored in your map via the REST API. The response will be a UTF8 encoded JSON value with the Content-Type header set to application/json. For more information, please take a look at the reference manual: Retrieving Entries from a Map for REST Client. Removal of ICompletableFuture In Hazelcast IMDG 3.x series, com.hazelcast.core.ICompletableFuture was introduced to enable reactive programming style. ICompletableFuture was intended as a temporary, JDK 6 compatible replacement for java.util.concurrent.CompletableFuture that was introduced in Java 8. Since Hazelcast 4.0 requires Java 8, the user-facing asynchronous Hazelcast API methods now have their return type changed from ICompletableFuture to Java 8’s java.util.concurrent.CompletionStage. By returning CompletionStage, the API has become more familiar and simpler. Integration with other third-party libraries and user-code is also easier since there is no conversion needed anymore. For more information, please take a look at the reference manual: Removal of ICompletableFuture. Refactoring of Migration Listener We’ve redesigned the MigrationListener API to be more intuitive. Previously, on each migration, a pair of migration started and migration completed or failed events were fired but this level of detail is not relevant to most users. Now, we send events when the overall migration process starts, when each partition is migrated and when the overall migration process completes. On each step, you can see how many migrations are planned, how many have completed, how much time has elapsed etc. For more information, please take a look at the reference manual: Refactoring of Migration Listener. Client backup acknowledgment improvements We’ve improved the performance of client operations involving sync backups by decreasing the number of network hops needed to complete the operation. This feature does not change any safety guarantees involving usage of client and is turned on by default. What it does do is provide much faster write operations. Using the default of 1 synchronous backup increases of around 32% have been observed in lab tests. For more information, please take a look at the reference manual: Configuring Backup Acknowledgment. HTTPs for Hazelcast bash scripts We added HTTPs support to cluster.sh, healthcheck.sh and cp-subsystem.sh management scripts. For more information, please take a look at the reference manual: Using the healthcheck.sh Script Using the cluster.sh Script Renaming methods, classes and configuration We’ve renamed methods, classes and configuration to improve clarity and to avoid naming conflicts: we’ve renamed getId() to getClassId() in IdentifiedDataSerializable. The getId() method of the IdentifiedDataSerializable interface is a method with a common name, meaning a naming conflict would happen frequently. For example, database entities also have a getId() method. For more information, please take a look at the reference manual: Renaming getID to getClassId in IdentifiedDataSerializable. we’ve renamed Quorum to Split Brain Protection. This should prevent any similarities between this feature and using a strongly-consistent system like the CP subsystem. For more information, please take a look at the reference manual: Renaming Quorum as Split Brain Protection. we’ve renamed group-name to cluster-name. For more information, please take a look at the reference manual: Renaming group-name as cluster-name. Deprecated classes removal There were a number of deprecated and outdated public API and SPI classes which were kept only for backward compatibility. Some of them are no longer used in Hazelcast itself, while others have a better alternative. We’ve now removed those classes. we’ve replaced the concurrency API implementations with the “unsafe” mode of CP subsystem (see CP Subsystem Unsafe Mode and Removal of Deprecated Concurrency API Implementations). we’ve removed the merge policies specific to each data structure. Now you can use the generic merge policies introduced in 3.10 (see Removal of Legacy Merge Policies) we’ve removed map reduce. For a more complete, versatile and powerful alternative, please take a look at Hazelcast Jet or the Aggregations that uses the Query infrastructure. Also see the reference manual (Removal of MapReduce). we also clearly separated lots of private API classes from public packages. Private API should now be located in packages containing “internal” or “impl”. With Hazelcast 4.0, we are now requiring Java 8 as a minimum to run Hazelcast IMDG. Serialization and performance improvements With Hazelcast 4.0, we’ve been able to do many improvements in serialization and performance. We’ve extended usage of UUID instead of String where applicable, added serialization support for many Java collections like ConcurrentHashMap, HashSet, LinkedHashMap and many others. Hazelcast Instances: Using Moby naming When starting Hazelcast members, each instance would get assigned a generated instance name. Different members are similarly named and when monitoring a cluster through the Management center it can get difficult to distinguish between different members. We now provide a more human-readable (and fun!) name using Moby – names such as hopeful_turing, jovial_chatelet or brave_driscoll. If you’re used to Docker process names this will be familiar to you. Since the instance name will be new after restart it will be easier to track member restarts in logs. For more information, please take a look at the reference manual: Checking the Name of the Instance for REST Client and the hazelcast.member.naming.moby.enabled property. CP Subsystem Persistence (Professional only) CP Subsystem Persistence enables CP members to persist their local state on stable storage. This capability significantly improves the overall reliability of CP Subsystem by recovering from member and cluster crash scenarios, without sacrificing safety guarantees. Our entire CP subsystem implementation, including the CP subsystem persistence feature, is fully covered by the Jepsen testing framework. For more information, please take a look at the reference manual: CP Subsystem Persistence. Encryption at Rest with key rotation (Enterprise only) IMap and ICache entries stored in the Hot Restart may contain sensitive information. This sensitive information may be present in the keys, values, or in both. With Encryption at Rest, entries stored in Hot Restart are encrypted using symmetric encryption. The encryption keys can be fetched from an external system (e.g. Hashicorp Vault) and you can rotate keys while the cluster is operational with no downtime, no performance degradation and with the added safety of gracefully handling situations where the member crashes during re-encryption. Plain old Java KeyStores can also be used. For more information, please take a look at the reference manual: Encryption at Rest. Intel Optane DC Based HD Memory (Enterprise only) Intel® Optane™ DC Persistent Memory is an innovative technology that provides large and affordable data stores. Hazelcast 4.0 offers integration with Intel Optane DC out-of-the-box. With minimum configuration and no code changes, data structures like IMap, ICache and NearCache can now use Optane as a storage layer for off-heap HD memory. For more information, please take a look at the reference manual: Using Persistent Memory. LDAP-based authentication (Enterprise only) Two new JAAS login modules are introduced to authenticate against LDAP servers either by regular or TLS protected connections. With these, information stored in LDAP can be used to verify credentials or load details required for authorization (user-role mapping). Prior to this, users would have to write their own LDAP login modules, this addition now provides an out of the box, ready to go experience for connecting Hazelcast to LDAP. For more information, please take a look at the reference manual: JAAS Authentication Cleanups. X509 Client Certificates (Enterprise Only) In 4.0 Hazelcast can now use X509 Certificates to provide Identity to Roles Based Authentication and Authorisation. Using a new out of the box JAAS login module, a certificate can be examined at connection and subject information can be used to provide a principal to RBAC. Disclaimer and Closing Words For information on upgrading and more information on changes in 4.0, see the migration guide. For another overview of the changes in IMDG 4.0 and the links to pull requests for those changes, see our brand new wiki page. We are eager to hear what you think., so please download Hazelcast IMDG 4.0-BETA-1 and try it out. We do not recommend running Beta versions in production, but we encourage everyone to grab it and play with the new features. We want to hear what you think so please stop by our Gitter channel and Google Group. If you would like to introduce some changes or contribute, please take a look at our or GitHub repository.