Announcing Hazelcast IMDG 3.8

I am very excited to announce the release of Hazelcast IMDG 3.8. My personal impression is that it is one of our best releases! It includes both new valuable features and many useful improvements in the underlying infrastructure. The testing period was very smooth – we encountered fewer failures and issues than in the previous release cycle, which makes me confident of its quality.

Let’s have a look at the new features:

Scheduled Executor Service

Scheduled Executor Service was one of the most requested features by our community. See: https://github.com/hazelcast/hazelcast/issues/115 Now, you will be able to schedule tasks in your cluster at a given moment in time or at fixed intervals.

Open Sourcing Continuous Query Cache

Continuous query cache has been supported since Hazelcast Enterprise 3.5. We decided to make it open source in 3.8 in order to make this feature available to our great open source community. Continuous query cache combines the cache content results with a stream of events to keep the cache up-to-date. This is especially beneficial when you need to query the distributed IMap in a very frequent and fast way. By using a continuous query cache, the result of the query will be always ready and local to the application. Have a look here: http://docs.hazelcast.org/docs/3.8-EA/manual/html-single/index.html#continuous-query-cache

Projection for Queries

In 3.8, queries will be able to return specific fields of an entry. Currently queries return whole objects. Using projections, you will be able to return only parts of objects (fields, computed fields, etc.). This will cut the network and serialization overhead and increase the throughput when it the goal is to return a subset of attributes only. The following API for IMap has been made available:

<R> Collection<R> project(Projection<Map.Entry<K, V>, R> projection);
   <R> Collection<R> project(Projection<Map.Entry<K, V>, R> projection, Predicate<K, V> predicate);

See: http://docs.hazelcast.org/docs/3.8/manual/html-single/index.html

Aggregation API Revisited

Before Hazelcast 3.8, aggregations were based on our Map-Reduce engine. We were unsatisfied with the performance of this engine, though. In 3.8, new aggregations have been implemented on top of the query infrastructure. As a result it is both much faster and also much simpler to use. You can pass an Aggregator to the aggregate() method in the IMap:

R aggregate(Aggregator<Map.Entry<K, V>, R> aggregator);
   R aggregate(Aggregator<Map.Entry<K, V>, R> aggregator, Predicate<K, V> predicate);

See more details below: http://docs.hazelcast.org/docs/3.8/manual/html-single/index.html#fast-aggregations

Improvements to Near Cache

We made two major improvements in the Near Cache module:

1. From now on the module guarantees eventual consistency. This is a big step forward from the weakly consistent Near-Cache from the previous versions.
2. The Client Near Cache can now persist keys to a file-system and reload them on a restart. This means you can have your Near Cache hot right after the application starts!

Use Code Deployment

User Code Deployment enables the user to load new classes to Hazelcast IMDG at runtime, without restarting the servers. It may be also called “distributed classloading.” If it is enabled in your cluster, a new class will be copied and loaded by other servers in the background. Client support will be added in the next release so it will be possible to load classes even from the clients. Note that we have released this feature as a beta for now, so please test it thoroughly and give us your valuable feedback. You can also expect API and behavior changes in the forthcoming releases.

HyperLogLog

HyperLogLog is the new data structure that has been introduced in 3.8. It is a probabilistic data structure used to “estimate” cardinalities of unique elements in huge sets. Some common use cases include:

• Calculating unique site visitors metric (real-time) daily, weekly, monthly, yearly or ever, based on IP or user.
• Measuring how a campaign performs (impressions, clicks etc) in the advertising world.

RingBuffer Store

The RingBuffer store has been implemented in a way analogous to the existing Queue store. If the RingBuffer store is enabled, you can get items with sequences which are no longer in the actual RingBuffer but are only in the RingBuffer Store. Also when a new RingBuffer is created, it can ask the RingBuffer Store for the largest sequence in the store and continue from there. This allows the RingBuffer to continue with the new sequence IDs and without overwriting existing sequence IDs that are in the ring buffer store. This feature will also allow users of the ReliableTopic to replay messages older than the messages in the underlying RingBuffer.

Split Brain Protection for Queue and Lock

Cluster quorum is an effort to improve correctness in the network partitioning scenarios. When the number of members in cluster drops below a predefined threshold the Queue and Lock operations will fail with a QuorumException. Current implementation still does not guarantee strict correctness – it works on a best-effort basis. Note that the quorum for IMap and ICache was already implemented in 3.5. See: http://docs.hazelcast.org/docs/3.8-EA/manual/html-single/index.html#split-brain-protection

Rolling Upgrades (Enterprise)

Rolling upgrade is the ability to upgrade the cluster nodes’ versions without any service interruption. Currently this is only supported between patch versions (e.g., from 3.7.1 to 3.7.3). Starting with Hazelcast IMDG 3.8 we will support rolling upgrades among minor versions, too. You will be able to upgrade your cluster nodes from 3.8 to 3.9 without any service interruption. You just need to restart one node at a time not to lose any data. Rolling upgrades are supported only in Hazelcast Enterprise. See: http://docs.hazelcast.org/docs/3.8-EA/manual/html-single/index.html#rolling-member-upgrades

Dynamic WAN Sync (Enterprise)

Using WAN replication you will be able to copy one cluster’s data to another cluster without any service interruption. You can start the sync process inside the WAN Sync interface in the Management Center. From now on you can also add a new WAN replication endpoint to a running cluster using the Management Center. So any time you can create a new WAN replication destination at runtime and create a snapshot of your current cluster.

Hot Restart with Incomplete Members (Enterprise)

Before 3.8 you needed exactly the same cluster topology to perform a hot-restart. This is a very strict requirement, since a single permanent node failure will lead to loss of all the data in the cluster. Hot Restart with Incomplete Members makes the procedure more flexible by allowing users to do a partial start, which basically means the cluster can be restarted with missing members.

More Flexible Hot Restart Deployments (Enterprise)

Before 3.8 it was required that servers use the same IP addresses before and after restart. This was a strong limitation in many use cases, such as moving data to another servers, replacing servers, having different cloud instances, etc. In 3.8 Hot Restart allows different servers (IP addresses) to take part in the restart phase.

Hot Backup Cluster (Enterprise)

This capability is bound to Hot Restart. If Hot Restart Store is enabled, you will be able to make a backup of your running cluster instantly. You can backup your cluster via Java/REST API or Management Center. This is useful when you want to create a snapshot of your cluster instantly at any time.

Closing Words

Bringing Hazelcast IMDG 3.8 to reality was a great adventure. It is always a pleasure to work on new releases for our great community and users. I hope you will enjoy using 3.8 at least as much as we enjoyed working on it! Please try it out – we are looking forward to your contributions, bug reports, feedback, and questions on our mail group and github.