Hazelcast IMDG

Hot Restart Store

Feature of IMDG Enterprise HD

Hazelcast Hot Restart Store at a Glance

In addition to the basic reliability options (write-through and write-behind),
Hazelcast® Enterprise HD IMDG has introduced a new high-performance option. This feature is known as Hot Restart Store. This persistence feature provides
fast cluster restarts by storing the states of the cluster members on disk in a format
specially designed for restart performance and to work in concert with SSDs.

Key Features

  • Dramatically reduces restart time for huge caches by providing very fast data reload
  • Ultra high-performance persistence store, optimized for SSD & mirrored in native memory
  • 502,254 writes per second per SSD node
  • Each node has its own independent store, so enables linear scaling across the cluster
  • Restarts in 76 seconds with 100GB per node (so 76s to reload 1TB on a 10 node cluster)
  • Data entirely loaded into RAM on reload – Hazelcast always operates at in-memory speed
  • Persistence can be configured per data structure for JCache, Map, Web Sessions and Hibernate
  • Supports partial restart which allows you to restart the cluster even with an incomplete member set
  • Hot backup via Management Center or Java/REST API allows you to take a snapshot of the Hot Restart Store at a certain point in time to perform testing, QA or reproduce an issue on production data
  • Supports real world operational scenarios such as: moving data to another server, replacing a server, and allowing different servers (IP addresses) to take part in the restart phase
Setup Reading Writing
Storage Type Data Size Value Size Level of par*lism Partition Threads Restart Time Read Throughput User Threads Write Throughput
SSD 100 GB 1007 B 4 8 82 sec 1.4 GB/s 16 425,000 ops/sec
SSD 50 GB 495 B 4 40 49 sec 1.2 GB/s 40 400,000 ops/sec
SSD 25 GB 239 B 8 40 39 sec 0.80 GB/s 40 502,254 ops/sec

Whether the restart is a planned shutdown (including rolling upgrades across a cluster) or a sudden cluster-wide crash (e.g. power outage or total loss of a data center), Hot Restart Store allows full recovery to the previous state of configuration and cluster data.

The initial release of Hot Restart Store supports the IMap and JCache interfaces, as
well as Web Sessions and Hibernate. Further data structures will be delivered in
subsequent releases. It is strongly recommended that Hot Restart Store be used only
on SSD drives, as magnetic HDD drives are much slower (typically ~10x). This
difference in capability will typically result in restart and recovery times that are not
compatible with high-performance use cases.

Hot Restart Store works by each node controlling its own local snapshot. This provides
linear scaling across the cluster, provided one of two best practice approaches
is followed:

  1. That each node is not configured with more stored data than the size of the
    node’s JVM heap. This is the preferred approach for on-heap storage.
  2. In the case of larger stores, High-Density Memory Store should be used. This uses off-heap memory storage,
    and allows the creation of much larger nodes. This solves some of the limitations
    of garbage-collected heaps – 100GB JVM heaps are not practical to collect.
    This has clear best practice implications for the design of high-performance clusters
    – each node should have a large amount of physical RAM. In the on-heap case, we
    should use a large JVM heap (not larger than physical RAM) and the Hot Restart Store
    size configured to be the same as the heap size. In the off-heap High-Density Memory Store case, as much
    memory as we can should be devoted to the High-Density store.