Open Source Projects:
Back to top

Hot Restart Store Performance Benchmark


This benchmark tests the write and read performance of the Hot Restart Store, introduced in Hazelcast® Enterprise HD 3.6. All benchmarks test the performance of one Hazelcast member running on a physical server. As the Hot Restart Store is local to each member, performance is linearly scalable.

The Hazelcast Hot Restart Store is novel in that it is only ever written to when the cluster is operational. It is never read from. It is only read during start up where we read the entire store into memory before enabling operations. All operational reads are therefore always to RAM.

Results Summary

There are two questions on performance that are important: write performance and read performance.

  • Can the Hot Restart Store keep up with writes in the cluster?
    • With SSD each member can write 220,000 to 310,000 writes per second. This is well above the typical workload for a single member.
    • With HDD each member write 35,000 to 45,000 writes per second. This may impact write performance of a typical cluster. HDD is recommended only for clusters with small amounts of data or slowly changing data such as reference data.
  • How long does it take to restart the cluster?
    • With SSD each member can load data at 1.22GB/s to 1.28GB/s giving a startup time for a 100GB per member of 78 seconds. So a 1TB cluster of 10 members would start up in 78 seconds.
    • With HDD each member can load data at .05GB/s to .08GB/s giving a startup time for 10GB per member of 3.5 minutes. HDD is recommended only for clusters with small amounts of persistent data.

Server Hardware

All tests were performed on HP ProLiant servers running RHEL 7

Server Platform HP ProLiant DL360p Gen8
CPU 2x Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz – with 10 cores per processor. Total 20 cores, 40 with hyper threading enabled.
Memory 768GB 2133 MHz memory 24x HP 32GB 4Rx4 PC4-2133P-L Kit

Storage Media

Two common types of media were tested: 10k HDD and Solid State Drives.

Label Mount Point File System Media Details
HDD /datadisk ext4 /dev/sdb
Physical Hard Disk.
1TB (10K rpm) Hot-Pluggable 2.5 inch
SDD /appdisk ext4 /dev/fioa HP Light Endurance PCIe Workload Accelerator

Test Configuration

Tests were performed using Hazelcast Simulator:

  • custom test class (CycleRunAndRestart)
  • single-member cluster
  • IMap with varying HD Memory sizes; 1KB values
  • For write testing, user threads call IMap methods concurrently
  • various numbers of operation threads. By default Hazelcast will use the number of CPU cores. Testing was done at various numbers to show the effect of more cores and hence operations threads. More threads allows more work to be done in parallel.
Setup Reading Writing
Storage Type Data Size Value Size Level of par*lism Partition Threads Restart Time Read Throughput User Threads Write Throughput
SSD 100 GiB 1007 B 4 8 82 sec 1.4 GiB/s 16 425,000 ops/sec
SSD 50 GiB 495 B 4 40 49 sec 1.2 GiB/s 40 400,000 ops/sec
SSD 25 GiB 239 B 8 40 39 sec 0.80 GiB/s 40 502,254 ops/sec