Hot Restart Store Performance Benchmark
Download NowThis benchmark tests the write and read performance of the Hot Restart Store, introduced in Hazelcast® Enterprise HD 3.6. All benchmarks test the performance of one Hazelcast member running on a physical server. As the Hot Restart Store is local to each member, performance is linearly scalable.
The Hazelcast Hot Restart Store is novel in that it is only ever written to when the cluster is operational. It is never read from. It is only read during start up where we read the entire store into memory before enabling operations. All operational reads are therefore always to RAM.
Results Summary
There are two questions on performance that are important: write performance and read performance.
- Can the Hot Restart Store keep up with writes in the cluster?
- With SSD each member can write 220,000 to 310,000 writes per second. This is well above the typical workload for a single member.
- With HDD each member write 35,000 to 45,000 writes per second. This may impact write performance of a typical cluster. HDD is recommended only for clusters with small amounts of data or slowly changing data such as reference data.
- How long does it take to restart the cluster?
- With SSD each member can load data at 1.22GB/s to 1.28GB/s giving a startup time for a 100GB per member of 78 seconds. So a 1TB cluster of 10 members would start up in 78 seconds.
- With HDD each member can load data at .05GB/s to .08GB/s giving a startup time for 10GB per member of 3.5 minutes. HDD is recommended only for clusters with small amounts of persistent data.
Server Hardware
All tests were performed on HP ProLiant servers running RHEL 7
Server Platform | HP ProLiant DL360p Gen8 |
CPU | 2x Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz – with 10 cores per processor. Total 20 cores, 40 with hyper threading enabled. |
Memory | 768GB 2133 MHz memory 24x HP 32GB 4Rx4 PC4-2133P-L Kit |
Storage Media
Two common types of media were tested: 10k HDD and Solid State Drives.
Label | Mount Point | File System | Media | Details |
---|---|---|---|---|
HDD | /datadisk | ext4 | /dev/sdb Physical Hard Disk. |
1TB (10K rpm) Hot-Pluggable 2.5 inch |
SDD | /appdisk | ext4 | /dev/fioa | HP Light Endurance PCIe Workload Accelerator |
Test Configuration
Tests were performed using Hazelcast Simulator:
- custom test class (CycleRunAndRestart)
- single-member cluster
- IMap with varying HD Memory sizes; 1KB values
- For write testing, user threads call IMap methods concurrently
- various numbers of operation threads. By default Hazelcast will use the number of CPU cores. Testing was done at various numbers to show the effect of more cores and hence operations threads. More threads allows more work to be done in parallel.
Setup | Reading | Writing | ||||||
---|---|---|---|---|---|---|---|---|
Storage Type | Data Size | Value Size | Level of par*lism | Partition Threads | Restart Time | Read Throughput | User Threads | Write Throughput |
SSD | 100 GiB | 1007 B | 4 | 8 | 82 sec | 1.4 GiB/s | 16 | 425,000 ops/sec |
SSD | 50 GiB | 495 B | 4 | 40 | 49 sec | 1.2 GiB/s | 40 | 400,000 ops/sec |
SSD | 25 GiB | 239 B | 8 | 40 | 39 sec | 0.80 GiB/s | 40 | 502,254 ops/sec |