Open Source Projects:
Pricing
Chat
Contact
Back to top

Redis Open Source 5.0.3 vs Hazelcast IMDG Enterprise 3.12

Benchmark

Background

This performance benchmark compares Redis Open Source 5.0.3 vs Hazelcast IMDG Enterprise 3.12. The purpose is to reproduce the low thread count benchmark conducted by Redis Labs in 2018, correcting for deficiencies in the Redis benchmark.

The things we changed were:

  • We used Java clients for both. The original benchmark used C++ with Redis. We added Redis with the Lettuce client to RadarGun
  • We used the same benchmarking tool with the same test configuration for each. The original benchmark uses memtier_benchmark. We added Redis support to RadarGun, the tool they used to test Hazelcast.
  • We used Hazelcast Enterprise HD. The original benchmark used open source Hazelcast with a 40GB heap which caused serious garbage collection problems.
  • We used the same payload size distribution for each. The original benchmark used fixed 10KB values with Hazelcast due to the limitations of RadarGun and 5KB average payload sizes for Redis. We added variable payload size support to RadarGun.
  • We used pipelining for both. The original benchmark used pipelining for Redis but not for Hazelcast. Hazelcast now supports pipelining. We added it to RadarGun.
  • We used async backup replication for both. The original performance benchmark used async for Redis and sync for Hazelcast.
  • We used Redis Open Source 5.0.3. The original benchmark used Redis Enterprise. We asked several times for a license for 96 shards to test it but this was not granted. The self-serve trial license is limited to 4 shards. We did our own testing of Redis Open Source and Redis Enterprise, and found they had the same performance.

Summary of Findings: Hazelcast vs Redis

Hazelcast Enterprise 3.12 was slightly faster than Redis.

We would also draw your attention to our previous benchmarks which have consistently found Hazelcast faster than Redis:

  • Redis 3.0.7 vs Hazelcast IMDG 3.6 Benchmark – shows Hazelcast is faster. Hazelcast performs better than Redis as the number of clients and the numbers of threads increase. This was a high thread count test.
  • Redis 3.2.8 vs Hazelcast IMDG 3.8 Benchmark – shows Hazelcast is faster. Once again this was a test at scale. This benchmark also had an extra configuration that showed the benefits of near cache. With a 10% near cache we were 5 times faster than Redis.

Hardware Configuration

Specification Description
Server & client machines
  • EC2 instance of type m4.10xlarge: Intel E5 160GB RAM 10Gb Xeon E5-2676
  • RAM: 160GB of memory
  • Network: 10Gbps Ethernet
Number of server machines 3
Number of client machines 3

Cluster Configuration

Parameter Hazelcast IMDG Enterprise Redis Open Source

Product version

3.12-BETA-1

5.0.3

Cluster configuration

3 members (1 member per machine)

96 shards (48 master +
48 slave shards, 16 master +
16 slave shards per machine

Server configuration

<?xml version=”1.0” encoding=”UTF-8”?>

<hazelcast xsi:schemaLocation=”http://www.hazelcast.com/

schema/config

http://www.hazelcast.com/schema/config/hazelcastconfig-

3.12.xsd”

xmlns=”http://www.hazelcast.com/schema/config”

xmlns:xsi=”http://www.w3.org/2001/XMLSchemainstance”>

<group>

<name>dev</name>

<password>dev-pass</password>

</group>

<network>

<port port-count=”200” auto-increment=”true”>5701</

port>

<join>

<multicast enabled=”false”/>

<tcp-ip enabled=”true”>

<!-- Add the member IP addresses here -->

<member></member>

</tcp-ip>

</join>

</network>

<map name=”default*”>

<in-memory-format>NATIVE</in-memory-format>

<backup-count>0</backup-count>

<async-backup-count>1</async-backup-count>

</map>

<native-memory allocator-type=”POOLED” enabled=”true”>

<size unit=”GIGABYTES” value=”30”/>

</native-memory>

</hazelcast>

No additional configuration in redis.conf file. System adjusted as per Redis process output suggestions:

  • TT Disabled Transparent Huge Pages (THP)
  • TT Adjusted number of maximum open files
  • TT Set overcommit_memory to 1

Clients

Hazelcast Java client

Lettuce 5.1.5
More comments on Redis clients on top of this page.

Benchmark Tool Configuration

Parameter Value

Tool

RadarGun 3.0.0 with modifications to support Redis.

Number of threads per client

64

Read:Write ratio

80:20

Number of objects

4 million

Key size

8 bytes

Value size

values with a random distribution of sizes between 100 bytes and 10,000 bytes

Benchmark configuration
(Hazelcast)

radargun/blob/redis_benchmark/extensions/cache/src/main/resources/benchmark-hazelcast-pipeline-312.xml

Benchmark configuration
(Redis)

radargun/blob/redis_benchmark/extensions/cache/src/main/resources/benchmark-hazelcast-pipeline-312.xml

Pipelining used

yes

Command line used

dist.sh -c <benchmark_xml_above> -t -m <master_ip>:2103 server1 server2
server3 client1 client2 client3

RadarGun Modifications

In order to provide an apples to apples comparison, we needed to run tests for both products using the same tool – RadarGun. However, RadarGun didn’t have capabilities for the setup that was presented by RedisLabs’ benchmark.

We implemented the support for all of the missing pieces, namely:

  • Variable payload sizes based on a random distribution within boundaries
  • Redis benchmark support using the Lettuce client (the Jedis client supports pipelining but not with Redis Clustering)
  • Pipelining support

The source code of the modifications can be found in the following GitHub repository: https://github.com/hazelcast/radargun/tree/redis_benchmark. We hope to push these changes to the official RadarGun repository.

Benchmark Results

From the charts, we see that Redis withLettuce client performed ~612 000 operations per second, whereas Hazelcast IMDG Enterprise did ~636 000 operations per second.
Performance benchmark results comparing Redis vs Hazelcast at scale.

From the charts, we see that Redis with Lettuce client performed ~612 000 operations per second, whereas Hazelcast IMDG Enterprise did ~636 000 operations per second.