Fake Benchmark: Corrected Benchmark shows Hazelcast 3.6 is much faster than GridGain/Apache Ignite 1.5

GridGain/Apache Ignite have released a benchmark which purports to show that they are faster than Hazelcast. This follows on from our Hazelcast/Ignite Benchmark using Yardstick, their own benchmark, which shows Hazelcast 3.6-EA was much faster than Ignite 1.4.1.

In looking into it we discovered that they had faked the test. Instead of configuring the tests the same they were using:

• sync backups in Hazelcast for replicas while configuring their own backups as async.
• map.put() instead of map.set() in Hazelcast, while using put() is equivalent to our set() operation because their put() does not have a return value.

[Updated 21/2/16 with further details]

Async vs Sync

They configure the test using the following script:

https://github.com/apacheignite/yardstick-ignite/blob/master/config/benchmark.properties

-b 1 stands for 1 backup

–sm FULL_SYNC stands for synchronization mode full sync. 

In the Benchmark as published on 16 February, they are using 1 backup and sync mode PRIMARY_SYNC.

This means that you wait for response of the primary and not the ack’s of the backups. This is comparable to Hazelcast async-backup. However, they are comparing their PRIMARY_SYNC to our sync backup and that is unfair. So if they want to make a fair comparison they should select FULL_SYNC (waiting for all primaries + backups).

And this is exactly what did they yesterday:

https://github.com/apacheignite/yardstick-ignite/commit/a5abf5a92429c03d175b53133a0ae2510d3b0f54

As you can see, they changed sync mode from PRIMARY_SYNC to FULL_SYNC. However, they have not updated their performance report. In Hazelcast switching to async has a huge impact on performance compared to sync; async is almost as fast as having no backup at all.

put void vs put with return value

They are comparing map.put (which returns a value) with their cache:

See this benchmark: https://ignite.apache.org/benchmarks/ignite-vs-hazelcast/atomic-put/001-1.5.0-b1-SNAPSHOT-3ac1504d-atomic-put-1-backup/Results.html

This is the Ignite test: [Benchmark removed by Apache Ignite]

Here is the definition of the cache: https://github.com/apacheignite/yardstick-ignite/blob/master/src/main/java/org/apache/ignite/yardstick/cache/IgniteCacheAbstractBenchmark.java

It is an IgniteCache. For documentation of the put:https://ignite.apache.org/releases/1.5.0-b1/javadoc/org/apache/ignite/IgniteCache.html#put%28K,%20V%29

As you can see it is a void.

The Hazelcast test being used is: HazelcastPutBenchmark: https://github.com/gridgain/yardstick-hazelcast/blob/master/src/main/java/org/yardstickframework/hazelcast/HazelcastPutBenchmark.java

As you can see the put is being called. See the below link for the javadoc for put; which shows a return value.

http://docs.hazelcast.org/docs/latest/javadoc/com/hazelcast/core/IMap.html#put%28K,%20V%29

So as you clearly can see, they are comparing a put without a return value, to a put with a return value. Since the value is small, I don’t think this will make a huge impact.

An Apples to Apples Test

We ran a comparison using our own benchmark tool correcting for these issues. And we were way faster. Just as we were against Apache Ignite 1.4.1.

We believe this behavior is unbecoming of an Apache project. In fact it is a breach of the Apache Incubator Code of Conduct which prohibits conduct which “Is libelous, defamatory, knowingly false or misrepresents another person”.  We believe their benchmark is an intentional misrepresentation.

We worked very hard to increase our performance over the past year, and would like to see competitors improving performance in their projects and products rather than faking tests.

Test Cluster

4 nodes, 20 clients, 64 threads in each client that produces load

Test Hardware

4 Servers (both nodes and clients are run on 4 servers)

2x Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz – with 10 cores per processor. Total 20 cores, 40 with hyper threading enabled.

768GB 2133 MHz memory 24x HP 32GB 4Rx4 PC4-2133P-L Kit

Heap Configuration member: “-Xms500m -Xmx1g” client: “-Xms100m -Xmx500m”

Cache Configuration

Hazelcast XML Configuration

<map name="hzAsyncOnheap*">
    <in-memory-format>BINARY</in-memory-format>
    <backup-count>0</backup-count>
    <async-backup-count>1</async-backup-count>
    </map>

GridGain/Apache Ignite XML Configuration

<bean class="org.apache.ignite.configuration.CacheConfiguration">
    <property name="name" value="atomic"/>
    <property name="cacheMode" value="PARTITIONED"/>
    <property name="atomicityMode" value="ATOMIC"/>
    <property name="swapEnabled" value="false"/>
    </bean>

Throughput Results

Hazelcast is 20-50% faster.

Red: Hazelcast 3.6 Blue: Gridgain/Apache Ignite 1.5

99 Percentile Latency Results

Much lower latencies for Hazelcast

Red: Hazelcast 3.6 Blue: Gridgain/Apache Ignite 1.5