Get Hazelcast

Fake Benchmark: Corrected Benchmark shows Hazelcast 3.6 is much faster than GridGain/Apache Ignite 1.5

February 18, 2016

GridGain/Apache Ignite have released a benchmark which purports to show that they are faster than Hazelcast. This follows on from our Hazelcast/Ignite Benchmark using Yardstick, their own benchmark, which shows Hazelcast 3.6-EA was much faster than Ignite 1.4.1.

In looking into it we discovered that they had faked the test. Instead of configuring the tests the same they were using:

  • sync backups in Hazelcast for replicas while configuring their own backups as async.
  • map.put() instead of map.set() in Hazelcast, while using put() is equivalent to our set() operation because their put() does not have a return value.

[Updated 21/2/16 with further details]

Async vs Sync

They configure the test using the following script:

-b 1 stands for 1 backup

sm FULL_SYNC stands for synchronization mode full sync. 

In the Benchmark as published on 16 February, they are using 1 backup and sync mode PRIMARY_SYNC.

This means that you wait for response of the primary and not the ack’s of the backups. This is comparable to Hazelcast async-backup. However, they are comparing their PRIMARY_SYNC to our sync backup and that is unfair. So if they want to make a fair comparison they should select FULL_SYNC (waiting for all primaries + backups).

And this is exactly what did they yesterday:

As you can see, they changed sync mode from PRIMARY_SYNC to FULL_SYNC. However, they have not updated their performance report. In Hazelcast switching to async has a huge impact on performance compared to sync; async is almost as fast as having no backup at all.

put void vs put with return value

They are comparing map.put (which returns a value) with their cache:

See this benchmark:

This is the Ignite test: [Benchmark removed by Apache Ignite]

Here is the definition of the cache:

It is an IgniteCache. For documentation of the put:,%20V%29

As you can see it is a void.

The Hazelcast test being used is: HazelcastPutBenchmark:

As you can see the put is being called. See the below link for the javadoc for put; which shows a return value.,%20V%29

So as you clearly can see, they are comparing a put without a return value, to a put with a return value. Since the value is small, I don’t think this will make a huge impact.

An Apples to Apples Test

We ran a comparison using our own benchmark tool correcting for these issues. And we were way faster. Just as we were against Apache Ignite 1.4.1.

We believe this behavior is unbecoming of an Apache project. In fact it is a breach of the Apache Incubator Code of Conduct which prohibits conduct which “Is libelous, defamatory, knowingly false or misrepresents another person”.  We believe their benchmark is an intentional misrepresentation.

We worked very hard to increase our performance over the past year, and would like to see competitors improving performance in their projects and products rather than faking tests.

Test Cluster

4 nodes, 20 clients, 64 threads in each client that produces load

Test Hardware

4 Servers (both nodes and clients are run on 4 servers)

2x Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz – with 10 cores per processor. Total 20 cores, 40 with hyper threading enabled.

768GB 2133 MHz memory 24x HP 32GB 4Rx4 PC4-2133P-L Kit

Heap Configuration member: “-Xms500m -Xmx1g” client: “-Xms100m -Xmx500m”

Cache Configuration

Hazelcast XML Configuration

<map name="hzAsyncOnheap*">

GridGain/Apache Ignite XML Configuration

<bean class="org.apache.ignite.configuration.CacheConfiguration">
    <property name="name" value="atomic"/>
    <property name="cacheMode" value="PARTITIONED"/>
    <property name="atomicityMode" value="ATOMIC"/>
    <property name="swapEnabled" value="false"/>

Throughput Results

Hazelcast is 20-50% faster.

Red: Hazelcast 3.6 Blue: Gridgain/Apache Ignite 1.5 image2016-2-18 12_22_17

99 Percentile Latency Results

Much lower latencies for Hazelcast

Red: Hazelcast 3.6 Blue: Gridgain/Apache Ignite 1.5

image2016-2-18 12_34_32
Visit the Hazelcast vs GridGain page for the latest on these benchmarks.
About the Author

About the Author

Greg Luck

Greg Luck

Chief Technical Officer

Greg Luck is a leading technology entrepreneur with more than 15 years of experience in high-performance in-memory computing. He is the founder and inventor of Ehcache, a widely used open source Java distributed cache that was acquired by Software AG (Terracotta) in 2009, where he served as CTO. Prior to that, Greg was the Chief Architect at Australian start-up that went public on the Australian Stock Exchange (ASX:WTF) in 2006. Greg is a current member of the Java Community Process (JCP) Executive Committee, and since 2007 has been the Specification Lead for JSR 107 (Java Specification Requests) JCACHE. Greg has a master's degree in Information Technology from Queensland University of Technology and a Bachelor of Commerce from the University of Queensland.

Latest Blogs

Hazelcast IMDG 4.0 is Released

Hazelcast IMDG 4.0 is Released

Announcing the Hazelcast Community License

Redis Load Handling vs Data Integrity: Tradeoffs in Distributed Data Store Design

View all blogs by the author

Subscribe to the blog

Follow us on: