HZ 3.3 Client Performance Almost Doubled

In Hazelcast 3.2 client/member performance is not the same as member/member performance. For example, if we get String values from an IMap using 4 machines so that each machine has 1 client and 1 cluster-member. And we use 40 threads per client, 8 char keys and 100 char values, then the clients do roughly 190k ops/second in total. But if the same load is applied directly on the members, bypassing the clients, then the performance is 350k ops/second. The big question is: What is causing this performance degradation?

To understand the cause, we need to look inside the threading model of Hazelcast 3.2. When a member sends an operation to another member, then on the receiving side the following happens:

1. The io-thread takes the operation of the wire, reads out the partition-id. If the partition-id is set, the partition-aware operation-thread responsible for that partition is looked up (a simple mod and an array access) and the operation is added to the workqueue of that thread.
2. The partition-aware operation-thread takes the operation from its workqueue and executes it.

However, when an operation is received from a client, then on the receiving side the following happens:

1. The io-thread takes the operation of the wire and puts it on a central queue.
2. A client support thread picks up the operation from the central queue. It reads out the partition-id and if set, adds it the workqueue of the correct partition-aware operation-thread.
3. The partition-aware operation-thread takes the operation from its workqueue and executes it.

So the big difference between client and member is that a client has an additional queue in the flow. And this causes the performance degradation:

• There are additional synchronization operations and context switches involved.
• This central-queue is a shared queue between all io-threads and client-support threads, and therefore it is a cause of contention. So the more load you put on the system, the bigger the contention and the bigger the performance degradation.

In Hazelcast 3.3 this performance and scalability issue has been fixed by removing this central queue and letting the client operations immediately be put on the correct partition-aware operation-queue. This caused the client/member performance to increase from 190k ops/second to 340k ops/second! Therefore, client/member performance in Hazelcast 3.3 is almost the same as member/member performance; although it also depends on the use-case of course. If you want to try it out, check out our latest 3.3-RC4-SNAPSHOT.

Fair comparison

The make the performance comparison fair between member/member and client/member, the benchmark prevents the member from making any local calls; all data it retrieves will always be on a remote member. When data is local the whole TCP/IP stack and a lot of serialization is skipped, and since a client doesn’t have any local state, it can never win from a member that is able to do a local call.

Dumb clients

For the hard-core guys: but what about dumb clients? A dumb client doesn’t know about which member has a partition, so most of its operations will be send to the wrong member. On the receiving side, we still read out the partition-id and if set, we still put the operation on the partition-aware operation-thread responsible for that partition; even if that partition is not local. If the partition is remote, the operation-thread sends the operation to the right machine by putting the operation on write-queue of the connection where it eventually is going to be sent by an io-thread. Using an asynchronous callback, the response is sent to the client. Therefore, sending an operation to a remote machine is non-blocking and can be done without hogging the operation thread.

What about 3.3.1?

In the Hazelcast 3.3.1 release we’ll add some additional performance improvements. One of them will be the removal of an additional response-queue which is going to lead to a significant performance improvement on client/member and member/member communication.  But more about that in another blogpost…