Hazelcast C++ Client Thread Structure

We have just released our newest version of Hazelcast C++ Client API. It has a lot of improvements and features compared to older releases and I would like to provide some insights into the thread structure of our client, including how it interacts with the user threads.

IO Thread

IO thread is one of the most important threads. It writes and reads client messages to and from the server. This thread is the busiest and on the critical path and should process IO bytes very fast. Therefore, on the reader side, it only creates new messages and fills bytes from the TCP buffer into the message buffer. When writing a message it registers the invocation in a hash map, unordered_map, where the key is the per connection unique correlation id. It matches the incoming responses from the server to the outstanding invocation via this correlation id. After registering the invocation, the IO thread writes the message bytes into the TCP buffer in an async manner when the connection is available for writes.

Below are the threads involved during a simple imap::get operation:

Invocation Thread Pool

Invocation Thread Pool decodes the received client_message so that the valuable IO thread resources are not used. We use boost::asio::thread_pool for the pool. We use the default constructor of this thread pool which spawns threads equal to twice the number of cores. The Invocation Thread Pool decodes the message and does any needed book keepings (e.g. put an entry into near-cache if near-cache was enabled, etc.)

Event Delivery Thread Pool

The thread structure is a little different for event delivery to the user listeners. Below is the diagram for event delivery:

The number of threads for event delivery is 5. For each received message, if the operation was executed on a specific partition in the cluster, this thread pool delivers the events received for the same partition to the user listener in the same order as they are received using the boost::asio::strand. This way, we guarantee orderly delivery of events. Note that the user listeners should not do any blocking operations since the number of threads for event delivery is limited to 5. If the user listener needs to perform long-lasting work, it should off-load the work to another thread of control. There is also a single-threaded registration thread pool that does the register/unregister of the listeners.

Internal thread Pool

In addition to the above threads, we also have an internal executor pool. The default thread count for the internal pool is 3. This thread pool executes periodic scheduled tasks, for example, 

• Checking heartbeat on the active connections to the server and send periodic ping requests if no message communication occurred at a certain time.
• Retrying a retryable invocation (the time is scheduled on this thread pool).
• Scheduling expiration timer for near cache expirations.
• If client statistics collection is enabled (disabled by default) in the configuration, retrieving and sending the statistics to the server periodically.

Connection Opening Thread Pool

If the client is in smart mode (default is enabled), then there is one more thread pool that checks connections to existing members in the cluster and opens a connection to each member. This thread pool uses 10 threads to open connections. It works with a timer that wakes up every second and checks the existence of connections to the latest member list in the cluster. It triggers a connection to the member if no connection exists. Connection opening blocks one of the threads in the pool. The pool opens connections to all members in parallel using during client start.

I tried to give some ideas on the existing threads in our client design on this short blog. This is really a very high-level overview, and there are a lot more details that are not discussed in this post for simplicity. You can always explore our codebase at Github for more details on the exact uses of these threads. Please keep in mind that the API for using Hazelcast cluster structures (imap, ilist, iqueue, topic, etc.) is always thread-safe.