Hazelcast with Hundreds of Nodes

by Enes Akar — Feb 13, 2014

One of the frequent question about hazelcast:

Will it scale up to hundreds of nodes?

Answer from Talip (founder of Hazelcast) in mail group:

Forming 100+ node cluster is not hard and it will be as stable as your networking infrastructure. Running distributed executions is fine too as long as your application can handle execution failures and you don’t make a distributed call within another distributed call. 

To get less affected by the split-brain issues, smaller clusters are recommended. Try to run less number of execution nodes (cluster members) and submit your executions from any number of clients. Clients can scale to thousands easily.

Read More

Hazelcast: When to increase Partition Count?

by Enes Akar — Dec 31, 2013

Hazelcast stores data in partitions. Default partition number is 271. 

You can increase the partition number, if data size per partition is bigger than 100 Mb.

Why?

By consistent hashing* algorithm, partitions are migrated when  nodes join or leave the cluster. While a partition is migrating, operations related to this partition are retried. So migration should be completed as soon as possible. Small partitions cause quick migrations and more responsive cluster.

* http://en.wikipedia.org/wiki/Consistent_hashing

Read More

Out of Memory and Hazelcast

by Enes Akar — Dec 30, 2013

OOM is a real trouble for distributed environment. A node having trouble may pull down whole cluster’s performance.

Hazelcast’ general policy on OOM condition is trying to isolate the problem and shutdown instance as soon as possible.

Here as you can see below, it will first try close connections, stopping threads and shutting down JVM instances. Logging the error is the last thing; so you may not see the OOM log.

This is the default action. Closing connections is the first thing being tried. Otherwise other nodes can still consider the problematic node alive till operations’ timeout. 

You can have your own action executed when OOM. You can do some cleanup, log error, call System.exit() etc.

Here an example to define a custom OOM handler:

Problem: Hazelcast can detect OOM, when hazelcast threads experience the problem. So if OOME is thrown by an external thread; it can not be handled by hazelcast.

Read More

For Faster Hazelcast Queries

by Enes Akar — Dec 27, 2013

Here some tips on improving hazelcast query performance:

1- Add indexes for queried fields. For queries with ranges (gt, lt, between) you can use ordered index.

IMap imap = Hazelcast.getMap("employees");
// ordered, since we have ranged queries for this field
imap.addIndex("age", true); 
// not ordered, because boolean field cannot have range
imap.addIndex("active", false);

2- Set optimizeQuery = true, in map config. By enabling this flag, hazelcast will cache a deserialized (object) form, and use it when necessary. Disadvantage of this approach is memory overhead of cached object.

MapConfig mapConfig = config.getMapConfig(“mapName”);
mapConfig.setOptimizeQueries(true);

3- Object “in memory format”: By default hazelcast stores objects in their serialized form. This is an optimization: an extra serialisation step is skipped for remote operations. But if the majority of a map’s operations is query; then you can consider changing in-memory-format to object. With object format, queries will be run on directly objects, getting rid of de-serialisation. Disadvantage of this approach: The remote get operations from this map will have a serialisation step. 

MapConfig mapConfig = config.getMapConfig(“mapName”);
mapConfig.setInMemoryFormat(InMemoryFormat.OBJECT);

Note: If you use 3rd approach (in-memory-format=OBJECT) then no need for optimizeQuery setting (2nd one) because objects will be already stored in deserialized. 

Read More

Overflow in Hazelcast Queue Store

by Enes Akar — Dec 26, 2013

One of the new features of Hazelcast 3 is the queue store.

Differently from map store; overflow is possible. If you set a memory Iimit; it starts persisting to disk and skipping the memory when queue reaches this limit.

Here the configuration:

<queue-store>
            <class-name>com.hazelcast.QueueStoreImpl</class-name>
            <properties>
                <property name="binary">false</property>
                <property name="memory-limit">1000</property>
                <property name="bulk-load">500</property>
            </properties>
</queue-store>

For above configuration:

- Items will be persisted in their Object form (deserialized one). 

- If the size of the queue has reached 1000 (memory-limit), next element will be just persisted to disk it will not be stored in memory.

- Items will be loaded from store in bulks with size 500.

Some more notes:

- Use memory-limit when you avoid over-usage of RAM in case of high load.

- If you want all items in memory (and store) use Integer.MAX_VALUE, if you want all items just in store then use 0 as memory-limit.

- If you do not reach the store externally (just hazelcast uses the store) then make the binary configuration to true. That will increase the performance as a step for deserialization will be skipped.

 Here you can see some examples of QueueStore implementation:

https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/test/java/com/hazelcast/queue/QueueStoreTest.java

Read More

Hazelcast Executor Service: Thread configuration

by Enes Akar — Dec 24, 2013
At below configuration; 
max 4 threads will serve for e1,
max 8 threads will serve for e2.
There can be max 40 tasks of e1 waiting in the queue. 
There can be max 80 tasks of e2 waiting in the queue. 
 
<executor-service name=”e1”>
        <pool-size>4</pool-size>
        <queue-capacity>40</queue-capacity>
    </executor-service>
<executor-service name=”e2”>
        <pool-size>8</pool-size>
        <queue-capacity>80</queue-capacity>
</executor-service>
Read More

Hazelcast Tip: Native Client versus Lite Member

by Enes Akar — Dec 26, 2012

There are two options to connect to Hazelcast cluster.

- Using native client
- Creating a lite member which is practically a node which does not store any data.
At first glance using lite member may seem more attractive as it knows all nodes and where any key-value pair is stored. But we see that in the long term native client is the more scalable and clean approach.
Here a table to summarize the difference between them:
       

.

      

.

      

.

      

.

   Native Client Lite Member

.

 Performance There is an extra hop as client asks to a member where the data is. Better performance as it knows where data is.

.

 Scalability More scalable as you can create any number of client without overhead to cluster operations. Not scalable, as each lite member becomes a node of the cluster that is an overhead for cluster operations.

.

 Flexibility and Location Flexible, can be anywhere in LAN or WAN. It must be on the same data center even on the same RAC

.

 Near Cache Currently not available Available

.

 Distributed executor Can send tasks but cannot receive and execute Can send and receive taks
Read More