Hazelcast Headless Monitoring ;)

Since it’s Halloween I thought I would offer up some useful tips on how to perform JMX monitoring for those DevOps types that don’t want to log into a web application or use a GUI of any sort, headless even!

Enable JMX on the Hazelcast Management Center

Please refer to the official documentation here: Clustered JMX via Management Center

This is done by adding startup parameters to the startup file:

{hazelcast_home}/mancenter/startMancenter.sh
    java -Dhazelcast.mc.jmx.enabled=true -Dhazelcast.mc.jmx.port=8081 -jar mancenter-3.7.war

You can test it using jconsole which is included with most java distributions. For a connection just use IP:PORT as identified by the port above and the ip address of the server hosting ManCenter.

JMXTerm

Now head over to JMXTerm to get jmxterm-{version}-uber.jar file which all of the following examples are based on.

In this example we can list all the nodes for a given cluster:

One very nice feature of JMXTerm is how it will list bean for a given domain. A domain is just the name of the MBean so in our case it is will be in the form of ManagementCenter[nameofcluster]. In my case I am using the default cluster name of dev

Now create a file named getMemberList.jmx which contains the following:

beans -d ManagementCenter[dev]

Now you can create a shell file that contains the following:

java -jar jmxterm-1.0-alpha-4-uber.jar -l localhost:8081 -v silent -n < getMemberList.jmx | grep Members

And this will yield a dynamic way of listing all Hazelcast members for a given cluster:

$./listMembers.sh ManagementCenter[dev]:name="192.168.1.184:5701",type=Members ManagementCenter[dev]:name="192.168.1.184:5702",type=Members ManagementCenter[dev]:name="192.168.1.184:5703",type=Members

Now we can use the output from this script to create the input for future scripts. In the following example we are going to query every member for HeapFreeMemory:

./listMembers.sh | awk -F\" {'print "get -d ManagementCenter[dev] -b name=" "\""$2"\"" ",type=Members HeapFreeMemory"'} > getHeapFreeMemory.jmx

Now we just pass getHeapFreeMemory.jmx into jmxterm:

java -jar jmxterm-1.0-alpha-4-uber.jar -l localhost:8081 -v silent -n < getHeapFreeMemory.jmx

And we get the available heap memory from each member of our cluster:

HeapFreeMemory = 20727296;

HeapFreeMemory = 20746592;

HeapFreeMemory = 20749472;

And here is an example of listing the Map sizes for a pair of maps:

get -d ManagementCenter[dev] -b name=Customers,type=Maps -s OwnedEntryCount
    
    get -d ManagementCenter[dev] -b name=Orders,type=Maps -s OwnedEntryCount

So now that we can query the cluster and distributed objects maybe you want to send this via email:

Create a memberList.txt file from our jmxterm script:

./listMembers.sh > memberList.txt

Now uuencode and sendmail the file to yourself:

uuencode ./memberList.txt | sendmail -s "current member list" [email protected]

There is a wealth of information available via the Hazelcast Management Center and when paired with JMXTerm DevOps becomes much easier!