Configuring Hazelcast in Non-Orchestrated Docker Environments

Why?

Our blog already contains numerous articles about running Hazelcast in containers, but they mainly focus on orchestrated platforms (Kubernetes, ECS, GCP, …).
According to a 2018 Docker adoption report, half of the deployments did run in non-orchestrated environments 2 years ago. I didn’t find a similar report from 2020, but I expect the proportion has changed to favor orchestrated environments. Nevertheless, there are still enough companies running simple Docker daemons on their own or using both approaches.

Let’s dive a little bit into Hazelcast deployments in plain Docker environments and discuss the traps it can lay in front of us.

What?

We will discuss how to properly run multiple Hazelcast member containers inside a single Docker host. Note, that a Hazelcast cluster is formed automatically with no config change. However, it becomes more interesting when we want to make members available from the outside world.
We will play with Docker network configuration, expose container ports, and use public address Hazelcast configuration.

Once we cover the single Docker daemon, we will show how to run members across multiple Docker hosts. The default Hazelcast discovery method (multicast) doesn’t work in such scenarios. We will need to use TCP discovery configuration and then we can play with Docker networking and try different approaches on how one member can reach one another.

How?

We will use the official Hazelcast IMDG Docker images in the below examples, but the solutions mentioned in the article are valid for other images that run Hazelcast members – either standalone or embedded in Java applications.

Scenario 1: Everything in one Docker host

The simplest case is when all Hazelcast members are started in one Docker host.

Hazelcast members and default networking

By default, the network mode used by the Docker daemon is the bridge network. It means each container has its network stack and they run in the same network – typically 172.17.0.0/16. Docker supports multicast in this network so the default Hazelcast member discovery method works there.

Let’s start 2 member containers:

docker run --rm --name member1 hazelcast/hazelcast:4.0.1
docker run --rm --name member2 hazelcast/hazelcast:4.0.1

When the first member starts, we should see in its Docker log a member list similar to the following:

Apr 17, 2020 6:36:12 AM com.hazelcast.internal.cluster.ClusterService
INFO: [172.17.0.2]:5701 [dev] [4.0.1] 

Members {size:1, ver:1} [
    Member [172.17.0.2]:5701 - b10c208e-22c0-449c-8ace-c8433dd58d5e this
]

During the second member start, the default multicast discovery method is used to find existing cluster members and the original member joins the discovered members. The log of the second member will contain a new cluster view:

Apr 17, 2020 6:36:59 AM com.hazelcast.internal.cluster.ClusterService
INFO: [172.17.0.3]:5701 [dev] [4.0.1] 

Members {size:2, ver:2} [
    Member [172.17.0.2]:5701 - b10c208e-22c0-449c-8ace-c8433dd58d5e
    Member [172.17.0.3]:5701 - 8d485381-379f-4c7c-9034-d8d5e1eb51ef this
]

So far so good. We can start or stop members and the cluster will be updated accordingly. If the members are used in embedded mode and Hazelcast API is called directly on them, then everything works smoothly and data are distributed across the cluster.

Hazelcast clients and default networking

The complexity comes with clients. They don’t use multicast discovery as members do. The default client configuration only tries to connect to the localhost address on ports 5701-5703. It means, a client finds and joins the cluster only when the client is started in the same container as a member. Otherwise, the client won’t find anyone to connect at the localhost address. Remember, the network mode bridge is used and every container has its network stack and IP address.

Let’s use the Hazelcast Client demo console application (included in Hazelcast JAR) to demonstrate it:

docker run -it --rm --name client hazelcast/hazelcast:4.0.1 
  java -cp '/opt/hazelcast/lib/*' 
  com.hazelcast.client.console.ClientConsoleApp

The container will print connection errors and it fails to start:

Apr 17, 2020 7:26:51 AM com.hazelcast.client.impl.connection.ClientConnectionManager
WARNING: hz.client_1 [dev] [4.0.1] Exception during initial connection to [127.0.0.1]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address /127.0.0.1:5701
Apr 17, 2020 7:26:51 AM com.hazelcast.client.impl.connection.ClientConnectionManager
INFO: hz.client_1 [dev] [4.0.1] Trying to connect to [127.0.0.1]:5703
Apr 17, 2020 7:26:51 AM com.hazelcast.client.impl.connection.ClientConnectionManager
WARNING: hz.client_1 [dev] [4.0.1] Exception during initial connection to [127.0.0.1]:5703: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address /127.0.0.1:5703
Apr 17, 2020 7:26:51 AM com.hazelcast.client.impl.connection.ClientConnectionManager
INFO: hz.client_1 [dev] [4.0.1] Trying to connect to [127.0.0.1]:5702
Apr 17, 2020 7:26:51 AM com.hazelcast.client.impl.connection.ClientConnectionManager
WARNING: hz.client_1 [dev] [4.0.1] Exception during initial connection to [127.0.0.1]:5702: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address /127.0.0.1:5702
Apr 17, 2020 7:26:51 AM com.hazelcast.client.impl.connection.ClientConnectionManager
WARNING: hz.client_1 [dev] [4.0.1] Unable to get live cluster connection, retry in 1000 ms, attempt: 1 , cluster connect timeout: 20000 seconds , max backoff millis: 30000

We have to work around the problem by explicitly defining IP addresses of cluster members (or at least one of them). Let’s prepare the client configuration on the Docker host and map it as a volume into the started container. We will create a new file hazelcast-client.xml which includes this network configuration:

<hazelcast-client 
                  xmlns_xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi_schemaLocation="http://www.hazelcast.com/schema/client-config
                  http://www.hazelcast.com/schema/client-config/hazelcast-client-config-4.0.xsd">
    <network>
        <cluster-members>
            <address>172.17.0.2</address>
            <address>172.17.0.3</address>
        </cluster-members>
    </network>
</hazelcast-client>

Command to start the client demo application will contain new arguments allowing to reference the configuration file:

docker run -v `pwd`:/mnt --rm -it --name client hazelcast/hazelcast:4.0.1 
  java -Dhazelcast.client.config=/mnt/hazelcast-client.xml 
  -cp "/opt/hazelcast/lib/*" com.hazelcast.client.console.ClientConsoleApp

The application will join the cluster successfully and we can play with the data in it (run help command to list the possible commands).

hazelcast[default] > m.put key1 value1
null
hazelcast[default] > m.put key1 value2
value1

Network mode host

When all member and client containers run in one Docker host and we allow them to share the network interface with the Docker host, we can use the default discovery method for both.

Members will bind to all IP addresses of the Docker host – also its localhost address. Each member will get its port number (starting with 5701). Clients who also share the Docker host’s network use localhost:5701 to bind to, and this time they are successful. A member is listening to this socket address (host and port).

docker run --rm --network host --name member1 hazelcast/hazelcast:4.0.1
docker run --rm --network host --name member2 hazelcast/hazelcast:4.0.1
docker run -it --rm --network host --name client hazelcast/hazelcast:4.0.1 
  java -cp '/opt/hazelcast/lib/*' com.hazelcast.client.console.ClientConsoleApp

Warning: This approach opens Hazelcast members to the public. Anybody who can reach the Docker host over the network can connect to Hazelcast ports of the running members.

Notes

• Hazelcast uses one backup for the data in the default configuration. It means, one member is the owner of the data and another member holds its backup. It protects data in the cluster when one of the Hazelcast members (containers) crashes.
• The deployments on a single Docker host don’t protect users against a failure of the Docker host.

Scenario 2: Hazelcast members in one Docker host, clients connecting from outside

Things start getting complicated when we need to connect the outside world to the Hazelcast cluster running in a single Docker network.

As we mentioned, the default Docker network mode is bridge. All containers run in a single, newly created network, which is not visible outside of the Docker host. We must publish the ports used by Hazelcast so that clients can access it. Moreover, clients use smart routing by default so they connect directly to a proper owner of the data when writing/reading/updating. Exposing Hazelcast members can be done with multiple approaches, let’s examine them.

Network mode host

The simplest would be to use the network mode host and share the Docker host’s network stack in the Hazelcast member containers. We have seen this approach in Scenario 1 already. We used the host network to allow clients to discover members in the same network. Now we use the host network to open Hazelcast ports on the Docker host machine without a port mapping used:

docker run --rm --network host --name member1 hazelcast/hazelcast:4.0.1
docker run --rm --network host --name member2 hazelcast/hazelcast:4.0.1

Then the IP address of the host will be used for all members. It’s similar to running Hazelcast members directly on the host machine without Docker involved:

Apr 17, 2020 8:25:26 AM com.hazelcast.internal.cluster.ClusterService
INFO: [10.8.0.10]:5702 [dev] [4.0.1] 

Members {size:2, ver:2} [
    Member [10.8.0.10]:5701 - 5c11453b-981c-4194-a73d-fa31973c0fa6
    Member [10.8.0.10]:5702 - aebac828-729e-4bde-9b4d-7fc470fbee5c this
]

Public IP address and port mapping

The host network mode usage is not always possible (or desired). In such a case we have to map ports from Docker host and inform Hazelcast about a different public IP address. Then each member can provide the correct member list to connecting clients.

# use correct IP address of your Docker host
HOST_IP=192.168.1.13
docker run --rm --name member1 
  -e JAVA_OPTS=-Dhazelcast.com.publicAddress=$HOST_IP:5701 
  -p 5701:5701 hazelcast/hazelcast:4.0.1
docker run --rm --name member2 
  -e JAVA_OPTS=-Dhazelcast.com.publicAddress=$HOST_IP:5702 
  -p 5702:5701 hazelcast/hazelcast:4.0.1

We should then see the public address in the containers logs:

Apr 17, 2020 8:50:25 AM com.hazelcast.internal.cluster.ClusterService
INFO: [192.168.1.13]:5702 [dev] [4.0.1] 

Members {size:2, ver:2} [
    Member [192.168.1.13]:5701 - a4d7c62b-3429-4493-887d-b1d86778679d
    Member [192.168.1.13]:5702 - dcdc0e82-d75f-48b5-ae96-b27f435553bb this
]

And we reference the public addresses in the client configuration file:

<network>
    <cluster-members>
        <address>192.168.1.13:5701</address>
        <address>192.168.1.13:5702</address>
    </cluster-members>
</network>

Single exposed member as a proxy

We could also expose just one member’s port and use it as a proxy to the whole cluster:

docker run --rm --name member1 
  -e JAVA_OPTS=-Dhazelcast.com.publicAddress=192.168.1.13:5701 
  -p 5701:5701 hazelcast/hazelcast:4.0.1
docker run --rm --name member2 hazelcast/hazelcast:4.0.1

Then we will see both the public (member1) and the in-container (member2) addresses in the log:

Apr 17, 2020 9:14:53 AM com.hazelcast.client.impl.spi.ClientClusterService
INFO: hz.client_1 [dev] [4.0.1] 

Members [2] {
    Member [192.168.1.13]:5701 - d78b9f39-e9c0-462d-b8e1-2cddeaff8a17
    Member [172.17.0.3]:5701 - ceebd3f5-debd-4f8a-b05f-8a48684c068d
}

On the client-side, we need to disable smart routing so the client always goes to the exposed member and it doesn’t try to connect to all members:

<network>
    <smart-routing>false</smart-routing>
    <cluster-members>
        <address>192.168.1.13:5701</address>
    </cluster-members>
</network>

The solution with one member used as a proxy can simplify some scenarios, but it has its downsides:

• Performance drop – all requests to data owners (members) have to go through a single member.
• High availability – if the proxy member is down, then the client can’t use the Hazelcast cluster at all.

Scenario 3: Hazelcast members spread across multiple Docker hosts

Don’t forget that the Hazelcast cluster should live within one LAN. So even if you run the cluster on multiple Docker hosts, they should be located in one local network. If you need to synchronize Hazelcast data across multiple data centers, then have a look at the WAN Replication feature that is part of Hazelcast Enterprise. which you will need a license to access.

Network mode host and multicast discovery

Similar to Scenario 2, the simplest solution is to start the containers with host networking mode on all Docker hosts.
If your local network supports multicast, then containerized members should discover each other and form the cluster.

Public IP address, port mapping, and TCP discovery method

Again, this one is similar to port mapping in Scenario 2. The difference now is the fact that we can’t use the default Hazelcast multicast discovery method as it will not work across more Docker hosts. Therefore we have to change the Hazelcast member configuration file too. Let’s use YAML configuration for brevity and create the following hazelcast.yml on the Docker hosts:

hazelcast:
  network:
    join:
      multicast:
        enabled: false
      tcp-ip:
        enabled: true
        member-list:
          - 192.168.1.12
          - 192.168.1.13

The addresses 192.168.1.12 and 192.168.1.13 in the example are public IP addresses of the Docker hosts. We don’t need to specify the port number as we will use the default Hazelcast port 5701.

Then we run a member on first Docker host:

docker run -v `pwd`:/mnt --rm --name member1 
  -e "JAVA_OPTS=-Dhazelcast.com.publicAddress=192.168.1.12 -Dhazelcast.config=/mnt/hazelcast.yml" 
  -p 5701:5701 hazelcast/hazelcast:4.0.1

and another member on the second host:

docker run -v `pwd`:/mnt --rm --name member2 
  -e "JAVA_OPTS=-Dhazelcast.com.publicAddress=192.168.1.13 -Dhazelcast.config=/mnt/hazelcast.yml" 
  -p 5701:5701 hazelcast/hazelcast:4.0.1

You should see a properly formed cluster in the member logs:

Apr 17, 2020 9:53:32 AM com.hazelcast.internal.cluster.ClusterService
INFO: [192.168.1.13]:5701 [dev] [4.0.1] 

Members {size:2, ver:2} [
    Member [192.168.1.12]:5701 - a3fc5ee4-df77-484a-82f6-1aa1adfaf3dc
    Member [192.168.1.13]:5701 - 9c1e813d-52df-44c2-9f39-9be9395c7ec6 this
]

Clients don’t need to disable smart routing as both members are reachable under the host public addresses.

Overlay network mode

Another possibility is to form a cluster of Hazelcast members in multiple Docker hosts is to use Docker overlay network mode. See https://docs.docker.com/network/overlay/ for details.

We will not go into the details here because the overlay network doesn’t help us much. It doesn’t support multicast, so we would still need to define TCP discovery and define the IP addresses of other members.

Weave Net

Similar to Docker overlay networks is the Weave Net solution from Weave Works.

The Weave Net supports multicast, so you don’t need any special Hazelcast configuration to make it form a cluster. It just works.

Weave Net can be installed by running following command on Docker host machines:

sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave

Then we can start first Hazelcast member on the first Docker host:

weave launch
eval $(weave env)
docker run --rm  --name member1 hazelcast/hazelcast:4.0.1

And join to it from the second member:

weave launch 192.168.1.12
eval $(weave env)
docker run --rm  --name member2 hazelcast/hazelcast:4.0.1

The members successfully form a cluster:

Apr 17, 2020 12:55:12 PM com.hazelcast.internal.cluster.ClusterService
INFO: [10.44.0.0]:5701 [dev] [4.0.1] 

Members {size:2, ver:2} [
    Member [10.32.0.1]:5701 - f0225f20-fc0f-4c8f-ae6a-deb5359a7df6
    Member [10.44.0.0]:5701 - 73e18a00-252d-42fe-90b1-ea958686c835 this
]

Summary

We discussed several ways of running the Hazelcast cluster in non-orchestrated Docker environment and showed how to properly expose the members to the outside world. We also mentioned networking options we can use to simplify the configuration in Docker. Now it’s your turn to give it a try.