How to Deploy Hazelcast Jet – Part II on Docker with Docker Compose

In the first part of the Deploying Hazelcast Jet series, we walk through how to install and deploy Hazelcast Jet on a bare metal machine which leverages a lot of manual steps. Today, we’ll take one more step towards a more modern deployment methodology. We will use Docker containers to show how you can create a Hazelcast Jet cluster with Docker and deploy Hazelcast Jet Jobs on top of it. Docker is the de-facto platform for developers and sysadmins to build, ship, and run distributed applications. Docker Compose is a tool to define and run multi-container applications. Hazelcast Jet provides a DevOps-friendly Docker image for deployment.

Hazelcast Jet Code Samples and Hazelcast Jet Demo Applications repositories contain several code samples and demo applications to showcase features and simplicity of Hazelcast Jet. In this blogpost, we will use CoGroup Transform code sample from Hazelcast Jet Code Samples repository and deploy it into a Hazelcast Jet cluster running inside Docker.

CoGroup Transform code sample demonstrates usage of the co-group operation, which joins two or more streams on a common key and performs aggregate operation on top of co-grouped items. In the code sample, we are simply simulating an e-commerce shop that generates three type of items for the customer behaviors: PageVisit, AddToCart, and Payment. All those three events share a common key that is userId and we use it to co-group those events. The aggregation in the code sample simply collects all the items inside an accumulator class, called ThreeBags, which is a container of three bags (collections), each with its own element type. As a user you are free to pass any aggregation of your own on this stage, instead of collecting them with ThreeBags. After the co-group operation, results are written to a Hazelcast IMap sink.

To deploy this CoGroup Transform code sample, we have two options of deployment with Docker.

First one is to use the official Hazelcast Jet Docker image to create Hazelast Jet Cluster and the Hazelcast Jet Bootstrap, an application which lets you to submit Hazelcast Jet Job JAR files to a running Hazelcast Jet cluster via Hazelcast Jet Client.. The Jet Jobs will be supplied to the Hazelcast Jet Bootstrap container by mapping the directory which contains Hazelcast Jet Job JAR files to the Docker container as a data volume. This way Hazelcast Jet Bootstrap Docker container will have an access to the files on the host machine. Then the job is submitted using the jet-submit.sh script which is a helper to submit a Jet job that was packaged as an executable JAR file. The script handles uploading the JAR file as a job resource so domain classes in the JAR file will be available to the Hazelcast Jet cluster. You can see in the chart below, how aforementioned scenario will look inside the Docker Environment.

Second one is to build the CoGroup Transform code sample as a Docker image which is based from official Hazelcast Jet Docker image. This approach allows you to have more power on what will be on your Docker container which contains the Hazelcast Jet job. However, it requires to write a custom Dockerfile, and build and publish it into a Docker Registry. This is a more advanced use-case but gives you more control over bits and pieces about your docker image environment. We will not cover this in this blogpost at the moment, but we can create a template project for this use case in the future. Again, you can see this scenario in the chart below.

Up to this point, we’ve discovered available deployment options. Now, we will get our hands dirty with the first scenario. The source code that we’ll be executing to achieve first scenario can be found in the Hazelcast Jet Docker Compose Code Sample. The repository contains source code for the Co-Group Transform along with the Docker Compose file (hazelcast.yml) and a make script for not to type commands again and again while interacting with the Docker Compose. Check out README for more information and the structure of the Co-Group Docker Compose code sample.

We will start our deployment with cloning the Hazelcast Jet Code Samples repository to access source codes and configuration files/scripts.

git clone -b 0.6-maintenance https://github.com/hazelcast/hazelcast-jet-code-samples.git

The repository contains tons of code samples showcasing features of Hazelcast Jet. Currently we are interested in the Docker Compose code sample and that resides in the directory called docker-compose.

Navigate to docker-compose directory;

cd docker-compose/

Now, we need to build the Co-Group Transform code sample to create an executable JAR file.

mvn clean package

This command will build the docker-compose-0.6-SNAPSHOT-jar-with-dependencies.jar JAR file and place it into jars/ directory which will be mounted to the Hazelcast Jet Bootstrap docker container that handles the job submission.

Now we can start our Hazelast Jet Cluster and Hazelcast Jet Bootstrap containers with the following command;

make up

After running make up you should see the console log output like below

/jars/docker-compose-0.6-SNAPSHOT-jar-with-dependencies.jar file will be submitted to the cluster....
docker-compose -f hazelcast.yml up -d
Creating dockercompose_hazelcast-jet_1        ... done
Creating hazelcast-jet-submit ... done
Creating hazelcast-jet-submit ... 

This step creates a single node Hazelcast Jet Cluster and a Hazelcast Jet Bootstrap container, and submits the job to the cluster. We can check individual cluster node logs and bootstrap logs.

To check Hazelcast Jet Cluster log, run the following;

make logServer

You should see the console log output like below, log is truncated for brevity;

….
hazelcast-jet_1         | Mar 27, 2018 10:54:33 AM com.hazelcast.internal.cluster.ClusterService
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] 
hazelcast-jet_1         | 
hazelcast-jet_1         | Members {size:1, ver:1} [
hazelcast-jet_1         |       Member [172.19.0.2]:5701 - b14bc0cb-06f9-44e6-97e1-4167b58aa46a this
hazelcast-jet_1         | ]
hazelcast-jet_1         | 
hazelcast-jet_1         | Mar 27, 2018 10:54:34 AM com.hazelcast.core.LifecycleService
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] [172.19.0.2]:5701 is STARTED
….

To check Hazelcast Jet Bootstrap log, run the following;

make logClient

You should see the console log output like below, log is truncated for brevity;

….
hazelcast-jet-submit    | # starting now....
hazelcast-jet-submit    | ########################################
hazelcast-jet-submit    | Process id 8 for jet instance is written to location:  /opt/hazelcast-jet/jet_instance.pid
hazelcast-jet-submit    | addToCart:
hazelcast-jet-submit    | AddToCart{quantity=21} Event{userId=11}
hazelcast-jet-submit    | AddToCart{quantity=22} Event{userId=11}
hazelcast-jet-submit    | AddToCart{quantity=23} Event{userId=12}
hazelcast-jet-submit    | AddToCart{quantity=24} Event{userId=12}
….

At this point we created single node Hazelcast Jet Cluster and Hazelcast Jet Bootstrap application to submit our job. But I can hear that single node is easy stuff, tell me about a real cluster. Well, thanks to Docker Compose and Hazelcast Jet we can scale our single node cluster into 5 node cluster with a single command.

Let’s run the magical command;

make scale

And follow the output;

$ make scale                                                                                                                       
docker-compose -f hazelcast.yml up --scale hazelcast-jet=5 -d
Starting dockercompose_hazelcast-jet_1 ... done
Creating dockercompose_hazelcast-jet_2 ... done
Creating dockercompose_hazelcast-jet_3 ... done
Creating dockercompose_hazelcast-jet_4 ... done
Creating dockercompose_hazelcast-jet_5 ... done
hazelcast-jet-submit is up-to-date

We just told Docker to scale our hazelcast-jet service into 5 nodes. You can replace the number five to any number and your cluster will scale to that number of nodes!

In order to check the Hazelcast Jet Cluster logs and verify that we actually have a 5-node cluster, run the following again;

make logServer

You should see some logs similar to the logs below that proves that we have a 5-node Hazelcast Jet Cluster, yay!

hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] 
hazelcast-jet_1         | 
hazelcast-jet_1         | Members {size:5, ver:5} [
hazelcast-jet_1         |       Member [172.19.0.2]:5701 - b14bc0cb-06f9-44e6-97e1-4167b58aa46a this
hazelcast-jet_1         |       Member [172.19.0.4]:5701 - 65e1ebc1-a633-4d08-a854-6efaf9f852a0
hazelcast-jet_1         |       Member [172.19.0.5]:5701 - 75d99932-0384-43a3-8e12-2e6a7d01d790
hazelcast-jet_1         |       Member [172.19.0.6]:5701 - 0d0ee4e1-41fd-4b66-9132-3c7793421eb7
hazelcast-jet_1         |       Member [172.19.0.7]:5701 - 6737da51-9727-4d25-be36-76aa1bc84af9
hazelcast-jet_1         | ]
hazelcast-jet_1         | 

Since our Co-Group Transfor Job is a short running batch job, we can re-submit and run it against the 5-node cluster. To achieve that, run the command below;

make restart

Check the logs that job is re-submitted against 5 node cluster.

make logServer | grep hazelcast-jet_1


hazelcast-jet_1         | Mar 28, 2018 8:17:35 AM com.hazelcast.internal.cluster.ClusterService
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] 
hazelcast-jet_1         | 
hazelcast-jet_1         | Members {size:5, ver:5} [
hazelcast-jet_1         |   Member [172.19.0.2]:5701 - c2375f4e-f352-4f87-a02d-1123462d371e this
hazelcast-jet_1         |   Member [172.19.0.3]:5701 - 26fff30b-f4ab-4842-8f2e-d2723998126c
hazelcast-jet_1         |   Member [172.19.0.5]:5701 - 253de0b3-99a3-47d6-830c-5d8f141ce13b
hazelcast-jet_1         |   Member [172.19.0.6]:5701 - a845f467-030d-4e8c-b9db-cca0ef87f2a1
hazelcast-jet_1         |   Member [172.19.0.4]:5701 - 8b960a80-a577-4c60-a805-71a570c0c1a0
hazelcast-jet_1         | ]
hazelcast-jet_1         | 
hazelcast-jet_1         | Mar 28, 2018 8:17:35 AM com.hazelcast.internal.partition.impl.MigrationManager
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] Re-partitioning cluster data... Migration queue size: 1084
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] Received auth from Connection[id=6, /172.19.0.2:5701->/172.19.0.7:33683, endpoint=null, alive=true, type=JAVA_CLIENT], successfully authenticated, principal: ClientPrincipal{uuid='087ae607-c5cd-49ae-bf0b-374d9046b01a', ownerUuid='a845f467-030d-4e8c-b9db-cca0ef87f2a1'}, owner connection: false, client version: 3.10-BETA-2
hazelcast-jet_1         | Mar 28, 2018 8:17:39 AM com.hazelcast.jet.impl.JobCoordinationService
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] Starting job 96d8-2204-510f-fb92 based on submit request from client
hazelcast-jet_1         | Mar 28, 2018 8:17:47 AM com.hazelcast.internal.partition.impl.MigrationThread
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] All migration tasks have been completed, queues are empty.
hazelcast-jet_1         | Mar 28, 2018 8:17:47 AM com.hazelcast.jet.impl.MasterContext
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] Start executing job 96d8-2204-510f-fb92, execution 7a81-42d9-872b-4072, status STARTING
hazelcast-jet_1         | dag
hazelcast-jet_1         |     .vertex("listSource(addToCart)")
hazelcast-jet_1         |     .vertex("listSource(pageVisit)")
hazelcast-jet_1         |     .vertex("listSource(payment)")
hazelcast-jet_1         |     .vertex("3-way cogroup-and-aggregate-step1")
hazelcast-jet_1         |     .vertex("3-way cogroup-and-aggregate-step2")
hazelcast-jet_1         |     .vertex("mapSink(result)")
hazelcast-jet_1         |     .edge(between("listSource(pageVisit)", "3-way cogroup-and-aggregate-step1").partitioned(?))
hazelcast-jet_1         |     .edge(from("listSource(addToCart)").to("3-way cogroup-and-aggregate-step1", 1).partitioned(?))
hazelcast-jet_1         |     .edge(from("listSource(payment)").to("3-way cogroup-and-aggregate-step1", 2).partitioned(?))
hazelcast-jet_1         |     .edge(between("3-way cogroup-and-aggregate-step1", "3-way cogroup-and-aggregate-step2").partitioned(?).distributed())
hazelcast-jet_1         |     .edge(between("3-way cogroup-and-aggregate-step2", "mapSink(result)"))
hazelcast-jet_1         | 
hazelcast-jet_1         | Mar 28, 2018 8:17:47 AM com.hazelcast.jet.impl.JobExecutionService
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] Execution plan for job 96d8-2204-510f-fb92, execution 7a81-42d9-872b-4072 initialized
hazelcast-jet_1         | Mar 28, 2018 8:17:48 AM com.hazelcast.jet.impl.operation.StartExecutionOperation
hazelcast-jet_1         | INFO: [172.19.0.2]:5701 [hazelcast-jet-docker-compose] [0.6-SNAPSHOT] Start execution of job 96d8-2204-510f-fb92, execution 7a81-42d9-872b-4072 from coordinator [172.19.0.2]:5701

Then we can also trace the Hazelcast Jet Bootstrap logs to see job results.

make logClient      


docker-compose -f hazelcast.yml logs hazelcast-jet-submit
Attaching to hazelcast-jet-submit
hazelcast-jet-submit    | ########################################
hazelcast-jet-submit    | # RUN_JAVA=
hazelcast-jet-submit    | # JAVA_OPTS=-Dhazelcast.client.config=/opt/hazelcast-jet/config/hazelcast-client.xml -Dhazelcast.member.address=hazelcast-jet -Dgroup.name=hazelcast-jet-docker-compose
hazelcast-jet-submit    | # CLASSPATH=/jars/docker-compose-0.6-SNAPSHOT-jar-with-dependencies.jar:/opt/hazelcast-jet/hazelcast-jet-0.6-SNAPSHOT.jar:/opt/hazelcast-jet/hazelcast-aws-2.1.0.jar:/*
hazelcast-jet-submit    | # starting now....
hazelcast-jet-submit    | ########################################
hazelcast-jet-submit    | Process id 9 for jet instance is written to location:  /opt/hazelcast-jet/jet_instance.pid
hazelcast-jet-submit    | log4j:WARN No appenders could be found for logger (com.hazelcast.jet.impl.config.XmlJetConfigLocator).
hazelcast-jet-submit    | log4j:WARN Please initialize the log4j system properly.
hazelcast-jet-submit    | addToCart:
hazelcast-jet-submit    | AddToCart{quantity=21} Event{userId=11}
hazelcast-jet-submit    | AddToCart{quantity=22} Event{userId=11}
hazelcast-jet-submit    | AddToCart{quantity=23} Event{userId=12}
hazelcast-jet-submit    | AddToCart{quantity=24} Event{userId=12}
hazelcast-jet-submit    | 
hazelcast-jet-submit    | payment:
hazelcast-jet-submit    | Payment{amount=31} Event{userId=11}
hazelcast-jet-submit    | Payment{amount=32} Event{userId=11}
hazelcast-jet-submit    | Payment{amount=33} Event{userId=12}
hazelcast-jet-submit    | Payment{amount=34} Event{userId=12}
hazelcast-jet-submit    | 
hazelcast-jet-submit    | pageVisit:
hazelcast-jet-submit    | PageVisit{loadTime=1} Event{userId=11}
hazelcast-jet-submit    | PageVisit{loadTime=2} Event{userId=11}
hazelcast-jet-submit    | PageVisit{loadTime=3} Event{userId=12}
hazelcast-jet-submit    | PageVisit{loadTime=4} Event{userId=12}
hazelcast-jet-submit    | 
hazelcast-jet-submit    | result:
hazelcast-jet-submit    | 12->ThreeBags{bag0=[PageVisit{loadTime=3} Event{userId=12}, PageVisit{loadTime=4} Event{userId=12}], bag1=[AddToCart{quantity=23} Event{userId=12}, AddToCart{quantity=24} Event{userId=12}], bag2=[Payment{amount=33} Event{userId=12}, Payment{amount=34} Event{userId=12}]}
hazelcast-jet-submit    | 11->ThreeBags{bag0=[PageVisit{loadTime=1} Event{userId=11}, PageVisit{loadTime=2} Event{userId=11}], bag1=[AddToCart{quantity=21} Event{userId=11}, AddToCart{quantity=22} Event{userId=11}], bag2=[Payment{amount=31} Event{userId=11}, Payment{amount=32} Event{userId=11}]}
hazelcast-jet-submit    | 
hazelcast-jet-submit    | CoGroupDirect results are valid
hazelcast-jet-submit    | result:
hazelcast-jet-submit    | 12->BagsByTag {Tag0=[PageVisit{loadTime=3} Event{userId=12}, PageVisit{loadTime=4} Event{userId=12}], Tag1=[AddToCart{quantity=23} Event{userId=12}, AddToCart{quantity=24} Event{userId=12}], Tag2=[Payment{amount=33} Event{userId=12}, Payment{amount=34} Event{userId=12}]}
hazelcast-jet-submit    | 11->BagsByTag {Tag0=[PageVisit{loadTime=1} Event{userId=11}, PageVisit{loadTime=2} Event{userId=11}], Tag1=[AddToCart{quantity=21} Event{userId=11}, AddToCart{quantity=22} Event{userId=11}], Tag2=[Payment{amount=31} Event{userId=11}, Payment{amount=32} Event{userId=11}]}
hazelcast-jet-submit    | 
hazelcast-jet-submit    | CoGroupBuild results are valid

After we’ve done with the cluster, we can kill it by running;

make down

At this point we’ve seen how to create and scale Hazelcast Jet cluster on Docker, and how to submit Hazelcast Jet jobs to that cluster.

Stay tuned for more posts in the Deploying Hazelcast Jet series.