Blog ›How to set up Hazelcast on AWS ECS

By Rafal Leszko

Cloud Software Engineer

Rafał is a passionate software engineer, trainer, conference speaker, and author of the book, Continuous Delivery with Docker and Jenkins. He specializes in Java development, cloud environments, and continuous delivery. Prior to joining Hazelcast, Rafał worked with a variety of companies and scientific organizations, including Google, CERN, and AGH University of Science and Technology.

View all blogs by the author

Jul 13, 2020

Back to Blog

How to set up Hazelcast on AWS ECS

Note: This blog was originally published in December 2018, but has since been updated with the latest information.

Amazon ECS (Elastic Container Service) is a container orchestration service that supports Docker containers. Hazelcast, starting from version 4.0.2, fully supports the AWS ECS environment in both models: AWS Fargate and EC2. Thanks to the Hazelcast AWS plugin, with just a few steps, you can set up your Hazelcast cluster. This post presents this process step-by-step and the source code for this example can be found here.

TL;DR

Below is the simplest Hazelcast configuration (hazelcast.yaml file) you can use for the automatic AWS ECS discovery.

hazelcast:
  network:
    join:
      multicast:
        enabled: false
      aws:
        enabled: true
    interfaces:
      enabled: true
      interfaces:
        - 10.0.*.*

A few comments:

  • make sure to change 10.0.*.* to the CIDR of your VPC network
  • make sure the IAM Role attached to your task has the following permissions:
    • ecs:ListTasks
    • ecs:DescribeTasks
    • ec2:DescribeNetworkInterfaces (needed only for public IP discovery)
  • you can use two parameters in the aws part to narrow down the discovery:
    • family
    • service-name
  • if you use Hazelcast embedded into your application, make sure you use hazelcast-all version > 4.0.2

You can read more about Hazelcast AWS ECS discovery in the official documentation.

Supported Features

Hazelcast AWS ECS integration supports the following features:

  • Dynamic auto-discovery – Hazelcast automatically discovers other members to form one cluster
  • Zone Aware – Hazelcast automatically stores its partition backups on a member that is located in a different availability zone, so in case of AWS zone failure, you don’t encounter any data loss
  • Fargate and EC2 – Hazelcast works correctly no matter if your ECS cluster is used in the Fargate or EC2 mode
  • Autoscaling  – Hazelcast works correctly if you scale up and down the instances and, assuming you configured health checks, you will never encounter any data loss

Step-by-step Guide

Hazelcast can be used in the embedded and client-server topologies. Both are well supported by the Hazelcast AWS plugin. Starting from scratch, you’ll need the following steps to create a Hazelcast cluster in the ECS environment.

  • Define Hazelcast ECS Configuration
  • Use Hazelcast Configuration
  • Create ECS Cluster
  • Create Security Group
  • Create IAM Role
  • Configure CloudWatch
  • Create Task Definition
  • Create ECS Service
  • Tear Down

Note that in this guide, we’ll use Fargate as the backend, but in the same way you can configure ECS with EC2.

Define Hazelcast ECS Configuration

To enable Hazelcast AWS ECS discovery, you need to have hazelcast-aws on your classpath, so:

  • if you deploy Hazelcast embedded, make sure you have hazelcast-all version > 4.0.2 in your dependencies
  • if you deploy Hazelcast server, make sure you use Hazelcast Docker image hazelcast/hazelcast version > 4.0.2

Then, here’s the minimal hazelcast.yaml configuration you can use:

hazelcast:
  network:
    join:
      multicast:
        enabled: false
      aws:
        enabled: true
    interfaces:
      enabled: true
      interfaces:
        - 10.0.*.*

Note that you need to change 10.0.*.* to the VPC you plan to use for your ECS Tasks. To list all your available VPCs, execute the following command.

aws ec2 describe-vpcs --query 'Vpcs[*].[VpcId,CidrBlock]'
[
    [
        "vpc-0ae005a4a2835bbb5",
        "172.18.0.0/16"
    ],
    [
        "vpc-0681043d6f49b039a",
        "10.0.0.0/16"
    ]
]

For the next steps, let’s assume that we always use the VPC with ID vpc-0681043d6f49b039a (CIDR vpc-10.0.0.0/16). For the sake of simplicity, let’s put it as an environment variable:

export VPC_ID='vpc-0681043d6f49b039a'

Now, you need to make sure Hazelcast uses this configuration.

Use Hazelcast Configuration

AWS ECS does not support any simple way of injecting configuration files into your container. In Kubernetes, you could define ConfigMap, but unfortunately, such a concept does not exist in ECS.

If you use Hazelcast embedded, then you can define Hazelcast configuration inside your application container. You can see an example of how to do it here. If you just want to play now, you can use the already built image hazelcastguides/hazelcast-embedded-ecs.

If you want to set up a Hazelcast server, then you have two options:

  1. build your own Hazelcast Docker image with Hazelcast configuration baked-in (as described here)
  2. use AWS EFS File System (as described here):
    • create AWS EFS Volume
    • copy hazelcast.yaml
    • mount EFS in your ECS Task Definition
    • specify env variable JAVA_OPTS=-Dhazelcast.config=<hz-config-file-path>

If you use Hazelcast in client-server mode, then you can use the following ECS Hazelcast Client Configuration inside your application to connect to your cluster.

No matter which option you chose, you should now have a Docker image with Hazelcast which you can use in further steps.

Create ECS Cluster

Assuming you have an AWS Account and AWS command line tool configured, you can create an ECS cluster with the following command.

aws ecs create-cluster --cluster-name test-cluster

We will use this cluster for all further steps in this guide.

Create  Security Group

By default, Hazelcast uses port 5701 for all the communication. Therefore, we need to create an appropriate security group.

aws ec2 create-security-group \
  --group-name hazelcast-security-group \
  --description "Hazelcast Security Group" \
  --vpc-id ${VPC_ID}
{
    "GroupId": "sg-0cf1116753b96cdce"
}

Let’s define the security group id as an environment variable.

export SECURITY_GROUP_ID='sg-0cf1116753b96cdce'

Next, we need to open port 5701 for TCP communication.

aws ec2 authorize-security-group-ingress \
  --group-id ${SECURITY_GROUP_ID} \
  --protocol tcp \
  --port 5701 \
  --cidr 0.0.0.0/0

That is enough for Hazelcast communication.

Create IAM Role

Hazelcast ECS Discovery uses AWS API to discover Hazelcast members. That is why we need to grant certain permissions.

First, create a file policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeNetworkInterfaces",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ecs:ListTasks",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ecs:DescribeTasks",
      "Resource": "*"
    }
  ]
}

Then, we can create an IAM Role we will later use by ECS tasks.

aws iam create-policy \
  --policy-name hazelcast-ecs-policy \
  --policy-document file://policy.json
{
    "Policy": {
        "PolicyName": "hazelcast-ecs-policy",
        "PolicyId": "ANPAZV4HIPQ4QUIH45BXB",
        "Arn": "arn:aws:iam::665466731577:policy/hazelcast-ecs-policy",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2020-06-29T09:52:05Z",
        "UpdateDate": "2020-06-29T09:52:05Z"
    }
}

Let’s define this policy ARN as an environment variable.

export POLICY_ARN='arn:aws:iam::665466731577:policy/hazelcast-ecs-policy'

Next, let’s define a new IAM Role. We need a file role-policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Then, we can execute the following command.

aws iam create-role \
  --role-name hazelcast-ecs-role \
  --assume-role-policy-document file://role-policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "hazelcast-ecs-role",
        "RoleId": "AROAZV4HIPQ47NJGCHJ2A",
        "Arn": "arn:aws:iam::665466731577:role/hazelcast-ecs-role",
        "CreateDate": "2020-06-29T09:58:06Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ecs-tasks.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Define an environment variable with the role ARN.

export TASK_ROLE_ARN='arn:aws:iam::665466731577:role/hazelcast-ecs-role'

Finally, let’s attach the defined policy to the created role.

aws iam attach-role-policy --role-name hazelcast-ecs-role --policy-arn ${POLICY_ARN}

Configure CloudWatch

If you want to read logs from Hazelcast (and your application), then you need to create AWS CloudWatch Group. Note that this step is not required, but highly recommended. If you skip it, you also need to remove CloudWatch entry from task-definition.json in the later step and then you won’t be able to see any logs.

To create a CloudWatch group, execute the following command.

aws logs create-log-group --log-group-name /ecs/hazelcast

To allow ECS tasks to write to the CloudWatch log group, you also need to create the following ECS role.

aws iam create-role \
  --role-name ecs-execution-role \
  --assume-role-policy-document file://role-policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "ecs-execution-role",
        "RoleId": "AROAZV4HIPQ4SGDIWGYRK",
        "Arn": "arn:aws:iam::665466731577:role/ecs-execution-role",
        "CreateDate": "2020-06-29T10:44:01Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ecs-tasks.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Export this role as an environment variable.

export EXECUTION_ROLE_ARN='arn:aws:iam::665466731577:role/ecs-execution-role'

Finally, attach AmazonECSTaskExecutionRolePolicy to the created role.

aws iam attach-role-policy \
  --role-name ecs-execution-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Create Task Definition

To create an ECS task definition, you need first to define the following task-definition.json file.

{
  "family": "hazelcast",
  "networkMode": "awsvpc",
  "taskRoleArn": "TASK_ROLE_ARN",
  "executionRoleArn": "EXECUTION_ROLE_ARN",
  "containerDefinitions": [
    {
      "name": "hazelcast",
      "image": "hazelcastguides/hazelcast-embedded-ecs",
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/hazelcast",
          "awslogs-region": "REGION",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "256",
  "memory": "1024"
}

Note that you need to modify the following parts:

  • TASK_ROLE_ARN – ARN of the role created in the “Create IAM Role” step
  • EXECUTION_ROLE_ARN – ARN of the Role created in the “Configure AWS CloudWatch” step
  • REGION – the region that you use for your CloudWatch and ECS Cluster
  • (optionally) image – Docker image you created in the previous step
Then, you can create the task definition with the following command.
aws ecs register-task-definition --cli-input-json file://task-definition.json

Create ECS Service

Having ECS Task Definition specified, we can finally create ECS Tasks or ECS Service (which in turn creates ECS tasks). First, choose the subnet in which your service should operate.

aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=${VPC_ID}" \
  --query 'Subnets[*].[SubnetId,CidrBlock]'
[
    [
        "subnet-0f042c997bad8e2b9",
        "10.0.1.0/24"
    ]
]

export SUBNET_ID='subnet-0f042c997bad8e2b9'

Then, create a service with 3 application replicas.

aws ecs create-service --cluster test-cluster \
  --service-name hazelcast \
  --task-definition hazelcast \
  --launch-type=FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=["${SUBNET_ID}"],securityGroups=["${SECURITY_GROUP_ID}"],assignPublicIp=ENABLED}" \
  --desired-count 3

You can check that the related tasks were created. If you don’t see any tasks, then wait a moment and check the tasks again.

aws ecs list-tasks --cluster test-cluster --service hazelcast
{
    "taskArns": [
        "arn:aws:ecs:eu-central-1:665466731577:task/2154b675-df19-459e-a95e-8466f5d4bb59",
        "arn:aws:ecs:eu-central-1:665466731577:task/7c3adb1d-6850-4f86-9eb1-7eed5bc75235",
        "arn:aws:ecs:eu-central-1:665466731577:task/99de6977-949d-489c-9c5a-44c763edfaaf"
    ]
}

Finally, you can also examine the logs of one of the tasks and you should see that the Hazelcast cluster was successfully formed.

aws logs get-log-events \
  --log-group-name /ecs/hazelcast \
  --log-stream-name ecs/hazelcast/2154b675-df19-459e-a95e-8466f5d4bb59 \
  --output text --query 'events[*].[message]'

...
Members {size:3, ver:3} [
        Member [10.0.1.4]:5701 - dae44d3d-882b-4f3a-aff3-09721b737276
        Member [10.0.1.6]:5701 - 3874314e-30f9-4f10-9f7d-11a0a32dc16b
        Member [10.0.1.220]:5701 - 9009d19a-63b1-4e90-b4de-48e877bb7086 this
]
...

Tear Down

To delete the service, execute the following commands.

aws ecs update-service \
  --cluster test-cluster \
  --service hazelcast \
  --desired-count 0
aws ecs delete-service \
  --cluster test-cluster \
  --service hazelcast

To delete all other resources we created in the post, execute the following commands.

aws ecs deregister-task-definition --task-definition hazelcast:1
aws iam detach-role-policy \
  --role-name ecs-execution-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam delete-role --role-name ecs-execution-role
aws logs delete-log-group --log-group-name /ecs/hazelcast
aws iam detach-role-policy --role-name hazelcast-ecs-role --policy-arn ${POLICY_ARN}
aws iam delete-role --role-name hazelcast-ecs-role
aws iam delete-policy --policy-arn ${POLICY_ARN}
aws ec2 delete-security-group --group-id ${SECURITY_GROUP_ID}
aws ecs delete-cluster --cluster test-cluster

Conclusion

Hazelcast supports automatic member discovery in AWS ECS for both models: AWS Fargate and EC2. You can also use it in embedded and client-server topologies. For anyone interested in more details I recommend reading the Hazelcast AWS documentation.

Keep Reading