Hazelcast with Istio Service Mesh

Istio is said to be the next thing if you follow the Kubernetes path, and service meshes are mentioned whenever you go to a cloud-native meetup or conference like KubeCon. However, you do not always leverage something for the sake of popularity. The Hazelcast-Istio support discussion was started by a user after a GitHub issue raised with the title “Does hazelcast-kubernetes support istio?”

Until recently, we were thinking that Hazelcast and Istio both work fine after closing the issue above but then another user from the community raised a comment claiming that it does not work. That grabbed the better pieces of our attention and we started investigating what is going on under the hood. Luckily enough, I had won a book from a raffle in Istio Meetup at All Things Open and it was exactly what I needed during that period: an early edition of Istio in Action by Christian E. Posta, Sandeep Parikh. This blog post is the result of those chained events, which mainly focuses on how to run Hazelcast with Istio (but it does not explain what service meshes are and how Istio works). If you are not familiar with Istio, you can start with the Istio website.

Requirements

Kubernetes Cluster

In this guide, I used a Google Kubernetes Engine cluster but you can use any Kubernetes cluster you choose. Following is a simple command to create a Kubernetes cluster if you have a GCP account and gcloud is installed on your system.

$ gcloud container clusters create hazelcast-istio-k8s-cluster --cluster-version=1.14.8-gke.12 --num-nodes=4

Istio 1.3.4

This code sample has been tested against Istio 1.3.4

Istio Installation

Firstly, let’s download Istio and configure the path for istioctl:

$ curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.3.4 sh -
$ export PATH=`pwd`/istio-1.3.4/bin:$PATH

Install Istio CRDs

$ cd istio-1.3.4
$ for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done

Verify that you have 23 CRDs installed.

$ kubectl get crds | grep 'istio.io' | wc -l
23

Install Istio with mTLS enabled. This installation applies mTLS between Envoy Service Proxies.

$ kubectl apply -f install/kubernetes/istio-demo-auth.yaml

Verify both istioctl client and Istio Control Plane has the same version.

$ istioctl version
client version: 1.3.4
control plane version: 1.3.4

Clone Hazelcast Istio Code Sample

We have installed a Kubernetes cluster and Istio so far. It is time to clone sample code repository.

$ git clone https://github.com/hazelcast-guides/hazelcast-istio.git
$ cd hazelcast-istio

Apply RBAC. RBAC is needed by hazelcast-kubernetes plugin discovery.

$ kubectl apply -f rbac.yaml

Enable Istio Sidecar AutoInjection.

$ kubectl label namespace default istio-injection=enabled

Hazelcast Embedded and Client-Server with Istio

This guide contains a basic Spring Boot microservice with Hazelcast in both embedded and client-server modes. The microservices demonstrate how Hazelcast can be used with Istio. Business logic in each example is the same to keep it simple. The /put operation puts a key-value pair to Hazelcast and the /get operation returns the value together with the Kubernetes pod name. The pod name is used to show that the value is returned from any pod inside the Kubernetes cluster. That means your value is retrieved from the distributed cache.

Hazelcast Embedded with Istio

Switch to embedded code sample directory via cd hazelcast-embedded/

The embedded code sample can be built and pushed to your own Docker Hub or another registry via the following command, but that is optional. If you want to build your own image then you can use jib, but do not forget to update hazelcast-embedded.yaml with your own image.

$ mvn compile com.google.cloud.tools:jib-maven-plugin:1.8.0:build -Dimage=YOUR-NAME/hazelcast-springboot-embedded:1.0

The pre-built image used in the embedded image is located here.

Deploy Hazelcast Embedded Sample

$ kubectl apply -f hazelcast-embedded.yaml
statefulset.apps/hazelcast-embedded created
service/hazelcast-embedded-headless created
service/springboot-service created

When you list the services used, you will see that you have two Kubernetes services: hazelcast-embedded-headless and springboot-service. hazelcast-embedded-headless is used to handle Hazelcast cluster discovery operation so it has no need for an IP address. springboot-service is a Kubernetes service that is used to receive HTTP requests and forward it to an underlying pod to process it.

$ kubectl get svc
NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
hazelcast-embedded-headless   ClusterIP   None           <none>        5701/TCP   9s
kubernetes                    ClusterIP   10.19.240.1    <none>        443/TCP    73m
springboot-service            ClusterIP   10.19.252.76   <none>        80/TCP     9s

Lets now put a key-value pair into the Hazelcast cluster through Spring Boot REST service and then call get operation in a loop to see the value is returned from different pods.

Firstly, let’s run a container with curl installed and set an environment variable to point to the Spring load balancer.

$ kubectl run curl --rm --image=radial/busyboxplus:curl -i --tty
$ SPRINGBOOT_SERVICE=10.19.252.76

Put a value to the cluster.

$ curl "${SPRINGBOOT_SERVICE}/put?key=1&value=2"
{"value":"2","podName":"hazelcast-embedded-2"}

Get the value from cluster in a loop and see that it is retrieved from different pod names.

$ while true; do curl "${SPRINGBOOT_SERVICE}/get?key=1"; sleep 2;echo; done
{"value":"2","podName":"hazelcast-embedded-1"}
{"value":"2","podName":"hazelcast-embedded-0"}
...

In this sample, you were able to deploy a Spring Boot-based microservice with Hazelcast embedded in an Istio environment. Let’s clean up deployments with the following command.

kubectl delete -f hazelcast-embedded.yaml

Hazelcast Client-Server with Istio

In the previous section, we learned how to use hazelcast embedded with Istio. Lets now focus on a Hazelcast client-server topology. First of all, switch to client-server folder with cd ../hazelcast-client-server

The client-server code sample can be built and pushed to your own Docker Hub or some other registry via the following command, but that is optional. If you want to build your own image then you can use jib to do that but do not forget to update hazelcast-springboot-client.yaml with your own image.

mvn compile com.google.cloud.tools:jib-maven-plugin:1.8.0:build -Dimage=YOUR-NAME/hazelcast-springboot-client:1.0

The pre-built image used in the client-server example is located here.

Deploy Hazelcast Cluster

kubectl apply -f hazelcast-cluster.yaml

You can see the 3-member cluster has been initiated with 3 pods. 2/2 in READY column means that there are 2 containers running in each POD. One is a Hazelcast member and the other is Envoy Proxy.

$ kubectl get pods
NAME                  READY   STATUS    RESTARTS   AGE
hazelcast-cluster-0   2/2     Running   0          60s
hazelcast-cluster-1   2/2     Running   0          44s
hazelcast-cluster-2   2/2     Running   0          27s

Deploy Spring Boot Application with Hazelcast Client

kubectl apply -f hazelcast-springboot-client.yaml

Check logs and see that spring boot service is connected to the cluster.

$ kubectl logs hazelcast-client-0 hazelcast-client -f
...
Members [3] {
Member [10.16.2.14]:5701 - 51274b4d-dc7f-4647-9ceb-c32bfc922c95
Member [10.16.1.15]:5701 - 465cfefa-9b26-472d-a204-addf3b82d40a
Member [10.16.2.15]:5701 - 67fdf27a-e7b7-4ed7-adf1-c00f785d2325
}
...

Let’s run a container with curl installed and set an environment variable to point to the Spring load balancer.

Let’s first find the IP of Spring Boot Service.

$ kubectl get svc springboot-service
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
springboot-service   ClusterIP   10.19.250.127   <none>        80/TCP    3m29s

Launch a curl container inside the Kubernetes cluster and set service IP as environment variable

$ kubectl run curl --rm --image=radial/busyboxplus:curl -i --tty
$ SPRINGBOOT_SERVICE=10.19.250.127

Put a value in the cluster.

$ curl "${SPRINGBOOT_SERVICE}/put?key=1&value=2"
{"value":"2","podName":"hazelcast-embedded-2"}

Get the value from cluster in a loop and see that it is retrieved from different pod names.

$ while true; do curl "${SPRINGBOOT_SERVICE}/get?key=1"; sleep 2;echo; done
{"value":"2","podName":"hazelcast-embedded-1"}
{"value":"2","podName":"hazelcast-embedded-0"}
...

In this sample, you were able to deploy a Spring Boot-based microservice with a Hazelcast client-server topology in an Istio environment. Let’s clean up deployments with the following command.

$ kubectl delete -f hazelcast-springboot-client.yaml
$ kubectl delete -f hazelcast-cluster.yaml

Conclusion

This guide demonstrates how to use Hazelcast embedded and client-server topologies in an mTLS-enabled Istio environment with automatic sidecar injection. Hazelcast continuously tries to support cloud-native technologies and verifies those environments as they evolve.

Fork Github Repository for all the source code and deployment configs and raise an issue if you see a problem.