Securing Hazelcast with Cert-Manager

Cert-Manager became a standard way for issuing and rotating certificates in Kubernetes and OpenShift environments. Simple to install. Simple to use. Well integrated with Vault and other secret managers. No surprise it’s the way to go if you want to set up secure communication between your applications!

In this blog post, I show how to secure Hazelcast communication using keys provisioned with cert-manager. I focus on all necessary steps, from installing cert-manager and issuing certificates to using them for the Hazelcast member-to-member and client-to-member communication.

All you need to follow this tutorial is:

• Kubernetes (or OpenShift) cluster
• Hazelcast Enterprise license key (get a free trial here).

Step 0: Installing Cert-Manager

For the latest information on how to install cert-manager please check its installation guide. In general, it’s enough to execute two simple commands. Note that you need to have admin access to the Kubernetes cluster (or log in as admin in case of OpenShift).

# For OpenShift use "oc" instead of "kubectl"
$ kubectl create namespace cert-manager
$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.yaml

At this point, you should have cert-manager installed into your cluster.

Step 1: Creating Issuer

Issuer is a resource that represents Certificate Authority (CA). It uses a private key stored as a Secret in order to issue other keys securing the TLS communication.

You can generate a private key by yourself and put it in the secret ca-key-pair. However, for the sake of simplicity in this tutorial, let’s use cert-manager Self-Signed Issuer to generate the private key for us.

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: ca-issuer
spec:
  isCA: true
  secretName: ca-key-pair
  commonName: ca-issuer
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io

We created Self-Signed Issuer and Certificate which in turn generates a secret ca-key-pair with the private key. Note, that instead of the configuration above, you could directly create the ca-key-pair secret.

Now, when the private key is ready, we can create an Issuer which will sign keys encrypting Hazelcast communication.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: hazelcast-server-certificate
spec:
  secretName: hazelcast-server-tls
  keyEncoding: pkcs8
  usages:
    - server auth
    - client auth
  dnsNames:
    - hazelcast-hazelcast-enterprise.default.svc.cluster.local
  issuerRef:
    name: ca-issuer
    kind: Issuer

Issuer ca-issuer uses private key stored in ca-key-pair to sign all keys we need for securing Hazelcast communication. With that in place, we are ready to configure the Hazelcast cluster.

Step 2: Starting Secure Hazelcast Cluster

Let’s now create a Certificate to secure the communication between Hazelcast members.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: hazelcast-server-certificate
spec:
  secretName: hazelcast-server-tls
  keyEncoding: pkcs8
  dnsNames:
    - hazelcast-hazelcast-enterprise.default.svc.cluster.local
  issuerRef:
    name: ca-issuer
    kind: Issuer

A few words about the Certificate we have just defined:

• It stores keys in the secret named hazelcast-server-tls
• Keys are signed by ca-issuer, which acts as Certificate Authority
• Hostname verification is disabled for Hazelcast communication, so the dnsNames property is ignored
• The same keys are shared among all Hazelcast members

Now, we need to define one more secret with the Hazelcast license key.

# For OpenShift use "oc" instead of "kubectl"
$ kubectl create secret generic hz-license-key --from-literal=key=<hz-license-key>

As mentioned at the beginning, if you don’t have a valid Hazelcast license key, please request a trial license via the Hazelcast website.

Let’s also define the Hazelcast configuration in hazelcast.yaml.

hazelcast:
  advanced-network:
    enabled: true
    join:
      multicast:
        enabled: false
      kubernetes:
        enabled: true
        service-name: ${serviceName}
        namespace: ${namespace}
    member-server-socket-endpoint-config:
      port:
        port: 5701
      ssl:
        enabled: true
        factory-class-name: com.hazelcast.nio.ssl.OpenSSLEngineFactory
        properties:
          mutualAuthentication: REQUIRED
          protocol: TLSv1.2
          trustCertCollectionFile: /data/secrets/ca.crt
          keyFile: /data/secrets/tls.key
          keyCertChainFile: /data/secrets/tls.crt
    client-server-socket-endpoint-config:
      port:
        port: 5702
      ssl:
        enabled: true
        factory-class-name: com.hazelcast.nio.ssl.OpenSSLEngineFactory
        properties:
          mutualAuthentication: REQUIRED
          protocol: TLSv1.2
          trustCertCollectionFile: /data/secrets/ca.crt
          keyFile: /data/secrets/tls.key
          keyCertChainFile: /data/secrets/tls.crt
    rest-server-socket-endpoint-config:
      port:
        port: 5703
      endpoint-groups:
        HEALTH_CHECK:
          enabled: true

A few words of comment:

• We use kubernetes configuration to enable automatic discovery between Hazelcast members
• Placeholders ${serviceName} and ${namespace} are filled with JVM parameters during runtime
• We use advanced-network with 3 different ports for accessing Hazelcast cluster:
  • 5701 for member to member communication (secured with SSL)
  • 5702 for client to member communication (secured with SSL)
  • 5703 for health checks used in livenessProbe and readinessProbe (not secured, because it’s not possible to use Kubernetes probes with SSL Mutual Authentication)
• Certificates (ca.crt, tls.key, tls.crt) are automatically injected into the secret hazelcast-server-tls

Note, that if we used network instead of advanced-network, then Hazelcast health checks would also be secured with the SSL Mutual Authentication. This would, in turn, prevent us from using livenessProbe and readinessProbe.

As the last preparation step, let’s put the Hazelcast configuration into a ConfigMap.

# For OpenShift use "oc" instead of "kubectl"
$ kubectl create configmap hazelcast --from-file=hazelcast.yaml

We are all set up. Now, we can start a Hazelcast cluster. There are different ways to do it, let’s examine three exclusive options: Helm Chart, Hazelcast Operator, and raw Kubernetes configuration.

Step 2.1 (Option 1) Hazelcast Cluster with Helm Chart

Helm is the simplest way to deploy a Hazelcast cluster. Assuming you have the Helm 3 command installed, you can create values.yaml with the following content.

secretsMountName: hazelcast-server-tls
livenessProbe:
  port: 5703
readinessProbe:
  port: 5703
hazelcast:
  licenseKeySecretName: hz-license-key
  existingConfigMap: hazelcast
securityContext:
  readOnlyRootFilesystem: false
mancenter:
  enabled: false

This configuration:

• Mounts Certificate keys hazelcast-server-tls into Hazelcast pods
• Sets livenessProbe/readinessProbe port to 5703 as defined earlier in the Hazelcast configuration
• Uses Secret hz-license-key as Hazelcast license key and ConfigMap hazelcast as Hazelcast configuration
• Disables securityContext.readOnlyRootFilesystem since OpenSSL library needs write access to the filesystem
• Disables Hazelcast Management Center

With such configuration in place, we can add Hazelcast Helm Chart repository and start Hazelcast Enterprise deployment.

$ helm repo add hazelcast https://hazelcast.github.io/charts/
$ helm repo update
$ helm install hazelcast -f values.yaml --version 3.1.0 hazelcast/hazelcast-enterprise

We successfully installed the Hazelcast cluster using Helm Chart. Let’s see how to achieve exactly the same using Hazelcast Operator.

Step 2.2 (Option 2) Hazelcast Cluster with Hazelcast Operator

Operators are more and more popular among Kubernetes and OpenShift users. To install and use Hazelcast Operator, please follow the instructions described in hazelcast/hazelcast-operator. Use the following listing as the resource file hazelcast.yaml.

apiVersion: hazelcast.com/v1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  secretsMountName: hazelcast-server-tls
  livenessProbe:
    port: 5703
  readinessProbe:
    port: 5703
  hazelcast:
    licenseKeySecretName: hz-license-key
    existingConfigMap: hazelcast
  securityContext:
    readOnlyRootFilesystem: false
  mancenter:
    enabled: false

Parameters are exactly the same as described in step 2.1 related to the Helm Chart method.

Step 2.3 (Option 3) Hazelcast Cluster with Kubernetes configuration

I strongly suggest using the Hazelcast Helm Chart or Hazelcast Operator methods, because they incorporate a lot of features that would simply be too long to describe here in the raw Kubernetes configuration. However, if you are forced to use pure Kubernetes or you are simply interested in details, then here’s the minimal configuration to start a Hazelcast cluster.

apiVersion: v1
kind: Service
metadata:
  name: hazelcast-hazelcast-enterprise
spec:
    type: ClusterIP
    clusterIP: None
    selector:
      app: hazelcast-enterprise
    ports:
    - protocol: TCP
      port: 5701
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hazelcast-hazelcast-enterprise
  labels:
    app: hazelcast-enterprise
spec:
  serviceName: hazelcast-hazelcast-enterprise
  replicas: 3
  selector:
    matchLabels:
      app: hazelcast-enterprise
  template:
    metadata:
      labels:
        app: hazelcast-enterprise
    spec:
      containers:
        - name: hazelcast
          image: hazelcast/hazelcast-enterprise:4.0
          volumeMounts:
          - mountPath: "/data/secrets"
            name: secrets
          - name: hazelcast-storage
            mountPath: /data/hazelcast
          ports:
          - name: hazelcast
            containerPort: 5701
          env:
          - name: HZ_LICENSE_KEY
            valueFrom:
              secretKeyRef:
                name: hz-license-key
                key: key
          - name: JAVA_OPTS
            value: "-Dhazelcast.config=/data/hazelcast/hazelcast.yaml -DserviceName=hazelcast-hazelcast-enterprise -Dnamespace=default"
      volumes:
        - name: secrets
          secret:
            secretName: hazelcast-server-tls
        - name: hazelcast-storage
          configMap:
            name: hazelcast

Explanation of all the configuration above is out of the scope for this blog post. However, if you’re interested in details, please check the Hazelcast Kubernetes plugin and Hazelcast Kubernetes Code Samples.

Step 3: Verifying Hazelcast Cluster

No matter which installation method you chose, you should now have a running Hazelcast cluster secured with keys from cert-manager. Let’s verify that everything is set up correctly.

You should have three running PODs.

# For OpenShift use "oc" instead of "kubectl"
$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
hazelcast-hazelcast-enterprise-0   1/1     Running   0          2m24s
hazelcast-hazelcast-enterprise-1   1/1     Running   0          101s
hazelcast-hazelcast-enterprise-2   1/1     Running   0          57s

They should use SSL for communication.

# For OpenShift use "oc" instead of "kubectl"
$ kubectl logs hazelcast-hazelcast-enterprise-0 | grep SSL
INFO: Using OpenSSL for SSL encryption.

They should all form one Hazelcast cluster.

# For OpenShift use "oc" instead of "kubectl"
$ kubectl logs hazelcast-hazelcast-enterprise-0
...
Members {size:3, ver:3} [
        Member [10.8.1.5]:5701 - 0053daac-7c4d-49a5-a145-c54e852e06e7 this
        Member [10.8.0.10]:5701 - be313027-c866-4af2-a1f6-f380f984f6e3
        Member [10.8.2.6]:5701 - 88c0d395-f68f-49b2-8a69-409b64c0ba3b
]

When we are sure our cluster started successfully, we can now configure the Hazelcast client application.

Step 4: Starting Hazelcast Client Application

We will start a sample client application in a few steps:

• Create client Certificate
• Create Hazelcast client configuration
• Create client application
• Deploy client application

Step 4.1: Creating Client Certificate

Technically, we could re-use the Certificate that we already created for the Hazelcast cluster. However, it’s better to issue separate keys for the client part. The Certificate resource looks very similar to what we saw before.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: hazelcast-client-certificate
spec:
  secretName: hazelcast-client-tls
  keyEncoding: pkcs8
  usages:
    - client auth
  dnsNames:
    - my-app.com
  issuerRef:
    name: ca-issuer
    kind: Issuer

As the next step, let’s create a Hazelcast client configuration.

Step 4.2: Creating Hazelcast Client Configuration

We can define hazelcast-client.yaml with the following content.

hazelcast-client:
  network:
    kubernetes:
      enabled: true
      service-dns: hazelcast-hazelcast-enterprise.default.svc.cluster.local
      service-port: 5702
    ssl:  
      enabled: true
      factory-class-name: com.hazelcast.nio.ssl.OpenSSLEngineFactory
      properties:
        protocol: TLSv1.2
        trustCertCollectionFile: /secrets/ca.crt
        keyFile: /secrets/tls.key
        keyCertChainFile: /secrets/tls.crt

A few words of explanation:

• We use Hazelcast Kubernetes plugin configuration to automatically discover the Hazelcast cluster defined with the service name hazelcast-hazelcast-enterprise in the default namespace
• We use the SSL configuration which is very similar to the server part

Let’s create a ConfigMap with the specified configuration.

# For OpenShift use "oc" instead of "kubectl"
kubectl create configmap hazelcast-client --from-file=hazelcast-client.yaml

With that configuration in place, we can finally create our client application.

Step 4.3: Creating Client Application

To present the client application, we can use any supported programming language. Here, let’s use Java and create a sample application which connects to the Hazelcast cluster. The application does nothing more than inserting a value to a Hazelcast map and reading it periodically.

That is the code of Main.java.

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.client.config.ClientConfig;
import com.hazelcast.client.config.YamlClientConfigBuilder;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.map.IMap;

import java.io.IOException;

public class Main {
    public static void main(String[] args)
                    throws IOException, InterruptedException {
        ClientConfig clientConfig = 
                    new YamlClientConfigBuilder("/config/hazelcast-client.yaml")
                                               .build();
        HazelcastInstance instance = 
                    HazelcastClient.newHazelcastClient(clientConfig);

        IMap map = instance.getMap("test-map");
        map.put("some-key", "some-value");

        while (true) {
            System.out.println(map.get("some-key"));
            Thread.sleep(1000);
        }
    }
}

And here’s the related Dockerfile.

FROM openjdk:11-jdk
COPY Main.java .
RUN wget https://dl.bintray.com/hazelcast/release/com/hazelcast/hazelcast-enterprise-all/4.0/hazelcast-enterprise-all-4.0.jar
RUN wget https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.30.Final/netty-tcnative-boringssl-static-2.0.30.Final.jar
RUN wget https://repo1.maven.org/maven2/io/netty/netty-all/4.1.48.Final/netty-all-4.1.48.Final.jar
ENV CLASSPATH /*
CMD java Main.java

The Dockerfile above downloads the necessary dependencies and starts the client application. Let’s build and push it into the Docker Hub registry.

docker build -t leszko/hazelcast-client-test .
docker push leszko/hazelcast-client-test

Replace leszko with your Docker Hub username. Please also make sure your repository is public (or define necessary Docker credentials in the Kubernetes configuration) before proceeding with the next step.

Step 4.4: Deploying Client Application

As the last step, let’s apply the following configuration.

apiVersion: v1
kind: Pod
metadata:
  name: hazelcast-client
spec:
  containers:
  - name: hazelcast-client
    image: leszko/hazelcast-client-test
    volumeMounts:
    - mountPath: "/secrets"
      name: secrets
    - mountPath: "/config"
      name: config
  volumes:
  - name: secrets
    secret:
      secretName: hazelcast-client-tls
  - name: config
    configMap:
      name: hazelcast-client

In the logs, we should see that the application successfully connected to the cluster.

# For OpenShift use "oc" instead of "kubectl"
$ kubectl logs pod/hazelcast-client
...

Members [3] {
        Member [10.8.1.5]:5702 - 0053daac-7c4d-49a5-a145-c54e852e06e7
        Member [10.8.0.10]:5702 - be313027-c866-4af2-a1f6-f380f984f6e3
        Member [10.8.2.6]:5702 - 88c0d395-f68f-49b2-8a69-409b64c0ba3b
}

Mar 23, 2020 9:05:19 AM com.hazelcast.client.impl.ClientExtension
INFO: SSL is enabled
Mar 23, 2020 9:05:19 AM com.hazelcast.nio.ssl.OpenSSLEngineFactory
INFO: Using OpenSSL for SSL encryption.
...
some-value
some-value
some-value
...

Conclusion

What I presented above is the correct way to configure Hazelcast security in the Kubernetes/OpenShift environment. As always with the security aspects some steps may look superfluous, for example, defining separate Certificates for server and client. Nevertheless, in the long run, that’s what you should do to provide the high security level for your services!