Secure Service Mesh Communication Across Kubernetes Clusters

You can secure service-to-service communication across multiple Kubernetes clusters with Consul's mesh gateway feature. Mesh gateways enable you to secure cross-datacenter communication that may be sent over the public internet with mTLS.

In this tutorial, you will deploy two Consul datacenters on separate Kubernetes clusters with Consul's service mesh, WAN federation, and mesh gateways configured. You will then deploy two services into the service mesh, one in each Consul datacenter. To securely connect the two services, you will configure the service sidecar proxies to route communication through the mesh gateways. Finally, you will test that the services are able to communicate.

»Prerequisites

To successfully complete this tutorial, you will need the following environments provisioned.

Two Kubernetes clusters running in environments that support external load balancers.
kubectl installed locally with two contexts, one for each Kubernetes cluster.
Helm or the Consul-K8S CLI 0.43.0+ installed locally.

Security Warning: This tutorial is not for production use. Please refer to the Kubernetes deployment guide to determine how you can secure Consul on Kubernetes in production. Additionally, we recommend you use a properly secured Kubernetes cluster or make sure that you understand and enable the recommended security features.

»Deploy Consul

You can deploy a complete Consul datacenter using the official Consul Helm chart or the Consul K8S CLI. Feel free to review the Consul Kubernetes installation documentation to learn more about these installation options.

»Create a custom values file

You will learn how to configure each component section of the custom Consul values file, and then examine and apply the complete configuration for each datacenter.

»Configure Consul service mesh

By default, all Consul agents will be added to the Consul service mesh and catalog. However, your Kubernetes services will still need sidecar proxies to secure communication.

The first step to enable Consul to automatically add a sidecar proxy to all the service pods is to install the resources necessary on the Kubernetes node. That is achieved by adding the following stanza to the values yaml files that will be applied to your datacenters.

connectInject:
  enabled: true

connectInject:
  enabled: true

Note: You will still need to configure the service to automatically deploy sidecar proxies at deployment time.

»Configure mesh gateways

Next, enable a mesh gateway. A mesh gateway is a proxy that provides an accessible IP address that other datacenters can reach. This also resolves issues with pod IP address ranges overlapping between datacenters. This stanza will also need to be included in the values yaml files that will be applied to your datacenters.

meshGateway:
  enabled: true

meshGateway:
  enabled: true

»Configure WAN federation

WAN federation connects Consul servers from multiple datacenters into the same WAN gossip pool. WAN federation enables services to discover each other across datacenters. The WAN federation configuration is slightly different for in primary versus secondary datacenters. This yaml snippet shows how to configure the primary data center. Notice the createFederationSecret entry. This should only be set in a primary datacenter. Later in this tutorial you will export the secret from the primary datacenter and inject it into the secondary datacenter. This will allow the secondary datacenter to automatically negotiate WAN federation with the primary.

Note Mesh gateways also require TLS encryption.

federation:
  enabled: true
  createFederationSecret: true
tls:
  enabled: true

federation:
  enabled: true
  createFederationSecret: true
tls:
  enabled: true

»Review the complete custom values file

Finish configuring dc1 and the Consul agents. Below is a complete Consul datacenter configuration file, dc1-values.yaml.

global:
  name: consul
  datacenter: dc1
  federation:
    enabled: true
    createFederationSecret: true
  tls:
    enabled: true
server:
  replicas: 1
ui:
  enabled: true
meshGateway:
  enabled: true
  replicas: 1
connectInject:
  enabled: true
controller:
  enabled: true

dc1-values.yaml

global:
  name: consul
  datacenter: dc1
  federation:
    enabled: true
    createFederationSecret: true
  tls:
    enabled: true
server:
  replicas: 1
ui:
  enabled: true
meshGateway:
  enabled: true
  replicas: 1
connectInject:
  enabled: true
controller:
  enabled: true

Warning: By default, the chart will install an insecure configuration of Consul. This provides a less complicated out-of-box experience for new users, but is not appropriate for a production setup. Review the Secure Consul and Registered Services on Kubernetes tutorial for instructions on how to secure your datacenter for production.

»Install Consul in your dc1 cluster

You can now deploy a complete Consul datacenter in your Kubernetes cluster using the official Consul Helm chart or the Consul K8S CLI.

Your kubectl context should be connected to the Kubernetes cluster where you are deploying Consul datacenter dc1.

$ kubectl config use-context dc1

Switched to context "dc1".

$ kubectl config use-context dc1

Switched to context "dc1".

$ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update
"hashicorp" has been added to your repositories

$ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update
"hashicorp" has been added to your repositories

$ helm install --values dc1-values.yaml consul hashicorp/consul --create-namespace --namespace consul --version "0.43.0"

$ helm install --values dc1-values.yaml consul hashicorp/consul --create-namespace --namespace consul --version "0.43.0"

Note: You can review the official Helm chart values to learn more about the default settings.

Verify that Consul is running in your Kubernetes cluster using kubectl. Consul setup is complete when all pods have a status of Running, as illustrated in the following output.

$ kubectl get pods --namespace consul
NAME                                           READY   STATUS    RESTARTS   AGE
consul-webhook-cert-manager-859c76cdf6-zr7gs   1/1     Running   0          2m32s
consul-controller-76d9cc46f4-gs45g             1/1     Running   0          2m32s
consul-connect-injector-746849588-dh2sz        1/1     Running   0          2m32s
consul-connect-injector-746849588-qdfh6        1/1     Running   0          2m32s
consul-server-0                                1/1     Running   0          2m31s
consul-client-xxm4c                            1/1     Running   0          2m32s
consul-mesh-gateway-7c684b895c-wjrj7           2/2     Running   0          2m32s

$ kubectl get pods --namespace consul
NAME                                           READY   STATUS    RESTARTS   AGE
consul-webhook-cert-manager-859c76cdf6-zr7gs   1/1     Running   0          2m32s
consul-controller-76d9cc46f4-gs45g             1/1     Running   0          2m32s
consul-connect-injector-746849588-dh2sz        1/1     Running   0          2m32s
consul-connect-injector-746849588-qdfh6        1/1     Running   0          2m32s
consul-server-0                                1/1     Running   0          2m31s
consul-client-xxm4c                            1/1     Running   0          2m32s
consul-mesh-gateway-7c684b895c-wjrj7           2/2     Running   0          2m32s

»Set mesh gateway mode

Mesh gateways can be configured per datacenter in one of three modes: local, remote, or none. The official documentation has full details on the implications of running in each mode.

For this tutorial, create a file named proxy-defaults.yaml that contains a CRD specification that globally configures all proxies to run in local mode.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  meshGateway:
    mode: local

proxy-defaults.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  meshGateway:
    mode: local

Use kubectl to apply the CRD.

$ kubectl apply -f proxy-defaults.yaml --namespace consul

proxydefaults.consul.hashicorp.com/global created

$ kubectl apply -f proxy-defaults.yaml --namespace consul

proxydefaults.consul.hashicorp.com/global created

»Export secrets

You will need to export the federation secret created with Consul datacenter dc1 to use with Consul datacenter dc2.

$ kubectl get secret consul-federation --namespace consul -o yaml > consul-federation-secret.yaml

$ kubectl get secret consul-federation --namespace consul -o yaml > consul-federation-secret.yaml

»Deploy Consul datacenter `dc2`

Now that you have deployed Consul datacenter dc1, you can configure and deploy dc2. Connect your kubectl context to dc2.

$ kubectl config use-context dc2

Switched to context "dc2".

$ kubectl config use-context dc2

Switched to context "dc2".

Create the namespace consul.

$ kubectl create namespace consul
namespace/consul created

$ kubectl create namespace consul
namespace/consul created

Create the federation secret in dc2.

$ kubectl apply -f consul-federation-secret.yaml --namespace consul

secret/consul-federation created

$ kubectl apply -f consul-federation-secret.yaml --namespace consul

secret/consul-federation created

Now that you have prepared your Kubernetes cluster, finish configuring dc2 and the Consul agents. The dc2 datacenter will need the following additional options configured:

caCert - contains the certificate of the CA to use for TLS communication, which is the Consul federation secret exported from Consul datacenter dc1.
caKey - contains the private key of the CA to use for TLS communication, which is the Consul federation secret exported from Consul datacenter dc1.
server config - contains the server information from Consul datacenter dc1 necessary to configure federation.

Create the customized chart dc2-values.yaml.

global:
  name: consul
  datacenter: dc2
  tls:
    enabled: true
    caCert:
      secretName: consul-federation
      secretKey: caCert
    caKey:
      secretName: consul-federation
      secretKey: caKey
  federation:
    enabled: true
    primaryDatacenter: dc1
server:
  replicas: 1
  extraVolumes:
    - type: secret
      name: consul-federation
      items:
        - key: serverConfigJSON
          path: config.json
      load: true
connectInject:
  enabled: true
controller:
  enabled: true
meshGateway:
  enabled: true
  replicas: 1

dc2-values.yaml

global:
  name: consul
  datacenter: dc2
  tls:
    enabled: true
    caCert:
      secretName: consul-federation
      secretKey: caCert
    caKey:
      secretName: consul-federation
      secretKey: caKey
  federation:
    enabled: true
    primaryDatacenter: dc1
server:
  replicas: 1
  extraVolumes:
    - type: secret
      name: consul-federation
      items:
        - key: serverConfigJSON
          path: config.json
      load: true
connectInject:
  enabled: true
controller:
  enabled: true
meshGateway:
  enabled: true
  replicas: 1

Finally, install Consul on dc2 with Helm or the Consul-K8S CLI.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

$ helm install --values dc2-values.yaml consul hashicorp/consul --namespace consul --version "0.43.0"

$ helm install --values dc2-values.yaml consul hashicorp/consul --namespace consul --version "0.43.0"

Note: You can review the official Helm chart values to learn more about the default settings.

Check that Consul is running in your Kubernetes cluster using kubectl. Consul setup is complete when all pods have a status of Running, as illustrated in the following output.

$ kubectl get pods --namespace consul
NAME                                           READY   STATUS    RESTARTS   AGE
consul-client-bsgmq                            1/1     Running   0          3m12s
consul-connect-injector-85b9bbf868-kkgzk       1/1     Running   0          3m11s
consul-connect-injector-85b9bbf868-n9dtz       1/1     Running   0          3m11s
consul-controller-5498c9b47c-55x47             1/1     Running   0          3m11s
consul-mesh-gateway-684f89549c-v9h94           2/2     Running   0          3m11s
consul-server-0                                1/1     Running   0          3m11s
consul-webhook-cert-manager-859c76cdf6-rcjz5   1/1     Running   0          3m11s

$ kubectl get pods --namespace consul
NAME                                           READY   STATUS    RESTARTS   AGE
consul-client-bsgmq                            1/1     Running   0          3m12s
consul-connect-injector-85b9bbf868-kkgzk       1/1     Running   0          3m11s
consul-connect-injector-85b9bbf868-n9dtz       1/1     Running   0          3m11s
consul-controller-5498c9b47c-55x47             1/1     Running   0          3m11s
consul-mesh-gateway-684f89549c-v9h94           2/2     Running   0          3m11s
consul-server-0                                1/1     Running   0          3m11s
consul-webhook-cert-manager-859c76cdf6-rcjz5   1/1     Running   0          3m11s

»Set mesh gateway mode

In dc2, you must also deploy a ProxyDefaults CRD that sets the mesh gateway mode. Use kubectl and the proxy-defaults.yaml file you created earlier to apply the CRD to dc2.

$ kubectl apply -f proxy-defaults.yaml --namespace consul

proxydefaults.consul.hashicorp.com/global created

$ kubectl apply -f proxy-defaults.yaml --namespace consul

proxydefaults.consul.hashicorp.com/global created

»Verify the datacenters are connected

To verify that the Consul datacenters are connected and WAN federated, you can use kubectl to execute a Consul CLI command to query for a list of servers in the WAN gossip pool. All the servers, from both datacenters, should be listed.

$ kubectl exec statefulset/consul-server --namespace consul -- consul members -wan

Node                 Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0.dc1  10.0.4.29:8302   alive   server  1.12.0  2         dc1  default    <all>
consul-server-0.dc2  10.0.6.127:8302  alive   server  1.12.0  2         dc2  default    <all>

$ kubectl exec statefulset/consul-server --namespace consul -- consul members -wan

Node                 Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0.dc1  10.0.4.29:8302   alive   server  1.12.0  2         dc1  default    <all>
consul-server-0.dc2  10.0.6.127:8302  alive   server  1.12.0  2         dc2  default    <all>

»Deploy microservices

Now that you have two connected Consul datacenters, you can deploy a service in each using kubectl.

»Deploy the `static-client` service

The static-client service in this tutorial represents a frontend service, for example a website.

Change contexts to communicate with Consul datacenter dc1.

$ kubectl config use-context dc1
Switched to context "dc1".

$ kubectl config use-context dc1
Switched to context "dc1".

The service definition includes one Consul specific annotations:

"consul.hashicorp.com/connect-inject": "true" - ensures that the service is deployed into the Consul service mesh with a sidecar proxy and automatically registered in the Consul catalog.
"consul.hashicorp.com/transparent-proxy": "false" - use explicit upstreams instead of transparent proxy to connect to dc2. Use a ServiceResolver if you want to use transparent proxy. Note that this example disables transparent proxy for all upstream connections. You can omit this annotation to mix and match upstream connections with both transparent proxy and explicit upstreams.
"consul.hashicorp.com/connect-service-upstreams": "static-server:8080:dc2" - directs requests to the static-server service in dc2 using an explicit upstream. You can access static-server on port 8080.

First, create a yaml file, static-client.yaml, to define the static-client service.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-client
---
apiVersion: v1
kind: Service
metadata:
  name: static-client
spec:
  selector:
    app: static-client
  ports:
    - port: 4321
      targetPort: 4321
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: static-client
  name: static-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-client
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: 'true'
        consul.hashicorp.com/transparent-proxy: 'false'
        consul.hashicorp.com/connect-service-upstreams: 'static-server:8080:dc2'
      labels:
        app: static-client
    spec:
      serviceAccountName: static-client
      containers:
        - name: static-client
          image: curlimages/curl:7.77.0
          ports:
            - containerPort: 4321
          command: ['/bin/sh', '-c', '--']
          args: ['while true; do sleep 30; done;']

static-client.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-client
---
apiVersion: v1
kind: Service
metadata:
  name: static-client
spec:
  selector:
    app: static-client
  ports:
    - port: 4321
      targetPort: 4321
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: static-client
  name: static-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-client
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: 'true'
        consul.hashicorp.com/transparent-proxy: 'false'
        consul.hashicorp.com/connect-service-upstreams: 'static-server:8080:dc2'
      labels:
        app: static-client
    spec:
      serviceAccountName: static-client
      containers:
        - name: static-client
          image: curlimages/curl:7.77.0
          ports:
            - containerPort: 4321
          command: ['/bin/sh', '-c', '--']
          args: ['while true; do sleep 30; done;']

The "consul.hashicorp.com/connect-inject": "true" annotation causes Consul to deploy a sidecar proxy alongside the static-client service. The sidecar proxy can both accept and establish connections using Consul.

Now, deploy the static-client service into Consul datacenter dc1.

$ kubectl apply -f static-client.yaml

serviceaccount/static-client created
service/static-client created
deployment.apps/static-client created

$ kubectl apply -f static-client.yaml

serviceaccount/static-client created
service/static-client created
deployment.apps/static-client created

Use kubectl to verify that the pod deployed, and is running successfully.

$ kubectl get pods -l="app=static-client"

NAME                            READY   STATUS    RESTARTS   AGE
static-client-b965bf9b4-zk87j   2/2     Running   0          5m14s

$ kubectl get pods -l="app=static-client"

NAME                            READY   STATUS    RESTARTS   AGE
static-client-b965bf9b4-zk87j   2/2     Running   0          5m14s

»Add a service intention

In order for the static-client service to be able to communicate with the upstream static-server that is in a different datacenter, you must define a ServiceIntentions config entry that allows communication between the two services.

Create a yaml file, service-intentions.yaml, to contain the ServiceIntentions CRD YAML.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: static-client-to-static-server
spec:
  destination:
    name: static-server
  sources:
    - name: static-client
      action: allow

service-intentions.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: static-client-to-static-server
spec:
  destination:
    name: static-server
  sources:
    - name: static-client
      action: allow

Use kubectl to apply the intention.

$ kubectl apply -f service-intentions.yaml

serviceintentions.consul.hashicorp.com/static-client-to-static-server created

$ kubectl apply -f service-intentions.yaml

serviceintentions.consul.hashicorp.com/static-client-to-static-server created

»Deploy the `static-server` service

The static-server service in this tutorial represents a backend service, for example, a database.

Change contexts to communicate with Consul datacenter dc2.

$ kubectl config use-context dc2
Switched to context "dc2".

$ kubectl config use-context dc2
Switched to context "dc2".

First, create a yaml file, static-server.yaml, to define the static-server service. Note, the static-server service also includes the consul.hashicorp.com/connect-inject annotation.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: v1
kind: Service
metadata:
  name: static-server
spec:
  selector:
    app: static-server
  ports:
    - port: 1234
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: static-server
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: 'true'
      labels:
        app: static-server
    spec:
      containers:
        - name: static-server
          image: hashicorp/http-echo:latest
          args:
            - -text="hello world"
            - -listen=:8080
          ports:
            - containerPort: 8080
              name: http

static-server.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: v1
kind: Service
metadata:
  name: static-server
spec:
  selector:
    app: static-server
  ports:
    - port: 1234
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: static-server
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: 'true'
      labels:
        app: static-server
    spec:
      containers:
        - name: static-server
          image: hashicorp/http-echo:latest
          args:
            - -text="hello world"
            - -listen=:8080
          ports:
            - containerPort: 8080
              name: http

Now, deploy the static-server service.

$ kubectl apply -f static-server.yaml

serviceaccount/static-server created
service/static-server created
deployment.apps/static-server created

$ kubectl apply -f static-server.yaml

serviceaccount/static-server created
service/static-server created
deployment.apps/static-server created

Use kubectl to check that the pod deployed, and is running successfully.

$ kubectl get pods -l="app=static-server"

NAME                             READY   STATUS    RESTARTS   AGE
static-server-76557c7487-qq2n5   2/2     Running   0          72s

$ kubectl get pods -l="app=static-server"

NAME                             READY   STATUS    RESTARTS   AGE
static-server-76557c7487-qq2n5   2/2     Running   0          72s

»Discover services across Kubernetes clusters

Servers participate in WAN gossip to share membership information, which allows servers to perform cross datacenter requests. To secure these requests, the WAN gossip is sent through the mesh gateways which encrypt the communication with mTLS. The requests include service queries.

To discover services across your Kubernetes clusters, you can use the Consul UI or CLI to query the available services.

First, connect to one of the servers in Consul datacenter dc2.

$ kubectl config use-context dc2

Switched to context "dc2".

$ kubectl config use-context dc2

Switched to context "dc2".

Use kubectl to execute a Consul CLI command that retrieves a list of all services in the other datacenter, dc1. The output will confirm that you are able to request service information from the WAN connected datacenter.

$ kubectl exec statefulset/consul-server --namespace consul -- consul catalog services -datacenter dc1

consul
mesh-gateway
static-client
static-client-sidecar-proxy

$ kubectl exec statefulset/consul-server --namespace consul -- consul catalog services -datacenter dc1

consul
mesh-gateway
static-client
static-client-sidecar-proxy

»Route service communication across Kubernetes clusters

Finally, verify that communication is routed through the mesh gateways. Use curl to verify that the server in dc2 can retrieve data from the client in dc1.

Connect to the Kubernetes cluster where Consul datacenter dc1 is running.

$ kubectl config use-context dc1

Switched to context "dc1".

$ kubectl config use-context dc1

Switched to context "dc1".

Using kubectl to connect to the client and request data from the server via the proxy.

$ kubectl exec deploy/static-client -c static-client -- curl -sS http://localhost:8080

“hello world”

$ kubectl exec deploy/static-client -c static-client -- curl -sS http://localhost:8080

“hello world”

»Clean up

Connect to Kubernetes cluster for dc2.

$ kubectl config use-context dc2

Switched to context "dc2".

$ kubectl config use-context dc2

Switched to context "dc2".

Remove static-server from dc2.

$ kubectl delete -f static-server.yaml

$ kubectl delete -f static-server.yaml

Remove proxy-defaults from dc2.

$ kubectl delete -f proxy-defaults.yaml

$ kubectl delete -f proxy-defaults.yaml

Delete the Helm release of Consul in dc2.

$ helm delete consul

$ helm delete consul

Note You can fully uninstall Consul by removing the volume claims and secrets.

Connect to Kubernetes cluster for dc1.

$ kubectl config use-context dc1

Switched to context "dc1".

$ kubectl config use-context dc1

Switched to context "dc1".

Remove static-client from dc1.

$ kubectl delete -f static-client.yaml

$ kubectl delete -f static-client.yaml

Remove proxy-defaults from dc1.

$ kubectl delete -f proxy-defaults.yaml

$ kubectl delete -f proxy-defaults.yaml

Delete the Helm release of Consul in dc1.

$ helm delete consul

$ helm delete consul

»Extended concepts

Now that you have securely connected services across multiple Kubernetes clusters with mesh gateways, you can extend the feature for service fail-over, blue/green deployments, and canary testing.

You can deploy multiple instances of the same services across multiple production Kubernetes clusters. When services become unavailable in one cluster, you can route traffic to healthy instances in another cluster.

With the L7 routing and splitting features, you can test service upgrades in stages across multiple Kubernetes clusters.

»Next steps

In this tutorial, you enabled two services in the Consul service mesh in separate Kubernetes clusters to securely communicate with each other over mesh gateways. You also secured WAN gossip server communication by routing traffic through the mesh gateways.

If you are ready to deploy Kubernetes into production, review the Reference Architecture and Deployment Guide.

Infrastructure

Security

Applications

Networking

cloud

»Prerequisites

»Deploy Consul

»Create a custom values file

»Configure Consul service mesh

»Configure mesh gateways

»Configure WAN federation

»Review the complete custom values file

»Install Consul in your dc1 cluster

»Set mesh gateway mode

»Export secrets

»Deploy Consul datacenter dc2

»Set mesh gateway mode

»Verify the datacenters are connected

»Deploy microservices

»Deploy the static-client service

»Add a service intention

»Deploy the static-server service

»Discover services across Kubernetes clusters

»Route service communication across Kubernetes clusters

»Clean up

»Extended concepts

»Next steps

»Deploy Consul datacenter `dc2`

»Deploy the `static-client` service

»Deploy the `static-server` service