Secure Service Mesh Communication Across Kubernetes Clusters

You can secure service-to-service communication across multiple Kubernetes clusters with Consul's mesh gateway feature. Mesh gateways enable you to secure cross-datacenter communication that may be sent over the public internet with mTLS.

Kubernetes Mesh Gateway Diagram

In this tutorial, you will deploy two Consul datacenters on separate Kubernetes clusters with Consul's service mesh, WAN federation, and mesh gateways configured. You will then deploy two services into the service mesh, one in each Consul datacenter. To securely connect the two services, you will configure the service sidecar proxies to route communication through the mesh gateways. Finally, you will test that the services are able to communicate.

»Prerequisites

To successfully complete this tutorial, you will need the following environments provisioned.

Two Kubernetes clusters running in environments that support external load balancers.
kubectl installed locally with two contexts, one for each Kubernetes cluster.
Helm 3 installed locally.

NOTE: An interactive hands-on lab is also available if you do not have a Consul environment to perform the steps described in this tutorial. Click the Show Tutorial button to launch the lab experience.

Security Warning This tutorial is not for production use. By default, the chart will install an insecure configuration of Consul. Please refer to the Kubernetes documentation to determine how you can secure Consul on Kubernetes in production. Additionally, it is highly recommended to use a properly secured Kubernetes cluster or make sure that you understand and enable the recommended security features.

»Configure Consul

You will use Helm 3 to install Consul on your existing Kubernetes clusters. In the Helm chart, you will need to configure your Consul servers, the Consul datacenter, and the following features:

Consul service mesh
mesh gateways
WAN federation

You will learn how to configure each component section of the Helm values file, and then examine and apply the complete configuration for each datacenter.

»Add The HashiCorp Helm Chart Repository

For this tutorial, Consul is being installed into the Kubernetes default namespace. To install into another namespace, add the -n flag to the kubectl and helm commands.

First, you will need to add the HashiCorp Helm chart.

$ helm repo add hashicorp https://helm.releases.hashicorp.com

The command output should indicate that the repo has been added correctly.

"hashicorp" has been added to your repositories

»Configure Consul service mesh

By default, all Consul agents will be added to the Consul service mesh and catalog. However, your Kubernetes services will still need sidecar proxies to secure communication.

The first step to enable Consul to automatically add a sidecar proxy to all the service pods is to install the resources necessary on the Kubernetes node. That is achieved by adding the following stanza to the values yaml files that will be applied to your datacenters.

connectInject:
  enabled: true

Note, you will still need to configure the service to automatically deploy sidecar proxies at deployment time.

»Configure mesh gateways

Next, enable a mesh gateway. A mesh gateway is a proxy that provides an accessible IP address that other datacenters can reach. This also resolves issues with pod IP address ranges overlapping between datacenters. This stanza will also need to be included in the values yaml files that will be applied to your datacenters.

meshGateway:
  enabled: true

»Configure WAN federation

WAN federation connects Consul servers from multiple datacenters into the same WAN gossip pool. WAN federation enables services to discover each other across datacenters. The WAN federation configuration is slightly different for in primary versus secondary datacenters. This yaml snippet shows how to configure the primary data center. Notice the createFederationSecret entry. This should only be set in a primary datacenter. Later in this tutorial you will export the secret from the primary datacenter and inject it into the secondary datacenter. This will allow the secondary datacenter to automatically negotiate WAN federation with the primary.

Note mesh gateways also require TLS encryption.

federation:
  enabled: true
  createFederationSecret: true
tls:
  enabled: true

»Deploy Consul datacenter "dc1"

Finally, finish configuring "dc1" and the Consul agents. Below is a complete Consul datacenter configuration file, helm-consul-dc1-values.yaml.

global:
  name: consul
  datacenter: dc1
  federation:
    enabled: true
    createFederationSecret: true
  tls:
    enabled: true
meshGateway:
  enabled: true
connectInject:
  enabled: true
controller:
  enabled: true

Your kubectl context should be connected to the Kubernetes cluster where you are deploying Consul datacenter "dc1".

$ kubectl config use-context dc1

Switched to context "dc1".

Use helm to install Consul with the hashicorp/consul chart.

$ helm install -f helm-consul-values.yaml consul hashicorp/consul --wait

This command will wait until everything is up and running, which may take a few minutes depending on your environment.

»Export secrets

You will need to export the federation secret created with Consul datacenter "dc1" to use with Consul datacenter "dc2".

$ kubectl get secret consul-federation -o yaml > consul-federation-secret.yaml

»Deploy Consul datacenter "dc2"

Now that you have deployed Consul datacenter "dc1", you can configure and deploy "dc2". Connect your kubectl context to "dc2".

$ kubectl config use-context dc2

Switched to context "dc2".

Create the federation secret in dc2

$ kubectl apply -f consul-federation-secret.yaml

The command output should indicated that the secret was created.

secret/consul-federation created

Now that you have prepared your Kubernetes cluster, finish configuring "dc2" and the Consul agents. "dc2" will need the following additional options configured:

caCert - contains the certificate of the CA to use for TLS communication, which is the the Consul federation secret exported from Consul datacenter "dc1".
caKey - contains the private key of the CA to use for TLS communication, which is the Consul federation secret exported from Consul datacenter "dc1".
server config - contains the server information from Consul datacenter "dc1" necessary to configure federation.

Create the customized chart, helm-consul-dc2-values.yaml.

global:
  datacenter: dc2
  tls:
    enabled: true
    caCert:
      secretName: consul-federation
      secretKey: caCert
    caKey:
      secretName: consul-federation
      secretKey: caKey
  federation:
    enabled: true
  name: consul
server:
  extraVolumes:
    - type: secret
      name: consul-federation
      items:
        - key: serverConfigJSON
          path: config.json
      load: true
connectInject:
  enabled: true
controller:
  enabled: true
meshGateway:
  enabled: true

Finally, use helm to install Consul.

$ helm install -f helm-consul-dc2-values.yaml hashicorp hashicorp/consul --wait

»Verify the datacenters are connected

To verify that the Consul datacenters are connected and WAN federated, you can use kubectl to execute a Consul CLI command to query for a list of servers in the WAN gossip pool.

$ kubectl exec statefulset/consul-server -- consul members -wan

All the servers, from both datacenters, should be listed.

Node                 Address              Status  Type    Build        Protocol  DC   Segment
consul-server-0.dc1  10.32.4.216:8302     alive   server  1.8.0        2         dc1  
consul-server-0.dc2  192.168.2.173:8302   alive   server  1.8.0        2         dc2  
consul-server-1.dc1  10.32.5.161:8302     alive   server  1.8.0        2         dc1  
consul-server-1.dc2  192.168.88.64:8302   alive   server  1.8.0        2         dc2  
consul-server-2.dc1  10.32.1.175:8302     alive   server  1.8.0        2         dc1  
consul-server-2.dc2  192.168.35.174:8302  alive   server  1.8.0        2         dc2

»Deploy microservices

Now that you have two connected Consul datacenters, you can deploy a service in each using kubectl.

»Deploy the "client" service

The "client" service in this tutorial represents a frontend service, for example a website.

Change contexts to communicate with Consul datacenter "dc1".

$ kubectl config use-context dc1

Switched to context "dc1".

The service definition includes two Consul specific annotations:

"consul.hashicorp.com/connect-inject": "true" - ensures that the service is deployed into the Consul service mesh with a sidecar proxy and automatically registered in the Consul catalog.
"consul.hashicorp.com/connect-service-upstreams": "static-server:1234:dc2" - explicitly declares that the upstream service is "server". Note that the annotation value contains a segment that matches the upstream global datacenter name configuration entry.

First, create a yaml file, client.yaml, to define the "client" service.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-client
---
apiVersion: v1
kind: Pod
metadata:
  name: static-client
  annotations:
    'consul.hashicorp.com/connect-inject': 'true'
    'consul.hashicorp.com/connect-service-upstreams': 'static-server:1234:dc2'
spec:
  containers:
    - name: static-client
      image: tutum/curl:latest
      command: ['/bin/sh', '-c', '--']
      args: ['while true; do sleep 30; done;']
  serviceAccountName: static-client

The "consul.hashicorp.com/connect-inject": "true" annotation generates both the "client" service and the sidecar proxy. The sidecar proxy can both accept and establish connections using Consul.

Finally, deploy the "client" service into Consul datacenter dc1.

$ kubectl apply -f client.yaml

The output will indicate that the service account and pods were created.

serviceaccount/static-client created
pod/static-client created

Use kubectl to check that the pods were deployed and are running successfully.

$ kubectl get pods | grep static-client

The output will indicate that three pods are running.

static-client   3/3     Running   0          1s

»Deploy the "server" service

The "server" service in this tutorial represents a backend service, for example a database.

Change contexts to communicate with Consul datacenter "dc2".

$ kubectl config use-context dc2

Switched to context "dc2".

First, create a yaml file, server.yaml, to define the "server" service. Note, the "server" service also includes the consul.hashicorp.com/connect-inject annotation.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: v1
kind: Pod
metadata:
  name: static-server
  annotations:
    'consul.hashicorp.com/connect-inject': 'true'
spec:
  containers:
    - name: static-server
      image: hashicorp/http-echo:latest
      args:
        - -text="hello world"
        - -listen=:8080
      ports:
        - containerPort: 8080
          name: http
  serviceAccountName: static-server

Finally, deploy the "server" service.

$ kubectl apply -f server.yaml

serviceaccount/static-server created
pod/static-server created

Use kubectl to check that the pods were deployed and are running successfully.

$ kubectl get pods | grep static-server

static-server   3/3     Running   0          1s

»Discover services across Kubernetes clusters

Servers participate in WAN gossip to share membership information, which allows servers to perform cross datacenter requests. To secure these requests, the WAN gossip is sent through the mesh gateways which encrypt the communication with mTLS. The requests include service queries.

To discover services across your Kubernetes clusters, you can use the Consul UI or CLI to query the available services.

First, connect to one of the servers in Consul datacenter "dc2".

$ kubectl config use-context dc2

Switched to context "dc2".

Use 'kubectl' to execute a Consul CLI command that retrieves a list of all services in the other datacenter, "dc1".

$ kubectl exec statefulset/consul-server -- consul catalog services -datacenter dc1

The output will confirm that you are able to request service information from the WAN connected datacenter.

consul
mesh-gateway
static-client
static-client-sidecar-proxy

»Route service communication across Kubernetes clusters

Finally, verify that communication is being routed through the mesh gateways. Use curl to verify that the server in "dc2" can retrieve data from the client in "dc1".

Connect to the Kubernetes cluster where Consul datacenter "dc1" is running.

$ kubectl config use-context dc1

Switched to context "dc1".

Using kubectl to connect to the client and request data from the server.

$ kubectl exec static-client -c static-client -- curl -sS http://localhost:1234

“hello world”

»Extended concepts

Now that you have securely connected services across multiple Kubernetes clusters with mesh gateways, you can extend the feature for service failover, blue/green deployments, and canary testing.

You can deploy multiple instances of the same services across multiple production Kubernetes clusters. When services become unavailable in one cluster, you can route traffic to healthy instances in another cluster.

With the L7 routing and splitting features, you can test service upgrades in stages across multiple Kubernetes clusters.

»Next steps

In this tutorial you enabled two services in the Consul service mesh, in separate Kubernetes clusters, to securely communicate with each other over mesh gateways. You also secured WAN gossip server communication by routing traffic through the mesh gateways.

If you are ready to deploy Kubernetes into production, review the Reference Architecture and Deployment Guide.

Infrastructure

Security

Networking

Applications