Consul Service Discovery and Mesh on Kubernetes in Docker (kind)

In this tutorial, you'll start a local Kubernetes cluster with kind. You will then deploy Consul with the official Helm chart. After deploying Consul, you will learn how to access the agents. You will then deploy two services that use Consul to discover each other and communicate over TLS via Consul service mesh.

Security Warning This tutorial is not for production use. By default, the chart will install an insecure configuration of Consul. Refer to the Secure Consul and Registered Services on Kubernetes tutorial to learn how you can secure Consul on Kubernetes in production. Additionally, it is highly recommended to use a properly secured Kubernetes cluster or make sure that you understand and enable the recommended security features.

»Prerequisites

First, you'll need to follow the directions for installing kind.

You will also need to install kubectl and helm.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install kubernetes-helm

Next, ensure that you have Consul 1.8.0 or higher installed.

$ consul version

Example output:

Consul v1..8.0
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

»Start a Kind cluster

Once kind is installed, you can spin up any number of clusters. By default, kind will name your cluster "kind", but you may name it anything you like by specifying the --name option. This tutorial assumes the cluster is named tutorial. Refer to the kind documentation for information about how to specify additional parameters using a yaml configuration file.

$ kind create cluster --name tutorial

The output will be similar to the following.

Creating cluster "tutorial" ...
 ✓ Ensuring node image (kindest/node:v1.18.2) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-tutorial"
You can now use your cluster with:

kubectl cluster-info --context kind-tutorial

Have a nice day! 👋

Kind does not ship with the kubernetes dashboard by default. If, you wish to install the Kubernetes Dashboard, refer to the Kubernetes Dashboard project for instructions on how to install and view it.

»Install Consul with the official Helm chart

Tip: You can deploy a complete Consul datacenter using the official Helm chart. By default, the chart will install a three-server Consul datacenter on Kubernetes, with a Consul client on each Kubernetes node. You can review the official Helm chart values to learn more about the default settings.

»Download the demo code and Helm chart

First, add the HashiCorp Helm Chart repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

"hashicorp" has been added to your repositories

»Create a custom values file

The chart comes with reasonable defaults, however, you will override a few values to help things go more smoothly with kind and enable useful features.

Create a custom values file called helm-consul-values.yaml with the following contents. Name the Consul datacenter and then enable the following:

Consul UI via a NodePort
secure communication between pods with connectInject

Finally, you'll configure your deployment to only have one Consul server (suitable for local development).

# Choose an optional name for the datacenter
global:
  datacenter: minidc

# Enable the Consul Web UI via a NodePort
ui:
  service:
    type: 'NodePort'

# Enable Connect for secure communication between nodes
connectInject:
  enabled: true
# Enable CRD controller
controller:
  enabled: true

client:
  enabled: true

# Use only one Consul server for local development
server:
  replicas: 1
  bootstrapExpect: 1
  disruptionBudget:
    enabled: true
    maxUnavailable: 0

»Deploy Consul on kind

Now, run helm install, providing your custom values file, the hashicorp/consul chart, and a name for your Consul installation. It will print a list of all the resources that were created.

$ helm install -f helm-consul-values.yaml hashicorp hashicorp/consul

The output will be similar to the following.


NAME: hashicorp
LAST DEPLOYED: Thu Sep  3 07:29:22 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Consul!

Now that you have deployed Consul, you should look over the docs on using
Consul with Kubernetes available here:

https://www.consul.io/docs/platform/k8s/index.html


Your release is named hashicorp.

To learn more about the release if you are using Helm 2, run:

  $ helm status hashicorp
  $ helm get hashicorp

To learn more about the release if you are using Helm 3, run:

  $ helm status hashicorp
  $ helm get all hashicorp

»Access the Consul UI

Verify Consul was deployed properly by accessing the Consul UI. Run kubectl get pods to list your pods. Find the pod with consul-server in the name.

$ kubectl get pods

NAME                                                              READY   STATUS    RESTARTS   AGE
hashicorp-consul-2778v                                            1/1     Running   0          10m
hashicorp-consul-connect-injector-webhook-deployment-cbd674lv2x   1/1     Running   0          10m
hashicorp-consul-server-0                                         1/1     Running   0          10m

Now, expose the Consul UI with kubectl port-forward with the hashicorp-consul-server pod name as the target.

$ kubectl port-forward hashicorp-consul-server-0 8500:8500

You can now visit the Consul UI at localhost:8500 in a browser on your development machine. You will observe a list of Consul's services, nodes, and other resources. Currently, you should only find the consul service listed.

Minikube Consul UI

»Access Consul with kubectl and the HTTP API

In addition to accessing Consul with the UI, you can manage Consul with the HTTP API or by directly connecting to the pod with kubectl.

»Use Kubectl to access the server

To access the pod and data directory you can start a shell session in the pod with kubectl exec.

$ kubectl exec -it hashicorp-consul-server-0 -- /bin/sh

/ #

This will allow you to navigate the file system and run Consul CLI commands on the pod. For example you can view the Consul version and member list.

$ consul version

Consul v1.8.2
Revision ba7d9435e
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

$ consul members

Node                       Address          Status  Type    Build  Protocol  DC      Segment
hashicorp-consul-server-0  10.244.0.8:8301  alive   server  1.8.2  2         minidc  
tutorial-control-plane     10.244.0.5:8301  alive   client  1.8.2  2         minidc

$ exit

»Consul HTTP API

You can use the Consul HTTP API by communicating with the local agent running on the Kubernetes node. Read the documentation to learn more about using the Consul HTTP API with Kubernetes.

»Deploy services with Kubernetes

Because you enabled the Connect injector in your helm-consul-values.yaml file, all the services using Consul service mesh will automatically be registered in the Consul catalog.

»Deploy two services

Now you can deploy your services as a two-tier application made of a backend data service that returns a number (the counting service) and a frontend dashboard that pulls from the counting service over HTTP and displays the number.

Create a pod definition and service account for the counting service named counting.yaml.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: counting
---
apiVersion: v1
kind: Pod
metadata:
  name: counting
  annotations:
    'consul.hashicorp.com/connect-inject': 'true'
spec:
  containers:
    - name: counting
      image: hashicorp/counting-service:0.0.2
      ports:
        - containerPort: 9001
          name: http
  serviceAccountName: counting

Next, create a pod definition and service account for the dashboard service and its load balancer named dashboard.yaml.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard
---
apiVersion: v1
kind: Pod
metadata:
  name: dashboard
  labels:
    app: 'dashboard'
  annotations:
    'consul.hashicorp.com/connect-inject': 'true'
    'consul.hashicorp.com/connect-service-upstreams': 'counting:9001'
spec:
  containers:
    - name: dashboard
      image: hashicorp/dashboard-service:0.0.4
      ports:
        - containerPort: 9002
          name: http
      env:
        - name: COUNTING_SERVICE_URL
          value: 'http://localhost:9001'
  serviceAccountName: dashboard
---
apiVersion: 'v1'
kind: 'Service'
metadata:
  name: 'dashboard-service-load-balancer'
  namespace: 'default'
  labels:
    app: 'dashboard'
spec:
  ports:
    - protocol: 'TCP'
      port: 80
      targetPort: 9002
  selector:
    app: 'dashboard'
  type: 'LoadBalancer'
  loadBalancerIP: ''

Use kubectl to deploy the counting service.

$ kubectl apply -f counting.yaml

serviceaccount/counting created
pod/counting created

Use kubectl to deploy the dashboard service.

$ kubectl apply -f dashboard.yaml

serviceaccount/dashboard created
pod/dashboard created
service/dashboard-service-load-balancer created

To verify the services were deployed, refresh the Consul UI until you observe that the counting and dashboard services and their proxies are running.

Services

»View the dashboard

To visit the dashboard, forward the pod's port where the dashboard service is running to your local machine on the same port by providing the pod name (dashboard), which you specified in the service definition YAML file.

$ kubectl port-forward dashboard 9002:9002

Forwarding from 127.0.0.1:9002 -> 9002
Forwarding from [::1]:9002 -> 9002

Visit http://localhost:9002 in your web browser. It will display the dashboard service running in a Kubernetes pod, with a number retrieved from the counting service using Consul service discovery, and transmitted securely over the network with mutual TLS via an Envoy proxy.

Application Dashboard

»Secure service communication with intentions

Consul service mesh secures service-to-service communication with authorization and encryption. Applications can use sidecar proxies to automatically establish mutual TLS connections for inbound and outbound network traffic without being aware of Consul at all.

Consul "intentions" provide you the ability to control which services are allowed to communicate. Next, you will use intentions to test the communication between the dashboard and counting services.

»Create an intention that denies communication

Connect to the Consul server using kubectl. Then use the Consul CLI to create an intention to prevent the dashboard service from reaching its upstream counting service using the -deny flag.

$ kubectl exec -it hashicorp-consul-server-0 -- /bin/sh

/ #

$ consul intention create -deny dashboard counting

Created: dashboard => counting (deny)

Verify the services are no longer allowed to communicate by returning to the dashboard UI. The service will display a message that the "Counting Services Unreachable" and the count will display as "-1".

You can use consul intention create to create "deny" and "allow" rules.

»Allow the application dashboard to communicate with the Counting service

Finally, remove the intention so that the services can communicate again.

$ consul intention delete dashboard counting

Intention deleted.

These action does not require a reboot. It takes effect so quickly that by the time you visit the application dashboard, you'll notice that it's successfully communicating with the backend counting service again.

»Extend your knowledge

»Rolling updates to Consul

While running Consul you may want to make configuration and deployment updates. You can use helm upgrade to increase the number of agents, enable additional features, or upgrade the Consul version. You can practice using helm upgrade by updating your custom values file to enable catalogSync.

syncCatalog:
  enabled: true

The catalog sync feature allows Consul to discover services deployed in Kubernetes, without an operator creating Consul registration files. With catalog sync and Connect inject enabled, you will no longer need to manually register services when they are hosted on a Kubernetes node.

Initiate the upgrade with the -f flag that passes in your new values file.

$ helm upgrade hashicorp -f helm-consul-values.yaml hashicorp/consul

Release "hashicorp" has been upgraded. Happy Helming!
LAST DEPLOYED: Wed Nov 13 10:07:29 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
hashicorp-consul-connect-injector-svc  19m
hashicorp-consul-dns                   19m
hashicorp-consul-server                19m
hashicorp-consul-ui                    19m

==> v1/Deployment
hashicorp-consul-connect-injector-webhook-deployment  19m
hashicorp-consul-sync-catalog                         0s

==> v1/Pod(related)

NAME                                                             READY  STATUS             RESTARTS  AGE
hashicorp-consul-fqzsz                                           1/1    Running            0         19m
hashicorp-consul-connect-injector-webhook-deployment-5c7d4pvh5h  1/1    Running            0         19m
hashicorp-consul-sync-catalog-769c859849-fnmh8                   0/1    ContainerCreating  0         0s
hashicorp-consul-server-0

Notice the basic Kubernetes services now appear in the Consul UI.

Consul UI with Kubernetes services

»Next steps

To learn more about Consul service mesh on Kubernetes, review the service mesh tutorials. To learn how to deploy Consul on a Kubernetes cluster, review the production deployment tutorial. To learn how to secure Consul and services for production, read the Secure Consul and Registered Services on Kubernetes tutorial.

Infrastructure

Security

Networking

Applications