Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Run Consul on Kubernetes

Consul and Kubernetes Deployment Guide

This tutorial covers the necessary steps to install and configure a new Consul datacenter on Kubernetes, as defined in the Consul Reference Architecture tutorial. By the end of this tutorial, you will be able to identify the installation prerequisites, customize the Helm chart to fit your environment requirements, and interact with your new Consul datacenter.

»Configure Kubernetes permissions to deploy Consul

Before deploying Consul, you will need to have a kubectl credentials that allow you to create and modify policies, deploy services, access the Kubernetes dashboard, create secrets, and create RBAC objects. You can find documentation for RBAC and service accounts for the following cloud providers.

Note, Consul can be deployed on any properly configured Kubernetes cluster in the cloud or on premises.

In order to view the Kubernetes dashboard, you will need to create a ClusterRoleBinding:

$ kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard

Finally, you may need to create Kubernetes secrets to store Consul data. You can reference these secrets in the customized Helm chart values file.

  • If you have purchased Enterprise Consul, the enterprise license file should be used with the official image, hashicorp/consul-enterprise:<latest version>-ent.

  • Enable encryption to secure gossip traffic within the Consul datacenter.

»Configure Helm chart

Now that you have prepared your Kubernetes cluster, you can customize the Helm chart. First, you will need to add the HashiCorp Helm Chart repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

The Helm chart is configured with default values. You can override these values by creating your own config.yaml file and passing it to helm install with the -f flag, e.g. helm install hashicorp hashicorp/consul -f config.yaml. Below we detail some of the parameters you should customize and provide an example file, however you should consider your particular production needs when configuring your chart.

»Global key

The values under the global key will affect all the other parameters in the chart.

To enable all of the Consul components in the Helm chart, set enabled to true. This means servers, clients, Consul DNS, and the Consul UI will be installed with their defaults. You should also set the following global parameters based on your specific environment requirements.

  • image is the name and tag of the Consul Docker image.
  • datacenter the name of your Consul datacenter.
  • domain the domain Consul uses for DNS queries.

For security, set the bootstrapACLs parameter to true. This will enable Kubernetes to initially setup Consul's ACL system.

Read the Consul Helm chart documentation to review all the global parameters.

»Consul UI

To enable the Consul web UI update the ui section to your values file and set enabled to true.

Note, you can also set up a LoadBalancer resource or other service type in Kubernetes to make it easier to access the UI.

»Consul servers

For production deployments, you will need to deploy 3 or 5 Consul servers for quorum and failure tolerance. For most deployments, 3 servers are adequate.

In the server section set both replicas and bootstrapExpect to 3. This will deploy three servers and cause Consul to wait to perform leader election until all three are healthy. The resources will depend on your environment; in the example at the end of the tutorial, the resources are set for a large environment.

»Affinity

To ensure the Consul servers are placed on different Kubernetes nodes, you will need to configure affinity. Otherwise, the failure of one Kubernetes node could cause the loss of multiple Consul servers, and result in quorum loss. The default values.yaml has affinity configured correctly.

»Enterprise license

If you have an Enterprise license you should reference the Kubernetes secret in the enterpriseLicense parameter.

Read the Consul Helm chart documentation to review all the server parameters

»Consul clients

A Consul client is deployed on every Kubernetes node, so you do not need to specify the number of clients for your deployments. You will need to specify resources. The resources in the example at the end of this tutorial should be sufficient for most production scenarios since Consul clients are designed for horizontal scalability.

Read the Consul Helm chart documentation to review all the client parameters

»Consul Connect injection security

To enable Consul Connect service mesh set enabled under the connectInject key to true. In the connectInject section you will also configure security features. Enabling the default parameter will allow the injector to automatically inject the Connect sidecar proxy into all pods. If you would prefer to manually annotate which pods to inject, you can set this to false. Setting the aclBindingRuleSelector parameter to serviceaccount.name!=default ensures that new services do not all receive the same token if you are only using a default service account. This setting is only necessary if you have enabled ACLs in the global section.

Read more about the Connect Inject parameters.

»Complete example

Your finished config.yaml file should resemble the following example. For more complete descriptions of all the available parameters, check the values.yaml file provided with the Helm chart and the reference documentation.

# config.yaml

# Configure global settings in this section.
global:
  # Specify the Consul image to use
  image: 'consul:1.5.0'
  domain: consul
  datacenter: primarydc
  # Bootstrap ACLs within Consul. This is highly recommended.
  bootstrapACLs: true
  # Gossip encryption
  gossipEncryption:
    secretName: 'encrypt-key'
    secretKey: 'key'
# Configure your Consul servers in this section.
server:
  # Specify three servers that wait until all are healthy to bootstrap the Consul cluster.
  replicas: 3
  bootstrapExpect: 3
  # Specify the resources that servers request for placement. These values will serve a large environment.
  resources: |
    requests:
      memory: "32Gi"
      cpu: "4"
      disk: "50Gi"
    limits:
      memory: "32Gi"
      cpu: "4"
      disk: "50Gi"
  # If using Enterprise, reference the Kubernetes secret that holds your license here
  enterpriseLicense:
    secretName: 'consul-license'
    secretKey: 'key'
  # Prevent Consul servers from co-location on Kubernetes nodes.
  affinity: |
    podAntiAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       - labelSelector:
           matchLabels:
             app: {{ template "consul.name" . }}
             release: "{{ .Release.Name }}"
             component: server
       topologyKey: kubernetes.io/hostname
# Configure Consul clients in this section
client:
  # Specify the resources that clients request for deployment.
  resources: |
    requests:
      memory: "8Gi"
      cpu: "2"
      disk: "15Gi"
    limits:
      memory: "8Gi"
      cpu: "2"
      disk: "15Gi"
# Enable and configure the Consul UI.
ui:
  enabled: true
# Configure security for Consul Connect pod injection
connectInject:
  enabled: true
  default: true
  aclBindingRuleSelector: “serviceaccount.name!=default

»Deploy Consul

Now that you have customized your config.yaml file, you can deploy Consul with Helm. This should only take a few minutes. The Consul pods should appear in the Kubernetes dashboard immediately and you can monitor the deployment process there.

$ helm install -f config.yaml hashicorp hashicorp/consul

To check the deployment process on the command line you can use kubectl.

$ kubectl get pods

»Next steps

In this tutorial, you configured Consul using the Helm chart for a production environment. This involved ensuring that your Kubernetes cluster had a properly distributed configuration for Consul servers, specifying enough resources for your agents, securing the datacenter with ACLs and gossip encryption, and enabling other Consul functionality including Connect service mesh and the Consul UI.

Now you can interact with your Consul datacenter through the UI or CLI.

If you exposed the UI using a load balancer it will be available at the LoadBalancer Ingress IP address and Port that is output from the following command. Note, you will need to replace consul server with the server name from your datacenter.

$ kubectl describe services hashicorp-consul-server

To access the Consul CLI, open a terminal session using the Kubernetes CLI.

$ kubectl exec <pod name> -it /bin/ash

To learn more about how to interact with your Consul datacenter or use it for service discovery, as a service mesh, or for service configuration, try one of Learn's Operations or Development collection. Follow the Security and Networking collection to learn more about securing your Consul datacenter.