Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Run Consul on Kubernetes

Secure Consul and Registered Services on Kubernetes

When operating pre-production environments it is common to initialize an environment without security features enabled, in order make the development process more transparent. Cleartext network traffic can be a useful debugging tool.

To secure an existing environment, or create an environment that is secured from the beginning, you can apply security controls using the official Consul Helm chart.

In this tutorial, you will:

  • Review the types of Consul service mesh traffic
  • Install an unsecured Consul service mesh on Kubernetes for development or debugging
  • Verify that gossip encryption, TLS, and ACLs are not enabled
  • Upgrade the installation to enable gossip encryption, TLS, and ACLs
  • Verify that gossip encryption, TLS, and ACLs are enabled
  • Deploy two example services to the service mesh
  • Configure zero-trust networking using Consul intentions

»Intended Audience

  • admins with privileged permissions to administer local, pre-production, and production clusters
  • developers with privileged permissions to administer at least local or pre-production clusters
  • contributors who wish to contribute to HashiCorp open source projects

»Prerequisites

To complete this tutorial in your own environment you will need the following:

This tutorial was tested using:

»Interactive lab

An interactive hands-on lab, covering the same concepts of this tutorial, is also available. If you do not want to set up the demo environment locally, click the Show Tutorial button to launch the demo environment.

»Types of Consul service mesh traffic

The diagram below depicts the conceptual model of an active Consul service mesh on Kubernetes. The number of nodes varies based on your configuration, but the fundamental architecture and communication flows are represented.

Consul on Kubernetes

  • Consul uses a gossip protocol to manage membership and broadcast messages to the service mesh.
  • Consul uses the remote procedure call (RPC) pattern for communication between client and server nodes. Each server provides an HTTP API that supports read and write operations on the catalog which tracks the status of nodes, services, and other state information.
  • Consul uses Access Control Lists (ACLs) to secure the UI, API, CLI, service communications, and agent communications. At the core, ACLs operate by grouping rules into policies, then associating one or more policies with a token.
  • Consul uses intentions to control which services may establish connections. Intentions can be managed via the API, CLI, or UI.

»Install an unsecured Consul service mesh

HashiCorp provides a first-class Consul on Kubernetes experience via the Consul Helm chart. Installing Consul on Kubernetes consists of three steps:

  1. Downloading the consul-helm chart repository to the development host
  2. Defining your desired configuration in a custom yaml file
  3. Applying the chart to your cluster using the Helm cli

»Download Helm repo

To download the Consul Helm repo issue the following command:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

You should receive the following output:

"hashicorp" has been added to your repositories

»Create an unsecured config file

To create a minimal, unsecured configuration file called dc1.yaml, execute the following command:

$ cat > dc1.yaml <<EOF
global:
  name: consul
  enabled: true
  datacenter: dc1

server:
  replicas: 1
  bootstrapExpect: 1

connectInject:
  enabled: true
EOF

»Apply the chart

Apply the chart using the following command. Note that tutorial has been passed as an argument. Helm requires a name be provided for each installation. You may name your installation anything you like, but in this tutorial tutorial is used. Make note of the name of your installation. It will be required to apply future upgrades. The install may take a minute or two to complete.

$ helm install -f ./dc1.yaml tutorial hashicorp/consul --wait

When the installation is complete, you should receive output similar to the following:

NAME: tutorial
LAST DEPLOYED: Wed Jul  8 15:56:47 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Consul!

Now that you have deployed Consul, you should look over the docs on using
Consul with Kubernetes available here:

https://www.consul.io/docs/platform/k8s/index.html

Your release is named tutorial.

To learn more about the release if you are using Helm 2, run:

  $ helm status tutorial
  $ helm get tutorial

To learn more about the release if you are using Helm 3, run:

  $ helm status tutorial
  $ helm get all tutorial

»Verify installation

You can verify the installation was successful by reviewing the status of pods using the following command:

$ watch kubectl get pods

Once pods have a status of Running as illustrated in the following output, type CTRL-C to stop the watch.

NAME                                                  READY   STATUS
consul-7d4h2                                          1/1     Running
consul-connect-injector-webhook-deployment            1/1     Running
consul-server-0                                       1/1     Running

At this point, the Consul service mesh has been installed in the Kubernetes cluster, but no security features have been enabled. This means that:

  • All gossip traffic between agents is in clear text
  • All RPC communications between agents is in clear text
  • There are no Access Controls in place
  • No intentions have been defined

»Verify security not enabled (optional)

To verify that network traffic is in cleartext inspect it.

»View server traffic

To view network traffic, connect to the Consul server container, and observe its traffic using the tcpdump program. You should now have a container named consul-server-0.

Connect to the server container with the following command:

$ kubectl exec -it consul-server-0 -- /bin/sh

The container images used by the Consul Helm chart are lightweight alpine images. They ship with limited tools. Issue the following commands to install tcpdump:

$ apk update && apk add tcpdump

You should receive output similar to the following:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
v3.9.6-44-g3992359a2b [http://dl-cdn.alpinelinux.org/alpine/v3.9/main]
v3.9.6-46-g228dc16f63 [http://dl-cdn.alpinelinux.org/alpine/v3.9/community]
OK: 9775 distinct packages available
OK: 10 MiB in 28 packages

Now start tcpdump to view traffic to the server container. This script limits the port range to the range of ports used by Consul.

$ tcpdump -an portrange 8300-8700 -A

Traffic occurs rapidly. The following output is an abbreviated example.

... m m.q.w...Node.consul-server-0.SeqNo....SourceAddr.
....SourceNode.control-plane.SourcePort. m
14:43:11.325622 IP 10.244.0.8.8301 > 10.244.0.5.8301: UDP, length 152
E....x@.@.}.

Inspect the output and observe that the traffic is in cleartext. Note the UDP operations, these entries are the gossip protocol at work. This proves that gossip encryption is not enabled.

Next, issue a Consul CLI command to prove two things:

  • RPC traffic is currently unencrypted
  • ACLs are not enabled

If the previous tcpdump process is still active, type CTRL-C to end it. This slightly modified version of the tcpdump command writes results to a log file. Start it now so that you can grep for interesting log entries later.

$ tcpdump -an portrange 8300-8700 -A > /tmp/tcpdump.log

You will receive the following output. Traffic is now being captured in a log file rather than being output to the terminal.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

Next, generate some Consul traffic in a different terminal using kubectl to exec a command against the Consul CLI on the client container. This traffic will orginate from the client to the server, and will prove that RPC traffic is in cleartext.

$ kubectl exec $(kubectl get pods -l component=client -o jsonpath='{.items[0].metadata.name}') -- consul catalog services

The command succeeds, but notice that you did not pass a -token option nor did you set the CONSUL_HTTP_TOKEN environment variable. One or the other is required when ACLs are enabled. This proves that ACLs are not enabled.

Now, from the terminal session on the server container type CTRL-C to stop the tcpdump process, and then search the log file for the CLI operation with the following command:

$ grep 'ServiceMethod' /tmp/tcpdump.log

The output will be similar to the following:

....A...Seq..ServiceMethod.Catalog.ListServices..AllowStale..Datacenter.dc1.Filter..MaxAge..MaxQueryTime..MaxStaleDuration..MinQueryIndex..MustRevalidate..NodeMetaFilters..RequireConsistent..Source..Datacenter..Ip..Node..Segment..StaleIfError..Token..UseCache.

Note that you are able to see the RPC operation in cleartext. This proves that RPC traffic is not encrypted.

Enter exit to end the terminal session on the server container.

$ exit

»Upgrade to a secured Consul service mesh

Next, upgrade your Consul service mesh installation to enable gossip encryption, TLS, and ACLs. You can upgrade the service mesh by updating your custom configuration yaml file settings, and then passing the new configuration to the helm upgrade CLI.

The following command will generate the secure-dc1.yaml file with gossip encryption, TLS, and ACLs enabled.

$ cat > secure-dc1.yaml <<EOF
global:
  name: consul
  enabled: true
  datacenter: dc1

  gossipEncryption:
    secretName: "consul-gossip-encryption-key"
    secretKey: "key"

  tls:
    enabled: true
    enableAutoEncrypt: true
    verify: true

  acls:
    manageSystemACLs: true

server:
  replicas: 1
  bootstrapExpect: 1

connectInject:
  enabled: true
EOF

Notice that the file now includes a gossipEncryption stanza.

gossipEncryption:
  # This stanza provides the helm chart with the name
  # of a Kubernetes secret to retrieve
  # and use as the gossip encryption key at runtime.
  # You must create a valid key and register it
  # as a secret with Kubernetes.
  secretName: 'consul-gossip-encryption-key'
  secretKey: 'key'

Use the following command to register a gossip encryption key as a Kubernetes secret that the helm chart can consume.

$ kubectl create secret generic consul-gossip-encryption-key --from-literal=key=$(consul keygen)

TLS is enabled by this stanza.

tls:
  enabled: true
  # By enabling TLS and setting `enableAutoEncrypt` to true,
  # the TLS system will configure itself. You
  # do not need to take any action beyond setting these values
  # in the config file.
  enableAutoEncrypt: true
  # The `verify` setting instructs Consul to verify the
  # authenticity of servers and clients.
  verify: true

ACLs are enabled by this stanza.

acls:
  # By setting `manageSystemACLs` to true, the ACL system
  # will configure itself. You do not need to take any
  # action beyond setting the value in the config file.
  manageSystemACLs: true

»Helm upgrade

The config file for the chart has been properly configured, and all necessary secrets have been registered with Kubernetes. Execute the following command to upgrade the installation with these changes. The upgrade may take a minute or two to complete.

$ helm upgrade -f ./secure-dc1.yaml tutorial hashicorp/consul --wait

»Verify the upgrade

Once the upgrade is complete, verify everything was successful by reviewing the status of pods using the following command:

$ watch kubectl get pods

Once all pods have a status of Running you can proceed to the next step.

»Verify security enabled

Now, you can verify that gossip encryption and TLS are enabled, and that ACLs are being enforced. Otherwise, you can skip ahead to Configuring Consul intentions.

In a separate terminal, forward port 8501 from the Consul server on Kubernetes so that you can interact with the Consul CLI from the development host.

$ kubectl port-forward --address 0.0.0.0 consul-server-0 8501:8501

Set the CONSUL_HTTP_ADDR environment variable to use the HTTPS address/port on the development host.

$ export CONSUL_HTTP_ADDR=https://127.0.0.1:8501

Export the CA file from Kubernetes so that you can pass it to the CLI.

$ kubectl get secret consul-ca-cert -o jsonpath="{.data['tls\.crt']}" | base64 --decode > ca.pem

Now, execute consul members and provide Consul with the ca-file option to verify TLS connections.

$ consul members -ca-file ca.pem

You now observe a list of all members of the service mesh.

Node               Address           Status  Type    Build  Protocol  DC
consul-server-0    10.244.0.13:8301  alive   server  1.8.0  2         dc1
dc1-control-plane  10.244.0.12:8301  alive   client  1.8.0  2         dc1

The actions you performed in this section of the tutorial prove that TLS is being enforced.

»Set an ACL token

Now, try launching a debug session.

$ consul debug -ca-file ca.pem

The command fails with the following message:

==> Capture validation failed: error querying target agent: Unexpected response code: 403 (Permission denied). verify connectivity and agent address

The 403 response proves that ACLs are being enforced. You have not yet supplied an ACL token, so the command fails. The consul members command worked because consul-helm created an anonymous token and set the following policy for it:

node_prefix "" {
   policy = "read"
}
service_prefix "" {
   policy = "read"
}

In order to perform operations that require a higher level of authority, you must provide a token with the necessary permissions. In this tutorial you will set the CONSUL_HTTP_TOKEN environment variable.

The consul-helm chart created several secrets during the initialization process and registered them with Kubernetes. For a list of all Kubernetes secrets issue the following command:

$ kubectl get secrets

Notice that one of the secrets is named consul-bootstrap-acl-token. To view the Kubernetes secret, execute the following command:

$ kubectl get secret consul-bootstrap-acl-token -o yaml | more

You should receive output similar to the following, though this example has been abbreviated.

apiVersion: v1
data:
  token: OTk5ZGU1MGUtY2Q2Zi1iZGZiLTlkZmQtMDgwYjc0YTJjM2Jh
kind: Secret

This secret contains the Consul ACL bootstrap token. The bootstrap token is a full access token that can perform any operation in the service mesh. In a production scenario, you should avoid using the bootstrap token, and instead create tokens with specific permissions. In this tutorial, you will use it for convenience.

The value of interest is the string in the data stanza's token field. That value is a base64 encoded string that contains the bootstrap token generated during the consul-helm ACL init process.

For this tutorial you can retrieve the value, decode it, and set it to the CONSUL_HTTP_TOKEN environment variable with the following command.

$ export CONSUL_HTTP_TOKEN=$(kubectl get secrets/consul-bootstrap-acl-token --template={{.data.token}} | base64 -d)

Try to start a debug session again with an ACL token set.

$ consul debug -ca-file ca.pem

The command succeeds. This proves that ACLs are being enforced. Type CTRL-C to end the debug session in the terminal.

»Verify that network traffic is encrypted

Now that you have proven that ACLs are enabled, and that TLS verification is being enforced, you will prove that all gossip and RPC traffic is encrypted.

Start a shell session on the server container.

$ kubectl exec -it consul-server-0 -- /bin/sh

Since the containers were recycled during the helm upgrade, you will have to add tcpdump again.

$ apk update && apk add tcpdump

Next, start tcpdump and observe the gossip traffic.

$ tcpdump -an portrange 8300-8700 -A

Notice that none of the traffic is in cleartext, as it was before. This proves that gossip traffic is now encrypted.

Type CTRL-C to stop the tcpdump session. Use the following command to restart tcpdump and pipe the results to a log file so that you can search for cleartext RPC traffic.

$ tcpdump -an portrange 8300-8700 -A > /tmp/tcpdump.log

From a different terminal, try to list services with the Consul cli.

$ kubectl exec $(kubectl get pods -l component=client -o jsonpath='{.items[0].metadata.name}') -- consul catalog services -token $(kubectl get secrets/consul-bootstrap-acl-token --template={{.data.token}} | base64 -d)

You should receive the following output.

consul

Switch back to server, type CTRL-C to stop tcpdump, and grep log for an RPC entry

$ grep 'ServiceMethod' /tmp/tcpdump.log

Notice that no rows were found this time. This proves that RCP traffic is now encrypted. Exit the terminal session on the server container.

$ exit

»Configure Consul intentions

Now, deploy two sample services, and manage them using Consul intentions.

»Deploy example services

To simulate an active environment you will deploy a static client service and an upstream backend service. First, issue the following command to create a file named server.yaml that will be used to create an http echo server on Kubernetes:

$ cat > server.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: v1
kind: Pod
metadata:
  name: static-server
  annotations:
    "consul.hashicorp.com/connect-inject": "true"
spec:
  containers:
    - name: static-server
      image: hashicorp/http-echo:latest
      args:
        - -text="hello world"
        - -listen=:8080
      ports:
        - containerPort: 8080
          name: http
  serviceAccountName: static-server
EOF

Next, deploy the sample backend service.

$ kubectl apply -f server.yaml

Next, create a file named client.yaml that defines the sample client service.

$ cat > client.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-client
---
apiVersion: v1
kind: Pod
metadata:
  name: static-client
  annotations:
    "consul.hashicorp.com/connect-inject": "true"
    "consul.hashicorp.com/connect-service-upstreams": "static-server:1234"
spec:
  containers:
    - name: static-client
      image: tutum/curl:latest
      command: [ "/bin/sh", "-c", "--" ]
      args: [ "while true; do sleep 30; done;" ]
  serviceAccountName: static-client
EOF

Next, deploy the sample client.

$ kubectl apply -f client.yaml

Finally, ensure the deployment was successful by executing the following command:

$ watch kubectl get pods

Make sure all pods/containers having a status of Running before proceeding to the next section.

NAME                                                  READY   STATUS
static-client                                         4/4     Running
static-server                                         4/4     Running
consul-connect-injector-webhook-deployment            1/1     Running
consul-fgxx5                                          1/1     Running
consul-server-0                                       1/1     Running

With manageSystemACLs set to true, the Consul Helm chart will, by default, create a deny all intention. This means that services will not be able to communicate until an explicit intention is defined that allows them to communicate.

Issue the following command to validate that the default deny all intention is enforced.

$ kubectl exec static-client -c static-client -- curl -s http://127.0.0.1:1234/

The command exits with a non-zero exit code. This proves the intention is enforced.

Defaulting container name to static-client.
Use 'kubectl describe pod/static-client -n default' to see all of the containers in this pod.
command terminated with exit code 7

Now, create an allow intention for client to server traffic.

$ consul intention create -ca-file ca.pem -allow static-client static-server

You should received output indicating the intention was created.

Created: static-client => static-server (allow)

Finally, validate the intention allows traffic from the client to the server. If this fails, wait a few seconds for the intention to be applied, and try again.

$ kubectl exec static-client -c static-client -- curl -s http://127.0.0.1:1234/

Notice the output now states "hello world".

"hello world"

This proves the intention is allowing traffic from the client to the server.

»Next steps

In this tutorial, you enabled Consul security controls for Consul on Kubernetes using the Consul Helm chart.

Specifically, you:

  • Installed an unsecured Consul service mesh on Kubernetes for development or debugging
  • Verified that gossip encryption, TLS, and ACLs were not enabled
  • Upgraded the installation to enable gossip encryption, TLS, and ACLs
  • Verified that gossip encryption, TLS, and ACLs were now enabled
  • Deployed two example services to the service mesh
  • Configured zero-trust networking using Consul intentions

Next, consider reviewing our L7 observability tutorial to learn more techniques for monitoring or debugging Consul on Kubernetes.