Manage Consul with Kubernetes Custom Resource Definitions (CRDs)

Configuration entries are Consul resources used to define datacenter-wide configuration defaults for various aspects of the service mesh, or to create service specific configurations. As of Consul 1.9, configuration entries can be created as Custom Resource Definitions (CRDs), and managed using kubectl.

In this tutorial you will:

• Learn how to work with config entries as Custom Resource Definitions (CRDs)
• Download the latest helm chart
• Install or upgrade Consul to enable CRDs
• Deploy an application workload
• Configure a service intention
• Modify a service intention
• Delete a service intention

»Prerequisites

To complete this tutorial you will need:

• Access to a Kubernetes cluster (Minikube v1.10.1+, kind v0.8.1+, or cloud-based k8s)
• kubectl
• helm v3.2.1+
• A text editor
• Command line access

»Working with config entry Custom Resource Definitions (CRDs)

Prior to Consul 1.9, when using configuration entries with Consul in Kubernetes, an operator would either need to exec into a running container, or configure a host that could interact with the datacenter using a local Consul binary. In versions older than 1.9, configuration entries have to be managed with the Consul CLI, the HTTP API, or provided to agents during startup as configuration files.

As of Consul 1.9, most configuration entries can be managed as Kubernetes Custom Resource Definitions (CRDs). You can now define most configuration entries as YAML, and register them with Consul using the familiar kubectl apply command.

The configuration entries currently available as CRDs for Consul on Kubernetes are:

• proxy-defaults - controls proxy configuration
• service-defaults - configures defaults for all the instances of a given service
• service-resolver - matches service instances with a specific Connect upstream discovery requests
• service-router - defines where to send layer 7 traffic based on the HTTP route
• service-splitter - defines how to divide requests for a single HTTP route based on percentages
• service-intentions - defines restrictions for specific service to service interactions

»Download Helm chart

$ helm repo add hashicorp https://helm.releases.hashicorp.com

$ helm repo add hashicorp https://helm.releases.hashicorp.com

"hashicorp" has been added to your repositories

"hashicorp" has been added to your repositories

»Verify chart version

To ensure you have version 0.32.0 of the Helm chart, search your local repo.

$ helm search repo hashicorp/consul

$ helm search repo hashicorp/consul

Example output:

NAME                CHART VERSION   APP VERSION DESCRIPTION
hashicorp/consul    0.32.0          1.10.0      Official HashiCorp Consul Chart

NAME                CHART VERSION   APP VERSION DESCRIPTIONhashicorp/consul    0.32.0          1.10.0      Official HashiCorp Consul Chart

If you do not see that you have the correct version of the chart locally, try updating your Helm repo.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository

$ helm repo updateHang tight while we grab the latest from your chart repositories......Successfully got an update from the "hashicorp" chart repository

»Download sample code

This tutorial comes with sample code you can use to test CRDs. From the command line, clone the GitHub repository that contains the configuration you will use with this tutorial.

$ git clone https://github.com/hashicorp/learn-consul-kubernetes.git

$ git clone https://github.com/hashicorp/learn-consul-kubernetes.git

Next, change into the directory containing the repository you just cloned.

$ cd learn-consul-kubernetes/custom-resource-definitions

$ cd learn-consul-kubernetes/custom-resource-definitions

Now, checkout the tagged version verified for this tutorial.

$ git checkout tags/v0.0.8

$ git checkout tags/v0.0.8

»Enable Consul CRDs

CRDs can be enabled in a Consul on Kubernetes datacenter by adding the following top level stanza to the Helm chart configuration file, and then installing or upgrading the datacenter using the chart with the updated configuration file.

controller:
  enabled: true

controller:  enabled: true

No other changes to the Helm chart configuration is required.

global:
  name: consul
  datacenter: dc1
server:
  # use 1 server
  replicas: 1
connectInject:
  enabled: true
  # inject an envoy sidecar into every new pod,
  # except for those with annotations that prevent injection
  default: true
# enable CRDs
controller:
  enabled: true

global:  name: consul  datacenter: dc1server:  # use 1 server  replicas: 1connectInject:  enabled: true  # inject an envoy sidecar into every new pod,  # except for those with annotations that prevent injection  default: true# enable CRDscontroller:  enabled: true

$ helm install -f ./config.yaml consul hashicorp/consul  --version "0.32.0" --wait

$ helm install -f ./config.yaml consul hashicorp/consul  --version "0.32.0" --wait

Example output:

NAME: consul
...OMITTED...
  $ helm status consul
  $ helm get all consul

NAME: consul...OMITTED...  $ helm status consul  $ helm get all consul

»Deploy a demo application workload

Now that you have a datacenter with CRDs enabled, you will deploy an demo application to test the features. Deploy the demo application using the kubectl apply command.

$ kubectl apply -f hashicups

$ kubectl apply -f hashicups

You should receive the following output.

service/frontend created
serviceaccount/frontend created
configmap/nginx-configmap created
deployment.apps/frontend created
service/postgres created
serviceaccount/postgres created
deployment.apps/postgres created
servicedefaults.consul.hashicorp.com/postgres created
service/product-api created
serviceaccount/product-api created
configmap/db-configmap created
deployment.apps/product-api created
service/public-api created
serviceaccount/public-api created
deployment.apps/public-api created

service/frontend createdserviceaccount/frontend createdconfigmap/nginx-configmap createddeployment.apps/frontend createdservice/postgres createdserviceaccount/postgres createddeployment.apps/postgres createdservicedefaults.consul.hashicorp.com/postgres createdservice/product-api createdserviceaccount/product-api createdconfigmap/db-configmap createddeployment.apps/product-api createdservice/public-api createdserviceaccount/public-api createddeployment.apps/public-api created

HashiCups may take a minute or more to deploy and start. Use the following command to watch your deployment.

$ watch kubectl get pods

$ watch kubectl get pods

The output of that command will be similar to the following. The application is ready for use when all pods show a status of Running.

NAME                                                          READY   STATUS    RESTARTS   AGE
consul-6gjth                                                  1/1     Running   0          9m18s
consul-connect-injector-webhook-deployment-7f74c84444-b89m9   1/1     Running   0          9m18s
consul-controller-69c8b5f65d-kw76c                            1/1     Running   0          9m18s
consul-server-0                                               1/1     Running   0          9m16s
consul-webhook-cert-manager-665b8d7997-qllc7                  1/1     Running   0          9m18s
frontend-cf78c4879-5rjk8                                      3/3     Running   0          3m28s
postgres-7d774895db-7xq7v                                     3/3     Running   0          3m28s
product-api-66d6865657-br2fp                                  3/3     Running   1          3m28s
public-api-69fb4d6ffc-8mqlx                                   3/3     Running   0          3m29s

NAME                                                          READY   STATUS    RESTARTS   AGEconsul-6gjth                                                  1/1     Running   0          9m18sconsul-connect-injector-webhook-deployment-7f74c84444-b89m9   1/1     Running   0          9m18sconsul-controller-69c8b5f65d-kw76c                            1/1     Running   0          9m18sconsul-server-0                                               1/1     Running   0          9m16sconsul-webhook-cert-manager-665b8d7997-qllc7                  1/1     Running   0          9m18sfrontend-cf78c4879-5rjk8                                      3/3     Running   0          3m28spostgres-7d774895db-7xq7v                                     3/3     Running   0          3m28sproduct-api-66d6865657-br2fp                                  3/3     Running   1          3m28spublic-api-69fb4d6ffc-8mqlx                                   3/3     Running   0          3m29s

Type CTRL-C to stop watching the pods once all pods have a status of Running.

You can test the application by viewing the user interface. Do this by forwarding the frontend deployment's port 80 to your development host.

$ kubectl port-forward deploy/frontend 8081:80
Forwarding from 127.0.0.1:8081 -> 80
Forwarding from [::1]:8081 -> 80

$ kubectl port-forward deploy/frontend 8081:80Forwarding from 127.0.0.1:8081 -> 80Forwarding from [::1]:8081 -> 80

Navigate to http://localhost:8081 in a browser window. You should observe the following screen.

»Configure a service intention

Also, new as of Consul 1.9, is the ability to define permissions based on application-layer (i.e. layer 7) request attributes such as the HTTP method, path, or the presence/absence of a header.

To create the service intention for this tutorial, you first need to define the protocols for the frontend and public-api services as HTTP because application aware intentions can only operate on HTTP services. To do so, you need to create ServiceDefaults config entries for the frontend and public-api services:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: frontend
spec:
  protocol: 'http'
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: public-api
spec:
  protocol: 'http'

apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: frontendspec:  protocol: 'http'---apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: public-apispec:  protocol: 'http'

Now you will define a ServiceIntentions config entry that manages the traffic between the frontend and public-api services. ServiceIntentions can have only a single destination, but can have multiple sources, and each source can have its own permissions specified. For example, this configuration denies all traffic from the frontend service to the health route. The health route is only for internal use and could be abused by external users, so traffic from the frontend service to this exact path is denied.

- action: deny
    http:
      pathExact: "/health"

- action: deny    http:      pathExact: "/health"

Next, the permissions for the primary allow action are specified. In the HashiCups application, it is required that the client passes an Authorization header. Also, the application only allows the GET, PUT, POST, and DELETE HTTP methods.

- action: allow
  http:
    pathPrefix: '/'
    methods:
      - GET
      - PUT
      - POST
      - DELETE
    header:
      - name: 'Authorization'
        present: true

- action: allow  http:    pathPrefix: '/'    methods:      - GET      - PUT      - POST      - DELETE    header:      - name: 'Authorization'        present: true

Finally, a catch-all deny action is specified. In this particular datacenter, there is no default deny policy defined, so you must define a deny action for this specific destination service.

- action: deny
  http:
    pathPrefix: '/'

- action: deny  http:    pathPrefix: '/'

By placing this at the end of the list, any traffic that doesn't match either of the previous actions will trigger the action, and be denied.

Putting it all together, your manifest should resemble the following:

# Application aware intentions can only be applied to services that use the HTTP protocol
# so you must first define service default config entries for the targeted
# services.
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: frontend
spec:
  protocol: 'http'
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: public-api
spec:
  protocol: 'http'
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: public-api
spec:
  # Name of the destination service affected by this ServiceIntentions entry
  destination:
    name: public-api
  # The set of traffic sources affected by this ServiceIntentions entry
  sources:
    # The first affected traffic source-maps to service name of inbound traffic source
    - name: frontend
      # The set of permissions to apply when frontend is the traffic source
      # The first permission to match in the list is terminal and stops further evaluation.
      permissions:
        # Add this to always deny traffic from the frontend service to /health route
        - action: deny
          http:
            pathExact: '/health'
        # This permission now defines the conditions that should be allowed
        # Allow traffic to all paths for the GET, PUT, POST, DELETE verbs as long as an
        # Authorization header is present
        - action: allow
          http:
            pathPrefix: '/'
            methods:
              - GET
              - PUT
              - POST
              - DELETE
            header:
              - name: 'Authorization'
                present: true
        # Define a deny intention for all other traffic
        - action: deny
          http:
            pathPrefix: '/'

# Application aware intentions can only be applied to services that use the HTTP protocol# so you must first define service default config entries for the targeted# services.apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: frontendspec:  protocol: 'http'---apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: public-apispec:  protocol: 'http'---apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceIntentionsmetadata:  name: public-apispec:  # Name of the destination service affected by this ServiceIntentions entry  destination:    name: public-api  # The set of traffic sources affected by this ServiceIntentions entry  sources:    # The first affected traffic source-maps to service name of inbound traffic source    - name: frontend      # The set of permissions to apply when frontend is the traffic source      # The first permission to match in the list is terminal and stops further evaluation.      permissions:        # Add this to always deny traffic from the frontend service to /health route        - action: deny          http:            pathExact: '/health'        # This permission now defines the conditions that should be allowed        # Allow traffic to all paths for the GET, PUT, POST, DELETE verbs as long as an        # Authorization header is present        - action: allow          http:            pathPrefix: '/'            methods:              - GET              - PUT              - POST              - DELETE            header:              - name: 'Authorization'                present: true        # Define a deny intention for all other traffic        - action: deny          http:            pathPrefix: '/'

Use kubectl along with the file named service-intentions.yaml in the repository you downloaded earlier to register the service intention resource with Kubernetes.

$ kubectl apply -f service-intentions.yaml
servicedefaults.consul.hashicorp.com/frontend created
servicedefaults.consul.hashicorp.com/public-api created
serviceintentions.consul.hashicorp.com/public-api created

$ kubectl apply -f service-intentions.yamlservicedefaults.consul.hashicorp.com/frontend createdservicedefaults.consul.hashicorp.com/public-api createdserviceintentions.consul.hashicorp.com/public-api created

Now you can interact with the service intention using kubectl just like you can any other kube-native resource. For example, you can get a list of all service intentions with this command:

$ kubectl get serviceintentions
NAME         SYNCED
public-api   True

$ kubectl get serviceintentionsNAME         SYNCEDpublic-api   True

You can also inspect details about the service intention using kubectl describe like so:

$ kubectl describe serviceintentions/public-api
Name:         public-api
Namespace:    default
Labels:       <none>
Annotations:  API Version:  consul.hashicorp.com/v1alpha1
Kind:         ServiceIntentions
...OMITTED...
Spec:
  Destination:
    Name:  public-api
  Sources:
    Name:  frontend
    Permissions:
      Action:  deny
      Http:
        Path Exact:  /health
      Action:        allow
      Http:
        Header:
          Name:     Authorization
          Present:  true
        Methods:
          GET
          PUT
          POST
          DELETE
        Path Prefix:  /
Status:
  Conditions:
    Last Transition Time:  2020-10-08T15:13:15Z
    Status:                True
    Type:                  Synced
Events:                    <none>

$ kubectl describe serviceintentions/public-apiName:         public-apiNamespace:    defaultLabels:       <none>Annotations:  API Version:  consul.hashicorp.com/v1alpha1Kind:         ServiceIntentions...OMITTED...Spec:  Destination:    Name:  public-api  Sources:    Name:  frontend    Permissions:      Action:  deny      Http:        Path Exact:  /health      Action:        allow      Http:        Header:          Name:     Authorization          Present:  true        Methods:          GET          PUT          POST          DELETE        Path Prefix:  /Status:  Conditions:    Last Transition Time:  2020-10-08T15:13:15Z    Status:                True    Type:                  SyncedEvents:                    <none>

This configuration should allow traffic, because the frontend service passes the required header and uses the GET HTTP method. Revisit the application in a browser tab at localhost:8081, and verify you can still visit the web page.

»Modify a service intention

Next, modify the service intention you created a moment ago.

$ sed -i '' 's/Authorization/api-token/g' service-intentions.yaml

$ sed -i '' 's/Authorization/api-token/g' service-intentions.yaml

$ sed '' 's/Authorization/api-token/g' service-intentions.yaml

$ sed '' 's/Authorization/api-token/g' service-intentions.yaml

This script updates the service intention. It changes the name of the required header to a value the application does not provide. Review the file and notice the "allow" action has been altered. The header key "Authorization" has been changed to "api-token".

header:
  - name: 'api-token'
    present: true

header:  - name: 'api-token'    present: true

Use kubectl apply to apply the modified configuration.

$ kubectl apply -f service-intentions.yaml

$ kubectl apply -f service-intentions.yaml

Example output:

servicedefaults.consul.hashicorp.com/frontend unchanged
servicedefaults.consul.hashicorp.com/public-api unchanged
serviceintentions.consul.hashicorp.com/public-api configured

servicedefaults.consul.hashicorp.com/frontend unchangedservicedefaults.consul.hashicorp.com/public-api unchangedserviceintentions.consul.hashicorp.com/public-api configured

Now, navigate to localhost:8081 in a browser window. You should observe that you are no longer able to successfully visit the page. If you inspect the network call in the browser's developer tools, you should observe that the request was denied with the following error message:

RBAC: access denied

RBAC: access denied

»View the service intention in the Consul UI

To access the Consul UI, forward the consul-server-0 pod's port 8500 to the development host, so that you can access the Consul UI in a browser.

$ kubectl port-forward consul-server-0 8500:8500

$ kubectl port-forward consul-server-0 8500:8500

Example output.

Forwarding from 127.0.0.1:8500 -> 8500
Forwarding from [::1]:8500 -> 8500

Forwarding from 127.0.0.1:8500 -> 8500Forwarding from [::1]:8500 -> 8500

Now visit http://localhost:8500 in a new browser tab. Click on "Intentions" in the top navigation bar, and you will observe a list of intentions. The screen have one entry as illustrated in this screenshot.

Notice there is a badge next to the frontend entry indicating that the service has L7 intentions applied. Also, noticed that there is a badge that says Managed by CRDs. Click on the row in the list, and you will be redirected to a page that contains the service intentions configuration details and looks like this:

»Delete a service intention

Now, use kubectl delete to remove the service intention.

$ kubectl delete -f service-intentions.yaml

$ kubectl delete -f service-intentions.yaml

Example output:

servicedefaults.consul.hashicorp.com "frontend" deleted
servicedefaults.consul.hashicorp.com "public-api" deleted
serviceintentions.consul.hashicorp.com "public-api" deleted

servicedefaults.consul.hashicorp.com "frontend" deletedservicedefaults.consul.hashicorp.com "public-api" deletedserviceintentions.consul.hashicorp.com "public-api" deleted

If you navigate back to the "Intentions" screen in the Consul UI, you will see that the list is now empty.

Navigate to localhost:8081 in a browser window. You should once again be able to visit the page.

»Next steps

In this tutorial you:

• Learned how to work with config entries as Custom Resource Definitions (CRDs)
• Downloaded the latest helm chart
• Installed/upgraded a cluster to enable Consul CRDs
• Deployed an application workload
• Configured a service intention
• Modified a service intention
• Deleted a service intention

To learn more about how to manage Consul config entries using Consul on Kubernetes visit the documentation.

To learn more about securing your service mesh, check out our Securing Agents and Registered Services tutorial.