Secure Service Communication with HCP Consul Service Mesh and Envoy

Consul service mesh secures service-to-service communication with authorization and encryption. Applications can use sidecar proxies in a service mesh configuration to automatically establish TLS connections for inbound and outbound connections without being aware of the network configuration and topology. In addition to securing your services, Consul service mesh can also intercept data about service-to-service communications and surface it to monitoring tools.

Hashicorp Cloud Platform (HCP) Consul is secure by default. This includes the Access Control List (ACL) system, mTLS encryption, and gossip encryption. These security features protect Consul communication against eavesdropping, tampering, and spoofing. See the Consul security model for more information.

In this tutorial, you will register two services and their sidecar proxies on a Consul client. You will then start the services and sidecar proxies. Finally, you will demonstrate that the service-to-service communication is going through the proxies by stopping traffic with an "intention".

While this tutorial uses elements that are not suitable for production environments-high privilege ACL tokens for Consul clients, mock services, and a publicly accessible Consul UI endpoint—it will teach you the process of deploying your own services using Consul service mesh. At the end of this tutorial, we present additional information about how to adapt this process for production environments.

»Prerequisites

This tutorial builds on the concepts and environment created in the Connect a Consul client to HCP Consul tutorial.

In addition to the standard Consul security group rules, create a security group rule for your Linux EC2 instance that allows inbound traffic on port 9002. This port will only be used for this exercise, so it may be safely removed afterwards.

»Download and configure mock service resources

Download and extract the following mock services that you will use for this tutorial.

• Counting Service (linux_amd64)
• Dashboard Service (linux_amd64)

Copy these services to your Linux EC2 instance and then make them executable.

$ chmod 755 counting-service_linux_amd64

$ chmod 755 dashboard-service_linux_amd64

»Verify Consul agent health

Connect to your Linux EC2 instance and set your ACL token as the CONSUL_HTTP_TOKEN environment variable for your terminal session. This variable will need to be set for any terminal session that interacts with the HCP Consul server.

$ export CONSUL_HTTP_TOKEN=YOURACLTOKEN

Use the consul members command to verify your agent status.

$ consul members
Node                       Address            Status  Type    Build      Protocol  DC   Segment
ip-172-25-20-84            172.25.20.84:8301  alive   server  1.9.1+ent  2         dc1  <all>
ip-10-0-1-51.ec2.internal  10.0.1.51:8301     alive   client  1.9.1      2         dc1  <default>

»Install Envoy

Envoy is an open source, distributed proxy designed for single services and applications. Consul configures Envoy to run alongside your applications and abstracts the network by providing common features in a platform-agnostic manner.

To install Envoy, you will use the getenvoy installer. This simplifies the installation process by removing the need to copy multiple files and scripts, set the correct environment variables, and run the scripts. Alternatively, you can obtain container-based builds of Envoy directly from the Envoy website, or you can obtain packages of Envoy binary builds from a third-party project, getenvoy.io.

Download Envoy's streamlined installer, getenvoy.

$ curl -L https://getenvoy.io/cli | sudo bash -s -- -b /usr/local/bin

Use getenvoy to install Envoy version 1.16.0.

$ getenvoy run standard:1.16.0 -- --version

Consul needs to be able to find the envoy binary in the $PATH to automatically start it without specifying the binary location. Copy the envoy binary in a location in your $PATH.

$ sudo cp ~/.getenvoy/builds/standard/1.16.0/linux_glibc/bin/envoy /usr/local/bin/

Check that Envoy is in your $PATH by running the following command.

$ envoy --version

»Register the services and sidecar proxies

Services have to be registered with Consul. Consul service mesh uses service registrations to determine where to route proxied traffic.

There are several ways to register services in Consul:

• In code, directly from a Consul-aware application

• With an orchestrator, like Nomad or Kubernetes

• Using configuration files that are loaded at node startup

• Programmatic registration using the HTTP API to register them with a JSON or HCL specification

• Manual registration using the CLI

For this tutorial, you will use the consul service register CLI command to load them into the catalog.

»Create the counting service definition

First, create a file in your working directory called counting.json. This file will define the Counting service and its sidecar proxy. The definition should include the name of the service, the port the service listens on, and a connect block with the sidecar_service block. If the sidecar_service block is empty, Consul will use default parameters. The definition also includes an optional service health check.

HCP Consul has ACLs enabled by default to secure access to the datacenter. To successfully register a service, you will need to add an ACL token within the service definition file.

{
    "service": {
      "name": "counting",
      "id": "counting-1",
      "port": 9003,
      "token": "YOURACLTOKEN",
      "check": {
        "http": "http://localhost:9003/health",
        "method": "GET",
        "interval": "1s",
        "timeout": "1s"
      },
      "connect": {
        "sidecar_service": {}
      }
    }
  }

»Create the dashboard service definition

Create the Dashboard service and proxy definition in the same way. First, create a file in your working directory named dashboard.json with the following content.

{
    "service": {
      "name": "dashboard",
      "port": 9002,
      "token": "YOURACLTOKEN",
      "connect": {
        "sidecar_service": {
          "proxy": {
              "upstreams": [{
                "destination_name": "counting",
                "local_bind_port": 5000
              }]
            }
        }
      },
      "check": {
        "id": "dashboard-check",
        "http": "http://localhost:9002/health",
        "method": "GET",
        "interval": "1s",
        "timeout": "1s"
      }
    }
  }

Notice that the dashboard definition also includes an upstream block. Upstreams are ports on the local host that will be proxied to an external service. The upstream block's local_bind_port value is the port your service will communicate with to reach the external, or upstream service that this service depend on. The destination name is the Consul service name that the local_bind_port will proxy to.

In this scenario, the dashboard service depends on the counting service. With this configuration, when the dashboard service connects to localhost:5000 it is proxied across the service mesh to the counting service.

»Register the services and proxies

Run the following commands from your working directory to submit the service definitions to your Consul agent.

$ consul services register counting.json
Registered service: counting

$ consul services register dashboard.json
Registered service: dashboard

»Verify the services are registered

Run consul catalog services to verify that your registered services and sidecar proxies are present in the Consul services catalog.

$ consul catalog services

consul
counting
counting-sidecar-proxy
dashboard
dashboard-sidecar-proxy

From the output you can notice that two extra services *-sidecar-proxy are automatically registered alongside the ones defined in the configuration files.

»Allow service communication

Intentions define access control for services in your service mesh and are used to control which services may establish connections. This paradigm is a vastly improved operator user experience compared to firewall rules and routing tables. Intentions and sidecar proxies are the foundations of zero-trust networking in Consul. Since HCP Consul is secure by default, the default intention behavior is deny.

Create an explicit intention to allow communication between the dashboard and counting service.

$ consul intention create dashboard counting
Created: dashboard => counting (allow)

»Start the services and sidecar proxies

Now that you have created all the necessary configuration to describe your service's connections, it's time to start your services and their sidecar proxies. We are using the & operator to run the services as background tasks. However, because they write to the console, it's best to run them in their own shell session.

Establish the local URL and start the dashboard service.

$ PORT=9002 COUNTING_SERVICE_URL="http://localhost:5000" ./dashboard-service_darwin_amd64 &

Start the counting service.

$ PORT=9003 ./counting-service_darwin_amd64 &

Next, start the sidecar proxies that will run as sidecar processes along with the service applications.

Start the Envoy sidecar proxy for the counting service.

$ consul connect envoy -sidecar-for counting-1 -admin-bind localhost:19001 > counting-proxy.log &

Start the Envoy sidecar proxy for the dashboard service.

$ consul connect envoy -sidecar-for dashboard > dashboard-proxy.log &

Note, the -sidecar-for argument takes a Consul service ID, not a service name.

»Check the dashboard interface

Open a browser and navigate to http://YOURPUBLICIP:9002.

You should see a screen similar to the following. There is a connection indicator in the top right that will turn green and say "Connected" when the dashboard service is in communication with the counting service.

»Test the sidecar proxy connections

To test that traffic is flowing through the sidecar proxies, you will control traffic with an intention.

First, deny the Dashboard service access to the Counting service.

$ consul intention create -deny -replace dashboard counting
Created: dashboard => counting (deny)

Refresh your browser. The connection indicator in the Dashboard UI will now say "Disconnected"

You can restore communication between the services by replacing the deny intention with an allow.

»Clean up

Once you are done with this tutorial, you can begin cleaning up by closing the terminal in which your counting-service, dashboard-service, and proxies are running. This should automatically stop these processes.

»Clean up services

Delete the intention from Consul.

$ consul intention delete dashboard counting
Intention deleted.

Deregister the counting service.

$ consul services deregister counting.json
Deregistered service: counting

Deregister the dashboard service.

$ consul services deregister dashboard.json
Deregistered service: dashboard

»Clean up unused ACL tokens

If additional ACL tokens were generated from the HCP portal, it is important to clean up any unused ACL tokens to keep your environment highly secured.

Navigate to your public Consul UI and enter the ACL token you've been using for this tutorial to gain access. For more information on accessing the Consul UI in HCP, see the Explore UI tutorial.

Identify which tokens can be safely removed, then delete those tokens.

»Next steps

Now that you have completed this tutorial, you are familiar with deploying services to the HCP Consul service mesh. You created and registered Consul service definitions that describe how two services communicate with each other. After starting the application and sidecar proxies, you used Consul service mesh intentions to control traffic flow between services.

To further your service mesh knowledge, we recommend exploring the following collections:

• Consul Service Mesh
• Consul Service Mesh on Kubernetes
• Microservices with Consul Service Mesh on Kubernetes