Skip to main content
Oct 19-21 HashiConf Global is live. Join Now
Type '/' to Search

Secure Service Communication with Consul Service Mesh and Envoy

This tutorial also appears in: CLI Quick Start, Interactive Labs and Service Mesh Security.

Consul service mesh secures service-to-service communication with authorization and encryption. Applications can use sidecar proxies in a service mesh configuration to automatically establish TLS connections for inbound and outbound connections without being aware of the network configuration and topology. In addition to securing your services, Consul service mesh can also intercept data about service-to-service communications and surface it to monitoring tools.

In this tutorial, you will register two services and their sidecar proxies in the Consul catalog. You will then start the services and sidecar proxies. Finally, you will demonstrate that the service-to-service communication is going through the proxies by stopping traffic with an "intention".

Flow diagram showing end user traffic being sent to the Dashboard Service at port 9002. The dashboard service makes requests for the counting service to the local sidecar proxy at port 5000. This traffic then traverses the service mesh over dynamic ports. The traffic exits the service mesh from the counting service's local proxy. The proxy sends this traffic to the counting service itself at port 9003.

While this tutorial uses elements that are not suitable for production environments – Consul dev agents and mock services – it will teach you the common process for deploying your own services using Consul service mesh. At the end of this tutorial, we also present additional information about adapting this process to more production-like environments.

»Prerequisites

  • A running Consul server with Consul service mesh enabled. For this tutorial you will use a local dev agent, which enables Consul Connect service mesh by default.

  • Envoy binary installed on the Consul agents. Consul includes a built-in Layer 4 (L4) proxy for testing and development but also offers first class support for Envoy as a sidecar proxy. This tutorial provides commands for both, with Envoy being the recommended proxy.

  • Two service applications which need to securely communicate. For this tutorial you will use two example service applications, a counter service and a dashboard. Download and unzip the executables to follow along.

»Interactive tutorial

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Note: The interactive tutorial does not use the exact commands listed in this tutorial. You can consider the lab as a way to extend the concepts explained in this tutorial.

»Install Envoy on the agent

Note: Consul ships with a built-in proxy that you can use to test the Consul service mesh, as well as follow the tutorial. The built-in proxy should only be used where it is not possible to run Envoy, such as an older or unsupported operating system.

You can obtain container-based builds of Envoy directly from the Envoy Website or obtain a package of Envoy binary builds from a third-party project, such as func-e.io.

The following steps will install Envoy 1.18.3. Check Consul documentation for the full list of supported versions.

Issue the following command to download and install the func-e utility, which fetches and installs Envoy.

$ curl -L https://func-e.io/install.sh | bash -s -- -b /usr/local/bin

$ curl -L https://func-e.io/install.sh | bash -s -- -b /usr/local/bin

NOTE: If your machine uses an ARM64 chip (i.e., Apple M1), then you will need to set the following environment variable before proceeding to the next step.

$ export FUNC_E_PLATFORM=darwin/amd64

$ export FUNC_E_PLATFORM=darwin/amd64

Issue the following command to download version 1.18.3 of Envoy.

$ func-e use 1.18.3

$ func-e use 1.18.3

Open a new tab in your terminal or stop the Envoy process by pressing Control+C to use the current tab.

Copy the envoy binary to a location in your $PATH. This enables Consul to automatically start Envoy without specifying the binary location.

$ sudo cp ~/.func-e/versions/1.18.3/bin/envoy /usr/local/bin/

$ sudo cp ~/.func-e/versions/1.18.3/bin/envoy /usr/local/bin/

Verify that Envoy is in your $PATH by running the following command.

$ envoy --version

$ envoy --version

»Verify Consul agent health

To ensure that Consul is running and accessible from the command line, use the consul members command to verify your agent status.

$ consul members
Node            Address         Status  Type    Build   Protocol  DC   Segment
hostname.local  127.0.0.1:8301  alive   server  1.10.0  2         dc1  <all>

$ consul membersNode            Address         Status  Type    Build   Protocol  DC   Segmenthostname.local  127.0.0.1:8301  alive   server  1.10.0  2         dc1  <all>

If you receive an error message, verify that a local Consul dev agent is running and try again.

»Register the services and sidecar proxies

Services have to be registered with Consul. Consul shares this information around the datacenter so that operators or other services can determine the location of a service. Consul service mesh also uses service registrations to determine where to send proxied traffic to.

There are several ways to register services in Consul:

For this tutorial, we will use the consul service register CLI command to load them into the catalog.

»Create the counting service definition

First, define the Counting service and its sidecar proxy in a file named counting.hcl. The definition should include the name of the service, the port the service listens on, and a connect block with the sidecar_service block. This block is empty so Consul will use default parameters. The definition also includes an optional service health check.

service {
  name = "counting"
  id = "counting-1"
  port = 9003

  connect {
    sidecar_service {}
  }

  check {
    id       = "counting-check"
    http     = "http://localhost:9003/health"
    method   = "GET"
    interval = "1s"
    timeout  = "1s"
  }
}
counting.hcl
service {  name = "counting"  id = "counting-1"  port = 9003
  connect {    sidecar_service {}  }
  check {    id       = "counting-check"    http     = "http://localhost:9003/health"    method   = "GET"    interval = "1s"    timeout  = "1s"  }}

Services and sidecar proxies can be defined in either HCL or JSON. There is a JSON version of the service definition in the demo-consul-101 project.

»Create the dashboard service definition

Create the Dashboard service and proxy definition in the same way. First, create a file named dashboard.hcl.

service {
  name = "dashboard"
  port = 9002

  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "counting"
            local_bind_port  = 5000
          }
        ]
      }
    }
  }

  check {
    id       = "dashboard-check"
    http     = "http://localhost:9002/health"
    method   = "GET"
    interval = "1s"
    timeout  = "1s"
  }
}
dashboard.hcl
service {  name = "dashboard"  port = 9002
  connect {    sidecar_service {      proxy {        upstreams = [          {            destination_name = "counting"            local_bind_port  = 5000          }        ]      }    }  }
  check {    id       = "dashboard-check"    http     = "http://localhost:9002/health"    method   = "GET"    interval = "1s"    timeout  = "1s"  }}

There is a JSON version of the service definition in the demo-consul-101 project.

Note that the dashboard definition also includes an upstream block. Upstreams are ports on the local host that will be proxied to the destination service. The upstream block's local_bind_port value is the port your service will communicate with to reach the service you depend on. The destination name is the Consul service name that the local_bind_port will proxy to.

In our scenario, the dashboard service depends on the counting service. With this configuration, when dashboard service connects to localhost:5000 it is proxied across the service mesh to the counting service.

When applying these concepts to your production environments, verify that the values in the proxy configuration's upstreams block correctly specify the destination service and port number to avoid potention issues. For example, specifying the same values for the listener's port and target service will result in a proxy loop that will cause the server to crash. Refer to the Upstream Configuration Reference for additional information.

»Register the services and proxies

Finally, you can submit the service definitions to your Consul agent. If you are using the JSON definitions, ensure that the filenames end in ".json" instead of ".hcl".

$ consul services register counting.hcl
Registered service: counting

$ consul services register counting.hclRegistered service: counting
$ consul services register dashboard.hcl
Registered service: dashboard

$ consul services register dashboard.hclRegistered service: dashboard

Challenge: After completing the tutorial, try doing it again using one of the other service registration mechanisms mentioned earlier in the tutorial to register the services.

»Verify the services are registered

Now that you have registered your services and sidecar proxies, run consul catalog services to verify that they are present.

$ consul catalog services

$ consul catalog services
consul
counting
counting-sidecar-proxy
dashboard
dashboard-sidecar-proxy

consulcountingcounting-sidecar-proxydashboarddashboard-sidecar-proxy

From the output you can notice that two extra services *-sidecar-proxy are automatically registered alongside the one defined in the configuration files.

»Create a Consul intention

Intentions define authorization policies for services in the service mesh and are used to control which services may establish connections. The default intention behavior is defined by the default ACL policy.

In this tutorial, this step is not necessary since the default ACL policy for the dev agent is "allow all", so connections across the services in the service mesh are automatically allowed as well. However, you will create explicit intentions as a part of deploying service mesh enabled services.

Best Practice: Creating an explicit intention helps protect your service against changes to the implied permissions. For example, a change in default_policy or the introduction of a global deny-all intention would impact services without explicit intentions defined.

First, create a file for a config entry definition named intention-config.hcl.

Kind = "service-intentions"
Name = "counting"
Sources = [
  {
    Name   = "dashboard"
    Action = "allow"
  }
]
intention-config.hcl
Kind = "service-intentions"Name = "counting"Sources = [  {    Name   = "dashboard"    Action = "allow"  }]

From the same directory where you saved this file, run the following command on your terminal to initialize these intention rules.

$ consul config write intention-config.hcl
Config entry written: service-intentions/counting

$ consul config write intention-config.hclConfig entry written: service-intentions/counting

Note To learn more about intentions, feel free to check out the Service Intentions Docs.

»Start the services and sidecar proxies

Now that you have created all the necessary configuration to describe your service's connections, it's time to start your services and their sidecar proxies. We are using the & operator to run the services as background tasks. However, because they write to the console, it's best to run them in their own shell session.

Establish the local URL and start the dashboard service.

$ PORT=9002 COUNTING_SERVICE_URL="http://localhost:5000" ./dashboard-service &

$ PORT=9002 COUNTING_SERVICE_URL="http://localhost:5000" ./dashboard-service &

Start the counting service.

$ PORT=9003 ./counting-service &

$ PORT=9003 ./counting-service &

Next, start the sidecar proxies that will run as sidecar processes along with the service applications.

Start the Envoy sidecar proxy for the counting service.

$ consul connect envoy -sidecar-for counting-1 -admin-bind localhost:19001 > counting-proxy.log &

$ consul connect envoy -sidecar-for counting-1 -admin-bind localhost:19001 > counting-proxy.log &

Start the Envoy sidecar proxy for the dashboard service.

$ consul connect envoy -sidecar-for dashboard > dashboard-proxy.log &

$ consul connect envoy -sidecar-for dashboard > dashboard-proxy.log &

Note: The -sidecar-for argument takes a Consul service ID, not a service name.

»Check the dashboard interface

Open a browser and navigate to http://localhost:9002.

You should see a screen similar to the following. There is a connection indicator in the top right that will turn green and say "Connected" when the dashboard service is in communication with the counting service.

Image of Dashboard UI. There is white text on a magenta background, with the page title "Dashboard" at the top left. There is a green indicator in the top right with the word connected in white. There is a large number 19 to show sample counting output. The node name that the counting service is running on, host01, is in very small monospaced type underneath the large numbers.

If your application is not connected, check that the Counting service is running and healthy by viewing it in the Consul UI at http://localhost:8500.

»Test the sidecar proxy connections

To test that traffic is flowing through the sidecar proxies, you will control traffic with an intention.

First, deny the Dashboard service access to the Counting service.

$ consul config delete -kind service-intentions -name counting
Config entry deleted: service-intentions/counting

$ consul config delete -kind service-intentions -name countingConfig entry deleted: service-intentions/counting

Refresh your browser, the connection indicator in the Dashboard UI will now say "Disconnected"

Image of Dashboard UI. There is white text on a magenta background, with the page title "Dashboard" at the top left. There is a red indicator in the top right with the words "Counting Service is Unreachable" in white. There is a large number -1 to show sample counting output. The word "Unreachable" surrounded by square brackets is in monospaced type underneath the large numbers.

You can restore communication between the services with the command you ran earlier to initialize your intentions.

$ consul config write intention-config.hcl
Config entry written: service-intentions/counting

$ consul config write intention-config.hclConfig entry written: service-intentions/counting

Back in the browser, verify that the dashboard reconnects to the counting service.

»Clean up

Once you are done with this tutorial, you can begin cleaning up by closing the terminal in which your counting-service, dashboard-service, and proxies are running. This should automatically stop these processes.

Delete the intention from Consul.

$ consul config delete -kind service-intentions -name counting
Config entry deleted: service-intentions/counting

$ consul config delete -kind service-intentions -name countingConfig entry deleted: service-intentions/counting

Deregister the counting service.

$ consul services deregister counting.hcl
Deregistered service: counting

$ consul services deregister counting.hclDeregistered service: counting

Deregister the dashboard service.

$ consul services deregister dashboard.hcl
Deregistered service: dashboard

$ consul services deregister dashboard.hclDeregistered service: dashboard

»Extend these concepts

When you want to apply this learning to a proof-of-concept or production environment, there are some additional considerations.

»Enable Connect and gRPC

When a Consul server is started with the -dev flag, it will automatically enable Consul service mesh and provide a default port for gRPC communication. These have to be configured explicitly for regular Consul agents.

Consul server agent configuration
HCL
ports {
  grpc = 8502
}

connect {
  enabled = true
}

ports {  grpc = 8502}
connect {  enabled = true}
{
  "ports": {
    "grpc": 8502
  },
  "connect": {
    "enabled": true
  }
}

{  "ports": {    "grpc": 8502  },  "connect": {    "enabled": true  }}

The connect section in the configuration is only needed for server agents. Client agents only need to configure the gRPC port.

»Next steps

Now that you have completed this tutorial, you have familiarized yourself with a basic service mesh enabled service deployment. You created and registered Consul service definitions that describe how two services communicate with each other. After starting the application and sidecar proxies, you used Consul service mesh intentions to control traffic flow between services. Finally, you learned about additional requirements required to take the concepts to a proof-of-concept environment.

stdin: is not a tty