Connect Services with Consul Service Mesh

»Interactive tutorial

An interactive, in-browser tutorial that covers the same concepts as this tutorial is available. If you do not want to set up the demo environment locally, click the Show Terminal button to launch the in-browser experience.

»Overview

In addition to providing IP addresses directly to services with the DNS interface or HTTP API, Consul can connect services to each other via sidecar proxies that you deploy locally with each service instance. This type of deployment, local sidecar proxies that control network traffic between service instances, is a service mesh.

Since sidecar proxies connect your registered services, Consul's service mesh feature is called Consul Connect sometimes in the documentation; the term was the old name for the feature and is currently deprecated while we update the tutorials and documentation to move away from it.

Consul service mesh lets you secure and observe communication between your services without modifying their code. Instead Consul configures sidecar proxies to establish mutual TLS between your services and either allow or deny communication between them based on their registered names. Because sidecar proxies control all service-to-service traffic, they can gather metrics about them and export them to a third party aggregator like Prometheus.

You can also natively integrate applications with Consul Connect for optimal performance and security.

Registering services that use the service mesh is similar to registering services normally. In this tutorial, you will:

1. Start a service.
2. Register it normally, but with an additional connect stanza.
3. Register a second proxy to communicate with the service.
4. Start sidecar proxies.
5. Practice blocking the connection to the service by creating an intention.

»Prerequisites

»Local environment

The only requirement for the tutorial is to have the Consul binary installed on the local environment.

»Start a Consul-unaware service

Begin by starting a service that is unaware of Consul. You will use socat to start a basic echo service, which will act as the "upstream" service in this tutorial. In production, this service would be a database, backend, or any service which another service relies on.

Socat is a decades-old Unix utility that lacks a concept of encryption or the TLS protocol. You will use it to demonstrate that Consul Connect service mesh takes care of these concerns for you. If socat isn't installed on your machine, it should be available via a package manager.

Start the socat service and specify that it will listen for TCP connections on port 8181.

$ socat -v tcp-l:8181,fork exec:"/bin/cat"

You can verify it is working by using nc (netcat) to connect directly to the echo service on the correct port. Once connected, type some text and press enter. The text you typed should be echoed back:

$ nc 127.0.0.1 8181

hello
hello
testing 123
testing 123

»Register the service and proxy with Consul

Next, register the service with Consul by writing a new service definition, like you did in the last tutorial. This time you will include a Connect stanza in the registration that will register a sidecar proxy to handle traffic for this backend service instance.

Add a file called socat.json to the consul.d directory with the following command (copy the whole code block except the $).

$ echo '{
  "service": {
    "name": "socat",
    "port": 8181,
    "connect": {
      "sidecar_service": {}
    }
  }
}' > ./consul.d/socat.json

Now run consul reload or send a SIGHUP signal to Consul so it will read the new configuration.

Take a look at the "connect" stanza in the registration you just added. This empty configuration notifies Consul to register a sidecar proxy for this process on a dynamically allocated port. It also creates reasonable defaults that Consul will use to configure the proxy once you start it via the CLI. Consul does not automatically start the proxy process for you. This is because Consul Connect service mesh allows you to chose the proxy you'd like to use.

Consul comes with a L4 proxy for testing purposes, and first-class support for Envoy, which you should use for production deployments and layer 7 traffic management. You'll use the L4 proxy in this tutorial, because, unlike Envoy, it comes with Consul and doesn't require any extra installation.

Start the proxy process in another terminal window using the consul connect proxy command, and specify which service instance and proxy registration it corresponds to.

$ consul connect proxy -sidecar-for socat

==> Consul Connect proxy starting...
    Configuration mode: Agent API
        Sidecar for ID: socat
              Proxy ID: socat-sidecar-proxy

==> Log data will now stream in as it occurs:

    2019/07/24 13:27:59 [INFO] Proxy loaded config and ready to serve
    2019/07/24 13:27:59 [INFO] TLS Identity: spiffe://287133f6-3d1e-8fb0-a0c5-fb9d5a95d53c.consul/ns/default/dc/dc1/svc/socat
    2019/07/24 13:27:59 [INFO] TLS Roots   : [Consul CA 7]
    2019/07/24 13:27:59 [INFO] public listener starting on 0.0.0.0:21000

»Register a dependent service and proxy

Next, register a downstream service called "web". Like the socat service definition, the configuration file for web will include a connect stanza that specifies a sidecar, but unlike the socat definition, the configuration here isn't empty. Instead it specifies web's upstream dependency on socat, and the port that the proxy will listen on.

$ echo '{
  "service": {
    "name": "web",
    "connect": {
      "sidecar_service": {
        "proxy": {
          "upstreams": [
            {
              "destination_name": "socat",
              "local_bind_port": 9191
            }
          ]
        }
      }
    }
  }
}' > ./consul.d/web.json

Use consul reload or SIGHUP to reload Consul with the new web service definition. This registers a sidecar proxy for the service web that will listen on port 9191 to establish mTLS connections to socat.

If we were running a real web service it would talk to its proxy on a loopback address. The proxy would encrypt its traffic and send it over the network to the sidecar proxy for the socat service. Socat's proxy would decrypt the traffic and send it locally to socat on a loopback address at port 8181. Because there is no web service running, you will pretend to be the web service by talking to its proxy on the port that we specified (9191).

Before you start the proxy process, verify that you aren't able to connect to the socat service on port 9191. The below command should exit immediately, because there is nothing listening on port 9191 (socat is listening on 8181).

$ nc 127.0.0.1 9191

Now start the web proxy using the configuration from the sidecar registration.

$ consul connect proxy -sidecar-for web

==> Consul Connect proxy starting...
    Configuration mode: Agent API
        Sidecar for ID: web
              Proxy ID: web-sidecar-proxy

==> Log data will now stream in as it occurs:

    2019/07/24 13:32:10 [INFO] 127.0.0.1:9191->service:default/socat starting on 127.0.0.1:9191
    2019/07/24 13:32:10 [INFO] Proxy loaded config and ready to serve
    2019/07/24 13:32:10 [INFO] TLS Identity: spiffe://287133f6-3d1e-8fb0-a0c5-fb9d5a95d53c.consul/ns/default/dc/dc1/svc/web
    2019/07/24 13:32:10 [INFO] TLS Roots   : [Consul CA 7]
    2019/07/24 13:32:10 [INFO] public listener starting on 0.0.0.0:21001

Note in the first log line that the proxy setup a local listener on port 9191 that will proxy to the socat service just as we configured in the sidecar registration. Subsequent log lines list the identity URL of the certificate loaded from the agent, identifying it as the "web" service, and the set of trusted root CAs that the proxy knows about.

Try connecting to socat again on port 9191. This time it should work and echo back your text.

$ nc 127.0.0.1 9191

hello
hello

Close the connection by typing Crl+c.

The communication between the web and socat proxies is encrypted and authorized over a mutual TLS connection, while communication between each service and its sidecar proxy is unencrypted. In production, services should only accept only loopback connections. Any traffic in and out of the machine should travel through the proxies and therefore would always be encrypted.

»Control communication with intentions

Intentions define which services are allowed communicate with which other services. The connections above succeeded because in development mode, the ACL system (and therefore the default intention policy) is "allow all" by default.

Create an intention to deny access from web to socat that specifies policy, and the source and destination services.

$ consul intention create -deny web socat

Created: web => socat (deny)

Now, try to connect again. The connection will fail.

$ nc 127.0.0.1 9191

Delete the intention.

$ consul intention delete web socat
Intention deleted.

Try the connection again, and it will succeed.

$ nc 127.0.0.1 9191

hello
hello

Intentions allow you to segment your network much like traditional firewalls, but they rely on the services' logical names (for example "web" or "socat") rather than the IP addresses of each individual service instance. Learn more about intentions in the documentation.

»Next steps

In this tutorial, you configured a service on a single agent and used Consul Connect service mesh for automatic connection authorization and encryption. To get hands-on experience with the other features of Consul Connect's service mesh, try the Get Started with Consul Service Mesh tutorials.

Next explore how to use Consul's key value (KV) store for service configuration by following the Store Data in Consul KV tutorial.