Development: Service Mesh

Connecting Services Across Datacenters

Consul Connect is a service mesh control plane that helps you to observe and secure service-to-service communication.

Connect secures network traffic between services using mutual TLS. "Intentions" allow you to control communication between individual services by allowing or denying communication between them. These security features work across datacenters, with automatic intention replication.

Mesh gateways, which connect services in different datacenters, take advantage of Server Name Indication (SNI), a TLS extension that allows them to see the destination of traffic without decrypting the payload. Using mesh gateways for inter-datacenter communication prevents each Connect proxy from needing an accessible IP address, and frees operators from worrying about IP address overlap between datacenters.

In this guide, you will configure Consul Connect, replicate intentions across multiple Consul datacenters, and use mesh gateways to enable inter-service traffic between them.

Specifically, you will:

  1. Enable Connect in both datacenters
  2. Deploy the two mesh gateways
  3. Register services and Connect sidecar proxies
  4. Configure intentions, which are automatically replicated
  5. Test that your services can communicate with each other

For the remainder of this guide we will refer to mesh gateways as "gateways". Anywhere in this guide where you see the word gateway, assume it is specifically a mesh gateway (as opposed to an API or other type of gateway).

Prerequisites

To complete this guide you will need two wide area network (WAN) joined Consul datacenters with access control list (ACL) replication enabled. If you are starting from scratch, follow these guides to set up your datacenters, or use them to check that you have the proper configuration:

You will also need to enable ACL replication, which you can do by following the ACL Replication for Multiple Datacenters guide with the following modification.

When creating the replication token for ACL management, it will need the following policy:

{
  "acl": "write",
  "operator": "write",
  "service_prefix": {
    "": {
      "policy": "read"
    }
  }
}

The replication token needs different permissions depending on what you want to accomplish. The above policy allows for ACL policy, role, and token replication with acl:write, CA replication with operator:write, and intention and configuration entry replication with service:*:read.

This guide uses Envoy as the sidecar proxies and gateways. You will need to install it alongside your Consul clients. Each gateway or sidecar proxy needs to get configuration and updates from a local Consul client.

Lastly you should set enable_central_service_config = true on your Consul clients, which will allow them to centrally configure the sidecar and mesh gateway proxies.

Enable Connect in Both Datacenters

Once you have your datacenters set up and ACL replication configured, it's time to enable Connect in each of them sequentially. Connect's certificate authority (which is distinct from the Consul certificate authority that you manage using the CLI) will automatically bootstrap as soon as a server with Connect enabled becomes the server cluster's leader. You can also use Vault as a Connect CA.

Enable Connect in the Primary Datacenter

Enable Connect in the primary data center and bootstrap the Connect CA by adding the following snippet to the server configuration for each of your servers.

connect {
  "enabled": true
}

Load the new configuration by restarting each server one at a time, making sure to maintain quorum. This will be a similar process to performing a rolling restart during upgrades.

Stop the first server by running the following leave command.

$ consul leave

Once the server shuts down restart it, and make sure that it is healthy and rejoins the other servers. Repeat this process until you've restarted all the servers with Connect enabled.

Enable Connect in the Secondary Datacenter

Once Connect is enabled in the primary datacenter, follow the same process to enable Connect in the secondary datacenter. Add the following configuration to the configuration for your servers, and restart them one at a time, making sure to maintain quorum.

connect {
  "enabled": true
}

The primary_datacenter setting that was required in order to enable ACL replication between datacenters also specifies which datacenter will write intentions and act as the root CA for Connect. Intentions are automatically replicated to the secondary datacenter.

Deploy Gateways

Connect mesh gateways proxy requests from services in one datacenter to services in another, so you will need to deploy your gateways on nodes that can reach each other over the network. As we mentioned in the prerequisites, you will need to make sure that both Envoy and Consul are installed on the gateway nodes. You won't want to run any services on these nodes other than Consul and Envoy because they necessarily will have access to the WAN.

Generate Tokens for the Gateways

You'll need to generate a token for each gateway that gives it read access to the entire catalog.

Create a file named mesh-gateway-policy.json containing the following content.

{
  "node_prefix": {
    "": {
      "policy": "read"
    }
  }
}
{
  "service_prefix": {
    "": {
      "policy": "read"
    }
  }
}
{
  "service": {
      "mesh-gateway": {
        "policy": "write"
      }
    }
}

Next, create and name a new ACL policy using the file you just made.

$ consul acl policy create \
  -name mesh-gateway \
  -rules @mesh-gateway-policy.json

Generate a token for the primary datacenter's gateway from the new policy.

$ consul acl token create \
  -description "mesh-gateway primary datacenter token" \
  -policy-name mesh-gateway

Now generate a token for the secondary datacenter's gateway.

$ consul acl token create \
  -description "mesh-gateway secondary datacenter token" \
  -policy-name mesh-gateway

You'll apply those tokens when you deploy the gateways.

Deploy the Gateway for Your Primary Datacenter

Register and start the gateway in your primary datacenter with the following command.

$ consul connect envoy -mesh-gateway -register \
                     -service "gateway-primary" \
                     -address "<your private address>:<port>" \
                     -wan-address "<your externally accessible address>:<port>"\
                     -admin-bind 127.0.0.1:19005 \
                     -token=<token for the primary dc gateway>

Deploy the Gateway for Your Secondary Datacenter

Register and start the gateway in your secondary datacenter with the following command.

$ consul connect envoy -mesh-gateway -register \
                     -service "gateway-secondary" \
                     -address "<your private address>:<port>" \
                     -wan-address "<your externally accessible address>:<port>"\
                     -admin-bind 127.0.0.1:19005 \
                     -token=<token for the secondary dc gateway>

Configure Sidecar Proxies to use Gateways

Next, create a centralized configuration file for all the sidecar proxies in both datacenters called proxy-defaults.json. This file will instruct the sidecar proxies to send all their inter-datacenter traffic through the gateways. It should contain the following:

{
  "Kind": "proxy-defaults",
  "Name": "global",
  "MeshGateway": "local"
}

Write the centralized configuration you just created with the following command.

$ consul config write proxy-defaults.json

Once this step is complete, you will have set up Consul Connect with gateways across multiple datacenters. Now you are ready to register the services that will use Connect.

Register Services

You can register a service to use a sidecar proxy by including a sidecar proxy stanza in its registration file. For this guide, you can use socat to act as a backend service and register a dummy service called web to represent the client service. Those names are used in our examples. If you have services that you would like to connect, feel free to use those instead.

Register a Back End Service in One Datacenter

In one datacenter register a backend service and add an Envoy sidecar proxy registration. To do this you will either create a new registration file or edit an existing one to include a sidecar proxy stanza. If you are using socat as your backend service, you will create a new file called socat.json that will contain the below snippet. Since you have ACLs enabled, you will have to create a token for the service.

{
  "service": {
    "name": "socat",
    "port": 8181,
    "token": "<token here>",
    "connect": { "sidecar_service": {} }
  }
}

Note the Connect stanza of the registration with the sidecar_service and token options. This is what you would add to an existing service registration if you are not using socat as an example.

Reload the client with the new or modified registration.

$ consul reload

Then start Envoy specifying which service it will proxy.

$ consul connect envoy -sidecar-for socat

If you are using socat as your example, start it now on the port you specified in your registration by running the following command.

$ socat -v tcp-l:8181,fork exec:"/bin/cat"

Check that the socat service is running by accessing it using netcat on the same node. It will echo back anything you type.

$ nc 127.0.0.1 8181
hello
hello
echo
echo

Stop the running netcat service by typing ctrl + c.

Register a Front End Service in the Other Datacenter

Now in your other datacenter, you will register a service (with a sidecar proxy) that calls your backend service. Your registration will need to list the backend service as your upstream. Like the backend service, you can use an example service, which will be called web, or append the connect stanza to an existing registration with some customization.

To use web as your front end service, create a registration file called web.json that contains the following snippet.

{
  "service": {
    "name": "web",
    "port": 8080,
    "token": "<token here>",
    "connect": {
      "sidecar_service": {
        "proxy": {
          "upstreams": [
            {
              "destination_name": "socat",
              "datacenter": "primary",
              "local_bind_port": 8181
            }
          ]
        }
      }
    }
  }
}

Note the Connect part of the registration, which specifies socat as an upstream. If you are using another service as a back end, replace socat with its name and the 8181 with its port.

Reload the client with the new or modified registration.

$ consul reload

Then start Envoy and specify which service it will proxy.

$ consul connect envoy -sidecar-for web

Configure Intentions

Now that your services both use Connect, you will need to configure intentions in order for them to communicate with each other. Add an intention to allow the front end service to access the back end service. For web and socat the command would look like this.

$ consul intention create web socat

Consul will automatically forward intentions initiated in the secondary datacenter to the primary datacenter, where the servers will write them. The servers in the primary datacenter will then automatically replicate the written intentions back to the secondary datacenter.

Test the Connection

Now that you have services using Connect, verify that they can contact each other. If you have been using the example web and socat services, from the node and datacenter where you registered the web service, start netcat talking to the web service's proxy on port 8181 and type something for it to echo.

$ nc 127.0.0.1 8181
hello
hello
echo
echo

Summary

In this guide you configured two WAN-joined datacenters to use Consul Connect, deployed gateways in each datacenter, and connected two services to each other across datacenters.

Gateways know where to route traffic because of Server Name Indication (SNI) where the client service sends the destination as part of the TLS handshake. Because gateways rely on TLS to discover the traffic's destination, they require Consul Connect to route traffic.

Next Steps

Now that you've seen how to deploy gateways to proxy inter-datacenter traffic, you can deploy multiple gateways for redundancy or availability. The gateways and proxies will automatically round-robin load balance traffic between the gateways.

If you are using Kubernetes you can configure Connect and deploy gateways for your Kubernetes cluster using the Helm chart. Learn more in the Consul's Kubernetes documentation

Visit the Consul documentation for a full list of configurations for Consul Connect, including mesh gateway configuration options.