Connect Services Across Datacenters with Mesh Gateways

Consul Connect is a service mesh control plane that helps you to observe and secure service-to-service communication.

Consul service mesh secures network traffic between services using mutual TLS. "Intentions" allow you to control communication between individual services by allowing or denying communication between them. These security features work across datacenters, with automatic replication of intentions.

Mesh gateways, which connect services in different datacenters, take advantage of Server Name Indication (SNI) – a TLS extension – to inspect the destination of a connection and route to a remote datacenter without decrypting the payload. Using mesh gateways for inter-datacenter communication eliminates the requirement for Connect sidecar proxies to have routable IP addresses across datacenters, freeing operators from complex IP address management to avoid IP overlap between datacenters.

In this tutorial, you will configure Consul service mesh, replicate intentions across multiple datacenters, and use mesh gateways to enable inter-datacenter service communication.

Specifically, you will:

Enable Consul service mesh in both datacenters
Deploy two mesh gateways
Register services and sidecar proxies
Configure intentions, which are automatically replicated
Test that your services can communicate with each other

This tutorial configures cross-datacenter data plane communication to go through both the local and remote mesh gateway by using the local mesh gateway mode, as illustrated below. Other mesh gateway modes are supported. This tutorial does not configure cross-datacenter control plane communication to go through mesh gateways.

Note: Consul's service mesh feature is called Consul Connect sometimes in the documentation and on tutorials; the term was the old name for the feature and is currently deprecated while we update the tutorials and documentation to move away from it.

»Prerequisites

To complete this tutorial you will need two wide area network (WAN) joined Consul datacenters with access control list (ACL) replication enabled. If you are starting from scratch, follow these tutorials to set up your datacenters, or use them to check that you have the proper configuration:

You will also need to enable ACL replication, which you can do by following the ACL Replication for Multiple Datacenters tutorial with the following modification.

When creating the replication token for ACL management, it will need the following rules:

Replication token policy permissions

acl = "write"
operator = "write"
service_prefix "" {
  policy = "read"
  intentions = "read"
}

acl = "write"
operator = "write"
service_prefix "" {
  policy = "read"
  intentions = "read"
}

The replication token needs different permissions depending on what you want to accomplish. The above policy allows for ACL policy, role, and token replication with acl:write, CA replication with operator:write, and intention and configuration entry replication with service:*:read.

This tutorial uses Envoy as the sidecar proxies and mesh gateways. You will need to install it alongside your Consul clients. Each mesh gateway or sidecar proxy receives its configuration and updates from the local Consul client.

Note: Consul's built-in proxy cannot be configured as a mesh gateway. Use Envoy instead.

Lastly, you should set enable_central_service_config = true on your Consul clients which will allow them to centrally configure the sidecar and mesh gateway proxies.

»Enable Consul service mesh in both datacenters

Once you have your datacenters set up and ACL replication configured, it's time to enable Connect service mesh in each of them sequentially. Consul Connect's certificate authority (which is distinct from the Consul certificate authority that you manage using the consul tls CLI) will automatically bootstrap as soon as a server with service mesh enabled becomes the server cluster's leader. You can also use Vault as a Connect CA.

Warning: If you are using this tutorial as a production playbook, we strongly recommend that you enable Consul service mesh in each of your datacenters by following the Connect in Production tutorial, which includes production security recommendations.

»Enable Consul service mesh in the primary datacenter

Enable Consul service mesh in the primary datacenter and bootstrap the Connect CA by adding the following snippet to the server configuration for each of your servers.

Consul agent configuration

connect {
  enabled = true
}

connect {
  enabled = true
}

Load the new configuration by restarting each server one at a time, making sure to maintain quorum. This will be a similar process to performing a rolling restart during upgrades.

Stop the first server by running the following leave command.

$ consul leave

$ consul leave

Once the server shuts down, restart it, and make sure that it is healthy and rejoins the other servers. Repeat this process until you've restarted all the servers with Connect enabled.

»Enable Consul service mesh in the secondary datacenter

Once Consul service mesh is enabled in the primary datacenter, follow the same process to enable it in the secondary datacenter. Add the following configuration to the configuration for your servers, and restart them one at a time, making sure to maintain quorum.

Configure primary datacenter in secondary DCs

connect {
  enabled = true
}
primary_datacenter = "primary"

connect {
  enabled = true
}
primary_datacenter = "primary"

The primary_datacenter setting is required in order to enable ACL replication between datacenters. It specifies which datacenter will write intentions and act as the root CA for Connect service mesh. Intentions are automatically replicated from the primary to secondary datacenters.

»Deploy mesh gateways

Connect mesh gateways proxy requests from services in one datacenter to services in another, so you will need to deploy your mesh gateways on nodes that can reach each other over the network. As we mentioned in the prerequisites, you will need to make sure that both Envoy and Consul are installed on the mesh gateway nodes. You won't necessarily want to run any services on these nodes other than Consul and Envoy because they will have access to the WAN.

»Generate tokens for the mesh gateways

You will need to generate a token for each mesh gateway that gives it read access to the entire catalog.

Create a file named mesh-gateway-policy.hcl containing the following policy.

service_prefix "gateway" {
   policy = "write"
}
service_prefix "" {
   policy = "read"
}
node_prefix "" {
   policy = "read"
}
agent_prefix "" {
   policy = "read"
}

mesh-gateway-policy.hcl

service_prefix "gateway" {
   policy = "write"
}
service_prefix "" {
   policy = "read"
}
node_prefix "" {
   policy = "read"
}
agent_prefix "" {
   policy = "read"
}

The service:gateway*:write rule allows the gateway to register itself with Consul. agent:*:read allows the gateway to determine the node on which it is operating, and the gRPC port configured in the agent. The service:*:read and node:*:read rules allow the gateway to discover other services and nodes across the federated datacenter environment.

Depending on your organization's security posture, you can further secure the above rule set by specifying a specific agent or agent prefix as opposed to leaving the agent prefix empty.

Next, create and name a new ACL policy using the file you just created.

$ consul acl policy create \
  -name mesh-gateway \
  -rules @mesh-gateway-policy.hcl

$ consul acl policy create \
  -name mesh-gateway \
  -rules @mesh-gateway-policy.hcl

Generate a token for the primary datacenter's gateway from the new policy.

$ consul acl token create \
  -description "mesh-gateway primary datacenter token" \
  -policy-name mesh-gateway

$ consul acl token create \
  -description "mesh-gateway primary datacenter token" \
  -policy-name mesh-gateway

Now generate a token for the secondary datacenter's gateway.

$ consul acl token create \
  -description "mesh-gateway secondary datacenter token" \
  -policy-name mesh-gateway

$ consul acl token create \
  -description "mesh-gateway secondary datacenter token" \
  -policy-name mesh-gateway

You'll apply those tokens when you deploy the mesh gateways.

»Mesh gateway service configuration

When configuring your mesh gateways, you will need to set the advertise address for the service to be reachable both locally and remotely.

The -address option defines the address and port to advertise for services within the local datacenter to use to reach the mesh gateway instance. The -wan-address option defines the address and port to advertise for services or mesh gateways within remote datacenters to use to reach the mesh gateway instance. This address must be externally accessible from the remote datacenter.

»Deploy the mesh gateway for your primary datacenter

$ consul connect envoy -gateway=mesh -register \
                     -service "gateway-primary" \
                     -address "<private address>:<port>" \
                     -wan-address "<externally accessible address>:<port>"\
                     -token=<token for the primary dc gateway>

$ consul connect envoy -gateway=mesh -register \
                     -service "gateway-primary" \
                     -address "<private address>:<port>" \
                     -wan-address "<externally accessible address>:<port>"\
                     -token=<token for the primary dc gateway>

»Deploy the mesh gateway for your secondary datacenter

$ consul connect envoy -gateway=mesh -register \
                     -service "gateway-secondary" \
                     -address "<private address>:<port>" \
                     -wan-address "<externally accessible address>:<port>"\
                     -token=<gateway token for the secondary dc gateway>

$ consul connect envoy -gateway=mesh -register \
                     -service "gateway-secondary" \
                     -address "<private address>:<port>" \
                     -wan-address "<externally accessible address>:<port>"\
                     -token=<gateway token for the secondary dc gateway>

»Configure sidecar proxies to use the mesh gateways

Next, create a proxy-defaults centralized configuration file for all the sidecar proxies in both datacenters. This file will instruct the sidecar proxies to send all inter-datacenter traffic through the gateways.

Kind = "proxy-defaults"
Name = "global"
MeshGateway {
  mode = "local"
}

proxy-defaults.hcl

Kind = "proxy-defaults"
Name = "global"
MeshGateway {
  mode = "local"
}

Write the centralized configuration to Consul with the following command.

$ consul config write proxy-defaults.hcl

$ consul config write proxy-defaults.hcl

Once this step is complete, you will have set up Consul service mesh with gateways across multiple datacenters. Now you are ready to register the services inside your service mesh.

»Register services

You can configure a service to use a sidecar proxy by including a sidecar_proxy stanza in its registration file. For this tutorial, you can use socat to act as a back end application for a registered dummy service called web which will represent the client service. These are example names. If you have services that you would like to connect, feel free to use those names instead.

Caution: Consul service mesh takes its default intention policy from Consul's default ACL policy. If you have set your default ACL policy to deny (as is recommended for secure operation) and are adding a connect section to already registered services, those services may lose connection to each other until you set an intention between them to allow communication.

»Register a back end service in one datacenter

In one datacenter register a back end service and add a sidecar proxy registration. To do this you will either create a new registration file or edit an existing one to include a sidecar proxy stanza. If you have ACLs enabled, you first need to create a token for the service. The token will be granted the permissions defined by the ACL service identities template which allows the service and its proxy to register with Consul, and discover other services in the environment.

$ consul acl token create -service-identity=socat:<datacenter name>

$ consul acl token create -service-identity=socat:<datacenter name>

Then provide the SecretID in the token field of the service registration.

If you are using socat as your back end service, you will create a for the socat service registration in the Consul agent's configuration directory containing the snippet below.

Socat service registration

service {
  name = "socat"
  port = 8181
  token = "<socat token's secret ID>"
  connect {
    sidecar_service {}
  }
}

socat.hcl

service {
  name = "socat"
  port = 8181
  token = "<socat token's secret ID>"
  connect {
    sidecar_service {}
  }
}

Note the token and connect.sidecar_service stanzas of the registration. These are the parameters you would add to an existing service registration if you are not using the above example socat definition.

Reload the client with the new or modified registration.

$ consul reload

$ consul reload

Then start Envoy specifying which service it will proxy.

$ consul connect envoy -sidecar-for socat

$ consul connect envoy -sidecar-for socat

If you are using socat as your example, start it now on the port you specified in your registration by running the following command.

$ socat -v tcp-l:8181,fork exec:"/bin/cat"

$ socat -v tcp-l:8181,fork exec:"/bin/cat"

Check that the socat service is running by accessing it using netcat on the same node. It will echo back anything you type.

$ nc 127.0.0.1 8181

$ nc 127.0.0.1 8181

hello
hello
echo
echo

hello
hello
echo
echo

Stop the running netcat service by typing ctrl + c.

»Register a front end service in the other datacenter

Now in your other datacenter, you will register a service (with a sidecar proxy) that calls your back end service. The registration will need to declare the back end service as an upstream. Like the back end service, you can use an example service, which will be called web, or append the connect stanza to an existing registration with some customization.

To use web as your front end service, create a service registration file in the Consul agent's configuration directory that contains the following snippet.

Web service registration

service {
  name = "web"
  port = 8080
  token = "<web token's secret ID>"
  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "socat"
            datacenter = "primary"
            local_bind_port = 8181
          }
        ]
      }
    }
  }
}

web.hcl

service {
  name = "web"
  port = 8080
  token = "<web token's secret ID>"
  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "socat"
            datacenter = "primary"
            local_bind_port = 8181
          }
        ]
      }
    }
  }
}

Note the Connect part of the registration, which specifies socat as an upstream. If you are using another service as a back end, replace socat with its name and port 8181 with the desired port.

Reload the client with the new or modified registration.

$ consul reload

$ consul reload

Then start Envoy and specify which service it will proxy.

$ consul connect envoy -sidecar-for web

$ consul connect envoy -sidecar-for web

»Configure intentions

Now that your services are configured to be part of Consul service mesh, you will need to create intentions to allow them to communicate. Add an intention to allow the front end service to access the back end service. For services named web and socat perform the following steps.

First, create a file for a config entry definition named intention-config.hcl.

Kind = "service-intentions"
Name = "socat"
Sources = [
  {
    Name   = "web"
    Action = "allow"
  }
]

intention-config.hcl

Kind = "service-intentions"
Name = "socat"
Sources = [
  {
    Name   = "web"
    Action = "allow"
  }
]

From the same directory where you saved this file, run the following command on your terminal to initialize these intention rules.

$ consul config write intention-config.hcl
Config entry written: service-intentions/socat

$ consul config write intention-config.hcl
Config entry written: service-intentions/socat

Consul will automatically forward intentions created in the secondary datacenter to the primary datacenter, where the servers will write them, and then automatically replicate the written intentions back to the secondary datacenter.

»Test the connection

Now that you have services deployed in your Consul service mesh, verify they can contact each other. If you have been using the example web and socat services, from the node and datacenter where you registered the web service, start netcat connecting to the web service's proxy on port 8181 and type something for it to echo.

$ nc 127.0.0.1 8181

$ nc 127.0.0.1 8181

hello
hello
echo
echo

hello
hello
echo
echo

»Next steps

In this tutorial you configured two WAN-joined datacenters to use Consul service mesh, deployed gateways in each datacenter, and connected two services to each other across datacenters.

Gateways know where to route traffic because of Server Name Indication (SNI), where the client service sends the destination information as part of the TLS handshake. Because mesh gateways rely on TLS to discover the traffic's destination, they require Consul service mesh to route traffic.

Now that you've learned how to deploy mesh gateways to proxy inter-datacenter traffic, you can deploy multiple mesh gateways for redundancy or high availability. The mesh gateways and proxies will automatically round-robin load balance traffic between the mesh gateways in the remote datacenter.

If you are using Kubernetes you can configure Consul service mesh and deploy mesh gateways for your Kubernetes cluster using the Helm chart. Learn more in the Consul's Kubernetes documentation

Visit the Consul documentation for a full list of configurations for Consul Connect service mesh, including mesh gateway configuration options.

Infrastructure

Security

Applications

Networking

cloud