Consul Connect is a service mesh control plane that helps you to observe and secure service-to-service communication.
Consul service mesh secures network traffic between services using mutual TLS. "Intentions" allow you to control communication between individual services by allowing or denying communication between them. These security features work across datacenters, with automatic replication of intentions.
Mesh gateways, which connect services in different datacenters, take advantage of Server Name Indication (SNI) – a TLS extension – to inspect the destination of a connection and route to a remote datacenter without decrypting the payload. Using mesh gateways for inter-datacenter communication eliminates the requirement for Connect sidecar proxies to have routable IP addresses across datacenters, freeing operators from complex IP address management to avoid IP overlap between datacenters.
In this tutorial, you will configure Consul service mesh, replicate intentions across multiple datacenters, and use mesh gateways to enable inter-datacenter service communication.
Specifically, you will:
- Enable Consul service mesh in both datacenters
- Deploy two mesh gateways
- Register services and sidecar proxies
- Configure intentions, which are automatically replicated
- Test that your services can communicate with each other
This tutorial configures cross-datacenter data plane communication to go through
both the local and remote mesh gateway by using the local mesh gateway mode,
as illustrated below. Other mesh gateway modes
are supported. This tutorial does not configure
cross-datacenter control plane communication
to go through mesh gateways.
Note: Consul's service mesh feature is called Consul Connect sometimes in the documentation and on tutorials; the term was the old name for the feature and is currently deprecated while we update the tutorials and documentation to move away from it.
»Prerequisites
To complete this tutorial you will need two wide area network (WAN) joined Consul datacenters with access control list (ACL) replication enabled. If you are starting from scratch, follow these tutorials to set up your datacenters, or use them to check that you have the proper configuration:
You will also need to enable ACL replication, which you can do by following the ACL Replication for Multiple Datacenters tutorial with the following modification.
When creating the replication token for ACL management, it will need the following rules:
- HCL
- JSON
The replication token needs different permissions depending on what you want to
accomplish. The above policy allows for ACL policy, role, and token replication
with acl:write, CA replication with operator:write, and intention and
configuration entry replication with service:*:read.
This tutorial uses Envoy as the sidecar proxies and mesh gateways. You will need to install it alongside your Consul clients. Each mesh gateway or sidecar proxy receives its configuration and updates from the local Consul client.
Note: Consul's built-in proxy cannot be configured as a mesh gateway. Use Envoy instead.
Lastly, you should set enable_central_service_config = true
on your Consul clients which will allow them to centrally configure the
sidecar and mesh gateway proxies.
»Enable Consul service mesh in both datacenters
Once you have your datacenters set up and ACL replication configured, it's time
to enable Connect service mesh in each of them sequentially. Consul Connect's certificate authority
(which is distinct from the Consul certificate authority that you manage using
the consul tls CLI) will automatically bootstrap as soon as a server with service mesh enabled
becomes the server cluster's leader. You can also use Vault as a Connect
CA.
Warning: If you are using this tutorial as a production playbook, we strongly recommend that you enable Consul service mesh in each of your datacenters by following the Connect in Production tutorial, which includes production security recommendations.
»Enable Consul service mesh in the primary datacenter
Enable Consul service mesh in the primary datacenter and bootstrap the Connect CA by adding the following snippet to the server configuration for each of your servers.
- HCL
- JSON
Load the new configuration by restarting each server one at a time, making sure to maintain quorum. This will be a similar process to performing a rolling restart during upgrades.
Stop the first server by running the following leave command.
Once the server shuts down, restart it, and make sure that it is healthy and rejoins the other servers. Repeat this process until you've restarted all the servers with Connect enabled.
»Enable Consul service mesh in the secondary datacenter
Once Consul service mesh is enabled in the primary datacenter, follow the same process to enable it in the secondary datacenter. Add the following configuration to the configuration for your servers, and restart them one at a time, making sure to maintain quorum.
- HCL
- JSON
The primary_datacenter
setting is required in order to enable ACL
replication between datacenters. It specifies which datacenter will write
intentions and act as the root CA for
Connect service mesh.
Intentions are automatically replicated from the primary to secondary datacenters.
»Deploy mesh gateways
Connect mesh gateways proxy requests from services in one datacenter to services in another, so you will need to deploy your mesh gateways on nodes that can reach each other over the network. As we mentioned in the prerequisites, you will need to make sure that both Envoy and Consul are installed on the mesh gateway nodes. You won't necessarily want to run any services on these nodes other than Consul and Envoy because they will have access to the WAN.
»Generate tokens for the mesh gateways
You will need to generate a token for each mesh gateway that gives it read access to the entire catalog.
Create a file named mesh-gateway-policy.hcl containing the following policy.
The service:gateway*:write rule allows the gateway to register itself with
Consul. agent:*:read allows the gateway to determine the node on which it is
operating, and the gRPC port configured in the agent. The service:*:read
and node:*:read rules allow the gateway to discover other services and
nodes across the federated datacenter environment.
Depending on your organization's security posture, you can further secure the above rule set by specifying a specific agent or agent prefix as opposed to leaving the agent prefix empty.
Next, create and name a new ACL policy using the file you just created.
Generate a token for the primary datacenter's gateway from the new policy.
Now generate a token for the secondary datacenter's gateway.
You'll apply those tokens when you deploy the mesh gateways.
»Mesh gateway service configuration
When configuring your mesh gateways, you will need to set the advertise address for the service to be reachable both locally and remotely.
The -address option defines the address and port to advertise for
services within the local datacenter to use to reach the mesh gateway instance. The
-wan-address option defines the address and port to advertise for services or
mesh gateways within remote datacenters to use to reach the mesh gateway instance. This
address must be externally accessible from the remote datacenter.
»Deploy the mesh gateway for your primary datacenter
Register and start the mesh gateway in your primary datacenter with the command below.
»Deploy the mesh gateway for your secondary datacenter
Register and start the mesh gateway in your secondary datacenter with the following command.
»Configure sidecar proxies to use the mesh gateways
Next, create a
proxy-defaults
centralized configuration file for all the sidecar proxies in both datacenters.
This file will instruct the sidecar proxies to send all inter-datacenter traffic
through the gateways.
Write the centralized configuration to Consul with the following command.
Once this step is complete, you will have set up Consul service mesh with gateways across multiple datacenters. Now you are ready to register the services inside your service mesh.
»Register services
You can configure a service to use a sidecar proxy by including a sidecar_proxy
stanza in its registration file. For this tutorial, you can use socat to act as a
back end application for a registered dummy service called web which will represent the client
service. These are example names. If you have services that you would like to
connect, feel free to use those names instead.
Caution: Consul service mesh takes its default intention policy from Consul's
default ACL policy. If you have set your default ACL policy to deny (as is recommended
for secure operation) and are adding a connect section to already registered services,
those services may lose connection to each other until you set an intention
between them to allow communication.
»Register a back end service in one datacenter
In one datacenter register a back end service and add a sidecar proxy registration. To do this you will either create a new registration file or edit an existing one to include a sidecar proxy stanza. If you have ACLs enabled, you first need to create a token for the service. The token will be granted the permissions defined by the ACL service identities template which allows the service and its proxy to register with Consul, and discover other services in the environment.
Then provide the SecretID in the token field of the service registration.
If you are using socat as your back end service, you will create a for the
socat service registration in the Consul agent's configuration directory containing the
snippet below.
- HCL
- JSON
Note the token and connect.sidecar_service stanzas of the registration.
These are the parameters you would add to an existing service
registration if you are not using the above example socat definition.
Reload the client with the new or modified registration.
Then start Envoy specifying which service it will proxy.
If you are using socat as your example, start it now on the port you specified in your registration by running the following command.
Check that the socat service is running by accessing it using netcat on the same node. It will echo back anything you type.
Stop the running netcat service by typing ctrl + c.
»Register a front end service in the other datacenter
Now in your other datacenter, you will register a service (with a sidecar proxy)
that calls your back end service. The registration will need to declare the back
end service as an upstream. Like the back end service, you can use an example
service, which will be called web, or append the connect stanza to an existing
registration with some customization.
To use web as your front end service, create a service registration file in the Consul agent's configuration directory that contains the following snippet.
- HCL
- JSON
Note the Connect part of the registration, which specifies socat as an
upstream. If you are using another service as a back end, replace socat with
its name and port 8181 with the desired port.
Reload the client with the new or modified registration.
Then start Envoy and specify which service it will proxy.
»Configure intentions
Now that your services are configured to be part of Consul service mesh,
you will need to create intentions to allow them to communicate.
Add an intention to allow the front end service to access the back end
service. For services named web and socat the command would be as follows.
Consul will automatically forward intentions created in the secondary datacenter to the primary datacenter, where the servers will write them, and then automatically replicate the written intentions back to the secondary datacenter.
»Test the connection
Now that you have services deployed in your Consul service mesh, verify they can contact each other. If you have been using the example web and socat services, from the node and datacenter where you registered the web service, start netcat connecting to the web service's proxy on port 8181 and type something for it to echo.
»Next steps
In this tutorial you configured two WAN-joined datacenters to use Consul service mesh, deployed gateways in each datacenter, and connected two services to each other across datacenters.
Gateways know where to route traffic because of Server Name Indication (SNI), where the client service sends the destination information as part of the TLS handshake. Because mesh gateways rely on TLS to discover the traffic's destination, they require Consul service mesh to route traffic.
Now that you've learned how to deploy mesh gateways to proxy inter-datacenter traffic, you can deploy multiple mesh gateways for redundancy or high availability. The mesh gateways and proxies will automatically round-robin load balance traffic between the mesh gateways in the remote datacenter.
If you are using Kubernetes you can configure Consul service mesh and deploy mesh gateways for your Kubernetes cluster using the Helm chart. Learn more in the Consul's Kubernetes documentation
Visit the Consul documentation for a full list of configurations for Consul Connect service mesh, including mesh gateway configuration options.
