ACL Replication for Multiple Datacenters | Consul

You can configure tokens, policies and roles to work across multiple datacenters. ACL replication has several benefits.

It enables authentication of nodes and services between multiple datacenters.
The secondary datacenter can provide failover for all ACL components created in the primary datacenter.
Sharing policies reduces redundancy for the operator.

»Prerequisites

Before starting this tutorial, each datacenter will need to have ACLs enabled, the process is outlined in the Securing Consul with ACLs tutorial. This tutorial includes the additional ACL replication configuration for the Consul agents not covered in the Securing Consul with ACL tutorial.

Additionally, Basic Federation with WAN Gossip is required.

»Introduction

In this tutorial, you will setup ACL replication. This is a multi-step process that includes:

Setting the primary_datacenter parameter on all Consul agents in the primary datacenter.
Creating the replication token.
Configuring the primary_datacenter parameter on all Consul agents in the secondary datacenter.
Enabling token replication on the servers in the secondary datacenter.
Applying the replication token to all the servers in the secondary datacenter.

You should complete this tutorial during the initial ACL bootstrapping process.

After ACLs are enabled you must have a privileged token to complete any operation on either datacenter. You can use the initial bootstrap token as your privileged token.

»Configure the primary datacenter

Note, if your primary datacenter uses the default datacenter name of dc1, you must set a different datacenter parameter on each secondary datacenter. Otherwise, both datacenters will be named dc1 and there will be conflicts.

»Consul servers and clients

You should explicitly set the primary_datacenter parameter on all servers and clients, even though replication is enabled by default on the primary datacenter. Your agent configuration should be similar to the example below.

datacenter = "primary_dc",
primary_datacenter= "primary_dc",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true
}

server.hcl

datacenter = "primary_dc",
primary_datacenter= "primary_dc",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true
}

The primary_datacenter parameter sets the primary datacenter to have authority for all ACL information. It should also be set on clients, so that the they can forward API requests to the servers.

Finally, start the agent.

$ consul agent -config-file=server.json

$ consul agent -config-file=server.json

Complete this process on all agents. If you are configuring ACLs for the first time, you will also need to complete the bootstrapping process now.

»Create the replication token for ACL management

Next, create the replication token for managing ACLs with the following privileges.

acl = "write" which will allow you to replicate tokens.
operator = "write" for replicating proxy-default configuration entries and enabling CA certificate signing in the secondary datacenter.
service_prefix, policy = "read" and intentions = "read" for replicating service-default configuration entries, CA, and intention data.

acl = "write"

operator = "write"

service_prefix "" {
  policy = "read"
  intentions = "read"
}

replication-policy.hcl

acl = "write"

operator = "write"

service_prefix "" {
  policy = "read"
  intentions = "read"
}

If you have the Consul Enterprise version and use namespaces, then use the following policy.

operator = "write"

service_prefix "" {
  policy = "read"
  intentions = "read"
}

namespace_prefix "" {
  acl = "write"

  service_prefix "" {
    policy = "read"
    intentions = "read"
  }
}

replication-policy.hcl

operator = "write"

service_prefix "" {
  policy = "read"
  intentions = "read"
}

namespace_prefix "" {
  acl = "write"

  service_prefix "" {
    policy = "read"
    intentions = "read"
  }
}

Now that you have the ACL rules defined, create a policy with those rules.

$ consul acl policy create -name replication -rules @replication-policy.hcl

$ consul acl policy create -name replication -rules @replication-policy.hcl

ID:           240f1d01-6517-78d3-ec32-1d237f92ab58
Name:         replication
Description: Datacenters:
Rules: acl = "write"

operator = "write"

service_prefix "" { policy = "read" intentions = "read" }

ID:           240f1d01-6517-78d3-ec32-1d237f92ab58
Name:         replication
Description: Datacenters:
Rules: acl = "write"

operator = "write"

service_prefix "" { policy = "read" intentions = "read" }

Finally, use your newly created policy to create the replication token.

$ consul acl token create -description "replication token" -policy-name replication

$ consul acl token create -description "replication token" -policy-name replication

AccessorID:   67d55dc1-b667-1835-42ab-64658d64a2ff
SecretID:     fc48e84d-3f4d-3646-4b6a-2bff7c4aaffb
Description:  replication token
Local:        false
Create Time:  2019-05-09 18:34:23.288392523 +0000 UTC
Policies:
  240f1d01-6517-78d3-ec32-1d237f92ab58 - replication

AccessorID:   67d55dc1-b667-1835-42ab-64658d64a2ff
SecretID:     fc48e84d-3f4d-3646-4b6a-2bff7c4aaffb
Description:  replication token
Local:        false
Create Time:  2019-05-09 18:34:23.288392523 +0000 UTC
Policies:
  240f1d01-6517-78d3-ec32-1d237f92ab58 - replication

»Enable ACL replication on the secondary datacenter

Once you have configured the primary datacenter and created the replication token, you can setup the secondary datacenter.

Note, your initial bootstrap token can be used for the necessary privileges to complete any action on the secondary servers.

»Configure the servers

You will need to set the primary_datacenter parameter to the name of your primary datacenter and enable_token_replication to true on all the servers.

datacenter = "dc_secondary",
primary_datacenter= "primary_dc",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true,
  enable_token_replication = true
}

server.hcl

datacenter = "dc_secondary",
primary_datacenter= "primary_dc",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true,
  enable_token_replication = true
}

Now you can start the agent.

$ consul agent -config-file=server.json

$ consul agent -config-file=server.json

Repeat this process on all the servers.

Warning: When enabling ACL replication on the secondary datacenter, data from the primary datacenter will be replicated to all secondary datacenters. This will cause all pre-existing policies, roles, intentions (if Consul service mesh is enabled), and global tokens (if token replication is enabled) in the secondary datacenter to be lost. For production environments, consider configuring ACL replication in your initial datacenter bootstrapping process.

»Apply the replication token to the servers

Finally, apply the replication token to all the servers using the CLI.

$ consul acl set-agent-token replication <token>

$ consul acl set-agent-token replication <token>

ACL token "replication" set successfully

ACL token "replication" set successfully

Once token replication has been enabled, you will also be able to create datacenter local tokens.

Repeat this process on all servers. If you are configuring ACLs for the first time, you will also need to set the agent token.

Note, the clients do not need the replication token.

»Configure the clients

For the clients, you will need to set the primary_datacenter parameter to the name of your primary datacenter and enable_token_replication to true.

datacenter = "dc_secondary",
primary_datacenter= "primary_dc",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true,
  enable_token_replication = true
}

client.hcl

datacenter = "dc_secondary",
primary_datacenter= "primary_dc",
acl {
  enabled = true,
  default_policy = "deny",
  down_policy = "extend-cache",
  enable_token_persistence = true,
  enable_token_replication = true
}

Now you can start the agent.

$ consul agent -config-file=client.json

$ consul agent -config-file=client.json

Repeat this process on all clients. If you are configuring ACLs for the first time, you will also need to set the agent token on all clients.

»Check replication

Now that you have set up ACL replication, you can use the HTTP API to check the configuration in the secondary datacenter.

$ curl http://localhost:8500/v1/acl/replication?pretty

$ curl http://localhost:8500/v1/acl/replication?pretty

{
  "Enabled": true,
  "Running": true,
  "SourceDatacenter": "primary_dc",
  "ReplicationType": "tokens",
  "ReplicatedIndex": 19,
  "ReplicatedTokenIndex": 22,
  "LastSuccess": "2019-05-09T18:54:09Z",
  "LastError": "0001-01-01T00:00:00Z"
}

{
  "Enabled": true,
  "Running": true,
  "SourceDatacenter": "primary_dc",
  "ReplicationType": "tokens",
  "ReplicatedIndex": 19,
  "ReplicatedTokenIndex": 22,
  "LastSuccess": "2019-05-09T18:54:09Z",
  "LastError": "0001-01-01T00:00:00Z"
}

Notice, the "ReplicationType" should be "tokens". This means tokens, policies, and roles are being replicated.

Note: The /v1/acl/replication endpoint can be called in the primary datacenter but will not return replication configuration.

»Next steps

In this tutorial, you setup token replication on multiple datacenters. You can complete this process on an existing datacenter, with minimal modifications.

If you have not configured the other secure features of Consul, TLS encryption and gossip encryption to secure agent communication, we recommend doing so now.

Infrastructure

Security

Applications

Networking

cloud