Secure Consul with Access Control Lists (ACLs)

Consul uses Access Control Lists (ACLs) to secure access to the UI, API, CLI, service communications, and agent communications. When securing your datacenter you should configure the ACLs first. At the core, ACLs operate by grouping rules into policies, then associating one or more policies with a token. The ACL documentation introduces basic concepts and syntax for the ACL system, and we recommend that you read it before you begin this tutorial.

In this tutorial, you will bootstrap the ACL system and then learn how to create tokens with minimum privileges for:

Servers and Clients
Services
DNS
Consul KV
Consul UI

Important: For best results, use this tutorial during the initial deployment of a Consul (version 1.4.3 or newer) datacenter. Specifically, you should have already installed all agents and configured initial service definitions, but you should not yet rely on Consul for any service discovery or service configuration operations.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

»Bootstrap the ACL system

You will bootstrap the ACL system in two steps, enabling ACLs and creating the bootstrap token.

»Enable ACLs on the agents

To enable ACLs, add the following ACL parameters to the agent's configuration file and then restart the Consul service. In order for ACL configuration to be applied correctly you will need to apply the same parameters to every server and every client in your datacenter. If you want to reduce Consul client restarts, you can enable the ACLs on them when you apply the token.

acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

agent.hcl

acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

Note: Token persistence was introduced in Consul 1.4.3. In older versions of Consul, you cannot persist tokens when using the HTTP API.

In this example, you configured the default policy of "deny", which means you are in a mode where only explicitly named resources are allowed to access the datacenter. You also enabled token persistence when using the HTTP API. With persistence enabled, tokens will be persisted to disk and reloaded when an agent restarts.

Note: If you are bootstrapping ACLs on an existing datacenter, enable the ACLs on the agents first with default_policy=allow. Default policy allow will enable ACLs, but will allow all operations, allowing the datacenter to function normally while you create the tokens and apply them. This will reduce downtime. You should update the configuration files on all the servers first and then initiate a rolling restart.

»Create the initial bootstrap token

To create the initial bootstrap token, use the acl bootstrap command on one of the servers.

$ consul acl bootstrap

$ consul acl bootstrap

The output gives you important information about the token, including the associated policy global-management and SecretID.

Note: By default, Consul assigns the global-management policy to the bootstrap token, which has unrestricted privileges. It is important to have one token with unrestricted privileges in case of emergencies; however you should only give a small number of administrators access to it. The SecretID is a UUID that you will use to identify the token when using the Consul CLI or HTTP API.

While you are setting up the ACL system, set the CONSUL_HTTP_TOKEN environment variable to the bootstrap token on one server. For this tutorial the example is on server "consul-server-one". This gives you the necessary privileges to continue creating policies and tokens. Set the environment variable temporarily with export, so that it will not persist once you've closed the session.

$ export CONSUL_HTTP_TOKEN=<your_token_here>

$ export CONSUL_HTTP_TOKEN=<your_token_here>

Now, all of the following commands in this tutorial can be completed on the same server, in this case server "consul-server-one".

»Apply individual tokens to agents

Adding tokens to agents is a three step process.

Create the agent policy.
Create the token with the newly created policy.
Add the token to the agent.

»Create the agent policy

We recommend creating agent policies that have write privileges for node related actions including registering itself in the catalog, updating node level health checks, and having write access on its agent configuration file. The example below has unrestricted privileges for node related actions for "consul-server-one" only.

node "consul-server-one" {
  policy = "write"
}

consul-server-one-policy.hcl

node "consul-server-one" {
  policy = "write"
}

When creating agent policies, review the node rules. Now that you have specified the policy, you can initialize it using the Consul CLI. To create a programmatic process, you could also use the HTTP API.

$ consul acl policy create \
  -name consul-server-one \
  -rules @consul-server-one-policy.hcl

$ consul acl policy create \
  -name consul-server-one \
  -rules @consul-server-one-policy.hcl

The command output will include the policy information.

Repeat this process for all servers and clients in the Consul datacenter. Each agent should have its own policy based on the node name, that grants write privileges to it.

»Create the agent token

After creating the per-agent policies, create individual tokens for all the agents. You will need to include the policy in the consul acl token create command.

$ consul acl token create -description "consul-server-one agent token" \
  -policy-name consul-server-one

$ consul acl token create -description "consul-server-one agent token" \
  -policy-name consul-server-one

This command returns the token information, which should include a description and policy information.

Repeat this process for each agent. It is the responsibility of the operator to save tokens in a secure location; we recommend Vault.

»Add the token to the agent

Finally, apply the tokens to the agents.

Start by applying the agent token to the servers and ensure they are working correctly before applying the agent token to the clients.

You can setup the token in three ways: CLI, API or using the agent configuration file.

Use the set-agent-token subcommand to setup the agent token.

To use this CLI command, set the CONSUL_HTTP_TOKEN environment variable to a token with acl:write permission. For this purpose, you could use the previously created bootstrap token or create a new token using the global-management policy.

$ export CONSUL_HTTP_TOKEN="<bootstrap token here>"

$ export CONSUL_HTTP_TOKEN="<bootstrap token here>"

Apply token to agent.

$ consul acl set-agent-token agent "<agent token here>"

$ consul acl set-agent-token agent "<agent token here>"

You will need to complete this process on every agent.

At this point, every agent that has a token can once again read and write information to Consul, but only for node-related actions. Actions for individual services are not yet allowed.

Note: If you are bootstrapping ACLs on and existing datacenter, remember to update the default policy to default_policy = deny and initiate another rolling restart after applying the token.

»Apply individual tokens to the services

The token creation and application process for services is similar to agents. Create a policy. Use that policy to create a token. Add the token to the service. Service tokens are necessary for agent anti-entropy, registering and de-registering the service, and registering and de-registering the service's checks.

Review the service rules before getting started.

Below is an example service definition that needs a token after bootstrapping the ACL system.

service {
  name = "dashboard",
  port = 9002,
  check {
    id = "dashboard-check",
    http = "http://localhost:9002/health",
    method = "GET",
    interval = "1s",
    timeout = "1s"
  }
}

service {
  name = "dashboard",
  port = 9002,
  check {
    id = "dashboard-check",
    http = "http://localhost:9002/health",
    method = "GET",
    interval = "1s",
    timeout = "1s"
  }
}

This service definition should be located in the configuration directory on one of the clients.

First, create the policy that will grant write privileges to only the "dashboard" service. This means the "dashboard" service can register itself, update it's health checks, and write any of the fields in the service definition.

service "dashboard" {
  policy = "write"
}

dashboard-policy.hcl

service "dashboard" {
  policy = "write"
}

Use the policy definition to initiate the policy.

$ consul acl policy create -name "dashboard-service" \
  -rules @dashboard-policy.hcl

$ consul acl policy create -name "dashboard-service" \
  -rules @dashboard-policy.hcl

Next, create a token with the policy.

$ consul acl token create -description "Token for Dashboard Service" \
  -policy-name dashboard-service

$ consul acl token create -description "Token for Dashboard Service" \
  -policy-name dashboard-service

The command will return information about the token, which should include a description and policy information. As usual, save the token to a secure location.

Finally, add the token to the service definition.

service {
  name = "dashboard",
  port = 9002,
  token = "57c5d69a-5f19-469b-0543-12a487eecc66",
  check {
    id = "dashboard-check",
    http = "http://localhost:9002/health",
    method = "GET",
    interval = "1s",
    timeout = "1s"
  }
}

service {
  name = "dashboard",
  port = 9002,
  token = "57c5d69a-5f19-469b-0543-12a487eecc66",
  check {
    id = "dashboard-check",
    http = "http://localhost:9002/health",
    method = "GET",
    interval = "1s",
    timeout = "1s"
  }
}

If the service is running, you will need to restart it. Unlike with agent tokens, there is no HTTP API endpoint to apply the token directly to the service. If the service is registered with a configuration file, you must also set the token in the configuration file. However, if you register a service with the HTTP API, you can pass the token in the header with X-Consul-Token and it will be used by the service.

»Tokens for Consul service mesh

If you are using a sidecar proxy as part of Consul service mesh, it will inherit the token from the service definition. Alternatively, if you are deploying a proxy with a stand-alone registration file you will need to create a separate token. You will need to provide write privileges for the service and proxy, and read privileges for the service_prefix and node_prefix. Below is an example of a policy that provides the necessary privileges for a proxy for the dashboard service. This policy will also allow the proxy to register itself with the servers.

service "dashboard" {
  policy = "write"
}

service "dashboard-sidecar-proxy" {
  policy = "write"
}

service_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

service "dashboard" {
  policy = "write"
}

service "dashboard-sidecar-proxy" {
  policy = "write"
}

service_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

Next, you should create the token and assign it to the proxy with the same process outlined for services.

»Token for DNS

Depending on your use case, the token used for DNS may need policy rules for nodes, services, and prepared queries. You should apply the token to the Consul agent serving DNS requests. When the DNS server makes a request to Consul, it will include the token in the request. Consul can either authorize or revoke the request, depending on the token's privileges. The token creation for DNS is the same three step process you used for agents and services, create a policy, create a token, apply the token.

Below is an example of a policy that provides read privileges for all services, nodes, and prepared queries.

node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "read"
}
# only needed if using prepared queries
query_prefix "" {
  policy = "read"
}

dns-request-policy.hcl

node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "read"
}
# only needed if using prepared queries
query_prefix "" {
  policy = "read"
}

First, create the policy.

$ consul acl policy create -name "dns-requests" -rules @dns-request-policy.hcl

$ consul acl policy create -name "dns-requests" -rules @dns-request-policy.hcl

Next, create the token.

$ consul acl token create -description "Token for DNS Requests" -policy-name dns-requests

$ consul acl token create -description "Token for DNS Requests" -policy-name dns-requests

Finally, apply the token to the Consul agent serving DNS request in default token ACL configuration parameter.

$ consul acl set-agent-token default "<dns token>"

$ consul acl set-agent-token default "<dns token>"

Note, if you have multiple agents serving DNS requests you can use the same policy to create individual tokens for all of them if they are using the same rules.

»Consul KV tokens

The process of creating tokens for Consul KV follows the same three step process as nodes and services. First create a policy, then a token, and finally apply or use the token. However, unlike tokens for nodes and services Consul KV has many varied use cases.

Services may need to access configuration data in the key-value store.
You may want to store distributed lock information for sessions.
Operators may need access to update configuration values in the key-value store.

The rules for KV have four policy levels; deny, write, read, and list. Let's review several examples of read and write.

Depending on the use case, the token will be applied differently. For services you will add the token to the HTTP client. For operators use, the operator will use the token when issuing commands, either with the CLI or API.

»Recursive reads

key_prefix "redis/" {
  policy = "read"
}

key_prefix "redis/" {
  policy = "read"
}

In the above example, you are allowing any key with the prefix redis/ to be read. If you issued the command consul kv get -recurse redis/ -token=<your token> you would get a list of key/values for redis/.

This type of policy is good for allowing operators to recursively read configuration parameters stored in the KV. Similarly, a "write" policy with the same prefix would allow you to update any keys that begin with "redis/".

»Write privileges for one key

key "dashboard-app" {
  policy = "write"
}

key "dashboard-app" {
  policy = "write"
}

In the above example, you are allowing read and write privileges to the dashboard-app key. This allows for get, delete, and put operations.

This type of token would allow an application to update and read a value in the KV store. It would also be useful for operators who need access to set specific keys.

»Read privileges for one key

key "counting-app" {
  policy = "read"
}

key "counting-app" {
  policy = "read"
}

In the above example, you are setting a read privileges for a single key, "counting-app". This allows for only get operations.

This type of token allows an application to read from a key to get the value. This is useful for configuration parameter updates.

»Consul UI token

Once you have bootstrapped the ACL system, access to the UI will be limited. The anonymous token grants UI access if no default token is set on the agents, and all operations will be denied, including viewing nodes and services.

You can re-enable UI features (with flexible levels of access) by creating and distributing tokens to operators. Once you have a token, you can use it in the login form to access the UI features:

After saving a new token, it will be shown in the Consul UI.

The browser stores tokens that you add to the UI. This allows you to distribute different levels of privileges to operators. Like other tokens, it's up to the operator to decide the per-token privileges.

Below is an example of policy that will allow the operator to have read access to the UI for services, nodes, key/values, and intentions. You need to have "acl = read" to view policies and tokens. Otherwise you will not be able to access the ACL section of the UI, not even to view the token you used to access the UI.

service_prefix "" {
  policy = "read"
}
key_prefix "" {
  policy = "read"
}
node_prefix "" {
  policy = "read"
}

operator-ui.hcl

service_prefix "" {
  policy = "read"
}
key_prefix "" {
  policy = "read"
}
node_prefix "" {
  policy = "read"
}

The ACL system is described here is one of several mechanisms available to secure Consul UI access. Additional mechanisms include enabling HTTPS access and restricting HTTP writes to a list of CIDRs. For more information on additional mechanisms, consult the "Secure UI Access" entry in the security model recommendations.

»Rotate a token

Eventually, you will need to rotate your tokens. To do so, you must create a new token, update any configuration that references the old token with the new token, and then delete the old token. You can rotate tokens used to register services and agents, for DNS queries, and for accessing Consul data.

To keep the same privileges as the previous token, use the clone command with the existing token's AccessorID.

$ consul acl token clone -description "Clone of <token_you_are_cloning>" -id 6a1253d2-1785-24fd-91c2-f8e78c745511

$ consul acl token clone -description "Clone of <token_you_are_cloning>" -id 6a1253d2-1785-24fd-91c2-f8e78c745511

A response will be returned similar to the one below.

Token cloned successfully.
AccessorID:   dcfa52ed-9288-b3ff-056d-255ef69d2d88
SecretID:     0005d17e-5bb2-7e8b-7bfa-15f2eee9ad14
Description:  Clone of Super User
Local:        false
Create Time:  2018-10-22 16:26:02.909096 -0400 EDT
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

Token cloned successfully.
AccessorID:   dcfa52ed-9288-b3ff-056d-255ef69d2d88
SecretID:     0005d17e-5bb2-7e8b-7bfa-15f2eee9ad14
Description:  Clone of Super User
Local:        false
Create Time:  2018-10-22 16:26:02.909096 -0400 EDT
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

Apply tokens to the agent or to the services as needed.

Finally, delete the old token from Consul.

consul acl token delete -id 6a1253d2-1785-24fd-91c2-f8e78c745511

consul acl token delete -id 6a1253d2-1785-24fd-91c2-f8e78c745511

This workflow can also be performed via the HTTP API.

»Next steps

In this tutorial, you bootstrapped the ACL system for Consul and applied tokens to agents and services. You assigned tokens for DNS, Consul KV, and the Consul UI, and learned how to rotate a token.

To learn more about Consul's security model read the internals documentation. You can find commands relating to ACLs in our reference documentation.

Infrastructure

Security

Applications

Networking

cloud