Bootstrapping ACLs on a new cluster requires a few steps, outlined below. For multi-region clusters, you will perform the bootstrapping process on the authoritative region and set up replication.
NOTE: A similar, interactive lab is also available for this entire collection if you do not have a Nomad environment to perform the steps described in this tutorial. Click the Show Tutorial button to launch the lab experience.
»Enable ACLs on Nomad servers
The APIs needed to manage policies and tokens are not enabled until ACLs are enabled. To begin, you need to enable the ACLs on the servers. If a multi-region setup is used, the authoritative region should be enabled first. For each server:
- Set the
enabled
value of theacl
stanza to true. Theacl
stanza is a top-level stanza.
acl {
enabled = true
}
»Configure for multiple regions
For multi-region configurations, you will also need to perform additional configuration. The regions must be joined before bootstrapping ACLs.
»Set the authoritative region
For all clusters in a multi-region setup, set the authoritative_region
value in the server
stanza. Later, you'll set the replication_token
value of the acl
stanza. You will learn how to generate management tokens
later in this tutorial.
»Reload your Nomad configuration
Take care to restart the servers one at a time and to ensure each server has
joined and is operating correctly before restarting another. You can use the
nomad server members
command to verify that a server is up and ready to
handle requests.
»Bootstrap and deploy anonymous policy
Once the ACL system is enabled, you need to generate the initial token. This first management token is used to bootstrap the system. Care should be taken not to lose all of your management tokens. If you do, you will need to re-bootstrap the ACL subsystem.
Warning: Bootstrapping the ACL subsystem will interrupt access to your cluster.
Once the
nomad acl bootstrap
command is run, Nomad's
default-deny policy will become enabled. You should have an acceptable anonymous
policy prepared and ready to submit immediately after bootstrapping.»Stage a permissive anonymous policy
Because Nomad starts with a default deny-all policy, you might need to create a transitional anonymous policy to allow traffic to your cluster while you are generating and distributing tokens to your users.
The ACL system uses a default-deny model. This means by default no permissions are granted. For clients making requests without ACL tokens, you may want to grant some basic level of access. This is done by setting rules on the special "anonymous" policy. This policy is applied to any requests made without a token.
Create a file named anonymous.policy
with this HCL content:
namespace "*" {
policy = "write"
capabilities = ["alloc-node-exec"]
}
agent {
policy = "write"
}
operator {
policy = "write"
}
quota {
policy = "write"
}
node {
policy = "write"
}
host_volume "*" {
policy = "write"
}
This is an allow-all policy specification. You can use this as a transitional anonymous policy, which will minimize time in which requests can not be submitted to the cluster once you bootstrap.
»Run the bootstrap command
Once the ACL system is enabled, use the nomad acl bootstrap
command:
$ nomad acl bootstrap
Accessor ID = 5b7fd453-d3f7-6814-81dc-fcfe6daedea5
Secret ID = 9184ec35-65d4-9258-61e3-0c066d0a45c5
Name = Bootstrap Token
Type = management
Global = true
Policies = n/a
Create Time = 2017-09-11 17:38:10.999089612 +0000 UTC
Create Index = 7
Modify Index = 7
Once the initial bootstrap is performed, it cannot be performed again unless the
reset procedure is complete. Make sure to save this AccessorID and SecretID.
The bootstrap token is a management
type token, meaning it can perform any
operation. It should be used to setup the ACL policies and create additional ACL
tokens. The bootstrap token can be deleted and is like any other token, care
should be taken to not revoke all management tokens.
»Providing a CLI token
If you attempt to run a command at this point without a token provided, you will encounter an error.
$ nomad status
Error querying jobs: Unexpected response code: 403 (Permission denied)
You can provide a token for CLI commands using the -token flag or by setting the
NOMAD_TOKEN environment variable. Replace BOOTSTRAP_SECRET_ID
in the following
command with the Secret ID you received above.
## Store our token secret ID
$ export NOMAD_TOKEN="BOOTSTRAP_SECRET_ID"
Running the nomad status
command will now complete successfully.
$ nomad status
No running jobs
»Deploy your anonymous policy
Next, install the anonymous policy with the nomad acl policy apply
command.
$ nomad acl policy apply -description "Anonymous policy (full-access)" anonymous anonymous.policy.hcl
Successfully wrote "anonymous" ACL policy!
Once this command has completed, requests to the cluster that do not present a token will use this policy.
You can also use the Nomad API to submit policies as JSON objects. Consult the
Nomad acl/policy
API documentation for more information.
»Verify anonymous requests succeed
At this point, it is good practice to verify that the anonymous policy is performing as expected.
You can unset your NOMAD_TOKEN environment variable to send unauthenticated requests to your cluster.
$ unset NOMAD_TOKEN
Verify that the token is unset.
$ echo ${NOMAD_TOKEN}
Check the status.
$ nomad status
No running jobs
Once you have provide your users with tokens, you can update this anonymous policy (don't forget the description) to be more restrictive or delete it completely to deny all requests from unauthenticated users.
»Enable ACLs on Nomad clients
To enforce client endpoints, you need to enable ACLs on clients as well. Do this
by setting the enabled
value of the acl
stanza to true. Once complete,
restart the client to read in the new configuration.
»Create management tokens for other regions
Once you have bootstrapped ACLs on the servers of the authoritative region, you can create the replication tokens for all of the non-authoritative regions in a multi-region configuration. These tokens must be management-type tokens since they are used to communicate with ACL API in the authoritative region.
Create the replication token with the nomad acl token create
command. Don't
forget to provide a management token via the NOMAD_TOKEN environment variable or
the -token
flag. As practice, this time use the -token
flag:
$ nomad acl token create -type="management" -global=true \
-name="Cluster A Replication Token" \
-token="c999c4c2-6146-1bac-eb47-3958bbffe9d8"
Accessor ID = ec175d30-26ea-4a54-4850-45f833acece5
Secret ID = 9e2bb5ed-b3af-6bf3-5bbc-16dc684c5c31
Name = Cluster A Replication Token
Type = management
Global = true
Policies = n/a
Create Time = 2020-01-08 21:12:43.32324673 +0000 UTC
Create Index = 2916
Modify Index = 2916
»Re-bootstrap ACL system
If all management tokens are lost, it is possible to reset the ACL bootstrap so that it can be performed again. First, you need to determine the reset index with the bootstrap endpoint:
$ nomad acl bootstrap
Error bootstrapping: Unexpected response code: 500 (ACL bootstrap already done (reset index: 7))
The error message contains the reset index
. To reset the ACL system, create
a file named acl-bootstrap-reset
containing the value of the "reset index".
This file should be placed in the data directory of the leader node:
$ echo 7 >> /nomad-data-dir/server/acl-bootstrap-reset
Once the reset file is in place, you can re-bootstrap the cluster:
$ nomad acl bootstrap
Accessor ID = 52d3353d-d7b9-d945-0591-1af608732b76
Secret ID = 4b0a41ca-6d32-1853-e64b-de0d347e4525
Name = Bootstrap Token
Type = management
Global = true
Policies = n/a
Create Time = 2017-09-11 18:38:11.929089612 +0000 UTC
Create Index = 11
Modify Index = 11
If you attempt to bootstrap again you will get a mismatch on the reset index:
$ nomad acl bootstrap
Error bootstrapping: Unexpected response code: 500 (Invalid bootstrap reset index (specified 7, reset index: 11))
This is because the reset file is in place, but with the former index. The reset file can be deleted. However, if it is left behind, Nomad will not reset the bootstrap unless the file's contents match the actual reset index.
»Next steps
For a single cluster, the process is complete. For multi-region clusters, you can generate tokens local to each region to scope policies to a specific region, or use tokens with a Global scope so that you can submit jobs from any region to any other region.
Now that you have learned about bootstrapping your Nomad cluster, you will learn more about how to create Nomad ACL policies.