When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.
»Challenge
Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand.
However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.
»Solution
Vault supports opt-in automatic unsealing via transit secrets engine. This feature enables operators to delegate the unsealing process to a trusted Vault environment to ease operations.
This tutorial demonstrates how to auto-unseal a Vault with Transit secrets engine.
To auto-unseal your Vault with cloud provider's key, refer to the following tutorials:
»Prerequisites
To perform the tasks described in this tutorial, you need to have a Vault 1.1 or later. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.
An interactive tutorial is also available below if you do not have a Vault environment to perform the steps described in this tutorial. Click the Show Terminal button to start.
»Scenario Introduction
For the purpose of demonstration, you are going to run two instances of Vault as described in the following diagram:
In reality, the Vault 1 and Vault 2 are two separate stand-alone Vault clusters where one protecting another's master key. Nonetheless, the steps described in this tutorial directly applies to your clustered environment. The main difference would be the location (address) of Vault 1 and Vault 2.
In this scenario, Vault 1 (http://127.0.0.1:8200
) is the encryption
service provider, and its transit key protects the Vault 2 server's master
key.
»Step 1: Configure Auto-unseal Key Provider (Vault 1)
The first step is to enable and configure the transit
secrets engine on
Vault 1.
Enable an audit device if it hasn't been enabled already so that you can examine the audit log later in Step 3.
$ vault audit enable file file_path=audit.log
Execute the following command to enable the
transit
secrets engine.$ vault secrets enable transit
Execute the following command to create an encryption key named, "autounseal".
# Create a key named 'autounseal' $ vault write -f transit/keys/autounseal
NOTE: To learn more about the
transit
secrets engine, refer to the Encryption as a Service: Transit Secrets Engine.Create a policy file named
autounseal.hcl
which permitsupdate
againsttransit/encrypt/autounseal
andtransit/decrypt/autounseal
paths.$ tee autounseal.hcl <<EOF path "transit/encrypt/autounseal" { capabilities = [ "update" ] } path "transit/decrypt/autounseal" { capabilities = [ "update" ] } EOF
Create a policy named
autounseal
.$ vault policy write autounseal autounseal.hcl
Create a policy to permit
update
againsttransit/encrypt/<key_name>
andtransit/decrypt/<key_name>
where the<key_name>
is the name of the encryption key you created in the previous step.Create a client token with
autounseal
policy attached and response wrap it with TTL of 120 seconds.$ vault token create -policy="autounseal" -wrap-ttl=120 Key Value --- ----- wrapping_token: s.qg6HS6WNOYQ2SjnyyMqdSbZ0 wrapping_accessor: 4mHRpfx5oZcHhWAg6VeTR4or wrapping_token_ttl: 2m wrapping_token_creation_time: 2019-04-15 15:28:30.598376 -0700 PDT wrapping_token_creation_path: auth/token/create wrapped_accessor: GlZflsyuG891yw3PYC4b7GrM
Pass the generated
wrapping_token
value to Vault 2.
»Step 2: Configure Auto-unseal (Vault 2)
Now, start a second Vault instance which listens to port 8100. The server
configuration file should define a seal
stanza with parameters properly set
based on the tasks you performed in Step 1.
Execute the following command to unwrap the secrets passed from Vault 1.
$ VAULT_TOKEN=<wrapping_token> vault unwrap
For example, if the wrapping token value was "s.qg6HS6WNOYQ2SjnyyMqdSbZ0", the command would look as follow:
$ VAULT_TOKEN="s.AFqDxN5jdiDQDNuodJxsC6dm" vault unwrap Key Value --- ----- token s.SJj086AW7ZaobvRmNoSjhVaj token_accessor 6lUHdlwRZfyRnHtNiz8ZB1Jx token_duration 768h token_renewable true token_policies ["autounseal" "default"] identity_policies [] policies ["autounseal" "default"]
The revealed token is the client token Vault 2 will use to connect with Vault 1.
Set
VAULT_TOKEN
environment variable whose value is the client token you just unwrapped.Example:
$ export VAULT_TOKEN="s.SJj086AW7ZaobvRmNoSjhVaj"
Create a server configuration file (
config-autounseal.hcl
) to start a second Vault instance (Vault 2).disable_mlock = true ui=true storage "file" { path = "/vault-2/data" } listener "tcp" { address = "127.0.0.1:8100" tls_disable = "true" } seal "transit" { address = "http://127.0.0.1:8200" disable_renewal = "false" key_name = "autounseal" mount_path = "transit/" tls_skip_verify = "true" }
Notice that the
address
points to the Vault server listening to port 8200 (Vault 1). Thekey_name
andmount_path
match to what you created in Step 1.NOTE: The
seal
stanza does not set thetoken
value since it's already set asVAULT_TOKEN
environment variable.Although the listener stanza disables TLS (
tls_disable = "true"
) for this tutorial, Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each Vault host.Start the vault server with the configuration file.
$ vault server -config=config-autounseal.hcl
Open another terminal and initialize your second Vault server (Vault 2).
$ VAULT_ADDR=http://127.0.0.1:8100 vault operator init
By passing the
VAULT_ADDR
, the subsequent command gets executed against the second Vault server (http://127.0.0.1:8100).Example output:
Recovery Key 1: iz1XWxe4CM+wrOGqRCx8ex8kB2XvGJEdfjhXFC+MA6Rc Recovery Key 2: rKZETr6IAy686IxfO3ZBKXPDAOkkwkpSepIME+bjeUT7 Recovery Key 3: 4XA/KJqFOm+jzbBkKQuRVePEYPrQe3H3TmFVmdlUjRFv Recovery Key 4: lfnaYoZufP0uhooO3mHDAKGNZB5HLP9HYYb+LAfKkUmd Recovery Key 5: L169hHj3DMpphGsOnS8TEz3Febvdx3vsG3Xr8kGWdUtW Initial Root Token: s.AWnDagUkKNNbvkENiL72wysn Success! Vault is initialized Recovery key initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above.
The initialization generates recovery keys (instead of unseal keys) when using auto-unseal. Some of the Vault operations still require Shamir keys. For example, to regenerate a root token, each key holder must enter their recovery key. Similar to unseal keys, you can specify the number of recovery keys and the threshold using the
-recovery-shares
and-recovery-threshold
flags. It is strongly recommended to initialize Vault with PGP.Check the Vault 2 server status. It is now successfully initialized and unsealed.
$ VAULT_ADDR=http://127.0.0.1:8100 vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 # ...snip...
Notice that it shows
Total Recovery Shares
instead ofTotal Shares
. The transit secrets engine is solely responsible for protecting the master key of Vault 2.
»Step 3: Verify Auto-Unseal
When you stop and start the Vault 2 server, it comes up in the
unsealed
state and ready for operations.
To verify that Vault 2 gets automatically unseal, press Ctrl + C to stop the Vault 2 server where it is running.
...snip... [INFO] core.cluster-listener: rpc listeners successfully shut down [INFO] core: cluster listeners successfully shut down [INFO] core: vault is sealed
Note that Vault 2 is now sealed.
Press the upper-arrow key, and execute the
vault server -config=config-autounseal.hcl
command again to start Vault 2 and see what happens.$ vault server -config=config-autounseal.hcl ==> Vault server configuration: Seal Type: transit Transit Address: http://127.0.0.1:8200 Transit Key Name: autounseal Transit Mount Path: transit/ Cgo: disabled Listener 1: tcp (addr: "0.0.0.0:8100", cluster address: "0.0.0.0:8101", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") Log Level: info Mlock: supported: true, enabled: false Storage: file Version: Vault v1.1.0 Version Sha: 36aa8c8dd1936e10ebd7a4c1d412ae0e6f7900bd ==> Vault server started! Log data will stream in below: [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set [INFO] core: stored unseal keys supported, attempting fetch [INFO] core: vault is unsealed # ...snip...
Notice that the Vault server is already unsealed. The Transit Address is set to your Vault 1 which is listening to port 8200 (http://127.0.0.1:8200).
Check the Vault 2 server status.
$ VAULT_ADDR=http://127.0.0.1:8100 vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 # ...snip...
Now, examine the audit log in Vault 1.
$ tail -f audit.log | jq # ...snip... "request": { "id": "a46719eb-eee0-92a4-2da6-6c7de77fd410", "operation": "update", "client_token": "hmac-sha256:ce8613487054dadb36a9d08da1f5a4bbee2fbfc1ef1ec5ebdeec696df7823e69", "client_token_accessor": "hmac-sha256:f3b6cb798605835e8a00bafa9e0e16fc0534b8923b31e499f2c8e694f6b69158", "namespace": { "id": "root", "path": "" }, "path": "transit/decrypt/autounseal", # ...snip... "remote_address": "127.0.0.1", "wrap_ttl": 0, "headers": {} }, # ...snip... }
You should see an
update
request against thetransit/decrypt/autounseal
path. Theremote_address
is127.0.0.1
in this example since Vault 1 and Vault 2 are both running locally. If the Vault 2 is running on a different host, the audit log will show the IP address of the Vault 2 host.
If a security incident forces you to seal the Vault server using the vault
operator seal
command, it requires the threshold number of recovery keys to
unseal Vault and bring it back to operation.