Auto-unseal using Transit Secrets Engine

When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.

»Challenge

Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand.

However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.

»Solution

Vault supports opt-in automatic unsealing via transit secrets engine. This feature enables operators to delegate the unsealing process to a trusted Vault environment to ease operations.

This tutorial demonstrates how to auto-unseal a Vault with Transit secrets engine.

To auto-unseal your Vault with cloud provider's key, refer to the following tutorials:

»Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault 1.1 or later. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

»Scenario Introduction

For the purpose of demonstration, you are going to run two instances of Vault as described in the following diagram:

In reality, the Vault 1 and Vault 2 are two separate stand-alone Vault clusters where one protecting another's root key. Nonetheless, the steps described in this tutorial directly applies to your clustered environment. The main difference would be the location (address) of Vault 1 and Vault 2.

In this scenario, Vault 1 (http://127.0.0.1:8200) is the encryption service provider, and its transit key protects the Vault 2 server's master key.

»Step 1: Configure Auto-unseal Key Provider (Vault 1)

The first step is to enable and configure the transit secrets engine on Vault 1.

Enable an audit device if it hasn't been enabled already so that you can examine the audit log later in Step 3.
```
$ vault audit enable file file_path=audit.log
```
```
$ vault audit enable file file_path=audit.log
```

Execute the following command to enable the transit secrets engine.

$ vault secrets enable transit

$ vault secrets enable transit

Execute the following command to create an encryption key named, "autounseal".
```
$ vault write -f transit/keys/autounseal
```
```
$ vault write -f transit/keys/autounseal
```
NOTE: To learn more about the transit secrets engine, refer to the Encryption as a Service: Transit Secrets Engine.

Create a policy file named autounseal.hcl which permits update against transit/encrypt/autounseal and transit/decrypt/autounseal paths.

$ tee autounseal.hcl <<EOF
path "transit/encrypt/autounseal" {
   capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
   capabilities = [ "update" ]
}
EOF

$ tee autounseal.hcl <<EOF
path "transit/encrypt/autounseal" {
   capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
   capabilities = [ "update" ]
}
EOF

Create a policy named autounseal.
```
$ vault policy write autounseal autounseal.hcl
```
```
$ vault policy write autounseal autounseal.hcl
```
Create a policy to permit update against transit/encrypt/<key_name> and transit/decrypt/<key_name> where the <key_name> is the name of the encryption key you created in the previous step.

Create a client token with autounseal policy attached and response wrap it with TTL of 120 seconds.

$ vault token create -policy="autounseal" -wrap-ttl=120

Key                              Value
---                              -----
wrapping_token:                  s.qg6HS6WNOYQ2SjnyyMqdSbZ0
wrapping_accessor:               4mHRpfx5oZcHhWAg6VeTR4or
wrapping_token_ttl:              2m
wrapping_token_creation_time:    2019-04-15 15:28:30.598376 -0700 PDT
wrapping_token_creation_path:    auth/token/create
wrapped_accessor:                GlZflsyuG891yw3PYC4b7GrM

$ vault token create -policy="autounseal" -wrap-ttl=120

Key                              Value
---                              -----
wrapping_token:                  s.qg6HS6WNOYQ2SjnyyMqdSbZ0
wrapping_accessor:               4mHRpfx5oZcHhWAg6VeTR4or
wrapping_token_ttl:              2m
wrapping_token_creation_time:    2019-04-15 15:28:30.598376 -0700 PDT
wrapping_token_creation_path:    auth/token/create
wrapped_accessor:                GlZflsyuG891yw3PYC4b7GrM

Pass the generated wrapping_token value to Vault 2.

»Step 2: Configure Auto-unseal (Vault 2)

Now, start a second Vault instance which listens to port 8100. The server configuration file should define a seal stanza with parameters properly set based on the tasks you performed in Step 1.

Execute the following command to unwrap the secrets passed from Vault 1.

$ VAULT_TOKEN=<wrapping_token> vault unwrap

$ VAULT_TOKEN=<wrapping_token> vault unwrap

For example, if the wrapping token value was "s.qg6HS6WNOYQ2SjnyyMqdSbZ0", the command would look as follow:

$ VAULT_TOKEN="s.AFqDxN5jdiDQDNuodJxsC6dm" vault unwrap

Key                  Value
---                  -----
token                s.SJj086AW7ZaobvRmNoSjhVaj
token_accessor       6lUHdlwRZfyRnHtNiz8ZB1Jx
token_duration       768h
token_renewable      true
token_policies       ["autounseal" "default"]
identity_policies    []
policies             ["autounseal" "default"]

$ VAULT_TOKEN="s.AFqDxN5jdiDQDNuodJxsC6dm" vault unwrap

Key                  Value
---                  -----
token                s.SJj086AW7ZaobvRmNoSjhVaj
token_accessor       6lUHdlwRZfyRnHtNiz8ZB1Jx
token_duration       768h
token_renewable      true
token_policies       ["autounseal" "default"]
identity_policies    []
policies             ["autounseal" "default"]

The revealed token is the client token Vault 2 will use to connect with Vault 1.

Set VAULT_TOKEN environment variable whose value is the client token you just unwrapped.

Example:

$ export VAULT_TOKEN="s.SJj086AW7ZaobvRmNoSjhVaj"

$ export VAULT_TOKEN="s.SJj086AW7ZaobvRmNoSjhVaj"

Create a server configuration file (config-autounseal.hcl) to start a second Vault instance (Vault 2).

disable_mlock = true
ui=true

storage "file" {
  path = "/vault-2/data"
}

listener "tcp" {
  address     = "127.0.0.1:8100"
  tls_disable = "true"
}

seal "transit" {
  address = "http://127.0.0.1:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
  tls_skip_verify = "true"
}

disable_mlock = true
ui=true

storage "file" {
  path = "/vault-2/data"
}

listener "tcp" {
  address     = "127.0.0.1:8100"
  tls_disable = "true"
}

seal "transit" {
  address = "http://127.0.0.1:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
  tls_skip_verify = "true"
}

Notice that the address points to the Vault server listening to port 8200 (Vault 1). The key_name and mount_path match to what you created in Step 1.

NOTE: The seal stanza does not set the token value since it's already set as VAULT_TOKEN environment variable.

Although the listener stanza disables TLS (tls_disable = "true") for this tutorial, Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each Vault host.

Start the vault server with the configuration file.

$ vault server -config=config-autounseal.hcl

$ vault server -config=config-autounseal.hcl

Open another terminal and initialize your second Vault server (Vault 2).

$ VAULT_ADDR=http://127.0.0.1:8100 vault operator init

$ VAULT_ADDR=http://127.0.0.1:8100 vault operator init

By passing the VAULT_ADDR, the subsequent command gets executed against the second Vault server (http://127.0.0.1:8100).

Example output:

Recovery Key 1: iz1XWxe4CM+wrOGqRCx8ex8kB2XvGJEdfjhXFC+MA6Rc
Recovery Key 2: rKZETr6IAy686IxfO3ZBKXPDAOkkwkpSepIME+bjeUT7
Recovery Key 3: 4XA/KJqFOm+jzbBkKQuRVePEYPrQe3H3TmFVmdlUjRFv
Recovery Key 4: lfnaYoZufP0uhooO3mHDAKGNZB5HLP9HYYb+LAfKkUmd
Recovery Key 5: L169hHj3DMpphGsOnS8TEz3Febvdx3vsG3Xr8kGWdUtW

Initial Root Token: s.AWnDagUkKNNbvkENiL72wysn

Success! Vault is initialized

Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

Recovery Key 1: iz1XWxe4CM+wrOGqRCx8ex8kB2XvGJEdfjhXFC+MA6Rc
Recovery Key 2: rKZETr6IAy686IxfO3ZBKXPDAOkkwkpSepIME+bjeUT7
Recovery Key 3: 4XA/KJqFOm+jzbBkKQuRVePEYPrQe3H3TmFVmdlUjRFv
Recovery Key 4: lfnaYoZufP0uhooO3mHDAKGNZB5HLP9HYYb+LAfKkUmd
Recovery Key 5: L169hHj3DMpphGsOnS8TEz3Febvdx3vsG3Xr8kGWdUtW

Initial Root Token: s.AWnDagUkKNNbvkENiL72wysn

Success! Vault is initialized

Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

The initialization generates recovery keys (instead of unseal keys) when using auto-unseal. Some of the Vault operations still require Shamir keys. For example, to regenerate a root token, each key holder must enter their recovery key. Similar to unseal keys, you can specify the number of recovery keys and the threshold using the -recovery-shares and -recovery-threshold flags. It is strongly recommended to initialize Vault with PGP.

Check the Vault 2 server status. It is now successfully initialized and unsealed.

$ VAULT_ADDR=http://127.0.0.1:8100 vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
# ...snip...

$ VAULT_ADDR=http://127.0.0.1:8100 vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
# ...snip...

Notice that it shows Total Recovery Shares instead of Total Shares. The transit secrets engine is solely responsible for protecting the root key of Vault 2.

»Step 3: Verify Auto-Unseal

When you stop and start the Vault 2 server, it comes up in the unsealed state and ready for operations.

To verify that Vault 2 gets automatically unseal, press Ctrl + C to stop the Vault 2 server where it is running.

...snip...
[INFO]  core.cluster-listener: rpc listeners successfully shut down
[INFO]  core: cluster listeners successfully shut down
[INFO]  core: vault is sealed

...snip...
[INFO]  core.cluster-listener: rpc listeners successfully shut down
[INFO]  core: cluster listeners successfully shut down
[INFO]  core: vault is sealed

Note that Vault 2 is now sealed.

Press the upper-arrow key, and execute the vault server -config=config-autounseal.hcl command again to start Vault 2 and see what happens.

$ vault server -config=config-autounseal.hcl

==> Vault server configuration:

                Seal Type: transit
          Transit Address: http://127.0.0.1:8200
        Transit Key Name: autounseal
      Transit Mount Path: transit/
                      Cgo: disabled
              Listener 1: tcp (addr: "0.0.0.0:8100", cluster address: "0.0.0.0:8101", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
                Log Level: info
                    Mlock: supported: true, enabled: false
                  Storage: file
                  Version: Vault v1.1.0
              Version Sha: 36aa8c8dd1936e10ebd7a4c1d412ae0e6f7900bd

==> Vault server started! Log data will stream in below:

[WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
[INFO]  core: stored unseal keys supported, attempting fetch
[INFO]  core: vault is unsealed
# ...snip...

$ vault server -config=config-autounseal.hcl

==> Vault server configuration:

                Seal Type: transit
          Transit Address: http://127.0.0.1:8200
        Transit Key Name: autounseal
      Transit Mount Path: transit/
                      Cgo: disabled
              Listener 1: tcp (addr: "0.0.0.0:8100", cluster address: "0.0.0.0:8101", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
                Log Level: info
                    Mlock: supported: true, enabled: false
                  Storage: file
                  Version: Vault v1.1.0
              Version Sha: 36aa8c8dd1936e10ebd7a4c1d412ae0e6f7900bd

==> Vault server started! Log data will stream in below:

[WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
[INFO]  core: stored unseal keys supported, attempting fetch
[INFO]  core: vault is unsealed
# ...snip...

Notice that the Vault server is already unsealed. The Transit Address is set to your Vault 1 which is listening to port 8200 (http://127.0.0.1:8200).

Check the Vault 2 server status.

$ VAULT_ADDR=http://127.0.0.1:8100 vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
# ...snip...

$ VAULT_ADDR=http://127.0.0.1:8100 vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
# ...snip...

Now, examine the audit log in Vault 1.

$ tail -f audit.log | jq

# ...snip...
"request": {
  "id": "a46719eb-eee0-92a4-2da6-6c7de77fd410",
  "operation": "update",
  "client_token": "hmac-sha256:ce8613487054dadb36a9d08da1f5a4bbee2fbfc1ef1ec5ebdeec696df7823e69",
  "client_token_accessor": "hmac-sha256:f3b6cb798605835e8a00bafa9e0e16fc0534b8923b31e499f2c8e694f6b69158",
  "namespace": {
    "id": "root",
    "path": ""
  },
  "path": "transit/decrypt/autounseal",
    # ...snip...
  "remote_address": "127.0.0.1",
  "wrap_ttl": 0,
  "headers": {}
},
# ...snip...
}

$ tail -f audit.log | jq

# ...snip...
"request": {
  "id": "a46719eb-eee0-92a4-2da6-6c7de77fd410",
  "operation": "update",
  "client_token": "hmac-sha256:ce8613487054dadb36a9d08da1f5a4bbee2fbfc1ef1ec5ebdeec696df7823e69",
  "client_token_accessor": "hmac-sha256:f3b6cb798605835e8a00bafa9e0e16fc0534b8923b31e499f2c8e694f6b69158",
  "namespace": {
    "id": "root",
    "path": ""
  },
  "path": "transit/decrypt/autounseal",
    # ...snip...
  "remote_address": "127.0.0.1",
  "wrap_ttl": 0,
  "headers": {}
},
# ...snip...
}

You should see an update request against the transit/decrypt/autounseal path. The remote_address is 127.0.0.1 in this example since Vault 1 and Vault 2 are both running locally. If the Vault 2 is running on a different host, the audit log will show the IP address of the Vault 2 host.

If a security incident forces you to seal the Vault server using the vault operator seal command, it requires the threshold number of recovery keys to unseal Vault and bring it back to operation.

Infrastructure

Security

Applications

Networking

cloud