Auto-unseal using GCP Cloud KMS

When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.

Unseal with Shamir's Secret Sharing

» Challenge

Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand. However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.

» Solution

Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, Amazon KMS, Azure Key Vault, and Google Cloud KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.

Unseal with Cloud KMS

This guide demonstrates an example of how to use Terraform to provision an instance that can utilize an encryption key from GCP Cloud KMS to unseal Vault.

» Prerequisites

This guide assumes the following:

» Download demo assets

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this guide.

» Steps

This guide demonstrates how to implement and use the Auto-unseal feature using GCP Cloud KMS. You are going to perform the following steps:

  1. Provision the Cloud Resources
  2. Test the Auto-unseal Feature
  3. Rotating the Unseal Key
  4. Clean Up

» Step 1: Provision the Cloud Resources

Be sure to set your working directory to where the /operations/gcp-kms-unseal folder is located.

The working directory should contain the provided Terraform files: Assets

The main.tf generates the following:

Provide necessary GCP account information in the terraform.tfvars.example and save it as terraform.tfvars. Overwrite the default variable values (variables.tf) as needed.

Example terrafrom.tfvars:

gcloud-project = "gcloud-vault-test"
account_file_path = "/Users/james/gcp/gcloud-vault-test.json"

gcloud-region = "us-central1"
gcloud-zone = "us-central1-c"
# key_ring = "test"
# crypto_key "vault-test"
# keyring_location = "global"

If you do not have a Cloud KMS key ring and crypto key to use, un-comment the key ring and key creation block in the main.tf file (line 77 - 81 and line 84 - 88).


# Create a KMS key ring
resource "google_kms_key_ring" "key_ring" {
   project  = "${var.gcloud-project}"
   name     = "${var.key_ring}"
   location = "${var.keyring_location}"

# Create a crypto key for the key ring
resource "google_kms_crypto_key" "crypto_key" {
   name            = "${var.crypto-key}"
   key_ring        = "${google_kms_key_ring.key_ring.self_link}"
   rotation_period = "100000s"

Also, un-comment line 92 and comment out line 93:

# Add the service account to the Keyring
resource "google_kms_key_ring_iam_binding" "vault_iam_kms_binding" {
   key_ring_id = "${google_kms_key_ring.key_ring.id}"
   # key_ring_id = "${var.gcloud-project}/${var.keyring_location}/${var.key_ring}"
   role = "roles/owner"

   members = [

If your Cloud KMS key ring (key_ring) and crypto key (crypto_key) names are different (either overwriting the default names or use an existing key ring and key), be sure to specify the correct values in terrafrom.tfvars.

Example terrafrom.tfvars:

gcloud-project = "gcloud-vault-test"
account_file_path = "/Users/james/gcp/gcloud-vault-test.json"

gcloud-region = "us-central1"
gcloud-zone = "us-central1-c"
key_ring = "my-key-ring-name"
crypto_key = "my-crypto-key-name"
# keyring_location = "global"

Now you are ready to provision the cloud resources using Terraform.

# Initialize the GCP provider plugins
$ terraform init

# Create an execution plan
$ terraform plan
Plan: 3 to add, 0 to change, 0 to destroy.

# Apply the changes
$ terraform apply
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

» Step 2: Test the Auto-unseal Feature

SSH into the generated compute instance. SSH

To verify that Vault has been installed, run vault status command:

$ vault status
Key                      Value
---                      -----
Recovery Seal Type       gcpckms
Initialized              false
Sealed                   true
Total Recovery Shares    0
Threshold                0
Unseal Progress          0/0
Unseal Nonce             n/a
Version                  n/a
HA Enabled               false

Notice that Initialized is false.

Run the vault operator init command to initialize the Vault server:

$ vault operator init -key-shares=1 -key-threshold=1

Recovery Key 1: VcPAM+orp6qbLxzaqeUS7pwqaubYLwbTbOdU2Ak+WqI=

Initial Root Token: s.6qANdVrojxyTzh9laAlBml55

Success! Vault is initialized

Recovery key initialized with 1 key shares and a key threshold of 1. Please
securely distribute the key shares printed above.

After a successful initialization, restart the Vault server:

$ sudo systemctl restart vault

Check the Vault status to verify that it has been started and unsealed.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.0.0
Cluster Name             vault-cluster-a5689781
Cluster ID               ef9303f3-3e16-1f55-3362-f5c8986df47e
HA Enabled               false

Notice that the Vault server is already unsealed (Sealed is false).

In the service log, you should find a trace where GCP KMS key is being fetched to unseal the Vault server.

$ sudo journalctl --no-pager -u vault

 ==> Vault server configuration:
       GCP KMS Crypto Key: vault-test
         GCP KMS Key Ring: test
          GCP KMS Project: gcloud-vault-test
           GCP KMS Region: global
                Seal Type: gcpckms
                      Cgo: disabled
               Listener 1: tcp (addr: "", cluster address: "", max_request_duration: "1m30s", max_re
quest_size: "33554432", tls: "disabled")
                Log Level: (not set)
                    Mlock: supported: true, enabled: false
                  Storage: file
                  Version: Vault v1.0.0
              Version Sha: c19cef14891751a23eaa9b41fd456d1f99e7e856
 ==> Vault server started! Log data will stream in below:
 2018-11-02T00:30:55.154Z [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if pos
sible, but this value should be manually set
 2018-11-02T00:30:55.163Z [INFO]  core: stored unseal keys supported, attempting fetch
 2018-11-02T00:30:55.199Z [INFO]  core: vault is unsealed

Review the Vault configuration file (/test/vault/config.hcl):

$ cat /test/vault/config.hcl

storage "file" {
  path = "/opt/vault"

listener "tcp" {
  address     = ""
  tls_disable = 1

seal "gcpckms" {
  project     = "gcloud-vault-test"
  region      = "global"
  key_ring    = "test"
  crypto_key  = "vault-test"

disable_mlock = true

Notice the Vault configuration file defines the gcpckms stanza which sets the GCP Cloud KMS key ring name, crypto key name, and its location as well as your GCP project ID.

» Step 3: Rotating the Unseal Key

When Vault is sealed with Shamir' keys, execute the vault operator rekey command to generate a new set of unseal keys. With Auto-unseal enabled, you can simply rotate the Cloud KMS key used to unseal Vault. One of the benefits of using Cloud KMS is its automatic key rotation feature which eliminates the need for a manual operation.

» Automatic rotation

To enable automatic rotation of a key, set the rotation schedule using the gcloud command-line tool:

$ gcloud kms keys update <KEY_NAME> \
         --location <LOCATION> \
         --keyring <KEYRING_NAME> \
         --rotation-period <ROTATION_PERIOD> \
         --next-rotation-time <NEXT_ROTATION_TIME>

Alternatively, you can manage it through GCP Console.

  1. From the navigation menu, select Security > Cryptographic Keys.
  2. Select the key ring used for Vault Auto-unseal.
  3. Select the key to rotate.
  4. Select EDIT ROTATION PERIOD. Key Rotation
  5. Select the desired rotation period, and then click SAVE. Key Rotation

» Manual rotation

To create a new key version and make it primary using the gcloud command-line tool:

$ gcloud kms keys versions create --location <LOCATION> \
         --keyring <KEYRING_NAME> \
         --key <KEY_NAME> --primary

Using the GCP Web Console, simply select ROTATE. Key Rotation

» Step 4: Clean Up

Once completed, execute the following commands to clean up:

$ terraform destroy -force

$ rm -rf .terraform terraform.tfstate*

» Troubleshooting

If vault status command returns an error (e.g. Error checking seal status: Get dial tcp getsockopt: connection refused), check the log.

$ sudo journalctl --no-pager -u vault

If the log does not exist, try running the Vault server manually.

$ sudo chmod -R 0755 /opt/vault
$ vault server -config=/test/vault/config.hcl

The output should describe what error has occurred. Be sure to verify that all parameter values are set correctly in the seal stanza of /test/vault/config.hcl

» Help and Reference