When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.
Challenge
Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand. However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.
Solution
Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, Amazon KMS, Azure Key Vault, and Google Cloud KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.
This guide demonstrates an example of how to use Terraform to provision an instance that can utilize an encryption key from GCP Cloud KMS to unseal Vault.
Prerequisites
This guide assumes the following:
- Google Cloud Platform account
- Terraform installed and basic understanding of its usage
Download demo assets
Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this guide.
Clone the repository:
$ git clone https://github.com/hashicorp/vault-guides.git
Or download the repository:
This repository contains supporting content for all of the Vault learn guides. The content specific to this guide can be found within a sub-directory.
Step 1: Provision the Cloud Resources
Be sure to set your working directory to where the
/operations/gcp-kms-unseal
folder is located.
The working directory should contain the provided Terraform files:
$ cd vault-guides/operations/gcp-kms-unseal
$ tree
.
├── README.md
├── main.tf
├── terraform.tfvars.example
├── variables.tf
└── versions.tf
The main.tf
generates the following:
- Service account with Cloud KMS IAM to be used by Compute Engine instances
- Compute Engine instance with Vault installed
- (Optional) Cloud KMS key ring and crypto key
Provide necessary GCP account information in the terraform.tfvars.example
and
save it as terraform.tfvars
. Overwrite the default variable values
(variables.tf
) as needed.
Example terraform.tfvars
:
gcloud-project = "gcloud-vault-test"
account_file_path = "/Users/james/gcp/gcloud-vault-test.json"
gcloud-region = "us-central1"
gcloud-zone = "us-central1-c"
It assumes that you have a Cloud KMS key ring named, "test" and a crypto key
named "vault-test". If your Cloud KMS key ring (key_ring
) and crypto key
(crypto_key
) names are different, be sure to set the correct values in
terraform.tfvars
.
Example terraform.tfvars
:
gcloud-project = "gcloud-vault-test"
account_file_path = "/Users/james/gcp/gcloud-vault-test.json"
gcloud-region = "us-central1"
gcloud-zone = "us-central1-c"
key_ring = "my-key-ring-name"
crypto_key = "my-crypto-key-name"
# keyring_location = "global"
Cloud KMS: If you do NOT have a Cloud KMS key ring and crypto key to
use, un-comment the key ring and key creation block in the main.tf
file (line
77 - 81 and line 84 - 88).
...
# Create a KMS key ring
resource "google_kms_key_ring" "key_ring" {
project = "${var.gcloud-project}"
name = "${var.key_ring}"
location = "${var.keyring_location}"
}
# Create a crypto key for the key ring
resource "google_kms_crypto_key" "crypto_key" {
name = "${var.crypto-key}"
key_ring = "${google_kms_key_ring.key_ring.self_link}"
rotation_period = "100000s"
}
...
NOTE: Terraform will create a Cloud KMS key ring named, test in
the global location, and a key named, vault-test. If you wish to
customize the key and key ring names, be sure to overwrite the key_ring
and
crypto_key
values in the terraform.tfvars
.
Also, un-comment line 92 and comment out line 93:
# Add the service account to the Keyring
resource "google_kms_key_ring_iam_binding" "vault_iam_kms_binding" {
key_ring_id = "${google_kms_key_ring.key_ring.id}"
# key_ring_id = "${var.gcloud-project}/${var.keyring_location}/${var.key_ring}"
role = "roles/owner"
members = [
"serviceAccount:${google_service_account.vault_kms_service_account.email}",
]
}
Now you are ready to provision the cloud resources using Terraform.
# Initialize the GCP provider plugins
$ terraform init
# Create an execution plan
$ terraform plan
...
Plan: 3 to add, 0 to change, 0 to destroy.
# Apply the changes
$ terraform apply
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Step 2: Test the Auto-unseal Feature
SSH into the generated compute instance.
To verify that Vault has been installed, run vault status
command:
$ vault status
Key Value
--- -----
Recovery Seal Type gcpckms
Initialized false
Sealed true
Total Recovery Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version n/a
HA Enabled false
Notice that Initialized is false
.
NOTE: If you received an error instead, refer to the troubleshooting section.
Run the vault operator init
command to initialize the Vault server:
$ vault operator init -recovery-shares=1 -recovery-threshold=1
Recovery Key 1: VcPAM+orp6qbLxzaqeUS7pwqaubYLwbTbOdU2Ak+WqI=
Initial Root Token: s.6qANdVrojxyTzh9laAlBml55
Success! Vault is initialized
Recovery key initialized with 1 key shares and a key threshold of 1. Please
securely distribute the key shares printed above.
After a successful initialization, check the Vault server status:
$ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 1
Threshold 1
Version 1.3.0
Cluster Name vault-cluster-1b2159d8
Cluster ID f29c09db-d17a-b3b0-f5ed-d494b4446043
HA Enabled false
Notice that the Vault server is already unsealed (Sealed is false
).
In the service log, you should find a trace where GCP KMS key is being fetched to unseal the Vault server.
$ sudo journalctl --no-pager -u vault
...
==> Vault server configuration:
GCP KMS Crypto Key: vault-test
GCP KMS Key Ring: test
GCP KMS Project: gcloud-vault-test
GCP KMS Region: global
Seal Type: gcpckms
Cgo: disabled
Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_re
quest_size: "33554432", tls: "disabled")
Log Level: (not set)
Mlock: supported: true, enabled: false
Storage: file
Version: Vault v1.3.0
Version Sha: c19cef14891751a23eaa9b41fd456d1f99e7e856
==> Vault server started! Log data will stream in below:
...
Restart the Vault server to ensure that Vault server gets automatically unsealed:
$ sudo systemctl restart vault
$ vault status
Review the Vault configuration file (/test/vault/config.hcl
):
$ cat /test/vault/config.hcl
storage "file" {
path = "/opt/vault"
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = 1
}
seal "gcpckms" {
project = "gcloud-vault-test"
region = "global"
key_ring = "test"
crypto_key = "vault-test"
}
disable_mlock = true
Notice the Vault configuration file defines the gcpckms
stanza which
sets the GCP Cloud KMS key ring name, crypto key name, and its location as well
as your GCP project ID.
The credentials
value is not needed. This is because the Vault server running on this Compute
Engine instance has it own service account with
Cloud KMS role.
Step 3: Rotating the Unseal Key
When Vault is sealed with Shamir' keys, execute the vault operator rekey
command to generate a new set of unseal keys. With Auto-unseal enabled, you can simply rotate the Cloud KMS key used to unseal Vault. One of the benefits of using Cloud KMS is its automatic key rotation feature which eliminates the need for a manual operation.
Automatic rotation
To enable automatic rotation of a key, set the rotation schedule using the gcloud
command-line tool:
$ gcloud kms keys update <KEY_NAME> \
--location <LOCATION> \
--keyring <KEYRING_NAME> \
--rotation-period <ROTATION_PERIOD> \
--next-rotation-time <NEXT_ROTATION_TIME>
Alternatively, you can manage it through GCP Console.
- From the navigation menu, select Security > Cryptographic Keys.
- Select the key ring used for Vault Auto-unseal.
- Select the key to rotate.
- Select EDIT ROTATION PERIOD.
- Select the desired rotation period, and then click SAVE.
Manual rotation
To create a new key version and make it primary using the gcloud
command-line tool:
$ gcloud kms keys versions create --location <LOCATION> \
--keyring <KEYRING_NAME> \
--key <KEY_NAME> --primary
Using the GCP Web Console, simply select ROTATE.
To learn more about Cloud KMS key rotation, refer to its online documentation.
Step 4: Clean Up
Once completed, execute the following commands to clean up:
$ terraform destroy -force
$ rm -rf .terraform terraform.tfstate*
Troubleshooting
If vault status
command returns an error (e.g. Error checking seal status: Get http://127.0.0.1:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: getsockopt: connection refused
), check the log.
$ sudo journalctl --no-pager -u vault
If the log does not exist, try running the Vault server manually.
$ sudo chmod -R 0755 /opt/vault
$ vault server -config=/test/vault/config.hcl
The output should describe what error has occurred. Be sure to verify that all
parameter values are set correctly in the seal
stanza of
/test/vault/config.hcl