HCPEarn a free $50 credit for HCP Vault or HCP Consul today. Get Started

Auto-unseal using Azure Key Vault

When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.

Unseal with Shamir's Secret Sharing

»Challenge

Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand. However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.

»Solution

Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, Amazon KMS, Azure Key Vault, and Google Cloud KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.

Unseal

»Prerequisites

This tutorial assumes the following:

»Terraform Azure Provider Prerequisites

A service principal is an application within Azure Active Directory which can be used to authenticate. Service principals are preferable to running an app using your own credentials. Follow the instruction in the Terraform documentation to create a service principal.

Be sure to add the following Azure Active Directory Graph API permission to your app.

API permissions

Store the credentials as Environment Variables:

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

The following Azure credentials will be needed to perform the steps in this tutorial:

»Download demo assets

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

»Step 1: Provision the Cloud Resources

Be sure to set your working directory to where the /operations/azure-keyvault-unseal folder is located.

The working directory should contain the provided Terraform files:

$ cd vault-guides/operations/azure-keyvault-unseal

$ tree
.
├── README.md
├── main.tf
├── setup.tpl
├── terraform.tfvars.example
├── variables.tf
└── versions.tf

0 directories, 6 files

$ cd vault-guides/operations/azure-keyvault-unseal
$ tree.├── README.md├── main.tf├── setup.tpl├── terraform.tfvars.example├── variables.tf└── versions.tf
0 directories, 6 files

The main.tf generates a new resource group with:

  • Virtual machine with Vault already installed
  • Azure Vault Key (Test-vault-xxxx)
  • A key (generated-key)

Set your Azure credentials in the terraform.tfvars.example and save it as terraform.tfvars. Overwrite the default variable values (variables.tf) as needed.

Example terraform.tfvars:

tenant_id="0000000-0000-0000-0000000000"
public_key = "ssh-rsa AAAA..."
client_id="0000000-000000-0000000000"
client_secret="AABBBCCCDDDDEEEFFF"
subscription_id="0000000-0000-0000-0000-0000000000"

tenant_id="0000000-0000-0000-0000000000"public_key = "ssh-rsa AAAA..."client_id="0000000-000000-0000000000"client_secret="AABBBCCCDDDDEEEFFF"subscription_id="0000000-0000-0000-0000-0000000000"

Now you are ready to provision the cloud resources using Terraform.

Initialize the Azure provider plugins.

$ terraform init

$ terraform init

Create an execution plan.

$ terraform plan
...
Plan: 12 to add, 0 to change, 0 to destroy.

$ terraform plan...Plan: 12 to add, 0 to change, 0 to destroy.

Apply the changes.

$ terraform apply -auto-approve
...
Outputs:

ip = 13.82.62.56
key_vault_name = Test-vault-1e5a88de
ssh-addr =
    Connect to your virtual machine via SSH:

    $ ssh azureuser@13.82.62.56

$ terraform apply -auto-approve...Outputs:
ip = 13.82.62.56key_vault_name = Test-vault-1e5a88dessh-addr =    Connect to your virtual machine via SSH:
    $ ssh azureuser@13.82.62.56

»Step 2: Test the Auto-unseal Feature

Execute the following command to retrieve your virtual machine information:

$ terraform output ssh-addr

$ terraform output ssh-addr

SSH to connect to your virtual machine with username, azureuser:

$ ssh azureuser@<IP_address>

$ ssh azureuser@<IP_address>

Run vault status command to check current status:

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       azurekeyvault
Initialized              false
Sealed                   true
Total Recovery Shares    0
Threshold                0
Unseal Progress          0/0
Unseal Nonce             n/a
Version                  n/a
HA Enabled               true

$ vault status
Key                      Value---                      -----Recovery Seal Type       azurekeyvaultInitialized              falseSealed                   trueTotal Recovery Shares    0Threshold                0Unseal Progress          0/0Unseal Nonce             n/aVersion                  n/aHA Enabled               true

Notice that Initialized is false.

Run the vault operator init command to initialize the Vault server so that you can unseal:

$ vault operator init

$ vault operator init

Check the Vault status to verify that it has been initialized and unsealed.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.3.0
Cluster Name             vault-cluster-092ba5de
Cluster ID               8b173565-7d74-fe5b-a199-a2b56b7019ee
HA Enabled               false

$ vault status
Key                      Value---                      -----Recovery Seal Type       shamirInitialized              trueSealed                   falseTotal Recovery Shares    5Threshold                3Version                  1.3.0Cluster Name             vault-cluster-092ba5deCluster ID               8b173565-7d74-fe5b-a199-a2b56b7019eeHA Enabled               false

Notice that the Vault server is already unsealed (Sealed is false).

In the service log, you should find a trace where Azure Vault key is being fetched to unseal the Vault server.

$ sudo journalctl --no-pager -u vault

...
==> Vault server configuration:
        Azure Environment: AzurePublicCloud
           Azure Key Name: generated-key
         Azure Vault Name: Test-vault-a414d041
                Seal Type: azurekeyvault
                      Cgo: disabled
               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
                Log Level: (not set)
                    Mlock: supported: true, enabled: false
                  Storage: file
                  Version: Vault v1.3.0
              Version Sha: 37a1dc9c477c1c68c022d2084550f25bf20cac33
 ==> Vault server started! Log data will stream in below:
 2019-01-22T19:43:29.114Z [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
 2019-01-22T19:43:29.121Z [INFO]  core: stored unseal keys supported, attempting fetch
 2019-01-22T19:43:29.162Z [INFO]  core: vault is unsealed
...

$ sudo journalctl --no-pager -u vault
...==> Vault server configuration:        Azure Environment: AzurePublicCloud           Azure Key Name: generated-key         Azure Vault Name: Test-vault-a414d041                Seal Type: azurekeyvault                      Cgo: disabled               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")                Log Level: (not set)                    Mlock: supported: true, enabled: false                  Storage: file                  Version: Vault v1.3.0              Version Sha: 37a1dc9c477c1c68c022d2084550f25bf20cac33 ==> Vault server started! Log data will stream in below: 2019-01-22T19:43:29.114Z [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2019-01-22T19:43:29.121Z [INFO]  core: stored unseal keys supported, attempting fetch 2019-01-22T19:43:29.162Z [INFO]  core: vault is unsealed...

Restart the Vault server to ensure that Vault server gets automatically unsealed:

$ sudo systemctl restart vault

$ vault status

$ sudo systemctl restart vault
$ vault status

Explorer the systemd configuration for Vault server which is located at /lib/systemd/system/vault.service:

$ cat /lib/systemd/system/vault.service

[Unit]
Description=Vault Agent
Requires=network-online.target
After=network-online.target
[Service]
Restart=on-failure
PermissionsStartOnly=true
ExecStartPre=/sbin/setcap 'cap_ipc_lock=+ep' /usr/local/bin/vault
ExecStart=/usr/local/bin/vault server -config /etc/vault.d/config.hcl
ExecReload=/bin/kill -HUP
KillSignal=SIGTERM
User=azureuser
Group=azureuser
[Install]
WantedBy=multi-user.target

$ cat /lib/systemd/system/vault.service
[Unit]Description=Vault AgentRequires=network-online.targetAfter=network-online.target[Service]Restart=on-failurePermissionsStartOnly=trueExecStartPre=/sbin/setcap 'cap_ipc_lock=+ep' /usr/local/bin/vaultExecStart=/usr/local/bin/vault server -config /etc/vault.d/config.hclExecReload=/bin/kill -HUPKillSignal=SIGTERMUser=azureuserGroup=azureuser[Install]WantedBy=multi-user.target

Review the Vault configuration file (/etc/vault.d/config.hcl):

$ cat /etc/vault.d/config.hcl

storage "file" {
  path = "/opt/vault"
}
listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true"
}
seal "azurekeyvault" {
  client_id      = "YOUR-APP-ID"
  client_secret  = "YOUR-APP-PASSWORD"
  tenant_id      = "YOUR-AZURE-TENANT-ID"
  vault_name     = "Test-vault-XXXXXX"
  key_name       = "generated-key"
}
ui=true
disable_mlock = true

$ cat /etc/vault.d/config.hcl
storage "file" {  path = "/opt/vault"}listener "tcp" {  address     = "0.0.0.0:8200"  tls_disable = "true"}seal "azurekeyvault" {  client_id      = "YOUR-APP-ID"  client_secret  = "YOUR-APP-PASSWORD"  tenant_id      = "YOUR-AZURE-TENANT-ID"  vault_name     = "Test-vault-XXXXXX"  key_name       = "generated-key"}ui=truedisable_mlock = true

Notice the Vault configuration file defines the azurekeyvault stanza with all parameter values properly populated: client ID, client secret, tenant ID, vault name (generated by Terraform), and Azure Key Vault key name.

»Key rotation

With auto-unseal enabled, set up Azure Key Vault with key rotation using the Azure Automation Account and Vault will recognize newly rotated keys since the key metadata is stored with the encrypted data to ensure the correct key is used during decryption operations.

»Step 3: Clean Up

Once completed, execute the following commands to clean up:

$ terraform destroy -auto-approve

$ rm -rf .terraform terraform.tfstate*

$ terraform destroy -auto-approve
$ rm -rf .terraform terraform.tfstate*

»Help and Reference

Back to Collection
Hashicorp
stdin: is not a tty