When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.
»Challenge
Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand. However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.
»Solution
Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, Amazon KMS, Azure Key Vault, and Google Cloud KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.
This tutorial demonstrates an example of using Azure Key Vault to auto-unseal a Vault server.
»Prerequisites
This tutorial assumes the following:
- You have a Microsoft Azure account
- Terraform installed and configured
»Terraform Azure Provider Prerequisites
A service principal is an application within Azure Active Directory which can be used to authenticate. Service principals are preferable to running an app using your own credentials. Follow the instruction in the Terraform documentation to create a service principal.
Be sure to add the following Azure Active Directory Graph API permission to your app.
Store the credentials as Environment Variables:
$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"
The following Azure credentials will be needed to perform the steps in this tutorial:
Subscription ID: Navigate to the Subscriptions blade within the Azure Portal and copy the SUBSCRIPTION ID
Tenant ID: Navigate to the Azure Active Directory > Properties in the Azure Portal, and copy the Directory ID which is your tenant ID
Client ID: Same as the Application ID
Client secret: The password (credential) set on your application
»Download demo assets
Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.
»Step 1: Provision the Cloud Resources
Be sure to set your working directory to where the
/operations/azure-keyvault-unseal
folder is located.
The working directory should contain the provided Terraform files:
$ cd vault-guides/operations/azure-keyvault-unseal
$ tree
.
├── README.md
├── main.tf
├── setup.tpl
├── terraform.tfvars.example
├── variables.tf
└── versions.tf
0 directories, 6 files
The main.tf
generates a new resource group with:
- Virtual machine with Vault already installed
- Azure Vault Key (
Test-vault-xxxx
) - A key (
generated-key
)
IMPORTANT: Ensure that your service principal app has appropriate role assignments to provision those Azure resources.
Set your Azure credentials in the terraform.tfvars.example
and save it as
terraform.tfvars
. Overwrite the default variable values (variables.tf
) as
needed.
Example terraform.tfvars
:
tenant_id="0000000-0000-0000-0000000000"
public_key = "ssh-rsa AAAA..."
client_id="0000000-000000-0000000000"
client_secret="AABBBCCCDDDDEEEFFF"
subscription_id="0000000-0000-0000-0000-0000000000"
Now you are ready to provision the cloud resources using Terraform.
Initialize the Azure provider plugins.
$ terraform init
Create an execution plan.
$ terraform plan
...
Plan: 12 to add, 0 to change, 0 to destroy.
Apply the changes.
$ terraform apply -auto-approve
...
Outputs:
ip = 13.82.62.56
key_vault_name = Test-vault-1e5a88de
ssh-addr =
Connect to your virtual machine via SSH:
$ ssh azureuser@13.82.62.56
»Step 2: Test the Auto-unseal Feature
Execute the following command to retrieve your virtual machine information:
$ terraform output ssh-addr
SSH to connect to your virtual machine with username, azureuser
:
$ ssh azureuser@<IP_address>
Run vault status
command to check current status:
$ vault status
Key Value
--- -----
Recovery Seal Type azurekeyvault
Initialized false
Sealed true
Total Recovery Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version n/a
HA Enabled true
Notice that Initialized is false
.
Run the vault operator init
command to
initialize the Vault
server so that you can unseal:
$ vault operator init
Check the Vault status to verify that it has been initialized and unsealed.
$ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.3.0
Cluster Name vault-cluster-092ba5de
Cluster ID 8b173565-7d74-fe5b-a199-a2b56b7019ee
HA Enabled false
Notice that the Vault server is already unsealed (Sealed is false
).
In the service log, you should find a trace where Azure Vault key is being fetched to unseal the Vault server.
$ sudo journalctl --no-pager -u vault
...
==> Vault server configuration:
Azure Environment: AzurePublicCloud
Azure Key Name: generated-key
Azure Vault Name: Test-vault-a414d041
Seal Type: azurekeyvault
Cgo: disabled
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: (not set)
Mlock: supported: true, enabled: false
Storage: file
Version: Vault v1.3.0
Version Sha: 37a1dc9c477c1c68c022d2084550f25bf20cac33
==> Vault server started! Log data will stream in below:
2019-01-22T19:43:29.114Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
2019-01-22T19:43:29.121Z [INFO] core: stored unseal keys supported, attempting fetch
2019-01-22T19:43:29.162Z [INFO] core: vault is unsealed
...
Restart the Vault server to ensure that Vault server gets automatically unsealed:
$ sudo systemctl restart vault
$ vault status
Explorer the systemd
configuration for Vault server which is located at
/lib/systemd/system/vault.service
:
$ cat /lib/systemd/system/vault.service
[Unit]
Description=Vault Agent
Requires=network-online.target
After=network-online.target
[Service]
Restart=on-failure
PermissionsStartOnly=true
ExecStartPre=/sbin/setcap 'cap_ipc_lock=+ep' /usr/local/bin/vault
ExecStart=/usr/local/bin/vault server -config /etc/vault.d/config.hcl
ExecReload=/bin/kill -HUP
KillSignal=SIGTERM
User=azureuser
Group=azureuser
[Install]
WantedBy=multi-user.target
Review the Vault configuration file (/etc/vault.d/config.hcl
):
$ cat /etc/vault.d/config.hcl
storage "file" {
path = "/opt/vault"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "true"
}
seal "azurekeyvault" {
client_id = "YOUR-APP-ID"
client_secret = "YOUR-APP-PASSWORD"
tenant_id = "YOUR-AZURE-TENANT-ID"
vault_name = "Test-vault-XXXXXX"
key_name = "generated-key"
}
ui=true
disable_mlock = true
Notice the Vault configuration file defines the azurekeyvault
stanza
with all parameter values properly populated: client ID, client secret, tenant
ID, vault name (generated by Terraform), and Azure Key Vault key name.
Although the listener stanza disables TLS (tls_disable = "true"
) for this
tutorial, Vault should always be used with
TLS
in production to provide secure communication between clients and the Vault
server. It requires a certificate file and key file on each Vault host.
»Key rotation
With auto-unseal enabled, set up Azure Key Vault with key rotation using the Azure Automation Account and Vault will recognize newly rotated keys since the key metadata is stored with the encrypted data to ensure the correct key is used during decryption operations.
»Step 3: Clean Up
Once completed, execute the following commands to clean up:
$ terraform destroy -auto-approve
$ rm -rf .terraform terraform.tfstate*