Tech Preview / Beta

[Tech Preview] Vault HA Cluster with Integrated Storage

Challenge

Vault offers a number of configurable storage options today (e.g. Consul, MySQL, DynamoDB, etc.) and these are called storage backend and responsible for persisting the encrypted data.

Management of a persistent storage backend can be challenging, as it frequently requires administration-level knowledge of one of these separate systems. This complexity can lead to challenges in configuring and using Vault and complicate efforts to perform root cause analysis on potential failures in Vault.

Solution

Use Vault's integrated storage to persist the encrypted data. The Raft algorithm used in Consul is embedded directly into Vault to provide this built-in storage.

Ref Arch

Integrated Storage supports Vault failover and multi-cluster replication without depending on an external storage provider and making it operationally much simpler. By eliminating the network hop of connecting from Vault to the external storage provider, it contributes to performance gain (reduces disk write/read overhead).

If Vault has an outage, Vault is the only product to diagnose rather than two (Vault and its external storage provider), allowing for faster time to recovery.

Prerequisites

This guide provides two options to explore the integrated storage:

  • Option 1: Create a Vault HA cluster locally
  • Option 2: Create a Vault HA cluster on AWS using Terraform

To perform Option 2, you will need:

  • AWS account and associated credentials that allow for the creation of resources
  • Terraform is installed

Download demo assets

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this guide.

$ git clone git@github.com:hashicorp/vault-guides.git

Download

Steps

There are two options to explore the integrated storage:

Chose either Option 1 or Option 2 to create a Vault HA cluster with integrated storage for persistence, and then refer to the following steps to explore further:

  1. Raft snapshots for Vault data recovery
  2. Remove a cluster member
  3. Clean up

Option 1: Create a Vault HA cluster locally

Set your working directory to where the /vault-guides/operations/raft-storage repo was cloned/downloaded.

The directory should contain the following files and folders: Assets

The /local-test/raft-cluster-demo.sh script performs the following:

  • Start a Vault server, vault_1 (http://127.0.0.1:8200) which is used to auto-unseal other Vault servers. This server will not participate in the HA cluster (line 22 through 46).
  • Start a cluster members for this demo: vault_2, vault_3 and vault_4
    • Initialize vault_2 (line 125)
    • K/V secrets engine is enabled and some test data was created on vault_2 (line 134 through 140)

Scenario

  1. Execute the /local-test/network.sh script to add 127.0.0.2, 127.0.0.3 and 127.0.0.4 loopback addresses.

    # Change the working directory to where 'local-test' folder is located
    $ cd local-test
    
    # Ensure that network.sh is executable
    $ chmod +x network.sh
    
    # Now, execute the network.sh script
    $ ./network.sh
    

    127.0.0.0/8 address block is assigned for use as the Internet host loopback address. (RFC3330)

  2. At line 20 of the raft-cluster-demo.sh script, the TEST_HOME is set to $HOME/raft-test. All the generated files and folders will be created under this directory. If you wish to output elsewhere, be sure to modify this variable.

    TEST_HOME=$HOME/raft-test
    

    Execute the script to start four instances of Vault.

    # Make sure that the raft-cluster-demo.sh script is executable
    $ chmod +x raft-cluster-demo.sh
    
    # Execute the script
    $ ./raft-cluster-demo.sh
    
  3. The script output shows the test data created on vault_2:

    ...
    ++ VAULT_ADDR=http://127.0.0.2:8200
    ++ vault login s.ArffzRknwpSupR8XbIriGNS3
    
         ...
    
    ++ vault_2 kv get kv/apikey
    ++ VAULT_ADDR=http://127.0.0.2:8200
    ++ vault kv get kv/apikey
    ====== Metadata ======
    Key              Value
    ---              -----
    created_time     2019-07-02T04:55:18.681533Z
    deletion_time    n/a
    destroyed        false
    version          1
    
    ===== Data =====
    Key       Value
    ---       -----
    webapp    ABB39KKPTWOR832JGNLS02
    

    Copy and save the root token which you will use to log in later.

  4. When the script execution completes, the TEST_HOME directory includes the following files and folder: Outputs

  5. Examine the vault_2 server configuration file (config-vault2.hcl):

    $ cat config-vault2.hcl
    
    storage "raft" {
      path    = "/Users/student/raft-test/vault-raft/"
      node_id = "node2"
    }
    listener "tcp" {
      address = "127.0.0.2:8200"
      cluster_address = "127.0.0.2:8201"
      tls_disable = true
    }
    seal "transit" {
      address            = "http://127.0.0.1:8200"
      token              = "s.SsnAI6fJZKv1N1QMWP2rANDB"
      disable_renewal    = "false"
    
      // Key configuration
      key_name           = "unseal_key"
      mount_path         = "transit/"
    }
    disable_mlock = true
    cluster_addr = "http://127.0.0.2:8201"
    

    To use the integrated storage, the storage stanza must be set to raft. The path specifies the path where Vault data will be stored ($TEST_HOME/vault-raft). The seal stanza is configured to use Transit Auto-unseal which is provided by vault_1.

    # Set the VAULT_ADDR to point to 127.0.0.2 (vault_2) instance
    $ export VAULT_ADDR="http://127.0.0.2:8200"
    
    $ vault status
    Key                      Value
    ---                      -----
    Recovery Seal Type       shamir
    Initialized              true
    Sealed                   false
    Total Recovery Shares    5
    Threshold                3
    Version                  1.2.0-beta2
    Cluster Name             vault-cluster-ce3ec2c2
    Cluster ID               d0fcef16-cfac-e378-2587-f3edbd890851
    HA Enabled               true
    HA Cluster               https://127.0.0.2:8201
    HA Mode                  active
    
  6. Execute the raft configuration command to see the Raft (integrated storage backend) configuration information:

    $ vault operator raft configuration -format=json | jq
    {
      ...
      "data": {
        "config": {
          "index": 1,
          "servers": [
            {
              "address": "127.0.0.2:8201",
              "leader": true,
              "node_id": "node2",
              "protocol_version": "3",
              "voter": true
            }
          ]
        }
      },
      "warnings": null
    }
    

    Currently, node2 is the only member and a leader.

  7. Open a new terminal, and check the status of vault_3:

    # Set the VAULT_ADDR to point to 127.0.0.3 (vault_3) instance
    $ export VAULT_ADDR=http://127.0.0.3:8200
    
    $ vault status
    Key                      Value
    ---                      -----
    Recovery Seal Type       transit
    Initialized              false
    Sealed                   true
    Total Recovery Shares    0
    Threshold                0
    ...
    
  8. Let's execute the raft join command to add vault_3 to the HA cluster.

    $ vault operator raft join http://127.0.0.2:8200
    
    Key       Value
    ---       -----
    Joined    true
    

    The http://127.0.0.2:8200 is the vault_2 server address which has been already initialized and auto-unsealed. This makes vault_2 be the active node and its storage behaves as the leader in this cluster.

    The vault_3 server log (vault3.log) includes the following message:

    ...
    [TRACE] core: found new active node information, refreshing
    [DEBUG] sealwrap: unwrapping entry: key=core/leader/5a5b92a3-da56-5850-ebcf-1026a278b069
    [DEBUG] core: parsing information for new active node: active_cluster_addr=https://127.0.0.2:8201 active_redirect_addr=http://127.0.0.2:8200
    [DEBUG] core: refreshing forwarding connection
    ...
    
  9. Open another terminal and add vault_4 to the HA cluster:

    # Set the VAULT_ADDR to point to 127.0.0.4 (vault_4) instance
    $ export VAULT_ADDR=http://127.0.0.4:8200
    
    $ vault operator raft join http://127.0.0.2:8200
    
    Key       Value
    ---       -----
    Joined    true
    

    Now, you have a cluster with three Vault nodes which persist the data in the integrated storage.

    Scenario

  10. Execute the raft configuration command to list the cluster members:

    $ vault operator raft configuration -format=json | jq
    {
      ...
      "data": {
        "config": {
          "index": 94,
          "servers": [
            {
              "address": "127.0.0.2:8201",
              "leader": true,
              "node_id": "node2",
              "protocol_version": "3",
              "voter": true
            },
            {
              "address": "127.0.0.3:8201",
              "leader": false,
              "node_id": "node3",
              "protocol_version": "3",
              "voter": true
            },
            {
              "address": "127.0.0.4:8201",
              "leader": false,
              "node_id": "node4",
              "protocol_version": "3",
              "voter": true
            }
          ]
        }
      },
      "warnings": null
    }
    

    You can run this command against any of the cluster memebers (vault_2, vault_3 and vault_4) and get the same output.

  11. At this point, you should be able to log into vault_3 and vault_4 using the root token you copied from the raft-cluster-demo.sh script output earlier and manage the secrets at kv/apikey.

    Example:

    # Copy the root token from the raft-cluster-demo.sh output
    $ vault login s.ArffzRknwpSupR8XbIriGNS3
    
    $ vault kv patch kv/apikey expiration="365 days"
    Key              Value
    ---              -----
    created_time     2019-07-02T05:50:39.038931Z
    deletion_time    n/a
    destroyed        false
    version          2
    

    To verify the updates, read the secrets from another node in the cluster.

    # Copy the root token from the raft-cluster-demo.sh output
    $ vault login s.ArffzRknwpSupR8XbIriGNS3
    
    $ vault kv get kv/apikey
    ====== Metadata ======
    Key              Value
    ---              -----
    created_time     2019-07-02T05:50:39.038931Z
    deletion_time    n/a
    destroyed        false
    version          2
    
    ======= Data =======
    Key           Value
    ---           -----
    expiration    365 days
    webapp        ABB39KKPTWOR832JGNLS02
    

Option 2: Create a Vault HA cluster on AWS using Terraform

Set your working directory to where the /vault-guides/operations/raft-storage repo was cloned/downloaded.

The terraform-aws folder contains the Terraform files which set up a learning environment. Assets

NOTE: The example Terraform in this repository is created for the demo purpose.

  1. Set your AWS credentials as environment variables:

    $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"
    $ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"
    
  2. Use the provided terraform.tfvars.example as a base to create terraform.tfvars and specify your key_name. Be sure to set the correct key pair name created in the AWS region that you are using.

    Example terraform.tfvars:

    # SSH key name to access EC2 instances (should already exist) on the AWS region
    key_name = "vault-test"
    
    # If you want to use a different AWS region
    aws_region = "us-west-1"
    availability_zones = "us-west-1a"
    
  3. Perform a terraform init to pull down the necessary provider resources. Then terraform plan to verify your changes and the resources that will be created. If all looks good, then perform a terraform apply to provision the resources.

    $ cd terraform-aws
    
    $ terraform init
    
    $ terraform plan
    
    $ terraform apply -auto-approve
    ...
    Apply complete! Resources: 20 added, 0 changed, 0 destroyed.
    
    Outputs:
    
    endpoints =
    Auto-unseal Provider IP (public):  13.57.200.247
    Auto-unseal Provider IP (private): 10.0.101.11
    
    For example:
      ssh -i vault-test.pem ubuntu@13.57.200.247
    
    Server node IPs (public):  54.215.244.175, 54.193.26.177, 54.153.90.97
    Server node IPs (private): 10.0.101.53, 10.0.101.15, 10.0.101.73
    
    For example:
       ssh -i vault-test.pem ubuntu@54.215.244.175
    

    The Terraform output will display the IP addresses of the provisioned Vault nodes.

  4. There are three Vault server nodes provisioned by Terraform. The Terraform output displays three server node IP addresses as well as private IP addresses. Cloud Resources

    Example:

    Server node IPs (public):  54.215.244.175, 54.193.26.177, 54.153.90.97
    Server node IPs (private): 10.0.101.53, 10.0.101.15, 10.0.101.73
    
    For example:
       ssh -i vault-test.pem ubuntu@54.215.244.175
    

    In this example, the server node with public IP of 54.215.244.175, its private IP address is 10.0.101.53.

    SSH into one of the three server nodes: ssh -i <path_to_key> ubuntu@<public_ip>

    $ ssh -i vault-test.pem ubuntu@54.215.244.175
    ...
    Are you sure you want to continue connecting (yes/no)? yes
    

    When you are prompted, enter "yes" to continue.

  5. Examine the server configuration file, /etc/vault.d/vault.hcl.

    $ sudo cat /etc/vault.d/vault.hcl
    
    storage "raft" {
      path    = "/vault/storage1"
      node_id = "node1"
    }
    
    listener "tcp" {
      address     = "0.0.0.0:8200"
      cluster_address     = "0.0.0.0:8201"
      tls_disable = true
    }
    
    seal "transit" {
      address            = "http://10.0.101.11:8200"
      token              = "root"
      disable_renewal    = "false"
    
      // Key configuration
      key_name           = "unseal_key"
      mount_path         = "transit/"
    }
    
    api_addr = "http://54.215.244.175:8200"
    cluster_addr = "http://10.0.101.53:8201"
    disable_mlock = true
    ui=true
    

    To use the integrated storage, the storage stanza must be set to raft. The path specifies the path where Vault data will be stored (/vault/storage1). The seal stanza is configured to use Transit Auto-unseal which is provided by the Auto-unseal Provider instance.

    Optionally, you can review the Vault service systemd file:

    $ sudo cat /etc/systemd/system/vault.service
    
  6. Start the Vault server:

    # Start the Vault service
    $ sudo systemctl start vault
    
    # Check the service log
    $ sudo journalctl --no-page -u vault
    
  7. Run the vault operator init command to initialize the Vault server, and save the generated unseal keys and initial root token in a file named, key.txt:

    $ vault operator init > key.txt
    
    $ vault status
    Key                      Value
    ---                      -----
    Recovery Seal Type       shamir
    Initialized              true
    Sealed                   false
    Total Recovery Shares    5
    Threshold                3
    ...
    

    Vault is now initialized and auto-unsealed. Therefore, this node becomes the active node of the HA cluster that you are going to build.

  8. Log into Vault using the generated initial root token which is stored in the key.txt file.

    $ vault login $(grep 'Initial Root Token:' key.txt | awk '{print $NF}')
    
  9. Enable the Key/Value v2 secrets engine and create some test data.

    # Enable k/v v2 at 'kv' path
    $ vault secrets enable -path=kv kv-v2
    
    # Create some test data, kv/apikey
    $ vault kv put kv/apikey webapp=ABB39KKPTWOR832JGNLS02
    
    # Verify
    $ vault kv get kv/apikey
    
    ====== Metadata ======
    Key              Value
    ---              -----
    created_time     2019-07-03T05:29:51.003359433Z
    deletion_time    n/a
    destroyed        false
    version          1
    
    ===== Data =====
    Key       Value
    ---       -----
    webapp    ABB39KKPTWOR832JGNLS02
    
  10. Check the integrated storage for this server.

    $ vault operator raft configuration -format=json | jq
    {
      ...
      "data": {
        "config": {
          "index": 1,
          "servers": [
            {
              "address": "10.0.101.53:8201",
              "leader": true,
              "node_id": "node1",
              "protocol_version": "3",
              "voter": true
            }
          ]
        ...
    }
    
  11. Open a new terminal and SSH into another Vault server node.

    Example:

    $ ssh -i vault-test.pem ubuntu@54.193.26.177
    
  12. Edit the server configuration file (/etc/vault.d/vault.hcl) to modify the storage stanza.

    $ sudo vi /etc/vault.d/vault.hcl
    

    In the storage stanza, modify /vault/storage1 to /vault/storage2. Also, modify node1 to node2. (Press i to insert text using vi.)

    storage "raft" {
      path    = "/vault/storage2"
      node_id = "node2"
    }
    ...
    

    Press Esc and then enter :wq! in vi to save and exit.

  13. Now, start the Vault service.

    $ sudo systemctl start vault
    
  14. Execute the raft join command to add the second node to the HA cluster. Since you already initialized and unsealed a node, pass its API address to join the HA cluster: vault operator raft join <leader_node_API_addr>

    Example:

    If the private IP address of node1 is 10.0.101.12, the command would look like the following:

    # vault operator raft join <leader_node_API_addr>
    $ vault operator raft join http://10.0.101.12:8200
    
    Key       Value
    ---       -----
    Joined    true
    

    The node1 has been already initialized and auto-unsealed, it becomes the active node and its storage behaves as the leader in this cluster.

    NOTE: In this scenario, Transit auto-unseal is used; therefore, this server gets automatically unsealed once it successfully joins the cluster.

    $ vault status
    Key                      Value
    ---                      -----
    Recovery Seal Type       shamir
    Initialized              true
    Sealed                   false
    ...
    
  15. Open another terminal and SSH into the third node:

    Example:

    $ ssh -i vault-test.pem ubuntu@54.153.90.97
    
  16. Edit the server configuration file (/etc/vault.d/vault.hcl) to modify the storage stanza.

    $ sudo vi /etc/vault.d/vault.hcl
    

    In the storage stanza, modify /vault/storage1 to /vault/storage3. Also, modify node1 to node3. (Press i to insert text using vi.)

    storage "raft" {
      path    = "/vault/storage3"
      node_id = "node3"
    }
    ...
    

    Press Esc and then enter :wq! in vi to save and exit.

  17. Now, start the Vault service.

    $ sudo systemctl start vault
    
  18. Execute the raft join command to add the third node to the HA cluster. Again, the target node to join is the first server you initialized and unsealed.

    Example:

    If the private IP address of node1 is 10.0.101.12, the command would look like the following:

    # vault operator raft join <leader_node_API_addr>
    $ vault operator raft join http://10.0.101.12:8200
    
    Key       Value
    ---       -----
    Joined    true
    
  19. Execute the raft configuration command to see the Raft configuration:

    $ vault operator raft configuration -format=json | jq
    {
      ...
          "servers": [
            {
              "address": "10.0.101.53:8201",
              "leader": true,
              "node_id": "node1",
              "protocol_version": "3",
              "voter": true
            },
            {
              "address": "10.0.101.15:8201",
              "leader": false,
              "node_id": "node2",
              "protocol_version": "3",
              "voter": true
            },
            {
              "address": "10.0.101.73:8201",
              "leader": false,
              "node_id": "node3",
              "protocol_version": "3",
              "voter": true
            }
          ]
        ...
    }
    

    You should see all three nodes (node1, node2 and node3) in the HA cluster.

    Scenario

    Now, you have a cluster with three Vault nodes which persist the data in the integrated storage.

    At this point, you should be able to log into the node2 and node3 using the root token generated earlier on Vault server node 1 (stored in key.txt) and manage the secrets at kv/apikey.

    # Retrieve the root token from key.txt on node 1
    #     echo $(grep 'Initial Root Token:' key.txt | awk '{print $NF}')
    $ vault login <root_token>
    
    # Update the kv/apikey
    $ vault kv patch kv/apikey expiration="365 days"
    Key              Value
    ---              -----
    created_time     2019-07-02T05:50:39.038931Z
    deletion_time    n/a
    destroyed        false
    version          2
    

    You should be able to read the updated secret from any of the Vault nodes.

    $ vault kv get kv/apikey
    ====== Metadata ======
    Key              Value
    ---              -----
    created_time     2019-07-02T05:50:39.038931Z
    deletion_time    n/a
    destroyed        false
    version          2
    
    ======= Data =======
    Key           Value
    ---           -----
    expiration    365 days
    webapp        ABB39KKPTWOR832JGNLS02
    

Step 3: Raft snapshots for Vault data recovery

Taking a snapshot of the integrated storage captures the current Vault data which can be used later to restore the data if ever becomes necessary.

Taking a snapshot

To take a snapshot, use the raft snapshot save command:

vault operator raft snapshot save <snapshot_name>

Example:

$ vault operator raft snapshot save 2019-JULY-01.snapshot

Now, delete the secrets at kv/apikey:

$ vault kv metadata delete kv/apikey

# Verify that the data no longer exists:
$ vault kv get kv/apikey
No value found at kv/data/apikey

Restoring data from a snapshot

To recover the lost data from a snapshot, use the raft snapshot restore command:

vault operator raft snapshot restore <snapshot_name>

Example:

$ vault operator raft snapshot restore 2019-JULY-01.snapshot

Now, you the data should be recoverable at kv/apikey:

$ vault kv get kv/apikey

====== Metadata ======
Key              Value
---              -----
created_time     2019-07-02T05:50:39.038931Z
deletion_time    n/a
destroyed        false
version          2

======= Data =======
Key           Value
---           -----
expiration    365 days
webapp        ABB39KKPTWOR832JGNLS02

Step 4: Remove a cluster member

To join the HA cluster, you executed the raft join <leader_addr> command. To remove a node from the cluster, execute the raft remove-peer command:

vault operator raft remove-peer <node_id>

Example:

$ vault operator raft remove-peer node3
Peer removed successfully!

Now, node3 is removed from the HA cluster.

Option 1 Example:

$ vault operator raft configuration -format=json | jq
{
  ...
      "servers": [
        {
          "address": "127.0.0.2:8201",
          "leader": true,
          "node_id": "node2",
          "protocol_version": "3",
          "voter": true
        },
        {
          "address": "127.0.0.4:8201",
          "leader": false,
          "node_id": "node4",
          "protocol_version": "3",
          "voter": true
        }
      ]
    ...
}

Option 2 Example:

$ vault operator raft configuration -format=json | jq
{
  ...
      "servers": [
        {
          "address": "10.0.101.53:8201",
          "leader": true,
          "node_id": "node1",
          "protocol_version": "3",
          "voter": true
        },
        {
          "address": "10.0.101.15:8201",
          "leader": false,
          "node_id": "node2",
          "protocol_version": "3",
          "voter": true
        }
      ]
    ...
}

Step 5: Clean up

When you are done exploring, you can clean up your environment.

Option 1: Clean up your localhost

Execute the following to terminate all Vault processes:

# Kill all vault processes
$ pkill vault

# Delete the TEST_HOME directory - point to the correct location if different
$ rm -r $HOME/raft-test

Option 2: Clean up the cloud resources

Execute the terraform destroy command to terminal all AWS elements:

# Destroy the AWS resources provisioned by Terraform
$ terraform destroy -auto-approve

# Delete the state file
$ rm *tfstate*

Help and Reference