Automate Upgrades with Vault Enterprise

Enterprise Only: The functionality described in this tutorial is available only in Vault Enterprise. To explore Vault Enterprise features, you can sign up for a free 30-day trial.

»Challenge

Vault version upgrade is always a delicate moment for any production environment and it is important to have best practices in place that simplify the process where possible.

»Solution

Vault Enterprise provides automated version upgrades with the autopilot feature when using Integrated Storage. The feature allows you to start new Vault nodes alongside the old ones and automatically switch to the new nodes, as soon as they are able to reach quorum.

This automates the leader election process and ensures that the new leader is elected among the new nodes so that removing the old nodes from the datacenter does not trigger a leader election.

»Prerequisites

To test the automated upgrades feature explained in this tutorial you will need:

A Vault Enterprise cluster with three nodes running Vault Enterprise 1.11.0 or later.
Three extra nodes with Vault Enterprise 1.11.0 or later to use as the new servers once the upgrade is concluded.

You will also need a text editor, the curl executable to test the API endpoints, and optionally the jq command to format the output for curl.

»Scenario introduction

To demonstrate the new autopilot behavior, start a cluster with 3 nodes that has no automatic upgrade (see the Step 1 diagram below). Then, start additional 3 nodes with automatic upgrade enabled and add them to the cluster (see the Step 2 diagram below).

You will run a script to start a cluster.

Scenario

vault_1 (http://127.0.0.1:8100) is initialized and unsealed. The root token creates a transit key that enables the other Vaults auto-unseal. This Vault server is not a part of the cluster.
vault_2 (http://127.0.0.1:8200) is initialized and unsealed. This Vault starts as the cluster leader.
vault_3 (http://127.0.0.1:8300) is started and automatically joins the cluster via retry_join.
vault_4 (http://127.0.0.1:8400) is started and automatically joins the cluster via retry_join.

If this is your first time setting up a Vault cluster with integrated storage, go through the Vault HA Cluster with Integrated Storage tutorial.

»Setup an initial cluster

Retrieve the configuration by cloning the hashicorp/learn-vault-raft repository from GitHub.
```
$ git clone https://github.com/hashicorp/learn-vault-raft.git
```
```
$ git clone https://github.com/hashicorp/learn-vault-raft.git
```
This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

Change the working directory to learn-vault-raft/raft-auto-upgrade/local.

$ cd learn-vault-raft/raft-auto-upgrade/local

$ cd learn-vault-raft/raft-auto-upgrade/local

Set the setup_1.sh file to executable.

$ chmod +x setup_1.sh

$ chmod +x setup_1.sh

Execute the setup_1.sh script to spin up a Vault cluster.

$ ./setup_1.sh

[vault_1] Creating configuration
  - creating /git/learn-vault-raft/raft-autopilot/local/config-vault_1.hcl
[vault_2] Creating configuration
  - creating /git/learn-vault-raft/raft-autopilot/local/config-vault_2.hcl
  - creating /git/learn-vault-raft/raft-autopilot/local/raft-vault_2

...snip...

[vault_3] starting Vault server @ http://127.0.0.1:8300
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

[vault_4] starting Vault server @ http://127.0.0.1:8400
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

$ ./setup_1.sh

[vault_1] Creating configuration
  - creating /git/learn-vault-raft/raft-autopilot/local/config-vault_1.hcl
[vault_2] Creating configuration
  - creating /git/learn-vault-raft/raft-autopilot/local/config-vault_2.hcl
  - creating /git/learn-vault-raft/raft-autopilot/local/raft-vault_2

...snip...

[vault_3] starting Vault server @ http://127.0.0.1:8300
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

[vault_4] starting Vault server @ http://127.0.0.1:8400
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

You can find the server configuration files and the log files in the working directory.

Use your preferred text editor and open the config-vault_2.hcl file to examine the generated server configuration for vault_2.

storage "raft" {
 path    = "/learn-vault-raft/raft-auto-upgrade/local/raft-vault_2/"
 node_id = "vault_2"
}

listener "tcp" {
 address = "127.0.0.1:8200"
 cluster_address = "127.0.0.1:8201"
 tls_disable = true
}

seal "transit" {
 address            = "http://127.0.0.1:8100"
 # token is read from VAULT_TOKEN env
 # token              = ""
 disable_renewal    = "false"
 key_name           = "unseal_key"
 mount_path         = "transit/"
}

disable_mlock = true
cluster_addr = "http://127.0.0.1:8201"

config-vault_2.hcl

storage "raft" {
 path    = "/learn-vault-raft/raft-auto-upgrade/local/raft-vault_2/"
 node_id = "vault_2"
}

listener "tcp" {
 address = "127.0.0.1:8200"
 cluster_address = "127.0.0.1:8201"
 tls_disable = true
}

seal "transit" {
 address            = "http://127.0.0.1:8100"
 # token is read from VAULT_TOKEN env
 # token              = ""
 disable_renewal    = "false"
 key_name           = "unseal_key"
 mount_path         = "transit/"
}

disable_mlock = true
cluster_addr = "http://127.0.0.1:8201"

Review the generated server configuration for vault_3.

storage "raft" {
 path    = "/learn-vault-raft/raft-auto-upgrade/local/raft-vault_3/"
 node_id = "vault_3"

 retry_join {
   leader_api_addr = "http://127.0.0.1:8200"
 }
}

...snip...

config-vault_3.hcl

storage "raft" {
 path    = "/learn-vault-raft/raft-auto-upgrade/local/raft-vault_3/"
 node_id = "vault_3"

 retry_join {
   leader_api_addr = "http://127.0.0.1:8200"
 }
}

...snip...

The retry_join block is configured that vault_3 and vault_4 nodes automatically joined the cluster.

Export an environment variable for the vault CLI to address the vault_2 server.

$ export VAULT_ADDR=http://127.0.0.1:8200

$ export VAULT_ADDR=http://127.0.0.1:8200

Verify the cluster members.

$ vault operator raft list-peers

Node       Address           State       Voter
----       -------           -----       -----
vault_2    127.0.0.1:8201    leader      true
vault_3    127.0.0.1:8301    follower    false
vault_4    127.0.0.1:8401    follower    false

$ vault operator raft list-peers

Node       Address           State       Voter
----       -------           -----       -----
vault_2    127.0.0.1:8201    leader      true
vault_3    127.0.0.1:8301    follower    false
vault_4    127.0.0.1:8401    follower    false

View the autopilot's upgrade state information.

$ curl -s --header "X-Vault-Token: $(cat root_token-vault_2)" \
    $VAULT_ADDR/v1/sys/storage/raft/autopilot/state | jq -r ".data.upgrade_info"

$ curl -s --header "X-Vault-Token: $(cat root_token-vault_2)" \
    $VAULT_ADDR/v1/sys/storage/raft/autopilot/state | jq -r ".data.upgrade_info"

Output:

{
  "status": "idle",
  "target_version": "1.11.0",
  "target_version_voters": [
    "vault_2",
    "vault_3",
    "vault_4"
  ]
}

{
  "status": "idle",
  "target_version": "1.11.0",
  "target_version_voters": [
    "vault_2",
    "vault_3",
    "vault_4"
  ]
}

Notice the Upgrade Info fields shows the Status to be idle.

If watch command (or similar) is installed, you can monitor the upgrade status as you proceed to adding more nodes.

$ watch -n 0.5 'curl -H "X-Vault-Token: $(cat root_token-vault_2)" $VAULT_ADDR/v1/sys/storage/raft/autopilot/state | jq -r ".data.upgrade_info"'

$ watch -n 0.5 'curl -H "X-Vault-Token: $(cat root_token-vault_2)" $VAULT_ADDR/v1/sys/storage/raft/autopilot/state | jq -r ".data.upgrade_info"'

This checks the autopilot state every half a second.

»Add new nodes

When autopilot detects that a node with a newer version of Vault has joined the cluster, it will wait to promote the new node as voter until enough newer-versioned nodes have been added to the cluster to reach quorum. When the count of new nodes equals or exceeds that of the old nodes, autopilot will begin promoting the new nodes to voters and demoting the old nodes to non-voters.

Use your preferred text editor and open the config-vault_5.hcl file to examine the generated server configuration for vault_5.

storage "raft" {
  path    = "/learn-vault-raft/raft-auto-upgrade/local/raft-vault_5/"
  node_id = "vault_5"
  autopilot_upgrade_version = "1.12.0.1"

  retry_join {
    leader_api_addr = "http://127.0.0.1:8200"
  }
}

...snip...

config-vault_5.hcl

storage "raft" {
  path    = "/learn-vault-raft/raft-auto-upgrade/local/raft-vault_5/"
  node_id = "vault_5"
  autopilot_upgrade_version = "1.12.0.1"

  retry_join {
    leader_api_addr = "http://127.0.0.1:8200"
  }
}

...snip...

To enable automatic version upgrade, add the autopilot_upgrade_version parameter in the storage stanza where its value is a SemVer compatible version string of your choosing.

Vault Configuration: The vault_5, vault_6 and vault_7 nodes have autopilot_upgrade_version parameter configured. This implies that those nodes have a specific target Vault version.

Set the setup_2.sh file to executable.

$ chmod +x setup_2.sh

$ chmod +x setup_2.sh

Execute the setup_2.sh script to add three additional nodes to the cluster.

$ ./setup_2.sh

[vault_5] starting Vault server @ http://127.0.0.1:8500
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

[vault_6] starting Vault server @ http://127.0.0.1:8600
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

[vault_7] starting Vault server @ http://127.0.0.1:8700
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

$ ./setup_2.sh

[vault_5] starting Vault server @ http://127.0.0.1:8500
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

[vault_6] starting Vault server @ http://127.0.0.1:8600
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

[vault_7] starting Vault server @ http://127.0.0.1:8700
Using [vault_1] root token (hvs.tqKc9An04pQY5H1uysw02Xn6) to retrieve transit key for auto-unseal

Monitor the autopilot's upgrade status as it progress.

$ curl -s --header "X-Vault-Token: $(cat root_token-vault_2)" \
    $VAULT_ADDR/v1/sys/storage/raft/autopilot/state | jq -r ".data.upgrade_info"

$ curl -s --header "X-Vault-Token: $(cat root_token-vault_2)" \
    $VAULT_ADDR/v1/sys/storage/raft/autopilot/state | jq -r ".data.upgrade_info"

Or,

$ watch -n 0.5 'curl -H "X-Vault-Token: $(cat root_token-vault_2)" $VAULT_ADDR/v1/sys/storage/raft/autopilot/state | jq -r ".data.upgrade_info"'

$ watch -n 0.5 'curl -H "X-Vault-Token: $(cat root_token-vault_2)" $VAULT_ADDR/v1/sys/storage/raft/autopilot/state | jq -r ".data.upgrade_info"'

The Status changes from idle to await-new-voters.

{
  "other_version_voters": [
    "vault_2",
    "vault_3",
    "vault_4"
  ],
  "status": "await-new-voters",
  "target_version": "1.12.0.1",
  "target_version_non_voters": [
    "vault_5",
    "vault_6"
  ]
}

{
  "other_version_voters": [
    "vault_2",
    "vault_3",
    "vault_4"
  ],
  "status": "await-new-voters",
  "target_version": "1.12.0.1",
  "target_version_non_voters": [
    "vault_5",
    "vault_6"
  ]
}

The status will change to promoting as autopilot promotes the 3 new nodes to be voters. Then the status will change to demoting, as autopilot demotes 2 out of the 3 old nodes to be non-voters. Then, the leader will change from vault_2 to vault_5.

{
  "other_version_non_voters": [
    "vault_3",
    "vault_4"
  ],
  "other_version_voters": [
    "vault_2"
  ],
  "status": "leader-transfer",
  "target_version": "1.12.0.1",
  "target_version_voters": [
    "vault_5",
    "vault_6",
    "vault_7"
  ]
}

{
  "other_version_non_voters": [
    "vault_3",
    "vault_4"
  ],
  "other_version_voters": [
    "vault_2"
  ],
  "status": "leader-transfer",
  "target_version": "1.12.0.1",
  "target_version_voters": [
    "vault_5",
    "vault_6",
    "vault_7"
  ]
}

Finally, the status chanes to await-server-removal.

{
  "other_version_non_voters": [
    "vault_2",
    "vault_3",
    "vault_4"
  ],
  "status": "await-server-removal",
  "target_version": "1.12.0.1",
  "target_version_voters": [
    "vault_5",
    "vault_6",
    "vault_7"
  ]
}

{
  "other_version_non_voters": [
    "vault_2",
    "vault_3",
    "vault_4"
  ],
  "status": "await-server-removal",
  "target_version": "1.12.0.1",
  "target_version_voters": [
    "vault_5",
    "vault_6",
    "vault_7"
  ]
}

Autopilot Statue: The progression of autopilot statuses during an upgrade looks like: idle → await-new-voters → demoting → promoting → leader-transfer → await-server-removal → idle.

»Remove non-voter nodes

Once the autopilot upgrade status changes to await-server-removal, you can remove the old non-voting nodes from the cluster.

List the current peers before removing any nodes.

$ vault operator raft list-peers

Node       Address           State       Voter
----       -------           -----       -----
vault_2    127.0.0.1:8201    follower    false
vault_3    127.0.0.1:8301    follower    false
vault_4    127.0.0.1:8401    follower    false
vault_5    127.0.0.1:8501    leader      true
vault_6    127.0.0.1:8601    follower    true
vault_7    127.0.0.1:8701    follower    true

$ vault operator raft list-peers

Node       Address           State       Voter
----       -------           -----       -----
vault_2    127.0.0.1:8201    follower    false
vault_3    127.0.0.1:8301    follower    false
vault_4    127.0.0.1:8401    follower    false
vault_5    127.0.0.1:8501    leader      true
vault_6    127.0.0.1:8601    follower    true
vault_7    127.0.0.1:8701    follower    true

Export an environment variable for the vault CLI to address the server.

$ export VAULT_ADDR=http://127.0.0.1:8500

$ export VAULT_ADDR=http://127.0.0.1:8500

Remove vault_2 from the cluster.

$ vault operator raft remove-peer vault_2
Peer removed successfully!

$ vault operator raft remove-peer vault_2
Peer removed successfully!

Remove vault_3 from the cluster.

$ vault operator raft remove-peer vault_3
Peer removed successfully!

$ vault operator raft remove-peer vault_3
Peer removed successfully!

Remove vault_4 from the cluster.

$ vault operator raft remove-peer vault_4
Peer removed successfully!

$ vault operator raft remove-peer vault_4
Peer removed successfully!

Verify that the non-voter nodes have been removed from the cluster.

$ vault operator raft list-peers

Node       Address           State       Voter
----       -------           -----       -----
vault_5    127.0.0.1:8501    leader      true
vault_6    127.0.0.1:8601    follower    true
vault_7    127.0.0.1:8701    follower    true

$ vault operator raft list-peers

Node       Address           State       Voter
----       -------           -----       -----
vault_5    127.0.0.1:8501    leader      true
vault_6    127.0.0.1:8601    follower    true
vault_7    127.0.0.1:8701    follower    true

»Autopilot configuration

By default, automated upgrade migrations is supported for Vault Enterprise.

$ vault operator raft autopilot get-config

$ vault operator raft autopilot get-config

Output:

Key                                   Value
---                                   -----
Cleanup Dead Servers                  false
Last Contact Threshold                10s
Dead Server Last Contact Threshold    24h0m0s
Server Stabilization Time             10s
Min Quorum                            0
Max Trailing Logs                     1000
Disable Upgrade Migration             false

Key                                   Value
---                                   -----
Cleanup Dead Servers                  false
Last Contact Threshold                10s
Dead Server Last Contact Threshold    24h0m0s
Server Stabilization Time             10s
Min Quorum                            0
Max Trailing Logs                     1000
Disable Upgrade Migration             false

To disable automated upgrade migrations, set the -disable-upgrade-migration parameter to true.

$ vault operator raft autopilot set-config -disable-upgrade-migration=true

$ vault operator raft autopilot set-config -disable-upgrade-migration=true

»Clean up

The cluster.sh script provides a clean operation that removes all services, configuration, and modifications to your local system.

Clean up your local workstation.

$ ./cluster.sh clean

Found 1 Vault service(s) matching that name
[vault_1] stopping

...snip...

Removing log file /git/learn-vault-raft/raft-autopilot/local/vault_5.log
Removing log file /git/learn-vault-raft/raft-autopilot/local/vault_6.log
Clean complete

$ ./cluster.sh clean

Found 1 Vault service(s) matching that name
[vault_1] stopping

...snip...

Removing log file /git/learn-vault-raft/raft-autopilot/local/vault_5.log
Removing log file /git/learn-vault-raft/raft-autopilot/local/vault_6.log
Clean complete

»Next steps

In this tutorial you upgraded your Vault datacenter by using autopilot's automated upgrades functionality. Automated upgrades lets you automatically upgrade a cluster of Vault nodes to a new version as updated server nodes join the cluster. Once the number of nodes on the new version is equal to or greater than the number of nodes on the old version, Autopilot will promote the newer versioned nodes to voters, demote the older versioned nodes to non-voters, and initiate a leadership transfer from the older version leader to one of the newer versioned nodes. After the leadership transfer completes, the older versioned non-voting nodes can be removed from the cluster.

Infrastructure

Security

Applications

Networking

cloud