Workshops
Book a 90-minute product workshop led by HashiCorp engineers and product experts during HashiConf Digital Reserve your spot

Operations

Vault Deployment Guide

During the installation of Vault you should also review and apply the recommendations provided in the Vault Production Hardening guide.

To provide a highly-available single cluster architecture, we recommend Vault be deployed to more than one host, as shown in the Vault Reference Architecture, and connected to a Consul cluster for persistent data storage.

Reference Diagram

Per the Vault Reference Architecture, you will need to have a 6 node (3 voting and 3 non-voting) Consul cluster for the Vault back-end storage. Provided the assumption that you have already built the Consul backend, the below setup steps should be completed on all Vault hosts.

  1. Download Vault
  2. Install Vault
  3. Configure systemd
  4. Configure Consul
  5. Configure Vault
  6. Start Vault

»Step 1: Download Vault

Precompiled Vault binaries are available for download at https://releases.hashicorp.com/vault/ and Vault Enterprise binaries are available for download by following the instructions made available to HashiCorp Vault customers.

You should perform checksum verification of the zip packages using the SHA256SUMS and SHA256SUMS.sig files available for the specific release version. HashiCorp provides a guide on checksum verification for precompiled binaries.

First, export some environment variables to specify the Vault downloads base URL and preferred Vault version for covenience and concise commands. Then use curl to download the package and SHA256 summary files.

$ export VAULT_URL="https://releases.hashicorp.com/vault" \
         VAULT_VERSION="1.3.2"
$ curl \
      --silent \
      --remote-name \
     "${VAULT_URL}/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip"

$ curl \
      --silent \
      --remote-name \
      "${VAULT_URL}/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_SHA256SUMS"

$ curl \
      --silent \
      --remote-name \
      "${VAULT_URL}/${VAULT_VERSION}/${VAULT_VERSION}/vault_${VAULT_VERSION}_SHA256SUMS.sig"

You should now have the 3 files present locally:

$ ls -1
vault_1.3.2_SHA256SUMS
vault_1.3.2_SHA256SUMS.sig
vault_1.3.2_linux_amd64.zip

»Step 2: Install Vault

Unzip the downloaded package and move the vault binary to /usr/local/bin/. Check vault is available on the system path.

$ unzip vault_${VAULT_VERSION}_linux_amd64.zip
$ sudo chown root:root vault
$ sudo mv vault /usr/local/bin/
$ vault --version

The vault command features opt-in autocompletion for flags, subcommands, and arguments (where supported). Enable autocompletion.

$ vault -autocomplete-install
$ complete -C /usr/local/bin/vault vault

Give Vault the ability to use the mlock syscall without running the process as root. The mlock syscall prevents memory from being swapped to disk.

$ sudo setcap cap_ipc_lock=+ep /usr/local/bin/vault

Create a unique, non-privileged system user to run Vault.

$ sudo useradd --system --home /etc/vault.d --shell /bin/false vault

»Step 3: Configure systemd

Systemd uses documented sane defaults so only non-default values must be set in the configuration file.

Create a Vault service file at /etc/systemd/system/vault.service.

$ sudo touch /etc/systemd/system/vault.service

Add the below configuration to the Vault service file:

[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

The following parameters are set for the [Unit] stanza:

  • Description - Free-form string describing the vault service
  • Documentation - Link to the vault documentation
  • Requires - Configure a requirement dependency on the network service
  • After - Configure an ordering dependency on the network service being started before the vault service
  • ConditionFileNotEmpty - Check for a non-zero sized configuration file before vault is started
  • StartLimitIntervalSec, StartLimitBurst - Limit vault to three start attempts in 60 seconds (support for systemd version 230+)

The following parameters are set for the [Service] stanza:

The following parameters are set for the [Install] stanza:

  • WantedBy - Creates a weak dependency on vault being started by the multi-user run level

»Step 4: Configure Consul

When using Consul as the storage backend for Vault, we recommend using Consul's ACL system to restrict access to the path where Vault stores data. This access restriction is an added security measure in addition to the encryption Vault uses to protect data written to the storage backend.

The Consul website provides documentation on bootstrapping the ACL system, but this is explored in more detail here.

Vault requires a Consul token with specific policy to limit the requests Vault can make to Consul endpoints.

»Set up ACL for Vault access to Consul <= 1.3.x

On a host running a Consul agent:

  1. Export the CONSUL_HTTP_ADDR environment variable:

    $ export CONSUL_HTTP_ADDR="127.0.0.1:8500"
    
  2. Update Consul configuration in /etc/consul.d/consul.hcl on all Consul Servers to use ACLs and choose a default policy of allow (like a soft limit) or deny (like a hard limit).

    acl_datacenter =  "dc1"
    acl_default_policy =  "deny"
    acl_down_policy =  "extend-cache"
    
  3. Restart Consul.

    $ sudo systemctl restart consul
    
  4. Generate the Consul ACL Management token.

    $ curl --request PUT https://"${CONSUL_HTTP_ADDR}"/v1/acl/bootstrap
    

    This returns a Consul ACL management token that should be saved as it will be needed in all subsequent Consul API calls.

  5. Export the CONSUL_HTTP_TOKEN environment variable and specify the token from step #4 as its value.

    $ export CONSUL_HTTP_TOKEN="<TOKEN from #4>"
    
  6. Generate the Consul ACL agent Token by first creating a JSON payload for the API.

    $ cat > payload.json <<'EOF'
    {
      "Name": "Agent Token",
      "Type": "client",
      "Rules": "node \"\" { policy = \"write\" } service \"\" { policy = \"read\" }"
    }
    EOF
    
  7. Then generate the token.

    $ curl  --request PUT  \
            --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
            --data @payload.json \
            https://"${CONSUL_HTTP_ADDR}"/v1/acl/create
    

    This returns a Consul agent token.

    {"ID":"7f35dea8-4bfa-700e-f34a-7874723f8e2e"}
    
  8. Update Consul configuration in /etc/consul.d/consul.hcl on all Consul servers and client agents.

    acl_agent_token = "<TOKEN from #7>"
    
  9. Generate a Consul ACL token for Vault to use by first creating a JSON payload for the API.

    $ cat > payload.json <<'EOF'
    {
       "Name": "Vault Token",
       "Type": "client",
       "Rules":
         "key \"vault/\" { policy = \"write\" } node \"\" { policy = \"write\" } service \"vault\" { policy = \"write\" } agent \"\" { policy = \"write\" } session \"\" { policy = \"write\" }"
    }
    EOF
    
  10. Then generate the token.

    $ curl --request PUT \
           --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
           --data @payload.json \
           https://"${CONSUL_HTTP_ADDR}"/v1/acl/create
    

    This returns a Consul agent token.

    {"ID":"5cdde2da-1107-a0e8-c9b2-bcc08436d09b"}
    
  11. Update the Consul storage stanza in the Vault configuration file (/etc/vault.d/vault.hcl) on all Vault servers with the Vault Token.

    storage "consul" {
      address = "[consul address]"
      path = "vault/"
      token = "<TOKEN from #10>"
    }
    

    Save the token ID for use in the Vault configuration.

»Set up ACL for Vault access to Consul >= 1.4.x

  1. Create a Consul configuration file (/etc/consul.d/acl.hcl) on all servers and clients to use ACLs, and choose a default policy of "allow" (similar to a soft limit) or "deny" (similar to a hard limit).

    acl {
     enabled = true
     default_policy = "deny"
     enable_token_persistence = true
    }
    
  2. Restart Consul

    $ sudo systemctl restart consul
    
  3. Working from one agent generate the Consul ACL Management token.

    $ consul acl bootstrap
    

    This returns a Consul management token SecretID that should be saved to use in all subsequent Consul API calls.

  4. Set CONSUL_MGMT_TOKEN env variable

    $ export CONSUL_MGMT_TOKEN="<TOKEN from #3>"
    
  5. Create a node policy file (node-policy.hcl) with write access for nodes related actions and read access for service related actions.

    agent_prefix "" {
     policy = "write"
    }
    node_prefix "" {
     policy = "write"
    }
    service_prefix "" {
     policy = "read"
    }
    session_prefix "" {
     policy = "read"
    }
    
  6. Generate the Consul node ACL Policy.

    $ consul acl policy create \
         -token=${CONSUL_MGMT_TOKEN} \
         -name node-policy \
         -rules @node-policy.hcl
    
  7. Create the node token.

    $ consul acl token create \
         -token=${CONSUL_MGMT_TOKEN} \
         -description "node token" \
         -policy-name node-policy
    
  8. On all Consul Servers add the node token

    $ consul acl set-agent-token \
         -token=${CONSUL_MGMT_TOKEN} \
         agent "<Node Token SecretID>"
    
  9. Create a Vault policy file (vault-policy.hcl) with write access to the KV store used for Vault storage (default is vault/) and write access to the vault service. This is based on the generic sample for Vault. Refere to the Consul ACL documentation for more details.

    key_prefix "vault/" {
     policy = "write"
    }
    node_prefix "" {
     policy = "write"
    }
    service "vault" {
     policy = "write"
    }
    agent_prefix "" {
     policy = "write"
    }
    session_prefix "" {
     policy = "write"
    }
    
  10. Generate the Consul Vault ACL Policy

    $ consul acl policy create \
         -token=${CONSUL_MGMT_TOKEN} \
         -name vault-policy \
         -rules @vault-policy.hcl
    
  11. Create Vault service token

    $ consul acl token create \
         -token=${CONSUL_MGMT_TOKEN} \
         -description "Token for Vault Service" \
         -policy-name vault-policy
    

    Save the SecretID for use in the Vault configuration.

    {"ID":"292bbccc-656b-00b7-3b0a-df01e54c105b"}
    

»Step 5: Configure Vault

Vault uses documented sane defaults so only non-default values must be set in the configuration file.

Create a configuration file at /etc/vault.d/vault.hcl:

$ sudo mkdir --parents /etc/vault.d
$ sudo touch /etc/vault.d/vault.hcl
$ sudo chown --recursive vault:vault /etc/vault.d
$ sudo chmod 640 /etc/vault.d/vault.hcl

»Listener stanza

The listener stanza configures the addresses and ports on which Vault will respond to requests.

Add the below configuration to the Vault configuration file (/etc/vault.d/vault.hcl):

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/path/to/fullchain.pem"
  tls_key_file  = "/path/to/privkey.pem"
}

The following parameters are set for the tcp listener stanza:

  • address (string: "127.0.0.1:8200") - Changing from the loopback address to allow external access to the Vault UI
  • tls_cert_file (string: <required-if-enabled>, reloads-on-SIGHUP) - Must be set when using TLS
  • tls_key_file (string: <required-if-enabled>, reloads-on-SIGHUP) - Must be set when using TLS

More information about tcp listener configuration.

»Seal stanza

The seal stanza in the Vault configuration specifies the seal type to use for additional data protection such as using hardware security module (HSM) or Cloud KMS solutions to encrypt and decrypt the Vault master key to automatically unseal Vault. This stanza is optional, and if this is not configured, Vault will use the Shamir algorithm to cryptographically split the master key.

Review the seal configuration documentation for more detail.

An example PKCS #11 compatible HSM example is:

seal "pkcs11" {
  lib            = "/usr/vault/lib/libCryptoki2_64.so"
  slot           = "0"
  pin            = "AAAA-BBBB-CCCC-DDDD"
  key_label      = "vault-hsm-key"
  hmac_key_label = "vault-hsm-hmac-key"
}

»Storage stanza

The storage stanza configures the storage backend, which represents the location for the durable storage of Vault's data.

Add the below configuration to the Vault configuration file:

storage "consul" {
  address = "[consul address]"
  path = "vault/"
  token = "<TOKEN from Step 4 - #10>"
}

Where the token is the token you generated in Step 4: Configure Consul at #10.

The following parameters are set for the consul storage stanza:

  • token (string: "") - Specify the Consul ACL token with permission to read and write from /vault in Consul's key-value store

More information about consul storage configuration.

»Telemetry stanza

The telemetry stanza specifies various configurations for Vault to publish metrics to upstream systems.

If you decide to configure Vault to publish telemetry data, you should review the telemetry configuration section of our documentation.

»High Availability Parameters

The api_addr parameter configures the API address used in high availability scenarios, when client redirection is used instead of request forwarding. Client redirection is the fallback method used when request forwarding is turned off or there is an error performing the forwarding. As such, a redirect address is always required for all HA setups.

This parameter value defaults to the address specified in the listener stanza, but Vault will log a [WARN] message if it is not explicitly configured.

Add the below configuration to the Vault configuration file:

api_addr = "{{ full URL to Vault API endpoint }}"

More information about high availability configuration.

»Vault UI

Vault features a web-based user interface, allowing you to easily create, read, update, and delete secrets, authenticate, unseal, and more using a graphical user interface, rather than the CLI or API.

Vault should not be deployed in a public internet facing environment, so enabling the Vault UI is typically of benefit to provide a more familiar experience to administrators who are not as comfortable working on the command line, or who do not have alternative access.

Optionally, add the below configuration to the Vault configuration file to enable the Vault UI:

ui = true

More information about configuring the Vault UI.

»Step 6: Start Vault

Enable and start Vault using the systemctl command responsible for controlling systemd managed services. Check the status of the vault service using systemctl.

$ sudo systemctl enable vault
$ sudo systemctl start vault
$ sudo systemctl status vault

»Help and Reference