Encryption as a Service: Transit Secrets Engine

Vault's transit secrets engine handles cryptographic functions on data-in-transit. Vault doesn't store the data sent to the secrets engine, so it can also be viewed as encryption as a service.

Although the transit secrets engine provides additional features (sign and verify data, generate hashes and HMACs of data, and act as a source of random bytes), its primary use case is to encrypt data. This relieves the burden of proper encryption/decryption from application developers and pushes the burden onto the admins of Vault.

»Challenge

Think of the following scenario:

Example Inc. recently made headlines for a massive data breach which exposed millions of their users' payment card accounts online. When they tracked down the problem they found that a new HVAC system with management software had been put into their data centers and had created vulnerabilities in their networks and exposed ports and IPs to the databases publicly.

»Solutions

The transit secrets engine enables security teams to fortify data during transit and at rest. So even if an intrusion occurs, your data is encrypted with AES-GCM with a 256-bit AES key or other supported key types. Even if an attacker were able to access the raw data, they would only have encrypted bits. This means attackers would need to compromise multiple systems before exfiltrating data.

This tutorial demonstrates the basics of the transit secrets engine.

Encryption as a Service

»Personas

The end-to-end scenario described in this tutorial involves two personas:

admin with privileged permissions to manage the encryption keys
apps with un-privileged permissions encrypt/decrypt secrets via APIs

»Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

NOTE: An interactive tutorial is also available if you do not have a Vault environment to perform the steps described in this tutorial. Click the Show Terminal button to start.

»Policy requirements

NOTE: For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions.

# Enable transit secrets engine
path "sys/mounts/transit" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To read enabled secrets engines
path "sys/mounts" {
  capabilities = [ "read" ]
}

# Manage the transit secrets engine
path "transit/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Enable transit secrets enginepath "sys/mounts/transit" {  capabilities = [ "create", "read", "update", "delete", "list" ]}
# To read enabled secrets enginespath "sys/mounts" {  capabilities = [ "read" ]}
# Manage the transit secrets enginepath "transit/*" {  capabilities = [ "create", "read", "update", "delete", "list" ]}

If you are not familiar with policies, complete the policies tutorial.

»Configure Transit secrets engine

(Persona: admin)

The transit secrets engine must be configured before it can perform its operations. This step is usually done by an admin or configuration management tool.

Enable the transit secrets engine by executing the following command.

$ vault secrets enable transit

$ vault secrets enable transit

By default, the secrets engine will mount at the name of the engine. If you wish to enable it at a different path, use the -path argument.

$ vault secrets enable -path=encryption transit

$ vault secrets enable -path=encryption transit

Now, create an encryption key ring named orders by executing the following command.

$ vault write -f transit/keys/orders

$ vault write -f transit/keys/orders

»Create a token for Vault clients

(Persona: admin)

Vault clients must authenticate with Vault and acquire a valid token with appropriate policies allowing to request data encryption and decryption using the specific key.

Enable new engine

When the transit secrets engine is enabled at transit, the policy must include the following.

path "transit/encrypt/<key_name>" {
   capabilities = [ "update" ]
}

path "transit/decrypt/<key_name>" {
   capabilities = [ "update" ]
}

path "transit/encrypt/<key_name>" {   capabilities = [ "update" ]}
path "transit/decrypt/<key_name>" {   capabilities = [ "update" ]}

This tutorial uses the vault token create command to generate a client token and skips the authentication step.

In practice, you can leverage the Vault Agent Auto-Auth method to authenticate with Vault and manage the lifecycle of the client token. To learn more about Vault Agent, visit the App Integration in Learn.

Create a policy named app-orders.

$ vault policy write app-orders -<<EOF
path "transit/encrypt/orders" {
   capabilities = [ "update" ]
}
path "transit/decrypt/orders" {
   capabilities = [ "update" ]
}
EOF

$ vault policy write app-orders -<<EOFpath "transit/encrypt/orders" {   capabilities = [ "update" ]}path "transit/decrypt/orders" {   capabilities = [ "update" ]}EOF

The policy is created or updated; if it already exists.

Create a token with app-orders policy attached.

$ vault token create -policy=app-orders

$ vault token create -policy=app-orders

Example output:

Key                  Value
---                  -----
token                s.l0FFVw1zEr7pZxyOLnZjU3V1
token_accessor       FlvgZs1VIPox0nDQdPbkYsS0
token_duration       768h
token_renewable      true
token_policies       ["app-orders" "default"]
identity_policies    []
policies             ["app-orders" "default"]

Key                  Value---                  -----token                s.l0FFVw1zEr7pZxyOLnZjU3V1token_accessor       FlvgZs1VIPox0nDQdPbkYsS0token_duration       768htoken_renewable      truetoken_policies       ["app-orders" "default"]identity_policies    []policies             ["app-orders" "default"]

Copy the generated token value (e.g. s.l0FFVw1zEr7pZxyOLnZjU3V1). You are going to use it to request data encryption and decryption.

»Encrypt secrets

(Persona: apps)

Once the transit secrets engine has been configured, any Vault client holding a valid token with the proper permissions can send data to encrypt.

Here, you are going to encrypt a plaintext credit card number (4111 1111 1111 1111).

NOTE: Vault can encrypt a binary file such as an image. When you send data to Vault for encryption, it must be in the form of base64-encoded plaintext for a safe transport.

To encrypt your secret, use the transit/encrypt endpoint.

$ vault write transit/encrypt/<key_ring_name>

$ vault write transit/encrypt/<key_ring_name>

Execute the following command to encrypt plaintext (using the shell to do the base64 encoding). Be sure to replace <client_token> with the token value you copied in the previous step.

$ VAULT_TOKEN=<client_token> vault write transit/encrypt/orders \
    plaintext=$(base64 <<< "4111 1111 1111 1111")

$ VAULT_TOKEN=<client_token> vault write transit/encrypt/orders \    plaintext=$(base64 <<< "4111 1111 1111 1111")

Example output:

Key           Value
---           -----
ciphertext    vault:v1:cZNHVx+sxdMErXRSuDa1q/pz49fXTn1PScKfhf+PIZPvy8xKfkytpwKcbC0fF2U=

Key           Value---           -----ciphertext    vault:v1:cZNHVx+sxdMErXRSuDa1q/pz49fXTn1PScKfhf+PIZPvy8xKfkytpwKcbC0fF2U=

Notice that the ciphertext starts with vault:v1:. This prefix indicates that this value is wrapped by vault and the version of the orders encryption key used was v1. Therefore, when you decrypt this ciphertext, Vault knows to use v1 of the key. Later, you are going to rotate the encryption key and learn how to re-wrap the ciphertext with the latest version of the encryption key.

Vault does NOT store any data encrypted via the transit/encrypt endpoint. The output you received is the ciphertext. You can store this ciphertext at the desired location (e.g. MySQL database) or pass it to another application.

»Decrypt ciphertext

(Persona: apps)

Any client holding a valid token with proper permissions can decrypt ciphertext generated by Vault. To decrypt the ciphertext, invoke the transit/decrypt endpoint.

Execute the following command to decrypt the ciphertext emitted in the encrypt secrets step. Be sure to replace <client_token> with the token value you copied in the previous step.

$ VAULT_TOKEN=<client_token> vault write transit/decrypt/orders \
    ciphertext="vault:v1:cZNHVx+sxdMErXRSuDa1q/pz49fXTn1PScKfhf+PIZPvy8xKfkytpwKcbC0fF2U="

Key          Value
---          -----
plaintext    Y3JlZGl0LWNhcmQtbnVtYmVyCg==

$ VAULT_TOKEN=<client_token> vault write transit/decrypt/orders \    ciphertext="vault:v1:cZNHVx+sxdMErXRSuDa1q/pz49fXTn1PScKfhf+PIZPvy8xKfkytpwKcbC0fF2U="
Key          Value---          -----plaintext    Y3JlZGl0LWNhcmQtbnVtYmVyCg==

The resulting data is base64-encoded and must be decoded to reveal the plaintext.

$ base64 --decode <<< "Y3JlZGl0LWNhcmQtbnVtYmVyCg=="
4111 1111 1111 1111

$ base64 --decode <<< "Y3JlZGl0LWNhcmQtbnVtYmVyCg=="4111 1111 1111 1111

»Rotate the encryption key

(Persona: admin)

One of the benefits of using the Vault transit secrets engine is its ability to easily rotate encryption keys. Keys can be rotated manually by a human or by an automated process which invokes the key rotation API endpoint through cron, a CI pipeline, a periodic Nomad batch job, Kubernetes Job, etc.

Vault maintains the versioned keyring and the admin can decide the minimum version allowed for decryption operations. When data is encrypted using Vault, the resulting ciphertext is prepended by the version of the key used to encrypt it.

To rotate the encryption key, invoke the transit/keys/<key_ring_name>/rotate endpoint.

$ vault write -f transit/keys/orders/rotate

$ vault write -f transit/keys/orders/rotate

Let's encrypt another data.

$ vault write transit/encrypt/orders plaintext=$(base64 <<< "4111 1111 1111 1111")

Key           Value
---           -----
ciphertext    vault:v2:45f9zW6cglbrzCjI0yCyC6DBYtSBSxnMgUn9B5aHcGEit71xefPEmmjMbrk3

$ vault write transit/encrypt/orders plaintext=$(base64 <<< "4111 1111 1111 1111")
Key           Value---           -----ciphertext    vault:v2:45f9zW6cglbrzCjI0yCyC6DBYtSBSxnMgUn9B5aHcGEit71xefPEmmjMbrk3

Compare the ciphertexts from the encrypt secrets step as generated on your machine.

ciphertext    vault:v1:cZNHVx+sxdMErXRSuDa1q/pz49fXTn1PScKfhf+PIZPvy8xKfkytpwKcbC0fF2U=

ciphertext    vault:v1:cZNHVx+sxdMErXRSuDa1q/pz49fXTn1PScKfhf+PIZPvy8xKfkytpwKcbC0fF2U=

Notice that the first ciphertext starts with "vault:v1:". After rotating the encryption key, the ciphertext starts with "vault:v2:". This indicates that the data is encrypted using the latest version of the key after the rotation.

Execute the following command to rewrap your ciphertext from the encrypt secrets step with the latest version of the encryption key.

$ vault write transit/rewrap/orders \
    ciphertext="vault:v1:cZNHVx+sxdMErXRSuDa1q/pz49fXTn1PScKfhf+PIZPvy8xKfkytpwKcbC0fF2U="

Key           Value
---           -----
ciphertext    vault:v2:kChHZ9w4ILRfw+DzO53IZ8m5PyB2yp2/tKbub34uB+iDqtDRB+NLCPrpzTtJHJ4=

$ vault write transit/rewrap/orders \    ciphertext="vault:v1:cZNHVx+sxdMErXRSuDa1q/pz49fXTn1PScKfhf+PIZPvy8xKfkytpwKcbC0fF2U="
Key           Value---           -----ciphertext    vault:v2:kChHZ9w4ILRfw+DzO53IZ8m5PyB2yp2/tKbub34uB+iDqtDRB+NLCPrpzTtJHJ4=

Notice that the resulting ciphertext now starts with "vault:v2:".

This operation does not reveal the plaintext data. But Vault will decrypt the value using the appropriate key in the keyring and then encrypt the resulting plaintext with the newest key in the keyring.

»Update key configuration

(Persona: admin)

Vault admin can update the encryption key configuration to specify the minimum version of ciphertext allowed to be decrypted, the minimum version of the key that can be used to encrypt the plaintext, the key is allowed to be deleted, etc.

This helps to further tighten the data encryption rule.

Execute the key rotation command a few times to generate multiple versions of the key.

$ vault write -f transit/keys/orders/rotate

$ vault write -f transit/keys/orders/rotate

Now, read the orders key information.

$ vault read transit/keys/orders

Key                       Value
---                       -----
...
keys                      map[6:1531439714 1:1531439594 2:1531439667 3:1531439714 4:1531439714 5:1531439714]
latest_version            6
min_decryption_version    1
min_encryption_version    0
...

$ vault read transit/keys/orders
Key                       Value---                       -----...keys                      map[6:1531439714 1:1531439594 2:1531439667 3:1531439714 4:1531439714 5:1531439714]latest_version            6min_decryption_version    1min_encryption_version    0...

In this example, the current version of the key is 6. However, there is no restriction about the minimum encryption key version, and any of the key versions can decrypt the data (min_decryption_version).

Run the following command to enforce the use of the encryption key at version 5 or later to decrypt the data.

$ vault write transit/keys/orders/config min_decryption_version=5

$ vault write transit/keys/orders/config min_decryption_version=5

Now, verify the orders key configuration.

$ vault read transit/keys/orders

Key                       Value
---                       -----
allow_plaintext_backup    false
deletion_allowed          false
derived                   false
exportable                false
keys                      map[5:1531811719 6:1531811721]
latest_version            6
min_decryption_version    5
min_encryption_version    0
# ...snip...

$ vault read transit/keys/orders
Key                       Value---                       -----allow_plaintext_backup    falsedeletion_allowed          falsederived                   falseexportable                falsekeys                      map[5:1531811719 6:1531811721]latest_version            6min_decryption_version    5min_encryption_version    0# ...snip...

»Generate data key

(Persona: admin)

When you encrypt your data, the encryption key used to encrypt the plaintext is referred to as a data key. This data key needs to be protected so that your encrypted data cannot be decrypted easily by an unauthorized party. In encrypt secrets step, you encrypted your data by specifying the key ring name (orders) and the actual data key used to encrypt the data was never presented to you.

In this step, you are going to use the transit/datakey endpoint which returns the plaintext of a named data key.

Why would I need the data key?

Think of a scenario where you have a 2GB base64 binary large object (blob) that needs to be encrypted. You probably don't want to send the 2GB over the network to Vault and receive the 2GB back. Instead, you can generate a data key and encrypt it locally and use the same data key to decrypt it locally when needed.

The idea with data keys is to allow applications to encrypt and decrypt data without round-tripping through Vault.

The data key is its own full key; you can't decrypt it with the transit key that it is wrapped with. However, because the data key is wrapped by a transit key, and thus protected, you can store it with the data. This way, you can control which Vault clients can decrypt the data through policies.

$ vault write -f transit/datakey/plaintext/orders

Key           Value
---           -----
ciphertext    vault:v1:Aj7SUcmCAEiBhNhYBZkXKzO4kuOIDCwDx6FXqpngC3WQeQ4Tw9JCRCXnsq7So+pBN0L6mt2GybRiaQvi
plaintext     uB8pBOQMqtAQUHtu6HpEsfOGWZ+KknaRUpDW1dtO6k0=

$ vault write -f transit/datakey/plaintext/orders
Key           Value---           -----ciphertext    vault:v1:Aj7SUcmCAEiBhNhYBZkXKzO4kuOIDCwDx6FXqpngC3WQeQ4Tw9JCRCXnsq7So+pBN0L6mt2GybRiaQviplaintext     uB8pBOQMqtAQUHtu6HpEsfOGWZ+KknaRUpDW1dtO6k0=

The response contains the plaintext of the data key as well as its ciphertext. Use the plaintext to encrypt your blob. Store the ciphertext of your data key wherever you want. You can even store it in the key/value secrets engine.

When you need to decrypt the blob, request Vault to decrypt the ciphertext of your data key (decrypt ciphertext) so that you can get the plaintext back to decrypt the blob locally. In other words, once your blob is encrypted, you don't have to persist the data key. You only need to keep the ciphertext version of the data key.

Infrastructure

Security

Networking

Applications

»Challenge

»Solutions

»Personas

»Prerequisites

»Policy requirements

»Configure Transit secrets engine

»Create a token for Vault clients

»Encrypt secrets

»Decrypt ciphertext

»Rotate the encryption key

»Update key configuration

»Generate data key

»Help and Reference