Vault's PKI secrets engine can dynamically generate X.509 certificates on demand. This allows services to acquire certificates without going through the usual manual process of generating a private key and Certificate Signing Request (CSR), submitting to a Certificate Authority (CA), and then waiting for the verification and signing process to complete.
»Personas
The steps described in this tutorial are typically performed by a security engineer.
»Challenge
Organizations should protect their website; however, the Traditional PKI process workflow takes a long time, which motivates organizations to create certificates which do not expire for a year or more.
»Solution
Use Vault to create X.509 certificates for usage in Mutual Transport Layer Security (MTLS) or other arbitrary PKI encryption. This solution can be used to create web server certificates, but if users do not import the CA chains, browsers will complain about the self-signed certificates.
Creating PKI certificates is generally a cumbersome process using traditional
tools like openssl or even more advanced frameworks like CFSSL. These tools
also require a human component to verify certificate distribution meets
organizational security policies.
Vault's PKI secrets engine makes this a lot simpler. The PKI secrets engine can be an intermediate-only certificate authority, which potentially allows for higher levels of security.
- Store CA outside of Vault (air gapped).
- Create CSRs for the intermediate CA.
- Sign CSR outside Vault and import intermediate CA.
- Issue leaf certificates from the Intermediate CA.
»Prerequisites
To perform the tasks described in this tutorial, you need:
A Vault environment. Refer to the Getting Started tutorial to install Vault.
The API and CLI versions of the example scenario use the
jqtool to parse JSON output. Installjqin your Vault environment to follow the examples which use this tool.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
»Policy requirements
NOTE: For the purposes of this tutorial, you can use a root token to
work with Vault. However, it is recommended that root tokens are only used for
just enough initial setup or in emergencies. As a best practice, use tokens
with appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following capabilities:
If you are not familiar with policies, complete the policies tutorial.
»Scenario introduction
In this tutorial, you are going to first generate a self-signed root certificate.
Then you are going to generate an intermediate certificate that is signed by
the root. You can then create a role and generate a certificate for the test.example.com domain.
You can choose to complete this tutorial at this point or you can continue with the advanced section on multiple issuer functionality beginning with step 7.
There, you will learn how enable a second root CA in the same secrets engine mount. You will also learn how to rotate the old root CA, and begin issuing certificates from the existing role with the new root CA. Finally, you will learn how to sunset the old CA.
In this tutorial, you will perform the following:
- Generate Root CA
- Generate Intermediate CA
- Create a Role
- Request Certificates
- Revoke Certificates
- Remove Expired Certificates
- Enable new Root CA (optional)
- Rotate old Root CA (optional)
- Set default issuer and issue certificate (optional)
- Sunset old Root CA (optional)
»Notice About Multi-Issuer Functionality
Vault version 1.11.0 and greater allows a single PKI mount to have multiple Certificate Authority (CA) certificates ("issuers") in a single mount. All issuers within a single mount are treated as a single Authority, meaning that:
- Certificate Revocation List (CRL) configuration is common to all issuers,
- All authority access URLs are common to all issuers,
- Issued certificates' serial numbers will be unique across all issuers.
However, since each issuer may have a distinct subject and keys, different issuers may have different CRLs.
It is strongly encouraged to limit the scope of CAs within a mount and not to mix different types of CAs (roots and intermediates).
Note: Some functionality will not work if a default issuer is not configured. Vault automatically selects the default issuer from the current issuing certificate on migration from an older Vault version (Vault < 1.11.0).
»Lab setup
In another terminal, start a Vault dev server with
rootas the root token.The Vault dev server defaults to running at
127.0.0.1:8200. The server is also initialized and unsealed.Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the
vaultCLI to address the Vault server.Export an environment variable for the
vaultCLI to authenticate with the Vault server.
The Vault server is ready.
»Step 1: Generate root CA
In this step, you are going to generate a self-signed root certificate using PKI secrets engine.
Enable the
pkisecrets engine at thepkipath.Successful output example:
Tune the
pkisecrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours.Successful output example:
Generate the example.com root CA, give it an issuer name, and save its certificate in the file
root_2022_ca.crt.This command is not expected to produce any output.
This generates a new self-signed CA certificate and private key. Vault will automatically revoke the generated root at the end of its lease period (TTL); the CA certificate will sign its own Certificate Revocation List (CRL).
List the issuer information for the root CA.
You can read the issuer with its ID to get the certificates and other metadata about the issuer. Let's skip the certificate output, but list the issuer metadata and usage information.
Example output:
Create a role for the root CA; creating this role allows for specifying an issuer when necessary for the purposes of this scenario. This also provides a simple way to transition from one issuer to another by referring to it by name.
NOTE: This role is only overly permissive in the names that it allows for the purposes of simplification in this tutorial.
Configure the CA and CRL URLs.
Successful output example:
»Step 2: Generate intermediate CA
Now, you are going to create an intermediate CA using the root CA you regenerated in the previous step.
First, enable the
pkisecrets engine at thepki_intpath.Successful output example:
Tune the
pki_intsecrets engine to issue certificates with a maximum time-to-live (TTL) of 43800 hours.Successful output example:
Execute the following command to generate an intermediate and save the CSR as
pki_intermediate.csr.This command is expected to produce no output.
Sign the intermediate certificate with the root CA private key, and save the generated certificate as
intermediate.cert.pem.This command is expected to produce no output.
Once the CSR is signed and the root CA returns a certificate, it can be imported back into Vault.
Successful output example:
»Step 3: Create a role
A role is a logical name that maps to a policy used to generate those credentials. It allows configuration parameters to control certificate common names, alternate names, the key uses that they are valid for, and more.
Here are a few noteworthy parameters:
| Param | Description |
|---|---|
allowed_domains | Specifies the domains of the role (used with allow_bare_domains and allow-subdomains options) |
allow_bare_domains | Specifies if clients can request certificates matching the value of the actual domains themselves |
allow_subdomains | Specifies if clients can request certificates with CNs that are subdomains of the CNs allowed by the other role options (NOTE: This includes wildcard subdomains.) |
allow_glob_domains | Allows names specified in allowed_domains to contain glob patterns (e.g. ftp*.example.com) |
In this step, you are going to create a role named example-dot-com.
Create a role named example-dot-com which allows subdomains, and specify the default issuer ref ID as the value of issuer_ref.
»Step 4: Request certificates
Keep certificate lifetimes short to align with Vault's philosophy of short-lived secrets.
Execute the following command to request a new certificate for the
test.example.com domain based on the example-dot-com role.
The response contains the PEM-encoded private key, key type and certificate serial number.
»Step 5: Revoke certificates
If a certificate must be revoked, you can easily perform the revocation action which will cause the CRL to be regenerated. When the CRL is regenerated, any expired certificates are removed from the CRL.
In certain circumstances, you may wish to revoke an issued certificate.
To revoke a certificate, execute the following command.
Example:
»Step 6: Remove expired certificates
Keep the storage backend and CRL by periodically removing certificates that have expired and are past a certain buffer period beyond their expiration time.
To remove revoked certificate and clean the CRL.
NOTE: The warning message is expected in every invocation of the tidy command, and serves to alert the operator that progress from the tidy operation can be observed in the Vault operational log.
You are now finished with the steps in the main scenario, and can move to the Cleanup section from here.
The following steps 7-10 are optional and for learning about Root CA rotation; these steps also require that you are using Vault version 1.11.0 or greater.
»Step 7: Rotate Root CA
Prior to version 1.11.0, it was not possible to enable more than one root CA in the same PKI secrets engine mount. This makes rotating the root CA challenging with respect to coordination of multiple PKI secrets engine mounts, and manually managing two root CAs while simultaneously deploying the new root CA to clients.
Vault version 1.11.0 introduces a set of new features in the PKI secrets engine, including multi-issuer capabilities which include multiple root CAs in the same secrets engine mount for the purpose of root CA rotation. There are also new CA migration helpers that ease the operational burden of migrating a root CA controlled by Vault.
To begin learning about this functionality, enable another root CA in the existing PKI secrets engine mount using the new rotate feature.
Example output:
Note that the certificate information contains multi-issuer information, such as the
issuer_refandissuer_namevalues.You can also list the issuers to confirm the addition of the new Root CA.
Both root CA issuers are now enabled in the same PKI secrets engine mount. While the new root CA is enabled, it is not yet actively used, and all subsequent issuances from the mount will use the original root CA issuer,
root-2022.TIP: At this point, the new root CA certificate can be deployed to all devices which require it as part of rotating out the old root CA.
Create a role for the new example.com root CA; creating this role allows for specifying an issuer when necessary. This also provides a simple way to transition from one issuer to another by referring to it by name.
NOTE: This role is only overly permissive in the names that it allows for the purposes of simplification in this tutorial.
»Step 8: Create a cross-signed intermediate
A signifcant step in the process of rotating an old Root CA involves handling devices which might be offline for long periods of time, but which need to eventually come online and fetch the new root certificate.
If you want to effectively handle this situation, you can do so with the use of a cross-signed Intermediate CA between the old root to the new root, such that clients of the old root can trust certificates which are signed by the new root.
The new cross-sign command provides this functionality to the Vault operator in a familar way, resembling the command syntax for creating a new intermediate CA with an existing key.
Create the cross-siged Intermediate CSR.
Example output:
Sign the resulting CSR with the old root CA.
Example output:
The result is a cross-signed certificate, but no issuer is created yet. Let's import the signed certificate to create the new issuer.
Import the cross-signed certificate.
Example output:
You now have 2 root CAs with a cross-signed intermediate CA providing a trust chain between them. You are ready to handle migration of devices which were not initially possible to migrate to the new root CA.
Read the issuer for the new root CA. You'll notice that there are now more certificates in the
ca_chainfield.Example output:
Specifically, there are 3 certificates in
ca_chain, as follows:The new root CA certificate (root-2023).
The new cross-signed intermediate CA certificate.
The old root CA certficate (root-2022).
If you do not want the chain composed in this way, you can specify a custom chain with the
manual_chainparameter when creating the intermediate CA. More information is available in the API documentation.
»Step 9: Set default issuer
After your defined cutoff point for the old root CA, and after you have distributed the new root CA to all devices, you'll want to switch the default issuer to the new root CA.
Use the root replace command to accomplish this, and specify the issuer name of the new root CA as the value to the default parameter.
From this point forward, Vault will issue new certificates based on the new root CA (root-2023).
»Step 10: Sunset old Root CA
The last step is to sunset the old root CA.
You can effectively sunset the old root CA by removing its ability to issue certificates, while preserving the ability to sign CRLs and revoke certificates.
Remove issuing-certificates from the Root CA's capabilities.
Example output:
You'll notice that "issuing-certificates" is removed from usage, and now only "crl-signing", and "read-only" are available on the old root CA. This means it cannot issue any further certificates going forward.
You can confirm this by attempting to issue a certificate directly from the old root CA.
The request fails.
»Cleanup
You can unset the two environment variables, VAULT_ADDR and VAULT_TOKEN, that you set in the beginning of this tutorial and remove the files you created to clean up.
Unset the environment variables.
Remove the files.
If you also followed the advanced steps, you'll need to remove some additional files.
You can terminate the Vault process from a command.
»Next steps
Check out the Streamline Certificate Management with HashiCorp Vault webinar recording.
Also, refer to the Vault PKI Secrets Engine Integration tutorial for an example of Nomad using the PKI secrets engine to generate and renew the X.509 certificates it uses. To automate the process, this tutorial leverages the Consul Template tool.
