SSH Secrets Engine: One-Time SSH Password

In a distributed cloud environment, tenant and system is increasingly important part of online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.

The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This tutorial demonstrates the one-time SSH password mode.

»Personas

The end-to-end scenario described in this tutorial involves two personas:

operations with privileged permissions to setup SSH secrets engine
client trusted entity to request SSH OTP from Vault

»Challenge

SSH servers provide a range of authentication methods, but most use password based authentication by default. If any user on the system has a fairly weak password, an attacker can more readily determine the password and create an unauthorized SSH connection.

»Solution

Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.

An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.

When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.

Since the Vault server is contacted during SSH connection establishment, every login attempt and the correlating Vault lease information can be logged to an audit device.

NOTE: Vault SSH Helper version 0.1.6 and higher supports the Vault Enterprise namespaces feature.

»Prerequisites

To perform the tasks described in this tutorial, you need to have:

HCP or OSS Vault environment
Vagrant installed with a configured provider such as VirtualBox
jq installed

»Policy requirements

NOTE: For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

# To configure the SSH secrets engine
path "ssh/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable the userpass auth method
path "sys/auth/userpass" {
  capabilities = [ "update" ]
}

# To create the policy for the test user
path "sys/policies/acl/test" {
  capabilities = [ "read", "update" ]
}

# To create the test user
path "auth/userpass/users/ubuntu" {
  capabilities = [ "create", "update" ]
}

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

# To configure the SSH secrets engine
path "ssh/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable the userpass auth method
path "sys/auth/userpass" {
  capabilities = [ "update" ]
}

# To create the policy for the test user
path "sys/policies/acl/test" {
  capabilities = [ "read", "update" ]
}

# To create the test user
path "auth/userpass/users/ubuntu" {
  capabilities = [ "create", "update" ]
}

If you are not familiar with policies, complete the policies tutorial.

»Lab setup

»Start Vault

Open a terminal session and determine your local workstations IP address.

$ ifconfig | grep inet

$ ifconfig | grep inet

Depending on your operating system and workstations configuration, you may have multiple addresses. Copy the address that matches your network. For example, if your network uses a subnet of 192.168.100.0/24 copy the 192.168.100.101 address.

Example output:

     inet 127.0.0.1 netmask 0xff000000
     inet6 ::1 prefixlen 128
     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
     inet6 fe80::aede:48ff:fe00:1122%en5 prefixlen 64 scopeid 0x4
     inet6 fe80::1c44:73de:cb90:5f84%en0 prefixlen 64 secured scopeid 0x6
     inet 192.168.100.101 netmask 0xffffff00 broadcast 192.168.100.255
     inet6 fe80::7ceb:9fff:fe93:555c%awdl0 prefixlen 64 scopeid 0x7
     inet6 fe80::7ceb:9fff:fe93:555c%llw0 prefixlen 64 scopeid 0x8

     inet 127.0.0.1 netmask 0xff000000
     inet6 ::1 prefixlen 128
     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
     inet6 fe80::aede:48ff:fe00:1122%en5 prefixlen 64 scopeid 0x4
     inet6 fe80::1c44:73de:cb90:5f84%en0 prefixlen 64 secured scopeid 0x6
     inet 192.168.100.101 netmask 0xffffff00 broadcast 192.168.100.255
     inet6 fe80::7ceb:9fff:fe93:555c%awdl0 prefixlen 64 scopeid 0x7
     inet6 fe80::7ceb:9fff:fe93:555c%llw0 prefixlen 64 scopeid 0x8

Start a Vault dev server with root as the root token and IP address copied from the previous step.
```
$ vault server -dev -dev-root-token-id root -dev-listen-address <copied-address>:8200
```
```
$ vault server -dev -dev-root-token-id root -dev-listen-address <copied-address>:8200
```
The Vault dev server defaults to running at the provided IP address on port 8200. The server is initialized and unsealed.
Insecure operation: Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

Open another terminal session, and export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://<ip-address-copied-from-previous-step>:8200

$ export VAULT_ADDR=http://<ip-address-copied-from-previous-step>:8200

Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```
```
$ export VAULT_TOKEN=root
```

NOTE: For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

The Vault server is ready.

»Start Ubuntu

Open another terminal session and initialize a new Ubuntu Vagrant box.

$ vagrant init generic/ubuntu2204

$ vagrant init generic/ubuntu2204

To successfully complete this tutorial, your Ubuntu box needs to be on the same network as your workstation. Edit the Vagrantfile to connect to your local network.

$ sed -ibak "s/# config.vm.network \"public_network\"/config.vm.network \"public_network\"/g" Vagrantfile

$ sed -ibak "s/# config.vm.network \"public_network\"/config.vm.network \"public_network\"/g" Vagrantfile

Start the Ubuntu box.
```
$ vagrant up
```
```
$ vagrant up
```

When prompted, select the number that corresponds to your active network interface. For example, if you are connected to your local network via your Wi-Fi (Wireless) adapter, enter the number 1 and press enter.

Example output:

==> default: Available bridged network interfaces:
1) en0: Wi-Fi (Wireless)
2) en5: USB Ethernet(?)
3) awdl0
4) llw0
5) en2: Thunderbolt 2
6) en1: Thunderbolt 1
7) en3: Thunderbolt 3
8) en4: Thunderbolt 4
9) bridge0
==> default: When choosing an interface, it is usually the one that is
==> default: being used to connect to the internet.
==> default:
    default: Which interface should the network bridge to?

==> default: Available bridged network interfaces:
1) en0: Wi-Fi (Wireless)
2) en5: USB Ethernet(?)
3) awdl0
4) llw0
5) en2: Thunderbolt 2
6) en1: Thunderbolt 1
7) en3: Thunderbolt 3
8) en4: Thunderbolt 4
9) bridge0
==> default: When choosing an interface, it is usually the one that is
==> default: being used to connect to the internet.
==> default:
    default: Which interface should the network bridge to?

You are ready to proceed with the tutorial.

»Setup the SSH secrets engine

(Persona: operations)

On the Vault server, you must enable the SSH secrets engine and create a role.

Return to the terminal session where you created the VAULT_ADDR environment variable.

Enable the SSH secrets engine.

$ vault secrets enable ssh
Success! Enabled the ssh secrets engine at: ssh/

$ vault secrets enable ssh
Success! Enabled the ssh secrets engine at: ssh/

Create a role named otp_key_role with key_type set to otp.

$ vault write ssh/roles/otp_key_role \
    key_type=otp \
    default_user=vagrant \
    cidr_list=0.0.0.0/0

$ vault write ssh/roles/otp_key_role \
    key_type=otp \
    default_user=vagrant \
    cidr_list=0.0.0.0/0

This role defaults to creating credentials for the vagrant user and will allow all remote hosts whose IP addresses fit within this CIDR range.

Security: The tutorial uses a very permissive cidr_list value of 0.0.0.0/0. In production, we recommend that roles are defined with a range that is granular to the range of remote hosts. We also recommend that you create one role for each username to ensure isolation between usernames.

»Setup the client authentication

(Persona: operations)

On the Vault server, you must create a policy to allow access to the SSH OTP role and then attach the policy to an authentication method.

Create a policy file named, test.hcl, that provides access to the ssh/creds/otp_key_role path.

$ tee test.hcl <<EOF
# To list SSH secrets paths
path "ssh/*" {
  capabilities = [ "list" ]
}
# To use the configured SSH secrets engine otp_key_role role
path "ssh/creds/otp_key_role" {
  capabilities = ["create", "read", "update"]
}
EOF

$ tee test.hcl <<EOF
# To list SSH secrets paths
path "ssh/*" {
  capabilities = [ "list" ]
}
# To use the configured SSH secrets engine otp_key_role role
path "ssh/creds/otp_key_role" {
  capabilities = ["create", "read", "update"]
}
EOF

The path ssh/creds/otp_key_role is the path to the role created in the Setup the SSH secrets engine section.

Create a policy named test with the policy defined in test.hcl.

$ vault policy write test ./test.hcl

$ vault policy write test ./test.hcl

Enable the userpass auth method.

$ vault auth enable userpass

$ vault auth enable userpass

Create a user named ubuntu with the password "training" assigned the test policy.

$ vault write auth/userpass/users/ubuntu password="training" policies="test"

$ vault write auth/userpass/users/ubuntu password="training" policies="test"

»Install vault-ssh-helper

(Persona: operations)

SELinux users: If you use SELinux in enforcing mode, you need to define and enable a module that allows vault-ssh-helper to open and write log files, and communicate with the Vault server. Creating and enabling the SELinux module is beyond the scope of this tutorial. If you do not require SELinux enforcement, then operating SELinux in permissive mode also allows vault-ssh-helper to function.

On each Remote host, you must:

Install vault-ssh-helper
Create a configuration file for the vault-ssh-helper
Modify both the Pluggable Authentication Module (PAM) sshd configuration file and the sshd configuration file
Restart the sshd service

NOTE: The example scenario in this tutorial uses the Ubuntu Linux distribution. If you use a different distribution, you will need to update paths and file locations to match your distribution.

Return to the terminal where you performed vagrant up and connect to Ubuntu.
```
$ vagrant ssh
```
```
$ vagrant ssh
```

Download and install version 0.2.1 of vault-ssh-helper from releases.hashicorp.com.

$ wget https://releases.hashicorp.com/vault-ssh-helper/0.2.1/vault-ssh-helper_0.2.1_linux_amd64.zip

$ wget https://releases.hashicorp.com/vault-ssh-helper/0.2.1/vault-ssh-helper_0.2.1_linux_amd64.zip

Unzip the binary from the archive into /usr/local/bin.

$ sudo unzip -q vault-ssh-helper_0.2.1_linux_amd64.zip -d /usr/local/bin

$ sudo unzip -q vault-ssh-helper_0.2.1_linux_amd64.zip -d /usr/local/bin

Set the vault-ssh-helper binary's permissions to executable.

$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper

$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper

Set the vault-ssh-helper binary's user and group to root.

$ sudo chown root:root /usr/local/bin/vault-ssh-helper

$ sudo chown root:root /usr/local/bin/vault-ssh-helper

Create a directory to store the configuration file.
```
$ sudo mkdir /etc/vault-ssh-helper.d/
```
```
$ sudo mkdir /etc/vault-ssh-helper.d/
```
The Vault SSH Helper reads a configuration at the path /etc/vault-ssh-helper.d/config.hcl with this format:
```
vault_addr = "<VAULT_EXTERNAL_ADDR>"
tls_skip_verify = false
ca_cert = "<PEM_ENCODED_CA_CERT>"
ssh_mount_point = "ssh"
namespace = "my_namespace"
allowed_roles = "*"
```
```
vault_addr = "<VAULT_EXTERNAL_ADDR>"
tls_skip_verify = false
ca_cert = "<PEM_ENCODED_CA_CERT>"
ssh_mount_point = "ssh"
namespace = "my_namespace"
allowed_roles = "*"
```
The vault_addr is the network address of the Vault server configured to generate the OTP.
tls_skip_verify enables or disables TLS verification.
ca_cert is the path to the PEM-encoded CA certificate files used to verify the Vault server's TLS certificate. When vault-ssh-helper is run with the -dev flag this is ignored.
ssh_mount_point is the Vault server path where the SSH secrets engine is enabled.
namespace is the namespace of the SSH mount point (Vault Enterprise and HCP Vault)
allowed_roles defines all * or a comma-separated list of allowed roles defined in the SSH secrets engines.
Refer to the documentation for the entire list of configuration properties.
Return to the terminal session where you created the VAULT_ADDR environment variable.
Retrieve and copy the value for VAULT_ADDR.
```
$ echo $VAULT_ADDR
```
```
$ echo $VAULT_ADDR
```
Return to the terminal connected to the Ubuntu box.

Create a variable named VAULT_EXTERNAL_ADDR that contains the Vault server's external network address copied in the previous step.

$ VAULT_EXTERNAL_ADDR=<VAULT_EXTERNAL_ADDR>

$ VAULT_EXTERNAL_ADDR=<VAULT_EXTERNAL_ADDR>

Example:

$ VAULT_EXTERNAL_ADDR=https://vault-cluster-public-vault-abcdefg.123456.z1.hashicorp.cloud:8200

$ VAULT_EXTERNAL_ADDR=https://vault-cluster-public-vault-abcdefg.123456.z1.hashicorp.cloud:8200

Create a Vault SSH Helper configuration file /etc/vault-ssh-helper.d/config.hcl.

$ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF
vault_addr = "$VAULT_EXTERNAL_ADDR"
tls_skip_verify = false
ssh_mount_point = "ssh"
allowed_roles = "*"
EOF

$ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF
vault_addr = "$VAULT_EXTERNAL_ADDR"
tls_skip_verify = false
ssh_mount_point = "ssh"
allowed_roles = "*"
EOF

Backup the original PAM sshd configuration configuration file.

$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig

$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig

Open the file in your preferred text editor.

$ sudo nano /etc/pam.d/sshd

$ sudo nano /etc/pam.d/sshd

The common-auth must be commented out or removed to disable the standard Unix authentication and replaced with authentication through vault-ssh-helper. Finally, a workaround for a bug that exists with some versions of pam_exec.so must also be included.

Refer to the documentation for details about these parameter settings.

Example:

# PAM configuration for the Secure Shell service

# Standard Un*x authentication.
#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay

...

# PAM configuration for the Secure Shell service

# Standard Un*x authentication.
#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay

...

The vault-ssh-helper is configured to run in development using the -dev parameter. The configuration file config.hcl is specified with -config parameter and while running it logs output to /var/log/vault-ssh.log using the -log parameter.

Backup the original sshd_config configuration file.

$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig

$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig

Modify the sshd configuration file.
```
$ sudo nano /etc/ssh/sshd_config
```
```
$ sudo nano /etc/ssh/sshd_config
```
Add or set the following:
```
KbdInteractiveAuthentication yes
UsePAM yes
PasswordAuthentication no
```
```
KbdInteractiveAuthentication yes
UsePAM yes
PasswordAuthentication no
```
This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.
Note: Older version of Ubuntu use ChallengeResponseAuthentication instead of KbdInteractiveAuthentication.
Caution: You should ensure that there is another means of access to the server (such as a previously established SSH key for the root user) when disabling password authentication so that you are not prevented from logging in at all.

Restart the sshd service.

$ sudo systemctl restart sshd

$ sudo systemctl restart sshd

Verify the configuration.

$ vault-ssh-helper -verify-only -dev -config /etc/vault-ssh-helper.d/config.hcl
==> WARNING: Dev mode is enabled!
[INFO] using SSH mount point: ssh
[INFO] vault-ssh-helper verification successful!

$ vault-ssh-helper -verify-only -dev -config /etc/vault-ssh-helper.d/config.hcl
==> WARNING: Dev mode is enabled!
[INFO] using SSH mount point: ssh
[INFO] vault-ssh-helper verification successful!

Note: These steps must be performed on all remote hosts.

Determine the IP address of the Ubuntu box.
```
$ ip -4 address show eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1

192.168.100.102
```
```
$ ip -4 address show eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1

192.168.100.102
```
Note: By following this tutorial, you should have two eth adapters. The eth0 adapter represents the internal VirtualBox network and the eth1 adapter represents the bridged connection to your local network.
Copy the IP address for eth1.
Disconnect from Ubuntu.
```
$ exit
```
```
$ exit
```

»Generate an OTP

(Persona: client)

A client configured to target a Vault server may now authenticate with the Vault server, generate an OTP and then use that OTP to establish an SSH connection.

Return to the terminal configured with the VAULT_ADDR environment variable.

Create a variable named REMOTE_HOST_IP that stores the Ubuntu box's IP address.

$ REMOTE_HOST_IP=<REMOTE_HOST_IP>

$ REMOTE_HOST_IP=<REMOTE_HOST_IP>

Example:

$ REMOTE_HOST_IP=192.168.100.102

$ REMOTE_HOST_IP=192.168.100.102

Authenticate with Vault using the userpass auth method with the username ubuntu and password training. Store the token in an environment variable.

$ UBUNTU_TOKEN=$(vault login -method=userpass username=ubuntu password=training -format=json | jq -r '.auth | .client_token')

$ UBUNTU_TOKEN=$(vault login -method=userpass username=ubuntu password=training -format=json | jq -r '.auth | .client_token')

Generate an OTP, through the otp_key_role, for remote host given its IP address.

$ VAULT_TOKEN=$UBUNTU_TOKEN vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP

$ VAULT_TOKEN=$UBUNTU_TOKEN vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP

Example Output:

$ vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP
Key                Value
---                -----
lease_id           ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a
lease_duration     768h
lease_renewable    false
ip                 192.168.100.102
key                f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df
key_type           otp
port               22
username           vagrant

$ vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP
Key                Value
---                -----
lease_id           ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a
lease_duration     768h
lease_renewable    false
ip                 192.168.100.102
key                f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df
key_type           otp
port               22
username           vagrant

The output displays a key. Its value is the OTP to use during SSH authentication.

»Initiate an SSH session

Connect to the Ubunut box as the vagrant user and the one time password provided by Vault.
```
$ ssh -o PubkeyAuthentication=no vagrant@$REMOTE_HOST_IP
```
```
$ ssh -o PubkeyAuthentication=no vagrant@$REMOTE_HOST_IP
```
Note: To demonstrate one time passwords, include the PubkeyAuthentication=no parameter to ensure your native SSH client does include any cached SSH keys.

When prompted enter yes to accept the fingerprint and for a Password: enter the OTP key from the previous step.

Example:

$ ssh -o PubkeyAuthentication=no vagrant@$REMOTE_HOST_IP

The authenticity of host '192.168.100.102 (192.168.100.102)' can't be established.
ECDSA key fingerprint is SHA256:N8CjNy6ahilQ3xvOO6FZKA3NkvpIOPi1LxpLvNx7xbQ.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Password: <Enter OTP>
Last login: Tue Aug 16 13:20:22 2022 from 10.0.2.2
vagrant@ubuntu2204:~$

$ ssh -o PubkeyAuthentication=no vagrant@$REMOTE_HOST_IP

The authenticity of host '192.168.100.102 (192.168.100.102)' can't be established.
ECDSA key fingerprint is SHA256:N8CjNy6ahilQ3xvOO6FZKA3NkvpIOPi1LxpLvNx7xbQ.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Password: <Enter OTP>
Last login: Tue Aug 16 13:20:22 2022 from 10.0.2.2
vagrant@ubuntu2204:~$

You are now connected to the Ubuntu box using the one time password from Vault.

»Cleanup

Disconnect from the Ubuntu box.
```
$ exit
```
```
$ exit
```
Switch to the terminal where you performed vagrant up and destroy the Ubuntu box.
```
$ vagrant destroy --force
```
```
$ vagrant destroy --force
```
Stop the Vault dev mode server or delete your HCP Vault server.

Infrastructure

Security

Applications

Networking

cloud

»Personas

»Challenge

»Solution

»Prerequisites

»Policy requirements

»Lab setup

»Start Vault

»Start Ubuntu

»Setup the SSH secrets engine

»Setup the client authentication

»Install vault-ssh-helper

»Generate an OTP

»Initiate an SSH session

»Cleanup

»Help and Reference