SSH Secrets Engine: One-Time SSH Password

In a distributed cloud environment, tenant and system is increasingly important part of online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.

The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This tutorial demonstrates the one-time SSH password mode.

»Personas

The end-to-end scenario described in this tutorial involves two personas:

operations with privileged permissions to setup SSH secrets engine
client trusted entity to request SSH OTP from Vault

»Challenge

SSH servers provide a range of authentication methods, but most use password based authentication by default. If any user on the system has a fairly weak password, an attacker can more readily determine the password and create an unauthorized SSH connection.

»Solution

Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.

SSH OTP Workflow

An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.

When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.

Since the Vault server is contacted during SSH connection establishment, every login attempt and the correlating Vault lease information can be logged to an audit device.

NOTE: Vault SSH Helper version 0.1.6 and higher supports the Vault Enterprise namespaces feature.

»Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault.

Online tutorial: An interactive tutorial is also available if you do not wish to install the following resources. Click the Show Terminal button to start.

»Policy requirements

NOTE: For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

# To configure the SSH secrets engine
path "ssh/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable the userpass auth method
path "sys/auth/userpass" {
  capabilities = [ "update" ]
}

# To create the policy for the test user
path "sys/policies/acl/test" {
  capabilities = [ "read", "update" ]
}

# To create the test user
path "auth/userpass/users/ubuntu" {
  capabilities = [ "create", "update" ]
}

# To enable secrets enginespath "sys/mounts/*" {  capabilities = [ "create", "read", "update", "delete" ]}
# To configure the SSH secrets enginepath "ssh/*" {  capabilities = [ "create", "read", "update", "delete", "list" ]}
# To enable the userpass auth methodpath "sys/auth/userpass" {  capabilities = [ "update" ]}
# To create the policy for the test userpath "sys/policies/acl/test" {  capabilities = [ "read", "update" ]}
# To create the test userpath "auth/userpass/users/ubuntu" {  capabilities = [ "create", "update" ]}

If you are not familiar with policies, complete the policies tutorial.

»Start Vault

In another terminal, start a Vault dev server with root as the root token.

$ vault server -dev -dev-root-token-id root

$ vault server -dev -dev-root-token-id root

The Vault dev server defaults to running at 127.0.0.1:8200. The server is initialized and unsealed.

Insecure operation: Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

Open another terminal session, and export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://127.0.0.1:8200

$ export VAULT_ADDR=http://127.0.0.1:8200

Export an environment variable for the vault CLI to authenticate with the Vault server.

$ export VAULT_TOKEN=root

$ export VAULT_TOKEN=root

NOTE: For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

The Vault server is ready.

»Setup the SSH secrets engine

(Persona: operations)

On the Vault server, you must enable the SSH secrets engine and create a role.

Enable the SSH secrets engine.

$ vault secrets enable ssh
Success! Enabled the ssh secrets engine at: ssh/

$ vault secrets enable sshSuccess! Enabled the ssh secrets engine at: ssh/

Create a role named otp_key_role with key_type set to otp.

$ vault write ssh/roles/otp_key_role \
    key_type=otp \
    default_user=ubuntu \
    cidr_list=0.0.0.0/0

$ vault write ssh/roles/otp_key_role \    key_type=otp \    default_user=ubuntu \    cidr_list=0.0.0.0/0

This role defaults to creating credentials for the ubuntu user and will allow all remote hosts whose IP addresses fit within this CIDR range.

Security: The tutorial uses a very permissive cidr_list value of 0.0.0.0/0. In production, we recommend that roles are defined with a range that is granular to the range of remote hosts. We also recommend that you create one role for each username to ensure isolation between usernames.

»Setup the client authentication

(Persona: operations)

On the Vault server, you must create a policy to allow access to the SSH OTP role and then attach the policy to an authentication method.

First, create a policy file named, test.hcl, that provides access to the ssh/creds/otp_key_role path.

$ tee test.hcl <<EOF
# To list SSH secrets paths
path "ssh/*" {
  capabilities = [ "list" ]
}
# To use the configured SSH secrets engine otp_key_role role
path "ssh/creds/otp_key_role" {
  capabilities = ["create", "read", "update"]
}
EOF

$ tee test.hcl <<EOF# To list SSH secrets pathspath "ssh/*" {  capabilities = [ "list" ]}# To use the configured SSH secrets engine otp_key_role rolepath "ssh/creds/otp_key_role" {  capabilities = ["create", "read", "update"]}EOF

The path ssh/creds/otp_key_role is the path to the role created in the Setup the SSH secrets engine section.

Create a policy named test with the policy defined in test.hcl.

$ vault policy write test ./test.hcl

$ vault policy write test ./test.hcl

Enable an authentication method and attach the policy.

Enable the userpass auth method.

$ vault auth enable userpass

$ vault auth enable userpass

Create a user named ubuntu with the password "training" assigned the test policy.

$ vault write auth/userpass/users/ubuntu password="training" policies="test"

$ vault write auth/userpass/users/ubuntu password="training" policies="test"

»Install vault-ssh-helper

(Persona: operations)

SELinux users: If you use SELinux in enforcing mode, you need to define and enable a module that allows vault-ssh-helper to open and write log files, and communicate with the Vault server. Creating and enabling the SELinux module is beyond the scope of this tutorial. If you do not require SELinux enforcement, then operating SELinux in permissive mode also allows vault-ssh-helper to function.

On each Remote host, you must install vault-ssh-helper create a configuration file and modify both the Pluggable Authentication Module (PAM) sshd configuration file and the sshd configuration file. And finally, restart the sshd service.

NOTE: The example scenario in this tutorial uses the Ubuntu Linux distribution. If you use a different distribution, you will need to update paths and file locations to match your distribution.

Download and install vault-ssh-helper from releases.hashicorp.com.

Download version 0.2.1 of vault-ssh-helper.

$ wget https://releases.hashicorp.com/vault-ssh-helper/0.2.1/vault-ssh-helper_0.2.1_linux_amd64.zip

$ wget https://releases.hashicorp.com/vault-ssh-helper/0.2.1/vault-ssh-helper_0.2.1_linux_amd64.zip

Unzip the binary from the archive into /usr/local/bin.

$ sudo unzip -q vault-ssh-helper_0.2.1_linux_amd64.zip -d /usr/local/bin

$ sudo unzip -q vault-ssh-helper_0.2.1_linux_amd64.zip -d /usr/local/bin

Set the vault-ssh-helper binary's permissions to executable.

$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper

$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper

Set the vault-ssh-helper binary's user and group to root.

$ sudo chown root:root /usr/local/bin/vault-ssh-helper

$ sudo chown root:root /usr/local/bin/vault-ssh-helper

Create a Vault SSH Helper configuration file.
Create a directory to store the configuration file.
```
$ sudo mkdir /etc/vault-ssh-helper.d/
```
```
$ sudo mkdir /etc/vault-ssh-helper.d/
```
The Vault SSH Helper reads a configuration at the path /etc/vault-ssh-helper.d/config.hcl with this format:
```
vault_addr = "<VAULT_EXTERNAL_ADDR>"
tls_skip_verify = false
ca_cert = "<PEM_ENCODED_CA_CERT>"
ssh_mount_point = "ssh"
namespace = "my_namespace"
allowed_roles = "*"
```
```
vault_addr = "<VAULT_EXTERNAL_ADDR>"tls_skip_verify = falseca_cert = "<PEM_ENCODED_CA_CERT>"ssh_mount_point = "ssh"namespace = "my_namespace"allowed_roles = "*"
```
The vault_addr is the network address of the Vault server configured to generate the OTP.
tls_skip_verify enables or disables TLS verification.
ca_cert is the path to the PEM-encoded CA certificate files used to verify the Vault server's TLS certificate. When vault-ssh-helper is run with the -dev flag this is ignored.
ssh_mount_point is the Vault server path where the SSH secrets engine is enabled.
namespace is the namespace of the SSH mount point (Vault Enterprise only)
allowed_roles defines all * or a comma-separated list of allowed roles defined in the SSH secrets engines.
Refer to the documentation for the entire list of configuration properties.
Create a variable named VAULT_EXTERNAL_ADDR that contains the Vault server's external network address with protocol and port (e.g. http://192.0.2.10:8200)
```
$ VAULT_EXTERNAL_ADDR=<VAULT_EXTERNAL_ADDR>
```
```
$ VAULT_EXTERNAL_ADDR=<VAULT_EXTERNAL_ADDR>
```
Create the configuration file /etc/vault-ssh-helper.d/config.hcl.
```
$ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF
vault_addr = "$VAULT_EXTERNAL_ADDR"
tls_skip_verify = false
ssh_mount_point = "ssh"
allowed_roles = "*"
EOF
```
```
$ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOFvault_addr = "$VAULT_EXTERNAL_ADDR"tls_skip_verify = falsessh_mount_point = "ssh"allowed_roles = "*"EOF
```

Modify the PAM sshd configuration file.

Backup the original configuration file.

$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig

$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig

Open the file in your preferred text editor.

$ sudo nano /etc/pam.d/sshd

$ sudo nano /etc/pam.d/sshd

The common-auth must be commented out or removed to disable the standard Unix authentication and replaced with authentication through vault-ssh-helper. Finally, a workaround for a bug that exists with some versions of pam_exec.so must also be included.

Refer to the documentation for details about these parameter settings.

Example:

# PAM configuration for the Secure Shell service

# Standard Un*x authentication.
#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay

...

# PAM configuration for the Secure Shell service
# Standard Un*x authentication.#@include common-authauth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hclauth optional pam_unix.so not_set_pass use_first_pass nodelay
...

The vault-ssh-helper runs in development mode, -dev and loads its configuration file. While running it logs output to /var/log/vault-ssh.log.

Modify the sshd configuration file.
Backup the original configuration file.
```
$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
```
```
$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
```
Open the file in your preferred text editor.
```
$ sudo nano /etc/ssh/sshd_config
```
```
$ sudo nano /etc/ssh/sshd_config
```
Add or set the following:
```
ChallengeResponseAuthentication yes
UsePAM yes
PasswordAuthentication no
```
```
ChallengeResponseAuthentication yesUsePAM yesPasswordAuthentication no
```
This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.
Caution: You should ensure that there is another means of access to the server (such as a previously established SSH key for the root user) when disabling password authentication so that you are not prevented from logging in at all.

Restart the sshd service.

$ sudo systemctl restart sshd

$ sudo systemctl restart sshd

Verify the configuration.

$ vault-ssh-helper -verify-only -dev -config /etc/vault-ssh-helper.d/config.hcl
==> WARNING: Dev mode is enabled!
[INFO] using SSH mount point: ssh
[INFO] using namespace: us
[INFO] vault-ssh-helper verification successful!

$ vault-ssh-helper -verify-only -dev -config /etc/vault-ssh-helper.d/config.hcl==> WARNING: Dev mode is enabled![INFO] using SSH mount point: ssh[INFO] using namespace: us[INFO] vault-ssh-helper verification successful!

All Remote Hosts: These steps must be performed on all remote hosts.

»Generate an OTP and establish a connection.

(Persona: client)

A client configured to target a Vault server may now authenticate with the Vault server, generate an OTP and then use that OTP to establish an SSH connection.

Create a variable named REMOTE_HOST_IP that stores the remote host's external network address.

$ REMOTE_HOST_IP=<REMOTE_HOST_IP>

$ REMOTE_HOST_IP=<REMOTE_HOST_IP>

Authenticate via userpass method with the username ubuntu and password training.

$ vault login -method=userpass username=ubuntu password=training
Key                    Value
---                    -----
token                  s.uK9oTdKXwPuooZ9gPzTfVwPB
token_accessor         vG3BxXC0arMiMZnnOdgDiODt
token_duration         768h
token_renewable        true
token_policies         ["default" "test"]
identity_policies      []
policies               ["default" "test"]
token_meta_username    ubuntu

$ vault login -method=userpass username=ubuntu password=trainingKey                    Value---                    -----token                  s.uK9oTdKXwPuooZ9gPzTfVwPBtoken_accessor         vG3BxXC0arMiMZnnOdgDiODttoken_duration         768htoken_renewable        truetoken_policies         ["default" "test"]identity_policies      []policies               ["default" "test"]token_meta_username    ubuntu

Generate an OTP, through the otp_key_role, for remote host given its IP address.

Generate an OTP.

$ vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP

$ vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP

Example Output:

For a remote host at 192.0.2.10.

$ vault write ssh/creds/otp_key_role ip=192.0.2.10
Key                Value
---                -----
lease_id           ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a
lease_duration     768h
lease_renewable    false
ip                 192.0.2.10
key                f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df
key_type           otp
port               22
username           ubuntu

$ vault write ssh/creds/otp_key_role ip=192.0.2.10Key                Value---                -----lease_id           ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0alease_duration     768hlease_renewable    falseip                 192.0.2.10key                f1cb47ad-6255-0be8-6bd8-5c4b3b01c8dfkey_type           otpport               22username           ubuntu

The output displays a key. Its value is the OTP to use during SSH authentication.

Initiate an SSH session with the ubuntu user for a remote host given its IP address.

$ ssh ubuntu@$REMOTE_HOST_IP
Password: <Enter OTP>

$ ssh ubuntu@$REMOTE_HOST_IPPassword: <Enter OTP>

When prompted for a Password: enter the OTP.

Example:

For a remote host at 192.0.2.10.

$ ssh ubuntu@192.0.2.10
Password: <Enter OTP>

$ ssh ubuntu@192.0.2.10Password: <Enter OTP>

NOTE: If sshpass is installed, you can create a new OTP and SSH into the remote host with single-line CLI command:

$ vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no ubuntu@$REMOTE_HOST_IP

$ vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no ubuntu@$REMOTE_HOST_IP

Infrastructure

Security

Networking

Applications

»Personas

»Challenge

»Solution

»Prerequisites

»Policy requirements

»Start Vault

»Setup the SSH secrets engine

»Setup the client authentication

»Install vault-ssh-helper

»Generate an OTP and establish a connection.

»Help and Reference