SSH Secrets Engine: One-Time SSH Password

In a distributed cloud environment, tenant and system is increasingly important part of online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.

The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This tutorial demonstrates the one-time SSH password mode.

»Personas

The end-to-end scenario described in this tutorial involves two personas:

operations with privileged permissions to setup SSH secrets engine
client trusted entity to request SSH OTP from Vault

»Challenge

By default, SSH servers use password authentication with optional public key authentication. If any user on the system has a fairly weak password, this allows an attacker to hijack the SSH connection.

»Solution

Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.

SSH OTP Workflow

An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.

When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.

Since the Vault server is contacted during SSH connection establishment, every login attempt and the correlating Vault lease information is logged to the audit secrets engine.

NOTE: Vault SSH Helper version 0.1.6 and higher supports the Vault Enterprise namespaces feature.

»Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault.

Online tutorial: An interactive tutorial is also available if you do not wish to install the following resources. Click the Show Terminal button to start.

»Policy requirements

NOTE: For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# To view in Web UI
path "sys/mounts" {
  capabilities = [ "read", "update" ]
}

# To configure the SSH secrets engine
path "ssh/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

If you are not familiar with policies, complete the policies tutorial.

»Setup the SSH secrets engine

(Persona: operations)

On the Vault server, you must enable the SSH secrets engine and create a role.

Enable the SSH secrets engine.

$ vault secrets enable ssh
Success! Enabled the ssh secrets engine at: ssh/

Create a role named otp_key_role with key_type set to otp.

$ vault write ssh/roles/otp_key_role key_type=otp \
    default_user=ubuntu \
    cidr_list=0.0.0.0/0
Success! Data written to: ssh/roles/otp_key_role

This role defaults to creating credentials for the ubuntu user and will allow all remote hosts whose IP addresses fit within this CIDR range.

Security: The tutorial uses a very permissive cidr_list value of 0.0.0.0/0. In production, we recommend that roles are defined with a range that is granular to the range of remote hosts. We also recommend that you create one role for each username to ensure isolation between usernames.

»Setup the client authentication

(Persona: operations)

On the Vault server, you must create a policy to allow access to the SSH OTP role and then attach the policy to an authentication method.

First, create a policy file named, test.hcl, that provides access to the ssh/creds/otp_key_role path.

$ tee test.hcl <<EOF
path "ssh/creds/otp_key_role" {
  capabilities = ["create", "read", "update"]
}
EOF

The path ssh/creds/otp_key_role is the path to the role created in the Setup the SSH secrets engine section.

Create a policy named test with the policy defined in test.hcl.
```
$ vault policy write test ./test.hcl
```
Enable an authentication method and attach the policy.
Enable the userpass auth method.
```
$ vault auth enable userpass
```
Create a user named bob with the password "training" assigned the test policy.
```
$ vault write auth/userpass/users/bob password="training" policies="test"
```

»Install vault-ssh-helper

(Persona: operations)

On each Remote host, you must install vault-ssh-helper create a configuration file and modify both the Pluggable Authentication Module (PAM) sshd configuration file and the sshd configuration file. And finally, restart the sshd service.

Download and install vault-ssh-helper from releases.hashicorp.com.

Download version 0.2.0 of vault-ssh-helper.

$ wget https://releases.hashicorp.com/vault-ssh-helper/0.2.0/vault-ssh-helper_0.2.0_linux_amd64.zip

Unzip the binary from the archive into /usr/local/bin.

$ sudo unzip -q vault-ssh-helper_0.2.0_linux_amd64.zip -d /usr/local/bin

Set the vault-ssh-helper binary's permissions to executable.

$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper

Set the vault-ssh-helper binary's user and group to root.

$ sudo chown root:root /usr/local/bin/vault-ssh-helper

Create a Vault SSH Helper configuration file.
Create a directory to store the configuration file.
```
$ sudo mkdir /etc/vault-ssh-helper.d/
```
Create the configuration file /etc/vault-ssh-helper.d/config.hcl.
```
vault_addr = "<VAULT_ADDRESS>"
tls_skip_verify = false
ca_cert = "<PEM_ENCODED_CA_CERT>"
ssh_mount_point = "ssh"
namespace = "my_namespace"
allowed_roles = "*"
```
The vault_addr is the network address of the Vault server configured to generate the OTP.
tls_skip_verify enables or disables TLS verification.
ca_cert is the path to the PEM-encoded CA certificate files used to verify the Vault server's TLS certiciate. When vault-ssh-hlper is run with the -dev flag this is ignored.
ssh_mount_point is the Vault server path where the SSH secrets engine is enabled.
namespace is the namespace of the SSH mount point (Vault Enterprise only)
allowed_roles defines all * or a comma-separated list of allowed roles defined in the SSH secrets engines.
Refer to the documentation for the entire list of configuration properties.
Example:
```
$ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF
vault_addr = "http://198.51.100.10:8200"
tls_skip_verify = false
ssh_mount_point = "ssh"
allowed_roles = "*"
EOF
```
Modify the PAM sshd configuration file.
Backup the original configuration file.
```
$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig
```
Open the file in your preferred text editor.
```
$ sudo nano /etc/pam.d/sshd
```
The common-auth must be commented out or removed to disable the standard Unix authentication and replaced with authentication through vault-ssh-helper. Finally, a workaround for a bug that exists with some versions of pam_exec.so must also be included.
Refer to the documentation for details about these parameter settings.
Example:
```
# PAM configuration for the Secure Shell service

# Standard Un*x authentication.
#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay

...
```
The vault-ssh-helper runs in development mode, -dev and loads its configuration file. While running it logs output to /var/log/vault-ssh.log.
Modify the sshd configuration file.
Backup the original configuration file.
```
$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
```
Open the file in your preferred text editor.
```
$ sudo nano /etc/ssh/sshd_config
```
Add or set the following:
```
ChallengeResponseAuthentication yes
UsePAM yes
PasswordAuthentication no
```
This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.
Caution: You should ensure that there is another means of access to the server (such as a previously established SSH key for the root user) when disabling password authentication so that you are not prevented from logging in at all.
Restart the sshd service.
```
$ sudo systemctl restart sshd
```

Verify the configuration.

$ vault-ssh-helper -verify-only -dev -config /etc/vault-ssh-helper.d/config.hcl
==> WARNING: Dev mode is enabled!
[INFO] using SSH mount point: ssh
[INFO] using namespace: us
[INFO] vault-ssh-helper verification successful!

All Remote Hosts: These steps must be performed on all remote hosts.

»Generate an OTP and establish a connection.

(Persona: client)

A client configured to target a Vault server may now authenticate with the Vault server, generate an OTP and then use that OTP to establish an SSH connection.

Authenticate via userpass method with the username bob and password training.

$ vault login -method=userpass username=bob password=training
Key                    Value
---                    -----
token                  s.uK9oTdKXwPuooZ9gPzTfVwPB
token_accessor         vG3BxXC0arMiMZnnOdgDiODt
token_duration         768h
token_renewable        true
token_policies         ["default" "test"]
identity_policies      []
policies               ["default" "test"]
token_meta_username    bob

Generate an OTP, through the otp_key_role, for remote host given its IP address.

$ vault write ssh/creds/otp_key_role ip=<REMOTE_HOST_IP>

Example:

For a remote host at 192.0.2.10.

$ vault write ssh/creds/otp_key_role ip=192.0.2.10
Key                Value
---                -----
lease_id           ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a
lease_duration     768h
lease_renewable    false
ip                 192.0.2.10
key                f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df
key_type           otp
port               22
username           ubuntu

The output displays a key. Its value is the OTP to use during SSH authentication.

Initiate an SSH session with the ubuntu user for a remote host given its IP address.

$ ssh ubuntu@<REMOTE_HOST_IP>
Password: <Enter OTP>

When prompted for a Password: enter the OTP.

Example:

For a remote host at 192.0.2.10.

$ ssh ubuntu@192.0.2.10
Password: <Enter OTP>

NOTE: If sshpass is installed, you can create a new OTP and SSH into the remote host with single-line CLI command:

$ vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no ubuntu@192.0.2.10

Infrastructure

Security

Networking

Applications

»Personas

»Challenge

»Solution

»Prerequisites

»Policy requirements

»Setup the SSH secrets engine

»Setup the client authentication

»Install vault-ssh-helper

»Generate an OTP and establish a connection.

»Help and Reference