Secrets Management

SSH Secrets Engine: One-Time SSH Password

In a distributed cloud environment, tenant and system is increasingly important part of the online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.

The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This guide demonstrates the one-time SSH password mode.

Personas

The end-to-end scenario described in this guide involves two personas:

  • operations with privileged permissions to setup SSH secrets engine
  • client trusted entity to request SSH OTP from Vault

Challenge

By default, SSH servers use password authentication with optional public key authentication. If any user on the system has a fairly weak password, this allows an attacker to hijack the SSH connection.

Solution

Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.

SSH OTP Workflow

An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.

When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.

Prerequisites

To perform the tasks described in this guide, you need to have a Vault environment. Refer to the Getting Started guide to install Vault.

Policy requirements

To perform all tasks demonstrated in this guide, your policy must include the following permissions:

# To view in Web UI
path "sys/mounts" {
  capabilities = [ "read", "update" ]
}

# To configure the SSH secrets engine
path "ssh/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

If you are not familiar with policies, complete the policies guide.

Step 1: Install vault-ssh-helper

The SSH secrets engine uses Vault SSH Helper to verify the OTP used during the SSH authentication. Therefore the helper agent must be installed onto every target hosts.

  1. Download and install the latest version of vault-ssh-helper from releases.hashicorp.com.

    Example:

    # Download the vault-ssh-helper
    $ wget https://releases.hashicorp.com/vault-ssh-helper/0.1.4/vault-ssh-helper_0.1.4_linux_amd64.zip
    
    # Unzip the vault-ssh-helper in /usr/local/bin
    $ sudo unzip -q vault-ssh-helper_0.1.4_linux_amd64.zip -d /usr/local/bin
    
    # Make sure that vault-ssh-helper is executable
    $ sudo chmod 0755 /usr/local/bin/vault-ssh-helper
    
    # Set the usr and group of vault-ssh-helper to root
    $ sudo chown root:root /usr/local/bin/vault-ssh-helper
    
  2. Create a Vault SSH Helper configuration file, /etc/vault-ssh-helper.d/config.hcl containing the following:

    vault_addr = "<VAULT_ADDRESS>"
    ssh_mount_point = "ssh"
    ca_cert = "/etc/vault-ssh-helper.d/vault.crt"
    tls_skip_verify = false
    allowed_roles = "*"
    

    Where the <VAULT_ADDRESS> is the address of the Vault server generating the OTP. In a production environment, use the PEM-encoded CA cert file specified with ca_cert parameter. This CA cert is used by the client to verify Vault server's TLS certificate.

    For this guide, use -dev to ignore the cert requirement for simplicity.

    Example:

    # Create a new directory for vault-ssh-helper
    $ sudo mkdir /etc/vault-ssh-helper.d/
    
    # Create the config file
    $ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF
    vault_addr = "http://198.51.100.10:8200"
    ssh_mount_point = "ssh"
    ca_cert = "-dev"
    tls_skip_verify = false
    allowed_roles = "*"
    EOF
    

    Refer to the documentation for the entire list of configuration properties.

  3. Modify the /etc/pam.d/sshd file.

    # Make a copy of the original
    $ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig
    
    # Use your preferred text editor
    $ sudo nano /etc/pam.d/sshd
    

    Comment out the @include common-auth and then add custom Vault steatements. (NOTE: common-auth is the standard Linux authentication module which is commented out in favor of using our custom configuration.)

    # PAM configuration for the Secure Shell service
    
    # Standard Un*x authentication.
    #@include common-auth
    auth requisite pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl
    auth optional pam_unix.so not_set_pass use_first_pass nodelay
    
    ...
    

    Refer to the documentation for details about these parameter settings.

  4. Modify the /etc/ssh/sshd_config file.

    # Make a copy of the original
    $ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
    
    # Use your preferred text editor
    $ sudo nano /etc/ssh/ssh_config
    

    Add the following:

    ChallengeResponseAuthentication yes
    PasswordAuthentication no
    UsePAM yes
    

    This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.

  5. Restart the SSH service:

    $ sudo systemctl restart sshd
    

Step 2: Setup the SSH Secrets Engine

On the Vault server, you must enable the SSH secrets engine before you can perform the operation. Then you are going to create a role named, otp_key_role.

CLI command / API call using cURL / Web UI

CLI command

First, enable the SSH secrets engine.

$ vault secrets enable ssh

Next, create a role.

$ vault write ssh/roles/otp_key_role key_type=otp \
        default_user=ubuntu \
        cidr_list=0.0.0.0/0

This creates otp_key_role with ubuntu as its default username for which a credential will be generated.

API call using cURL

Enable ssh secrets engine using /sys/mounts endpoint:

$ curl --header "X-Vault-Token: <TOKEN>" \
       --request POST \
       --data <PARAMETERS> \
       <VAULT_ADDRESS>/v1/sys/mounts/<PATH>

Where <TOKEN> is your valid token, and <PARAMETERS> holds configuration parameters of the secrets engine.

Example:

The following example enables SSH secrets engine at sys/mounts/ssh path, and passed the secrets engine type (ssh) in the request payload.

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data '{"type":"ssh"}' \
       https://127.0.0.1:8200/v1/sys/mounts/ssh

Now, create a role using the ssh/roles/otp_key_role endpoint.

$ tee payload.json <<EOF
{
  "key_type": "otp",
  "default_user": "ubuntu",
  "cidr_list": "0.0.0.0/0"
}
EOF

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json
       https://127.0.0.1:8200/v1/ssh/roles/otp_key_role

This creates otp_key_role with ubuntu as its default username for which a credential will be generated.

Web UI

Open a web browser and launch the Vault UI (e.g. http://127.0.0.1:8200/ui) and then login.

  1. Select Enable new engine and select SSH from Secrets engine type drop-down list.

  2. Click Enable Engine.

  3. Select Create role.

  4. Enter otp_key_role in the Role name field, select otp from the Key type drop-down list, and then enter ubuntu in the Default user field.

  5. Select More options to expand the optional parameter fields, and then enter 0.0.0.0/0 in the CIDR list field. Create Role

  6. Click Create role.

Step 3: Setup the client authentication

(Persona: operations)

The client must have permissions against the ssh/creds/otp_key_role path to request an OTP for otp_key_role. First, create a policy file named, test.hcl.

$ tee test.hcl <<EOF
path "ssh/creds/otp_key_role" {
  capabilities = ["create", "read", "update"]
}
EOF

CLI command / API call using cURL / Web UI

  1. Create a test policy on the Vault server.

    $ vault policy write test ./test.hcl
    
  2. For this guide, enable the userpass auth method and create a username bob with password, training.

    # Enable the userpass auth method
    $ vault auth enable userpass
    
    # Create a user, 'bob'
    $ vault write auth/userpass/users/bob password="training" policies="test"
    

API call using cURL

  1. Create a test policy on the Vault server.

    # Create the API request payload, payload-2.json
    $ tee payload.json <<EOF
    {
      "policy": "path \"ssh/creds/otp_key_role\" {\n capabilities = [ \"create\", \"read\", \"update\", \"list\" ]\n }"
    }
    EOF
    
    # Create a test policy
    $ curl --header "X-Vault-Token: ..." \
           --request PUT \
           --data @payload.json \
           http://127.0.0.1:8200/v1/sys/policies/acl/test
    
  2. Enable the userpass auth method.

    $ curl --header "X-Vault-Token: ..." \
           --request POST \
           --data '{"type": "userpass"}' \
           http://127.0.0.1:8200/v1/sys/auth/userpass
    
  3. Create a username bob with password, training.

    $ curl --header "X-Vault-Token: ..." \
           --request POST \
           --data '{"password": "training", "policies": "test"}' \
           http://127.0.0.1:8200/v1/auth/userpass/users/bob
    

Web UI

  1. Open a web browser and launch the Vault UI (e.g. http://127.0.0.1:8200/ui) and then login.

  2. Click the Policies tab, and then select Create ACL policy.

  3. Toggle Upload file sliding switch, and click Choose a file to select your test.hcl file you authored. This loads the policy and sets the Name to be test. Create Policy

  4. Click Create Policy to complete.

  5. Click the Access tab, and select Enable new method.

    Enable Auth Method

  6. Select the Username & Password radio button.

  7. Click Next.

  8. Leave the path as default and click Enable Method.

  9. Click the Vault CLI shell icon (>_) to open a command shell. Execute vault write auth/userpass/users/bob password=training policies=test in the CLI shell to create a new user, bob:

    Create bob

  10. Click the icon (>_) again to hide the shell.

Step 4: Request an OTP

(Persona: client)

To test, authenticate as bob using the userpass and request an OTP.

CLI command / API call using cURL / Web UI

CLI command

  1. First, authenticate with Vault as bob.

    $ vault login -method=userpass username=bob password=training
    Key                    Value
    ---                    -----
    token                  s.uK9oTdKXwPuooZ9gPzTfVwPB
    token_accessor         vG3BxXC0arMiMZnnOdgDiODt
    token_duration         768h
    token_renewable        true
    token_policies         ["default" "test"]
    identity_policies      []
    policies               ["default" "test"]
    token_meta_username    bob
    
  2. To generate an OTP credential for an IP of the remote host belongs to the otp_key_role:

    $ vault write ssh/creds/otp_key_role ip=<REMOTE_HOST_IP>
    

    Example:

    $ vault write ssh/creds/otp_key_role ip=192.0.2.10
    Key                Value
    ---                -----
    lease_id           ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a
    lease_duration     768h
    lease_renewable    false
    ip                 192.0.2.10
    key                f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df
    key_type           otp
    port               22
    username           ubuntu
    

    The key value is the OTP to use during SSH authentication.

API call using cURL

  1. First, login as bob.

    $ curl --request POST \
           --data '{"password": "training"}' \
           http://127.0.0.1:8200/v1/auth/userpass/login/bob
    {
     ...
     "auth": {
       "client_token": "b3c2ac10-9f8f-4e64-9a1c-337236ba20f6",
       "accessor": "92204429-6555-772e-cf51-52492d7f1686",
       "policies": [
         "base",
         "default",
         "test"
       ],
       "token_policies": [
          "default",
          "test"
        ],
       ...
    
  2. To generate an OTP credential for an IP of the remote host belongs to the otp_key_role:

    $ curl --header "X-Vault-Token: ..." \
           --request POST \
           --data '{"ip": "<REMOTE_HOST_IP>"}'
           https://127.0.0.1:8200/v1/ssh/creds/otp_key_role
    

    Example:

    $ curl --header "X-Vault-Token: ..." \
           --request POST \
           --data '{"ip": "192.0.2.10"}'
           https://127.0.0.1:8200/v1/ssh/creds/otp_key_role  | jq
    {
       ...
       "data": {
         "ip": "192.0.20.10",
         "key": "6e472878-721a-b066-2cec-1bed0127ad44",
         "key_type": "otp",
         "port": 22,
         "username": "ubuntu"
       },
       ...
    }
    

    The key value is the OTP to use during SSH authentication.

Web UI

  1. Open a web browser and launch the Vault UI (e.g. http://127.0.0.1:8200/ui). If you are already logged in, sign out.

  2. At the Sign in to Vault, select the Userpass tab, and enter bob in the Username field, and training in the Password field.

  3. Select ssh under Secrets Engines.

  4. Select otp_key_role and enter ubuntu in the Username field, and enter the target host's IP address (e.g. 192.0.2.10) in the IP Address field.

  5. Click Generate.

  6. Click Copy credentials. This copies the OTP (key value).

Step 5: Establish an SSH session

Use the OTP generated at Step 4 to authenticate:

$ ssh ubuntu@192.0.2.10
Password: <Enter OTP>
$ vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no ubuntu@192.0.2.10

Help and Reference