Security

SSH Secrets Engine: One-Time SSH Password

In a distributed cloud environment, tenant and system is increasingly important part of the online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.

The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This guide demonstrates the one-time SSH password mode.

Personas

The end-to-end scenario described in this guide involves two personas:

  • operations with privileged permissions to setup SSH secrets engine
  • client trusted entity to request SSH OTP from Vault

Challenge

By default, SSH servers use password authentication with optional public key authentication. If any user on the system has a fairly weak password, this allows an attacker to hijack the SSH connection.

Solution

Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.

SSH OTP Workflow

An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.

When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.

Prerequisites

To perform the tasks described in this guide, you need to have a Vault environment. Refer to the Getting Started guide to install Vault.

Policy requirements

To perform all tasks demonstrated in this guide, your policy must include the following permissions:

# To view in Web UI
path "sys/mounts" {
  capabilities = [ "read", "update" ]
}

# To configure the SSH secrets engine
path "ssh/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

If you are not familiar with policies, complete the policies guide.

Step 1: Install vault-ssh-helper

The SSH secrets engine uses Vault SSH Helper to verify the OTP used during the SSH authentication. Therefore the helper agent must be installed onto every target hosts.

  1. Download and install the latest version of vault-ssh-helper from releases.hashicorp.com.

    Download the vault-ssh-helper.

    $ wget https://releases.hashicorp.com/vault-ssh-helper/0.1.4/vault-ssh-helper_0.1.4_linux_amd64.zip
    

    Unzip the vault-ssh-helper in /usr/local/bin.

    $ sudo unzip -q vault-ssh-helper_0.1.4_linux_amd64.zip -d /usr/local/bin
    

    Make sure that vault-ssh-helper is executable.

    $ sudo chmod 0755 /usr/local/bin/vault-ssh-helper
    

    Set the usr and group of vault-ssh-helper to root.

    $ sudo chown root:root /usr/local/bin/vault-ssh-helper
    
  2. Create a Vault SSH Helper configuration file, /etc/vault-ssh-helper.d/config.hcl containing the following.

    vault_addr = "<VAULT_ADDRESS>"
    ssh_mount_point = "ssh"
    ca_cert = "/etc/vault-ssh-helper.d/vault.crt"
    tls_skip_verify = false
    allowed_roles = "*"
    

    Where the <VAULT_ADDRESS> is the address of the Vault server generating the OTP. In a production environment, use the PEM-encoded CA cert file specified with ca_cert parameter. This CA cert is used by the client to verify Vault server's TLS certificate.

    For this guide, use -dev to ignore the cert requirement for simplicity.

    Example:

    First, create a new directory for vault-ssh-helper.

    $ sudo mkdir /etc/vault-ssh-helper.d/
    

    Create the config file.

    $ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF
    vault_addr = "http://198.51.100.10:8200"
    ssh_mount_point = "ssh"
    ca_cert = "-dev"
    tls_skip_verify = false
    allowed_roles = "*"
    EOF
    

    Refer to the documentation for the entire list of configuration properties.

  3. Modify the /etc/pam.d/sshd file.

    Make a copy of the original before making any change.

    $ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig
    

    Open the file in your preferred text editor.

    $ sudo nano /etc/pam.d/sshd
    

    Comment out the @include common-auth and then add custom Vault steatements. (NOTE: common-auth is the standard Linux authentication module which is commented out in favor of using our custom configuration.)

    # PAM configuration for the Secure Shell service
    
    # Standard Un*x authentication.
    #@include common-auth
    auth requisite pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl
    auth optional pam_unix.so not_set_pass use_first_pass nodelay
    
    ...
    

    Refer to the documentation for details about these parameter settings.

  4. Modify the /etc/ssh/sshd_config file.

    Make a copy of the original before making any change.

    $ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
    

    Open the file in your preferred text editor.

    $ sudo nano /etc/ssh/ssh_config
    

    Add the following:

    ChallengeResponseAuthentication yes
    PasswordAuthentication no
    UsePAM yes
    

    This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.

  5. Restart the SSH service.

    $ sudo systemctl restart sshd
    

Step 2: Setup the SSH Secrets Engine

On the Vault server, you must enable the SSH secrets engine before you can perform the operation. Then you are going to create a role named, otp_key_role.

First, enable the SSH secrets engine.

$ vault secrets enable ssh

Next, create a role.

$ vault write ssh/roles/otp_key_role key_type=otp \
        default_user=ubuntu \
        cidr_list=0.0.0.0/0

This creates otp_key_role with ubuntu as its default username for which a credential will be generated.

Step 3: Setup the client authentication

(Persona: operations)

The client must have permissions against the ssh/creds/otp_key_role path to request an OTP for otp_key_role. First, create a policy file named, test.hcl.

$ tee test.hcl <<EOF
path "ssh/creds/otp_key_role" {
  capabilities = ["create", "read", "update"]
}
EOF
  1. Create a test policy on the Vault server.

    $ vault policy write test ./test.hcl
    
  2. For this guide, enable the userpass auth method and create a username bob with password, training.

    Enable the userpass auth method.

    $ vault auth enable userpass
    

    Create a user, bob.

    $ vault write auth/userpass/users/bob password="training" policies="test"
    

Step 4: Request an OTP

(Persona: client)

To test, authenticate as bob using the userpass and request an OTP.

  1. First, authenticate with Vault as bob.

    $ vault login -method=userpass username=bob password=training
    Key                    Value
    ---                    -----
    token                  s.uK9oTdKXwPuooZ9gPzTfVwPB
    token_accessor         vG3BxXC0arMiMZnnOdgDiODt
    token_duration         768h
    token_renewable        true
    token_policies         ["default" "test"]
    identity_policies      []
    policies               ["default" "test"]
    token_meta_username    bob
    
  2. To generate an OTP credential for an IP of the remote host belongs to the otp_key_role:

    $ vault write ssh/creds/otp_key_role ip=<REMOTE_HOST_IP>
    

    Example:

    $ vault write ssh/creds/otp_key_role ip=192.0.2.10
    Key                Value
    ---                -----
    lease_id           ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a
    lease_duration     768h
    lease_renewable    false
    ip                 192.0.2.10
    key                f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df
    key_type           otp
    port               22
    username           ubuntu
    

    The key value is the OTP to use during SSH authentication.

Step 5: Establish an SSH session

Use the OTP generated at Step 4 to authenticate:

$ ssh ubuntu@192.0.2.10
Password: <Enter OTP>
$ vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no ubuntu@192.0.2.10

Help and Reference