In a distributed cloud environment, tenant and system is increasingly important part of online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.
The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This tutorial demonstrates the one-time SSH password mode.
»Personas
The end-to-end scenario described in this tutorial involves two personas:
operations
with privileged permissions to setup SSH secrets engineclient
trusted entity to request SSH OTP from Vault
»Challenge
By default, SSH servers use password authentication with optional public key authentication. If any user on the system has a fairly weak password, this allows an attacker to hijack the SSH connection.
»Solution
Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.
An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.
When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.
Since the Vault server is contacted during SSH connection establishment, every login attempt and the correlating Vault lease information is logged to the audit secrets engine.
NOTE: Vault SSH Helper version 0.1.6 and higher supports the Vault Enterprise namespaces feature.
»Prerequisites
To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault.
Online tutorial: An interactive tutorial is also available if you do not wish to install the following resources. Click the Show Terminal button to start.
»Policy requirements
NOTE: For the purpose of this tutorial, you can use root
token to work
with Vault. However, it is recommended that root tokens are only used for
initial setup or in emergencies. As a best practice, use tokens with
appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:
# To view in Web UI
path "sys/mounts" {
capabilities = [ "read", "update" ]
}
# To configure the SSH secrets engine
path "ssh/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
# To enable secrets engines
path "sys/mounts/*" {
capabilities = [ "create", "read", "update", "delete" ]
}
If you are not familiar with policies, complete the policies tutorial.
»Setup the SSH secrets engine
(Persona: operations)
On the Vault server, you must enable the SSH secrets engine and create a role.
Enable the SSH secrets engine.
$ vault secrets enable ssh
Success! Enabled the ssh secrets engine at: ssh/
Create a role named otp_key_role
with key_type
set to otp
.
$ vault write ssh/roles/otp_key_role key_type=otp \
default_user=ubuntu \
cidr_list=0.0.0.0/0
Success! Data written to: ssh/roles/otp_key_role
This role defaults to creating credentials for the ubuntu
user and will allow
all remote hosts whose IP addresses fit within this CIDR range.
Security: The tutorial uses a very permissive cidr_list
value of
0.0.0.0/0
. In production, we recommend that roles are defined with a range
that is granular to the range of remote hosts. We also recommend that you create
one role for each username to ensure isolation between usernames.
»Setup the client authentication
(Persona: operations)
On the Vault server, you must create a policy to allow access to the SSH OTP role and then attach the policy to an authentication method.
First, create a policy file named, test.hcl
, that provides access to the
ssh/creds/otp_key_role
path.
$ tee test.hcl <<EOF
path "ssh/creds/otp_key_role" {
capabilities = ["create", "read", "update"]
}
EOF
The path ssh/creds/otp_key_role
is the path to the role created in the
Setup the SSH secrets engine section.
Create a policy named
test
with the policy defined intest.hcl
.$ vault policy write test ./test.hcl
Enable an authentication method and attach the policy.
Enable the
userpass
auth method.$ vault auth enable userpass
Create a user named
bob
with the password "training" assigned thetest
policy.$ vault write auth/userpass/users/bob password="training" policies="test"
»Install vault-ssh-helper
(Persona: operations)
On each Remote host, you must install vault-ssh-helper create a configuration file and modify both the Pluggable Authentication Module (PAM) sshd configuration file and the sshd configuration file. And finally, restart the sshd service.
Download and install vault-ssh-helper from releases.hashicorp.com.
Download version
0.2.0
ofvault-ssh-helper
.$ wget https://releases.hashicorp.com/vault-ssh-helper/0.2.0/vault-ssh-helper_0.2.0_linux_amd64.zip
Unzip the binary from the archive into
/usr/local/bin
.$ sudo unzip -q vault-ssh-helper_0.2.0_linux_amd64.zip -d /usr/local/bin
Set the
vault-ssh-helper
binary's permissions to executable.$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper
Set the
vault-ssh-helper
binary's user and group toroot
.$ sudo chown root:root /usr/local/bin/vault-ssh-helper
Create a Vault SSH Helper configuration file.
Create a directory to store the configuration file.
$ sudo mkdir /etc/vault-ssh-helper.d/
Create the configuration file
/etc/vault-ssh-helper.d/config.hcl
.vault_addr = "<VAULT_ADDRESS>" tls_skip_verify = false ca_cert = "<PEM_ENCODED_CA_CERT>" ssh_mount_point = "ssh" namespace = "my_namespace" allowed_roles = "*"
The
vault_addr
is the network address of the Vault server configured to generate the OTP.tls_skip_verify
enables or disables TLS verification.ca_cert
is the path to the PEM-encoded CA certificate files used to verify the Vault server's TLS certificate. Whenvault-ssh-helper
is run with the-dev
flag this is ignored.ssh_mount_point
is the Vault server path where the SSH secrets engine is enabled.namespace
is the namespace of the SSH mount point (Vault Enterprise only)allowed_roles
defines all*
or a comma-separated list of allowed roles defined in the SSH secrets engines.Refer to the documentation for the entire list of configuration properties.
Example:
$ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF vault_addr = "http://198.51.100.10:8200" tls_skip_verify = false ssh_mount_point = "ssh" allowed_roles = "*" EOF
Modify the PAM sshd configuration file.
Backup the original configuration file.
$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig
Open the file in your preferred text editor.
$ sudo nano /etc/pam.d/sshd
The
common-auth
must be commented out or removed to disable the standard Unix authentication and replaced with authentication throughvault-ssh-helper
. Finally, a workaround for a bug that exists with some versions ofpam_exec.so
must also be included.Refer to the documentation for details about these parameter settings.
Example:
# PAM configuration for the Secure Shell service # Standard Un*x authentication. #@include common-auth auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl auth optional pam_unix.so not_set_pass use_first_pass nodelay ...
The
vault-ssh-helper
runs in development mode,-dev
and loads its configuration file. While running it logs output to/var/log/vault-ssh.log
.Modify the sshd configuration file.
Backup the original configuration file.
$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
Open the file in your preferred text editor.
$ sudo nano /etc/ssh/sshd_config
Add or set the following:
ChallengeResponseAuthentication yes UsePAM yes PasswordAuthentication no
This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.
Caution: You should ensure that there is another means of access to the server (such as a previously established SSH key for the root user) when disabling password authentication so that you are not prevented from logging in at all.
Restart the
sshd
service.$ sudo systemctl restart sshd
Verify the configuration.
$ vault-ssh-helper -verify-only -dev -config /etc/vault-ssh-helper.d/config.hcl ==> WARNING: Dev mode is enabled! [INFO] using SSH mount point: ssh [INFO] using namespace: us [INFO] vault-ssh-helper verification successful!
All Remote Hosts: These steps must be performed on all remote hosts.
»Generate an OTP and establish a connection.
(Persona: client)
A client configured to target a Vault server may now authenticate with the Vault server, generate an OTP and then use that OTP to establish an SSH connection.
Authenticate via
userpass
method with the usernamebob
and passwordtraining
.$ vault login -method=userpass username=bob password=training Key Value --- ----- token s.uK9oTdKXwPuooZ9gPzTfVwPB token_accessor vG3BxXC0arMiMZnnOdgDiODt token_duration 768h token_renewable true token_policies ["default" "test"] identity_policies [] policies ["default" "test"] token_meta_username bob
Generate an OTP, through the
otp_key_role
, for remote host given its IP address.$ vault write ssh/creds/otp_key_role ip=<REMOTE_HOST_IP>
Example:
For a remote host at
192.0.2.10
.$ vault write ssh/creds/otp_key_role ip=192.0.2.10 Key Value --- ----- lease_id ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a lease_duration 768h lease_renewable false ip 192.0.2.10 key f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df key_type otp port 22 username ubuntu
The output displays a
key
. Its value is the OTP to use during SSH authentication.
Initiate an SSH session with the ubuntu
user for a remote host given its IP
address.
$ ssh ubuntu@<REMOTE_HOST_IP>
Password: <Enter OTP>
When prompted for a Password:
enter the OTP.
Example:
For a remote host at 192.0.2.10
.
$ ssh ubuntu@192.0.2.10
Password: <Enter OTP>
NOTE: If sshpass
is
installed, you can create a new OTP and SSH into the remote host with
single-line CLI command:
$ vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no ubuntu@192.0.2.10