In a distributed cloud environment, tenant and system is increasingly important part of online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.
The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This tutorial demonstrates the one-time SSH password mode.
»Personas
The end-to-end scenario described in this tutorial involves two personas:
operations
with privileged permissions to setup SSH secrets engineclient
trusted entity to request SSH OTP from Vault
»Challenge
SSH servers provide a range of authentication methods, but most use password based authentication by default. If any user on the system has a fairly weak password, an attacker can more readily determine the password and create an unauthorized SSH connection.
»Solution
Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.
An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.
When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.
Since the Vault server is contacted during SSH connection establishment, every login attempt and the correlating Vault lease information can be logged to an audit device.
NOTE: Vault SSH Helper version 0.1.6 and higher supports the Vault Enterprise namespaces feature.
»Prerequisites
To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault.
»Policy requirements
NOTE: For the purpose of this tutorial, you can use root
token to work
with Vault. However, it is recommended that root tokens are only used for
initial setup or in emergencies. As a best practice, use tokens with
appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:
If you are not familiar with policies, complete the policies tutorial.
»Start Vault
In another terminal, start a Vault dev server with root
as the root token.
The Vault dev server defaults to running at 127.0.0.1:8200
. The server is
initialized and unsealed.
Insecure operation: Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.
Open another terminal session, and export an environment variable for the vault
CLI to address the Vault server.
Export an environment variable for the vault
CLI to authenticate with the
Vault server.
NOTE: For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.
The Vault server is ready.
»Setup the SSH secrets engine
(Persona: operations)
On the Vault server, you must enable the SSH secrets engine and create a role.
Enable the SSH secrets engine.
Create a role named otp_key_role
with key_type
set to otp
.
This role defaults to creating credentials for the ubuntu
user and will allow
all remote hosts whose IP addresses fit within this CIDR range.
Security: The tutorial uses a very permissive cidr_list
value of
0.0.0.0/0
. In production, we recommend that roles are defined with a range
that is granular to the range of remote hosts. We also recommend that you create
one role for each username to ensure isolation between usernames.
»Setup the client authentication
(Persona: operations)
On the Vault server, you must create a policy to allow access to the SSH OTP role and then attach the policy to an authentication method.
First, create a policy file named, test.hcl
, that provides access to the
ssh/creds/otp_key_role
path.
The path ssh/creds/otp_key_role
is the path to the role created in the
Setup the SSH secrets engine section.
Create a policy named
test
with the policy defined intest.hcl
.Enable an authentication method and attach the policy.
Enable the
userpass
auth method.Create a user named
ubuntu
with the password "training" assigned thetest
policy.
»Install vault-ssh-helper
(Persona: operations)
SELinux users: If you use SELinux in enforcing mode, you need to define and enable a module that allows vault-ssh-helper
to open and write log files, and communicate with the Vault server. Creating and enabling the SELinux module is beyond the scope of this tutorial. If you do not require SELinux enforcement, then operating SELinux in permissive mode also allows vault-ssh-helper
to function.
On each Remote host, you must install vault-ssh-helper create a configuration file and modify both the Pluggable Authentication Module (PAM) sshd configuration file and the sshd configuration file. And finally, restart the sshd service.
NOTE: The example scenario in this tutorial uses the Ubuntu Linux distribution. If you use a different distribution, you will need to update paths and file locations to match your distribution.
Download and install vault-ssh-helper from releases.hashicorp.com.
Download version
0.2.1
ofvault-ssh-helper
.Unzip the binary from the archive into
/usr/local/bin
.Set the
vault-ssh-helper
binary's permissions to executable.Set the
vault-ssh-helper
binary's user and group toroot
.Create a Vault SSH Helper configuration file.
Create a directory to store the configuration file.
The Vault SSH Helper reads a configuration at the path
/etc/vault-ssh-helper.d/config.hcl
with this format:The
vault_addr
is the network address of the Vault server configured to generate the OTP.tls_skip_verify
enables or disables TLS verification.ca_cert
is the path to the PEM-encoded CA certificate files used to verify the Vault server's TLS certificate. Whenvault-ssh-helper
is run with the-dev
flag this is ignored.ssh_mount_point
is the Vault server path where the SSH secrets engine is enabled.namespace
is the namespace of the SSH mount point (Vault Enterprise only)allowed_roles
defines all*
or a comma-separated list of allowed roles defined in the SSH secrets engines.Refer to the documentation for the entire list of configuration properties.
Create a variable named
VAULT_EXTERNAL_ADDR
that contains the Vault server's external network address with protocol and port (e.g.http://192.0.2.10:8200
)Create the configuration file
/etc/vault-ssh-helper.d/config.hcl
.Modify the PAM sshd configuration file.
Backup the original configuration file.
Open the file in your preferred text editor.
The
common-auth
must be commented out or removed to disable the standard Unix authentication and replaced with authentication throughvault-ssh-helper
. Finally, a workaround for a bug that exists with some versions ofpam_exec.so
must also be included.Refer to the documentation for details about these parameter settings.
Example:
The
vault-ssh-helper
runs in development mode,-dev
and loads its configuration file. While running it logs output to/var/log/vault-ssh.log
.Modify the sshd configuration file.
Backup the original configuration file.
Open the file in your preferred text editor.
Add or set the following:
This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.
Caution: You should ensure that there is another means of access to the server (such as a previously established SSH key for the root user) when disabling password authentication so that you are not prevented from logging in at all.
Restart the
sshd
service.Verify the configuration.
All Remote Hosts: These steps must be performed on all remote hosts.
»Generate an OTP and establish a connection.
(Persona: client)
A client configured to target a Vault server may now authenticate with the Vault server, generate an OTP and then use that OTP to establish an SSH connection.
Create a variable named REMOTE_HOST_IP
that stores the remote host's
external network address.
Authenticate via
userpass
method with the usernameubuntu
and passwordtraining
.Generate an OTP, through the
otp_key_role
, for remote host given its IP address.Generate an OTP.
Example Output:
For a remote host at
192.0.2.10
.The output displays a
key
. Its value is the OTP to use during SSH authentication.
Initiate an SSH session with the ubuntu
user for a remote host given its IP
address.
When prompted for a Password:
enter the OTP.
Example:
For a remote host at 192.0.2.10
.
NOTE: If sshpass
is
installed, you can create a new OTP and SSH into the remote host with
single-line CLI command: