SSH Certificates

Table of contents
  1. SSH authentication methods
  2. SSH certificates
  3. Host certificate
    1. Configure a host CA
    2. Sign the host key with CA
    3. Save host CA public key on client
  4. Client certificate
    1. Configure a client CA
    2. Sign the client key with CA
    3. Save client CA public key on host
  5. Host sshd_config settings
  6. Test connection
  7. Debugging
    1. From the client side
    2. From the host side
      1. Linux
      2. macOS
    3. To check certificate metadata
  8. SSH certificate extensions

SSH authentication methods

There are two popular ways to SSH into a server.

  • Use a system password
  • Use SSH key

Using an SSH key adds more security layers compared to using a system password.

However, even with an SSH key, there still exists a problem that once an SSH key is placed on the authorized_keys of the server, it is permanent unless someone actively removes it.

This may not matter if you’re the one and only person trying to SSH into a server, but if the number of people that need to be given or revoked access increases, this can become a hassle.

Using SSH certificates can alleviate these issues.

SSH certificates

SSH certificate authentication goes in both directions. The server/host presents its certificate to the user/client, and vice versa.

The whole idea of a certificate is simple:

  • Both parties agree on some authority and trust what it signs.

Such authority is called a Certificate Authority (CA).

Technically, our CA is just a pair of cryptographic keys.

So instead of trusting each user keys, parties will decide trust the CA that will sign those keys. This eliminates having to accumulate public keys in authorized_keys.

In addition, CAs can set a expiration date on their signatures.

So if you decide to give someone a server access for only a single day, but don’t want to bother remembering to come back after a day to remove his/her key from your authorized_keys, you can have the CA sign the key to only be valid for a day.

In the following example, I will be using Vault to manage these CAs.

Host certificate

Usually people dismiss the need for host certificates. However it is a good security layer to prevent SSHing into a bad machine.

When you’re SSHing without a certificate, you’ve probably seen something like this.

$ ssh server
The authenticity of host 'badsiteindisguise.com' can't be established.
...cryptographic key fingerprint...
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Usually you just end up typing yes, which adds the fingerprint to ~/.ssh/known_hosts and you never see the prompt again.

However, just like the prompt says, are you sure that this site can be trusted? Are you sure that there wasn’t a man in the middle that redirected you to one of his bad machines?

By placing a trusted host CA’s public key on the client before the first SSH, this security risk can be avoided.

Configure a host CA

Create a key pair for host CA:

ssh-keygen -t ed25519 -C "hostca" -f hostca

Which produces hostca and hostca.pub.

vault write ssh-host-signer/config/ca generate_signing_key=true currently only generates rsa keys, which is deprecated in newer OpenSSH. To use different crypto algorithms such as ed25519, you have to generate one and upload it to Vault.

Mount the SSH secrets engine on Vault:

vault secrets enable -path=ssh-host-signer ssh

The path can be anything you like, just make sure you are logged in to Vault and has the policy to read/write to that path.

Upload the CA key pair:

vault write ssh-host-signer/config/ca \
  private_key=@hostca \
  public_key=@hostca.pub

The @ points to the file. Use "..." to copy and paste the values.

Extend the host key certificate TTL (time-to-live):

vault secrets tune -max-lease-ttl=87600h ssh-host-signer

87600h is 10 years.

Create a role to sign host keys:

vault write ssh-host-signer/roles/hostrole \
  key_type=ca \
  ttl=87600h \
  allow_host_certificates=true \
  allowed_domains="example.com,something.com" \
  allow_bare_domains=true \
  allow_subdomains=true

Vault uses these roles to sign keys.

The above configuration basically says it can sign host certificates for domains of example.com, *.example.com, something.com, *.something.com, and the certificate will be valid for 10 years.

Sign the host key with CA

Sign and save the resulting certificate on the server:

vault write -field=signed_key ssh-host-signer/sign/hostrole \
  cert_type=host \
  public_key=@/etc/ssh/ssh_host_ed25519_key.pub > /etc/ssh/ssh_host_ed25519_key-cert.pub

Optionally set permission on the certificate:

sudo chmod 0640 /etc/ssh/ssh_host_ed25519_key-cert.pub

If the host key doesn’t exist, create one using ssh-keygen.

Update /etc/ssh/sshd_config:

HostKey /etc/ssh/ssh_ed25519_key
HostCertificate /etc/ssh/ssh_ed25519_key-cert.pub

Save host CA public key on client

From the client, get the public key of ssh-host-signer CA:

# If using API endpoint
curl <vault-api-url>/v1/ssh-host-signer/public-key

# If client has direct access to the Vault server
vault read -field=public_key ssh-host-signer/config/ca

Save the result to client’s ~/.ssh/known_hosts:

@cert-authority *.example.com,*.something.com ssh-ed25519 ...

If you have already logged in to the server before the host certificate was set up, remove the corresponding fingerprint in known_hosts.

Try SSHing to the server with a password or a regular public key.

Assuming you have never SSHed to the server before or have removed the previous fingerprint, SSH should not show you the Are you sure you want to continue prompt.

If it does, the host certificate is not set up correctly.

Client certificate

Instead of having the client’s public key saved to host’s authorized_keys, we will have the client use a certificate to authenticate.

Some of the process is actually very similar to above. Mostly the only difference is to use client or user instead of host.

Configure a client CA

Create a key pair for client CA:

ssh-keygen -t ed25519 -C "clientca" -f clientca

Which produces clientca and clientca.pub.

You can actually use the same pair of keys you used for host CA.

Mount the SSH secrets engine on Vault:

vault secrets enable -path=ssh-client-signer ssh

Upload the CA key pair:

vault write ssh-client-signer/config/ca \
  private_key=@clientca \
  public_key=@clientca.pub

Create a role to sign client keys:

vault write ssh-client-signer/roles/clientrole -<<"EOH"
{
  "allow_user_certificates": true,
  "allowed_users": "my-user",
  "allowed_extensions": "permit-pty,permit-port-forwarding,permit-x11-forwarding,permit-agent-forwarding,permit-user-rc",
  "default_extensions": [
    {
      "permit-pty": ""
    }
  ],
  "key_type": "ca",
  "default_user": "my-user",
  "ttl": "30m0s"
}
EOH
  • allowed_users: Comma separated list of allowed username
  • allowed_extensions: Comma separated list of extensions that client can request in their certificate
  • default_extensions: Default extensions given when this role signs a certificate
  • default_user: Username to use when one isn’t specified
  • ttl: Client certificate expires after ttl.

Sign the client key with CA

Create an SSH key if one doesn’t already exist:

ssh-keygen -t ed25519 -C "user@example.com" -f client_key

Sign and save the resulting certificate on the client:

  • To accept default
vault write -field=signed_key ssh-client-signer/sign/clientrole \
  public_key=@client_key.pub > client_key-cert.pub
  • To customize
vault write ssh-client-signer/sign/my-role -<<"EOH"
{
  "public_key": "ssh-ed25519 ...",
  "valid_principals": "my-user",
  "extensions": {
    "permit-pty": "",
    "permit-port-forwarding": ""
  }
}
EOH

Then copy and paste the certificate to client_key-cert.pub.

If your certificate ends in the <same_base>-cert.pub suffix, OpenSSH will automatically detect it so you won’t have to pass in your certificate as an identity file in addition to the private key.

Save client CA public key on host

From the host, get the public key of ssh-client-signer CA and save it to /etc/ssh/trusted-user-ca-keys.pem:

# If using API endpoint
curl -o /etc/ssh/trusted-user-ca-keys.pem http://127.0.0.1:8200/v1/ssh-client-signer/public_key

# If host has direct access to the Vault server
vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem

Now modify /etc/ssh/sshd_config on host:

# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

Host sshd_config settings

To disable SSH password authentication,

# /etc/ssh/sshd_config
PasswordAuthentication no
ChallengeResponseAuthentication no

Recap:

HostKey /etc/ssh/ssh_ed25519_key
HostCertificate /etc/ssh/ssh_ed25519_key-cert.pub

# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

Test connection

Now the client should be able to authenticate to the server with certificates:

# If the certificate ends in '-cert.pub' with the same base name
ssh -i ~/.ssh/client_key my-user@example.com

# If the certificate has a different naming scheme
ssh -i ~/.ssh/client-certificate.pub -i ~/.ssh/client_key my-user@example.com

Connection is a success if you don’t see any fingerprint validation prompt and was able to connect without adding client_key.pub to host’s authorized_keys.

Debugging

From the client side

Add -vvv to get a verbose log output:

ssh -vvv -i ~/.ssh/client_key my-user@example.com

From the host side

Linux

Set the LogLevel in /etc/ssh/sshd_config to VERBOSE.

Then inspect /var/log/auth.log.

macOS

log show --process sshd --last <num> --debug --info

See log show help for details.

To check certificate metadata

ssh-keygen -Lf ssh_key-cert.pub

SSH certificate extensions

Some of the basic extensions (names are self explanatory):

  • permit-pty: Allow interactive shell
  • permit-port-forwarding: Allow SSH tunnels
  • permit-x11-forwarding
  • permit-agent-forwarding
  • permit-user-rc

References: