Zero-Trust Infrastructure: Deploying an OpenSSH Ephemeral Certificate Authority with FIDO2 WebAuthn Hardware Keys and Automated Host Attestation

In classic Linux administration, secure remote access begins with generating an asymmetric SSH keypair—such as an Ed25519 key (as covered in our foundational guides on generating Ed25519 SSH keys on Linux and importing Ed25519 keys for PuTTY on Windows)—and copying the public key to a remote server’s ~/.ssh/authorized_keys file. While this model is significantly more secure than password authentication, it breaks down catastrophically as engineering organizations scale.

This operational failure is known as SSH Key Sprawl:

• Public keys are permanently copied to hundreds of virtual machines, cloud instances, and bastions.
• Keys have no native expiration: a key generated five years ago remains fully active until manually purged.
• When an engineer leaves a company, rotating or revoking their public key requires executing Ansible playbooks or scripts across every active host in the fleet. A single missed host creates a persistent backdoor.
• Private keys frequently sit on developer laptops unencrypted or protected only by weak passphrases, making them vulnerable to exfiltration.
• Users face constant “The authenticity of host can’t be established” prompts, conditioning engineers to blindly accept fingerprint mismatches and enabling Man-In-The-Middle (MITM) attacks.

To resolve these vulnerabilities, enterprise zero-trust architectures replace static public keys with an Ephemeral OpenSSH Certificate Authority (SSH CA) combined with hardware-bound FIDO2 security keys (ed25519-sk).

Under this architecture, servers trust only the CA’s public key. Engineers authenticate each morning, undergo identity attestation, and receive a short-lived certificate (valid for 8 to 12 hours) tied to their physical security key. When the certificate expires, access ceases automatically—with zero changes required on individual target servers.

---

1. How OpenSSH Certificates Differ from X.509/TLS

Before configuring the system, it is vital to understand OpenSSH’s native certificate protocol.

Unlike web servers that utilize heavyweight X.509 certificates encoded in ASN.1 with complex trust chains and CRL/OCSP infrastructure, OpenSSH features an independent, lightweight certificate specification natively implemented in OpenSSH 5.4+ and enhanced with FIDO2 (sk-ssh-ed25519@openssh.com) in OpenSSH 8.2+.

Key characteristics of OpenSSH certificates:

1. Flat Hierarchy: There are no intermediate CAs. A certificate is directly signed by a root User CA or Host CA key.
2. Built-in Principals: Certificates explicitly specify the valid usernames an identity can assume (e.g. sysadmin, deployer, ubuntu), preventing privilege escalation.
3. Hard Expiration Bounds: Certificates embed strict validity windows (valid: from 2026-11-02T08:00:00 to 2026-11-02T18:00:00). Once expired, the OpenSSH daemon instantly denies the connection.
4. Audit Identification (Key ID): Every certificate embeds an immutable string identifier (such as an engineer’s email or identity provider UID), which the SSH daemon logs in syslog and auditd on every single session.
5. Critical Options & Extensions: Certificates can enforce source IP subnet restrictions (source-address 10.0.0.0/8), disable port forwarding, or enforce forced commands.

---

2. Infrastructure Topology & CA Generation

An enterprise deployment separates certificates into two distinct authorities:

• User CA (ca_user_key): Signs engineer and service identity certificates.
• Host CA (ca_host_key): Signs server host keys to establish mutual trust and eliminate host fingerprint prompts.

Step 1: Dedicated CA Key Generation

On a secure, dedicated signing node or bastion (isolated from general access), generate the master Ed25519 CA keys:

# Create dedicated CA directory with strict POSIX permissions
sudo mkdir -p /etc/ssh/ca
sudo chmod 700 /etc/ssh/ca
cd /etc/ssh/ca

# Generate the User Certificate Authority master key
sudo ssh-keygen -t ed25519 \
  -C "ca-user@andre.systems-2026" \
  -f /etc/ssh/ca/ca_user_key

# Generate the Host Certificate Authority master key
sudo ssh-keygen -t ed25519 \
  -C "ca-host@andre.systems-2026" \
  -f /etc/ssh/ca/ca_host_key

# Set airtight file permissions
sudo chmod 600 /etc/ssh/ca/ca_user_key /etc/ssh/ca/ca_host_key
sudo chmod 644 /etc/ssh/ca/ca_user_key.pub /etc/ssh/ca/ca_host_key.pub

Note on Production Hardening: In enterprise production environments, the CA private keys should reside either within a Hardware Security Module (HSM), a HashiCorp Vault transit engine, or on dedicated air-gapped YubiKeys with PIN + touch requirements.

---

3. Hardware-Bound User Keys with FIDO2 (ed25519-sk)

To satisfy Zero-Trust requirements, user private keys must never exist as exportable files on a laptop SSD. OpenSSH supports FIDO2 / CTAP2 hardware authenticators via the ed25519-sk key type.

When generating an ed25519-sk key, the private key material remains locked inside the hardware token’s secure element. OpenSSH on the client only retains a “key handle” that cannot be used on another physical token.

Generating the Resident Hardware Key on the Client

On the engineer’s workstation (Linux, macOS, or Windows 11 OpenSSH):

# Insert your YubiKey 5 or FIDO2-compliant security key
ssh-keygen -t ed25519-sk \
  -O resident \
  -O verify-required \
  -C "andre@andre.systems [YubiKey-5C-NFC]" \
  -f ~/.ssh/id_ed25519_sk

Key flags explained:

• -t ed25519-sk: Instructs OpenSSH to use the FIDO2 hardware token interface with Ed25519 curves.
• -O resident: Stores the key handle directly on the hardware token. If an engineer moves to a new workstation, running ssh-add -K automatically reconstructs the public key and handle without transferring files!
• -O verify-required: Enforces that the user must enter the hardware key’s FIDO2 PIN before the token responds to an SSH challenge, providing multi-factor authentication (something you have + something you know).

Verify the generated key files:

ls -la ~/.ssh/id_ed25519_sk*

You will see:

• ~/.ssh/id_ed25519_sk: The lightweight key handle.
• ~/.ssh/id_ed25519_sk.pub: The public key descriptor.

---

4. Signing Ephemeral User Certificates

The engineer submits their public key (id_ed25519_sk.pub) to the CA service for cryptographic signing.

The CA executes ssh-keygen -s with targeted constraints:

ssh-keygen -s /etc/ssh/ca/ca_user_key \
  -I "andre@andre.systems#emp_8834" \
  -n "sysadmin,ubuntu,rocky,deployer" \
  -V -5m:+8h \
  -z 20261102001 \
  -O permit-pty \
  -O permit-user-rc \
  -O source-address="10.0.0.0/8,192.168.0.0/16,172.16.0.0/12" \
  ~/.ssh/id_ed25519_sk.pub

Breakdown of the Signing Parameters:

Flag	Purpose & Security Rationale
-s /etc/ssh/ca/ca_user_key	Signs the certificate using the User CA master private key.
-I "andre@andre.systems#emp_8834"	Key ID: Injected into system audit logs (/var/log/secure) whenever this cert connects.
-n "sysadmin,ubuntu,rocky,deployer"	Principals: The exact operating system usernames this certificate is authorized to log in as.
-V -5m:+8h	Validity Window: Valid starting 5 minutes ago (safeguard against minor NTP clock drift) and expires in exactly 8 hours.
-z 20261102001	Serial Number: Unique 64-bit integer used for audit tracking and Key Revocation Lists (KRL).
-O source-address="..."	IP White-listing: The certificate is only valid when connections originate from designated corporate subnets or VPNs.
-O permit-pty	Allows interactive terminal allocation. Notice -O permit-port-forwarding is intentionally omitted unless required.

This command generates ~/.ssh/id_ed25519_sk-cert.pub.

Inspecting the Generated Certificate

Inspect the cryptographic details of the certificate using ssh-keygen -L:

ssh-keygen -L -f ~/.ssh/id_ed25519_sk-cert.pub

Expected diagnostic output:

~/.ssh/id_ed25519_sk-cert.pub:
        Type: sk-ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: SK-ED25519-CERT SHA256:4y+L82eN7hQ9m0A1zXk8P3vB7c9T2r1u5o4p0q9y8z0
        Signing CA: ED25519 SHA256:k7V3b8n1m9X0y2Z4a6c8e0g2i4k6m8o0q2s4u6w8y0z (ca-user@andre.systems-2026)
        Key ID: "andre@andre.systems#emp_8834"
        Serial: 20261102001
        Valid: from 2026-11-02T08:25:00 to 2026-11-02T16:30:00
        Principals: 
                sysadmin
                ubuntu
                rocky
                deployer
        Critical Options: 
                source-address 10.0.0.0/8,192.168.0.0/16,172.16.0.0/12
        Extensions: 
                permit-pty
                permit-user-rc

Notice how every constraint is cryptographically signed. If an attacker modifies the Key ID, extends the validity date, or adds root to the Principals list, the CA signature verification fails immediately.

---

5. Hardening Target Linux Servers (sshd_config)

Now configure target servers in your fleet (Ubuntu 22.04/24.04, Rocky Linux 9, RHEL 9, Debian 12) to accept User CA certificates and map principals to system accounts.

Step 1: Install the User CA Public Key

Copy /etc/ssh/ca/ca_user_key.pub from the CA node to every server in the fleet:

sudo cp ca_user_key.pub /etc/ssh/trusted-user-ca-keys.pub
sudo chmod 644 /etc/ssh/trusted-user-ca-keys.pub
sudo chown root:root /etc/ssh/trusted-user-ca-keys.pub

Step 2: Establish Authorized Principals Mapping

Create the authorized principals directory:

sudo mkdir -p /etc/ssh/auth_principals
sudo chmod 755 /etc/ssh/auth_principals

Each file in this directory corresponds to a local Linux user account (/etc/ssh/auth_principals/%u) and lists which certificate principals can log into that account.

For the sysadmin local user:

sudo tee /etc/ssh/auth_principals/sysadmin << 'EOF'
sysadmin
sre-lead
infrastructure-admin
EOF
sudo chmod 644 /etc/ssh/auth_principals/sysadmin

For the deployer local user:

sudo tee /etc/ssh/auth_principals/deployer << 'EOF'
deployer
ci-runner
release-engineer
EOF
sudo chmod 644 /etc/ssh/auth_principals/deployer

If an engineer presents a certificate containing only the deployer principal, OpenSSH will permit access to the deployer account but actively reject login attempts to sysadmin or root.

Step 3: Hardened sshd_config.d Drop-In Configuration

Create a dedicated drop-in file at /etc/ssh/sshd_config.d/00-zero-trust-ca.conf:

# /etc/ssh/sshd_config.d/00-zero-trust-ca.conf
# Enterprise Zero-Trust OpenSSH CA Hardening

# 1. Enable Public Key & CA Authentication
PubkeyAuthentication yes
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

# 2. Phase Out Legacy Static Keys (Set to /dev/null once fully transitioned)
# AuthorizedKeysFile /dev/null
AuthorizedKeysFile .ssh/authorized_keys

# 3. Key Revocation List (KRL)
RevokedKeys /etc/ssh/revoked_keys

# 4. Eliminate Insecure Authentication Vectors
PasswordAuthentication no
KbdInteractiveAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication no
PermitEmptyPasswords no

# 5. Restrict Root Access
PermitRootLogin no

# 6. Audit & Log Configuration
LogLevel VERBOSE

Note on LogLevel VERBOSE: Setting LogLevel VERBOSE instructs sshd to record the full certificate Key ID, serial number, and CA fingerprint in /var/log/secure or /var/log/auth.log on every login attempt.

Verify configuration syntax before restarting sshd:

# Test sshd syntax without interrupting active connections
sudo sshd -t

If the test exits cleanly with code 0, reload sshd:

# On systemd systems (Ubuntu, Debian, Rocky Linux, RHEL)
sudo systemctl reload sshd

---

6. Host Attestation: Eliminating “Host Key Verification Failed”

In traditional SSH, the first time you connect to a server, you encounter:

The authenticity of host 'bastion01.andre.systems (10.0.10.5)' can't be established.
ED25519 key fingerprint is SHA256:abc123...
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Because engineers cannot verify random 43-character base64 hashes in their heads, they blindly type yes. If an attacker performs ARP spoofing or DNS poisoning, the engineer gets a scary warning—or if connecting for the first time, accepts the attacker’s key undetected.

With an OpenSSH Host CA, servers present a signed host certificate during the initial key exchange. If the client trusts the Host CA, the connection is authenticated transparently with zero interactive prompts.

Step 1: Sign the Server’s Host Key on the CA

On the server node (bastion01.internal), inspect its public host key: /etc/ssh/ssh_host_ed25519_key.pub.

Send this public key to the CA node and sign it with -h (Host Certificate flag):

ssh-keygen -s /etc/ssh/ca/ca_host_key \
  -I "bastion01.internal" \
  -h \
  -n "bastion01,bastion01.internal,10.0.10.5,bastion01.andre.systems" \
  -V -5m:+52w \
  -z 20261102101 \
  /tmp/ssh_host_ed25519_key.pub

The resulting file is ssh_host_ed25519_key-cert.pub. Copy it back to the target server’s /etc/ssh/ directory.

Step 2: Configure Server Host Certificate in sshd

In /etc/ssh/sshd_config.d/00-zero-trust-ca.conf, register the certificate:

# Server Host Certificate
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Reload sshd:

sudo systemctl reload sshd

Step 3: Configure Clients to Trust the Host CA

On client workstations (or globally via /etc/ssh/ssh_known_hosts or centrally via configuration management), add a single @cert-authority record:

# ~/.ssh/known_hosts or /etc/ssh/ssh_known_hosts
@cert-authority *.internal,*.andre.systems,10.0.* ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... [Host CA Public Key Content]

When an engineer connects to ssh sysadmin@bastion01.internal, OpenSSH validates the server’s signed host certificate against the @cert-authority record. The connection establishes instantly without any prompt! If a host certificate is invalid or missing, OpenSSH aborts immediately, neutralizing MITM attacks.

---

7. Revocation Management: OpenSSH Key Revocation Lists (KRL)

Even with short 8-hour certificates, immediate revocation is required if a hardware token is lost or compromised mid-day.

OpenSSH provides the Key Revocation List (KRL) mechanism via binary database files (ssh-keygen -k). Unlike flat text files, OpenSSH KRLs use compact bitmap filters capable of checking millions of revoked keys and serial numbers in microseconds.

Creating and Updating a KRL

On the CA node:

# Initialize a new KRL or add a compromised public key / cert
ssh-keygen -k -f /etc/ssh/ca/revoked_keys ~/.ssh/id_ed25519_sk-cert.pub

# Alternatively, revoke by CA serial number:
# (Revoke serial number 20261102001 issued by ca_user_key)
ssh-keygen -k -f /etc/ssh/ca/revoked_keys -s /etc/ssh/ca/ca_user_key.pub serial:20261102001

Inspecting the KRL

Check which certificates or serials are currently revoked:

ssh-keygen -Q -l -f /etc/ssh/ca/revoked_keys

Distribute /etc/ssh/ca/revoked_keys to /etc/ssh/revoked_keys on target servers. When a user presents a revoked certificate, sshd denies access with:

Authentication refused: revoked certificate

---

8. Automating the Daily Engineer Workflow

To make this architecture seamless for developers and systems engineers, automate morning attestation with a concise script.

Create an automation wrapper script ssh-cert-login.sh:

#!/usr/bin/env bash
set -euo pipefail

CA_ENDPOINT="https://ca.internal.andre.systems/api/v1/sign-user-key"
KEY_HANDLE="$HOME/.ssh/id_ed25519_sk"
PUB_KEY="$KEY_HANDLE.pub"
CERT_FILE="$HOME/.ssh/id_ed25519_sk-cert.pub"

echo "=== Enterprise Zero-Trust SSH Morning Login ==="

# Check if hardware key exists
if [[ ! -f "$PUB_KEY" ]]; then
  echo "Error: Hardware key public handle not found at $PUB_KEY"
  exit 1
fi

# Check if current certificate is still valid
if [[ -f "$CERT_FILE" ]]; then
  if ssh-keygen -L -f "$CERT_FILE" | grep -q "Valid: from"; then
    VALID_TO=$(ssh-keygen -L -f "$CERT_FILE" | grep "Valid: from" | awk '{print $4}')
    CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
    if [[ "$CURRENT_TIME" < "$VALID_TO" ]]; then
      echo "✓ Existing certificate is valid until $VALID_TO (UTC). No refresh required."
      exit 0
    fi
  fi
fi

echo "Requesting new 8-hour ephemeral certificate..."
echo "Please tap your physical FIDO2 hardware key when prompted..."

# Post public key to CA attestation endpoint
RESPONSE=$(curl -s -f -X POST "$CA_ENDPOINT" \
  -H "Authorization: Bearer $(get-idp-oidc-token)" \
  --data-binary "@$PUB_KEY")

echo "$RESPONSE" > "$CERT_FILE"
chmod 644 "$CERT_FILE"

# Add key to ssh-agent
ssh-add "$KEY_HANDLE"

echo "✓ Successfully acquired certificate for user: $(whoami)"
ssh-keygen -L -f "$CERT_FILE" | grep -E "(Key ID|Valid:|Principals)"

Add the identity to the SSH agent for seamless shell sessions:

ssh-add -K

---

9. Verification & System Audit Logging

Let’s test an authenticated connection with verbose debugging:

ssh -vvv -i ~/.ssh/id_ed25519_sk sysadmin@bastion01.internal

Examine the debug trace during authentication:

debug1: Offering public key: /home/andre/.ssh/id_ed25519_sk-cert.pub ED25519-CERT-SK explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /home/andre/.ssh/id_ed25519_sk-cert.pub ED25519-CERT-SK explicit
debug3: sign_and_send_pubkey: using user-presence ED25519-SK
Confirm user presence for key ED25519-SK SHA256:4y+L82e...
[User taps physical YubiKey]
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to bastion01.internal ([10.0.10.5]:22) using "publickey".

Now inspect the server’s /var/log/secure (Rocky/RHEL) or /var/log/auth.log (Ubuntu/Debian):

sudo tail -n 20 /var/log/secure | grep "Accepted publickey"

Look at the extraordinary audit trail:

Nov  2 09:14:22 bastion01 sshd[14201]: Accepted publickey for sysadmin from 10.0.1.50 port 52140 ssh2: \
  ED25519-CERT-SK SHA256:4y+L82eN7hQ9m0A1zXk8P3vB7c9T2r1u5o4p0q9y8z0 \
  ID andre@andre.systems#emp_8834 (serial 20261102001) \
  CA ED25519 SHA256:k7V3b8n1m9X0y2Z4a6c8e0g2i4k6m8o0q2s4u6w8y0z

Notice what is captured in a single log line:

1. Target system account (sysadmin).
2. Client IP address (10.0.1.50).
3. Hardware key fingerprint (SHA256:4y+L82...).
4. Individual engineer identity (andre@andre.systems#emp_8834).
5. Certificate serial number (20261102001).
6. Signing authority fingerprint (CA ED25519 SHA256:k7V3b8...).

This satisfies SOC 2, ISO 27001, and NIST 800-53 auditing requirements with zero ambiguity.

---

10. Production Troubleshooting Matrix

Symptom	Root Cause	Solution
Certificate not yet valid or Certificate has expired	Server and CA system clocks have drifted out of sync.	Ensure NTP (chrony or systemd-timesyncd) is synchronized across all hosts. Always sign certificates with a negative start offset (e.g. -V -5m:+8h).
Certificate does not contain valid principal for user	The certificate’s -n list does not match the target username or the server’s AuthorizedPrincipalsFile does not include the principal.	Run ssh-keygen -L -f cert.pub to inspect principals. Update /etc/ssh/auth_principals/%u on the server or reissue certificate with proper principal.
Host key verification failed	Server presented a certificate whose -n names do not match the exact hostname or IP the client used to connect.	Ensure the host certificate is signed with all FQDNs, short hostnames, and IP addresses used to access the node (-n "bastion,bastion.internal,10.0.10.5").
sign_and_send_pubkey: signing failed: agent refused operation	YubiKey FIDO2 PIN required or user did not touch the token in time.	Re-insert security key and ensure PIN prompt is answered before timeout.
Authentication refused: bad ownership or modes for file	/etc/ssh/trusted-user-ca-keys.pub or /etc/ssh/auth_principals have writable permissions for non-root users.	Execute chmod 644 /etc/ssh/trusted-user-ca-keys.pub and chmod 755 /etc/ssh/auth_principals.

---

Summary & Architecture Checklist

By migrating from static Ed25519 keys to an Ephemeral OpenSSH Certificate Authority with FIDO2 Hardware Attestation, you eliminate key sprawl, enforce mandatory multi-factor hardware security, protect against MITM attacks, and gain an audit logging infrastructure that scales across thousands of hosts.

Production Implementation Checklist:

• Master User CA and Host CA Ed25519 keys generated and isolated on secure infrastructure.
• Engineers equipped with FIDO2 hardware tokens using resident, PIN-protected keys (ed25519-sk).
• Target Linux hosts configured with TrustedUserCAKeys and AuthorizedPrincipalsFile.
• Fleet servers signed with Host CA certificates and distributed via client @cert-authority known hosts.
• Key Revocation List (KRL) configured for rapid mid-session revocation.
• Legacy ~/.ssh/authorized_keys deprecated and phased out.