▶ Watch (3:12)

When a machine loses network, standard SSH fails. Engineers connect through a KVM or BMC that presents a serial console or a text-only screen. The only interface is reading output and typing responses. No binary protocol fits in that channel. Cloudflare’s infrastructure operations team uses this path to recover broken servers. Authentication often defaults to a shared root password. That password must be revoked across every image when someone leaves the SRE group. Old images stay accessible, and the revoked user can still log in.

OTP Approaches and Their Shortcomings

▶ Watch (11:29)

One-time passwords improve on static passwords. Time-based OTP requires clock sync, which a crashed server may lack. Counter-based OTP desyncs when the user mistypes a response. Challenge-based OTP avoids state sync but still stores a shared secret on the server. Any SRE with root access can read every user’s seed and impersonate them. Shared secrets introduce the same theft and revocation problems as passwords.

Public-Key OTP via ECDH

▶ Watch (20:41)

Replacing the shared secret with public-key cryptography solves the seed-theft problem. The server generates an ephemeral ECDH key pair and sends the public key as a challenge. The client computes a shared secret using its SSH private key and the server’s public key. The client sends the first 8 bytes (12 base64 characters) back. The server verifies using the user’s public key from authorized_keys. No shared secret ever lives on the server.

Implementing with Linux PAM

▶ Watch (27:33)

Korchagin implemented the scheme as a Linux PAM authentication module. PAM allows per‑service configuration. The module for the login service (serial console) sits before the standard pam_unix module. On success it skips the password prompt. On failure it falls through to the existing authentication. The module reads the user’s authorized_keys file, just like SSH. The server challenge is the base64‑encoded ephemeral public key.

Demo: Software Key and YubiKey PIV

▶ Watch (38:20)

In the demo the user types a username. The server presents a 45‑character base64 challenge. On the client the user runs ssh-login-response with the private key path. The tool outputs a 12‑character OTP. With a YubiKey in PIV mode the tool uses the hardware key and asks for a PIN, providing 2FA. The server side is identical because it sees only the public key.

Clippy: Improving the User Experience

▶ Watch (43:37)

Operators found running the command‑line tool tedious. Korchagin built Clippy, a wrapper around SSH. It intercepts the server’s challenge prompt, detects the hostname and challenge, and automatically generates the response. The user hits any key and touches the YubiKey. Clippy types the OTP into the terminal, eliminating copy‑paste. Two keystrokes replace a multi‑step workflow.

Q&A

Concern about IoT software vulnerability with out-of-band networks? Korchagin suggested taking control of the software, citing the OpenBMC project. ▶ Watch (48:30)

Notable Quotes

The answer is everywhere. Ignat Korchagin · ▶ Watch (8:49)

nobody cares that it’s better security. Ignat Korchagin · ▶ Watch (43:22)

public key-based OTPs can offer superior manageability Ignat Korchagin · ▶ Watch (46:50)

Key Takeaways

  • Emergency serial console access often relies on shared passwords with poor revocation.
  • ECDH-based OTP reuses existing SSH keys and produces short 12-character responses.
  • Linux PAM allows per-service authentication policies, enabling drop-in OTP modules.

About the Speaker(s)

Ignat Korchagin is a systems engineer at Cloudflare working mostly on Linux, platforms and hardware security. Ignat’s interests are cryptography, hacking, and low-level programming. Before Cloudflare, Ignat worked as a senior security engineer for Samsung Electronics’ Mobile Communications Division. His solutions may be found in many older Samsung smart phones and tablets. Ignat started his career as a security researcher in the Ukrainian government’s communications services.