Category: Linux

  • SSH hardening beyond “disable root login”

    SSH hardening beyond “disable root login”

    Part 1 of 4 in the Hardening a Linux server series.

    Every hardening guide starts with PermitRootLogin no, and plenty of them stop there. That setting is fine. It’s also the least interesting thing about securing SSH, because an attacker who can log in as a sudo-capable user with a password has root anyway. The things that actually decide how exposed your SSH daemon is are how users prove who they are, how the server proves who it is, which crypto you still accept, and who is allowed to try at all.

    Start from what sshd is really doing

    Sequence diagram: laptop sends a public key to an SSH CA, receives an 8 hour certificate, presents it to sshd, which checks the CA signature, principals, revocation list and validity before granting the session

    Before changing anything, look at the effective configuration rather than the file. Distributions now ship Include /etc/ssh/sshd_config.d/*.conf near the top of sshd_config, and for most keywords sshd uses the first value it reads, not the last. I’ve watched people append PasswordAuthentication no to the bottom of the main file while a cloud-init drop-in earlier in the include order said yes, and the drop-in won.

    sudo sshd -T | grep -Ei 'passwordauth|kbdinteractive|permitroot|pubkeyauth|authenticationmethods'
    sudo sshd -T -C user=deploy,host=10.0.0.5,addr=10.0.0.5

    The second form evaluates Match blocks for a specific connection, which is the only sane way to test conditional rules. Put your own settings in a drop-in with a low number, like /etc/ssh/sshd_config.d/00-hardening.conf, so they load first.

    Keys, and better keys

    Passwords on an internet-facing sshd are a credential-stuffing target forever. Turn off both password paths, since keyboard-interactive will happily do password auth through PAM even when PasswordAuthentication is off:

    PasswordAuthentication no
    KbdInteractiveAuthentication no
    PubkeyAuthentication yes
    PermitRootLogin no
    MaxAuthTries 3
    LoginGraceTime 20

    (KbdInteractiveAuthentication replaced ChallengeResponseAuthentication in OpenSSH 8.7. Older configs use the old name, which is still accepted as an alias.)

    For user keys, Ed25519 is the default choice now. The bigger upgrade is hardware-backed keys. Since OpenSSH 8.2 you can generate ed25519-sk keys that live on a FIDO2 token, so the private key file on disk is useless without the physical device and a touch:

    ssh-keygen -t ed25519-sk -O resident -O verify-required -C "alice@laptop"

    verify-required (OpenSSH 8.4+) demands a PIN as well as a touch. Malware on a laptop can steal an ordinary key file in a second. With an -sk key it has to wait for the user to tap the token at the right moment, which is a much harder problem.

    Certificates instead of authorized_keys sprawl

    The real weakness of plain public keys is lifecycle. authorized_keys files accumulate keys from people who left two years ago, and nobody can say which key belongs to whom. OpenSSH has had its own certificate format since 5.4, and it’s underused. A CA signs a user’s public key with an identity, a list of principals and a validity window:

    ssh-keygen -s user_ca -I [email protected] -n alice,deploy -V +8h id_ed25519.pub

    The server trusts the CA rather than individual keys:

    TrustedUserCAKeys /etc/ssh/user_ca.pub
    AuthorizedPrincipalsFile /etc/ssh/principals/%u
    RevokedKeys /etc/ssh/revoked_keys

    /etc/ssh/principals/deploy lists which principals may log in as deploy. Now offboarding is mostly automatic, because certificates expire in hours, and the -I identity shows up in the auth log so you know which human used the shared account. Keep an emergency revocation path anyway (ssh-keygen -k builds a key revocation list), and protect the CA key like the crown jewels it is, ideally in a hardware token or a signing service.

    Host certificates solve the other half. Sign host keys with ssh-keygen -h, add one @cert-authority *.example.com line to clients’ known_hosts, and the “authenticity of host can’t be established” prompt that everyone trains themselves to click through goes away. Trust on first use is a bet that the first connection wasn’t intercepted. Certificates remove the bet.

    Prune the crypto, but don’t fight the defaults

    Modern OpenSSH defaults are good, and hand-written cipher lists copied from 2016 blog posts are often worse than the defaults they replace. OpenSSH 8.8 already disabled ssh-rsa signatures using SHA-1. OpenSSH 9.0 made the hybrid post-quantum [email protected] key exchange the default, and 10.0 moved to mlkem768x25519-sha256.

    Where explicit lists still pay off is when a specific weakness lands. Terrapin (CVE-2023-48795) was a prefix-truncation attack against the SSH transport that worked when [email protected], or a CBC cipher paired with an encrypt-then-MAC MAC, was negotiated. The fix was “strict kex” in OpenSSH 9.6, which needs both sides patched. Until clients caught up, removing the affected algorithms was a reasonable stopgap. Audit what you expose with ssh-audit rather than guessing:

    ssh-audit -p 22 server.example.com

    Decide who gets to try

    Authentication strength matters less when most accounts can’t log in at all. An allowlist is short and readable:

    AllowGroups ssh-users
    AllowTcpForwarding no
    AllowAgentForwarding no
    AllowStreamLocalForwarding no
    X11Forwarding no
    
    Match Group ssh-admins Address 10.20.0.0/16
        AllowTcpForwarding yes
    
    Match User backup
        ForceCommand /usr/local/bin/backup-receive
        PermitTTY no

    Forwarding is how an SSH foothold turns into a pivot into the rest of the network, so default it off and turn TCP forwarding back on only for admins coming from the internal range. There’s also DisableForwarding yes (OpenSSH 7.4+), which kills every kind of forwarding in one line, but it overrides the individual switches, so a Match block can’t selectively re-enable one of them without also setting DisableForwarding no there. Explicit switches are easier to reason about.

    Network placement still matters. Port knocking and moving sshd to port 2222 cut log noise, nothing more. Putting sshd behind a VPN or a bastion, or at least restricting port 22 to known address ranges at the firewall, actually shrinks who can reach the parser. And the parser has had bad days. regreSSHion (CVE-2024-6387) was a signal-handler race in sshd on glibc Linux that allowed unauthenticated remote code execution, affecting OpenSSH 8.5p1 through 9.7p1 until 9.8p1 fixed it. The temporary mitigation was LoginGraceTime 0, which trades RCE risk for an easy connection-exhaustion DoS. The xz backdoor (CVE-2024-3094) showed that sshd’s attack surface includes whatever libraries your distribution links into it. Nobody’s config file would have stopped either of those. Reachability would have.

    OpenSSH 9.8 also added PerSourcePenalties, which is on by default and makes sshd refuse connections for a while from addresses that repeatedly fail auth or crash a child. It’s a built-in, rough fail2ban.

    Log enough to answer questions later

    Set LogLevel VERBOSE. At that level sshd logs the fingerprint of the key that authenticated each session, and for certificates the key ID and serial. Without it, “which key did this login use?” has no answer. Part 3 of this series covers auditd and shipping those logs somewhere an intruder can’t edit them.

    What I’d actually do on a new server, in order: allowlist users, kill passwords, move people to certificates, keep forwarding off, limit reachability, and patch sshd quickly when the next CVE lands. Disabling root login comes for free along the way.

    Other parts of the Hardening a Linux server series: Part 2: Least privilege with systemd sandboxing, Part 3: auditd and what to log, Part 4: Patching without fear.

  • Horcrux dev log 3: back after nine months, 92 failed CI runs

    Horcrux dev log 3: back after nine months, 92 failed CI runs

    Part 3 in the Horcrux dev log series.

    Backdated, written in November 2026 from git and CI records. Part 2 stopped at an unfinished ISO on December 31, 2025.

    I came back on September 24, 2026 with two goals: get CI to really run, and build Horcrux the Gentoo way.

    The Gentoo goal came from a mistake. At some point the Dockerfile had turned into standard Rust boilerplate: FROM rust, cargo build, FROM debian-slim. It had drifted into boilerplate that ignored what the project is for. I fixed it on September 24 and wrote a rule into the repo notes so it can’t come back. The rule in the repo notes reads:

    A Dockerfile doing FROM rust:* -> cargo build -> FROM debian:*-slim
    is wrong. It is a regression. Fixed once already on 2026-09-24.

    Now the Dockerfile builds through real Portage against the project’s own ebuild, and the ebuild is the canonical install path.

    The numbers

    Pulled from the Forgejo API, September 24 to 30:

    workflow runs: 121
      success 12
      failure 92
      cancelled 11
      skipped 6
    ci.yml: 8 success, 71 failure
    docs.yml: 4 success, 19 failure

    88 commits that week, a lot of them fixes for whatever the last run complained about.

    What kept breaking

    The ebuild first. It depended on virtual/rust, which doesn’t exist (the package is dev-lang/rust), so emerge said “there are no ebuilds to satisfy”. dev-util/trunk doesn’t exist either, so the UI bundler gets installed with cargo during the build. qemu’s REQUIRED_USE says vte? ( gtk ) and I’d enabled vte alone. The Xen tools ebuild in that Gentoo snapshot died in src_prepare on an unset version variable; Horcrux only does KVM, so I dropped the Xen flag instead of fighting it. A pile of qemu and spice USE flags dead-ended in a masked libglvnd, and a headless VM host has no use for audio or GL anyway.

    Then the runners. Actions that weren’t mirrored on the registry my Forgejo uses. upload-artifact v4 not supported (“GHESNotSupportedError”). A runner with a full disk. A clippy step killed by the runner. A 30 minute ephemeral timeout that killed legitimate Gentoo builds. A full emerge that hit a 3 hour limit until I gave the CI image light USE flags. Several commit messages just say “retrigger” because the last failure was the infrastructure and not the code.

    Fake stuff that came out

    Real tests started finding placeholders. The login form wasn’t wired to the auth API. Disk usage in the metrics was a hardcoded 65%; it now reads statvfs. The Kubernetes join code accepted a made-up CA cert hash and now refuses it. HA fencing for IPMI, iLO, iDRAC and SNMP PDUs, parallel volume restore and live-migration pre-flight checks were stubs and got real code on the 30th. I don’t know yet how any of that behaves on actual hardware. release.yml only built x86_64 although the README claimed three architectures, so it now cross-compiles aarch64 and riscv64.

    On the 25th and 26th the API integration suite reached 22 of 22. It’s the first green test run I can find in the history.

    State

    The Gentoo build works in CI with trimmed USE flags, the tests run, and about one run in ten was green. The README still overpromised, and I rewrote its status line on October 21 (part 5).

    Design decisions

    Gentoo first. My reason is freedom to choose. USE flags are the main part: they let an installer choose which virtualization backends, container runtimes and NAS protocols get compiled in, and the repo notes call that the main advantage over Proxmox. Compiling for specific hardware is the secondary reason. For background, I was a Debian developer from 1999 to 2006 and I’ve used Gentoo since. I’m blunt about it in the repo notes: generic Debian or Ubuntu conventions are the wrong default here.

    Dropping Xen. The commit says Horcrux is a headless KVM-only VM host and has no need for Xen guest support. The same reasoning removed audio and GL from qemu: no local display on a headless host.

    Architectures: amd64, arm64 and riscv64. The README dropped ppc64le the same day as out of scope. That’s my call, and I don’t have the reason written down. The riscv64 notes do say it needs real hardware for validation.

    Light USE flags in CI: the full emerge hit a 3 hour limit, so the CI smoke-test image builds with fewer flags. A speed trade, and it means CI doesn’t build the full-feature package.

    Other parts: Part 1, Part 2, Part 4, Part 5, Part 6. The series continues with the latest Horcrux dev log post.

  • Patronus dev log 3: back in September, with 43 red CI runs and no green ones

    Patronus dev log 3: back in September, with 43 red CI runs and no green ones

    Part 3 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 2 ended with nine months of silence.

    I picked Patronus back up the same day as Horcrux, September 24, 2026. The first job was the same: make CI run. Patronus got a Rust CI workflow that day (fmt, clippy, tests, an eBPF check, a full-workspace build). It failed. Over September 24 to 30 the Forgejo record shows 57 runs: 43 failures, 7 cancelled, 7 skipped, 0 successes. Of those, rust-ci.yml had 42 failures.

    The first failures were the dull kind: a missing libmnl-dev, a missing libelf, no linker on the runner image, and a real bug. An eBPF map-create call took its name as i8 and needed u8, which was blocking the detection and CNI crates from building at all.

    The Dockerfile again

    Same mistake as in Horcrux, found the same day. The Dockerfile had become a generic FROM rust build on a Debian slim image. The repo notes now say so, in short:

    A generic FROM rust -> cargo build -> debian slim Dockerfile is wrong.
    It crept in once and was fixed on 2026-09-24.

    Patronus builds through real Portage against its own overlay. The ebuild kept breaking, in familiar ways: virtual/rust doesn’t exist (same bug as Horcrux, the commit says so), acct-user and acct-group variable names were wrong, a USE flag was called carp in the code and ucarp in IUSE, the meta USE flags defaulted on with no sub-backend, and the ebuild forwarded cargo features that don’t exist (aliases, tc, prometheus). Building Rust from source inside the Docker build took 2 hours 45 minutes, so I switched to dev-lang/rust-bin. And the Gentoo Docker build got capped at 60 minutes and made non-blocking while I sorted it out.

    On the 29th the installer picked up root-on-ZFS, preferred when the ZFS tools are present.

    Design decisions

    Gentoo and USE flags: Gentoo is the core design point. My reason is freedom to choose. USE flags come first because they let you pick which features (web UI, eBPF/XDP, SD-WAN, detection, cloud integrations) get compiled in, and the repo notes call that the main advantage over pfSense and OPNsense. Hardware specific compiles are the second reason. A container build must pass USE flags through to emerge rather than compile a fixed set.

    rust-bin over source Rust: the from-source build took 2h45m (the commit’s number).

    Architecture: the repo notes have one rule: one SQLite store as the single source of truth, one “Applier” per subsystem, one apply step at boot, and no extra in-memory list, fixture or config file that only one process knows about.

    State on September 30

    57 runs, none green, and about 60 commits. The Gentoo build was non-blocking at the end of the week and still no run was green.

    Other parts: Part 1, Part 2, Part 4, Part 5.