Category: Security

  • SSH hardening beyond “disable root login”

    SSH hardening beyond “disable root login”

    Part 1 of 4 in the Hardening a Linux server series.

    Every hardening guide starts with PermitRootLogin no, and plenty of them stop there. That setting is fine. It’s also the least interesting thing about securing SSH, because an attacker who can log in as a sudo-capable user with a password has root anyway. The things that actually decide how exposed your SSH daemon is are how users prove who they are, how the server proves who it is, which crypto you still accept, and who is allowed to try at all.

    Start from what sshd is really doing

    Sequence diagram: laptop sends a public key to an SSH CA, receives an 8 hour certificate, presents it to sshd, which checks the CA signature, principals, revocation list and validity before granting the session

    Before changing anything, look at the effective configuration rather than the file. Distributions now ship Include /etc/ssh/sshd_config.d/*.conf near the top of sshd_config, and for most keywords sshd uses the first value it reads, not the last. I’ve watched people append PasswordAuthentication no to the bottom of the main file while a cloud-init drop-in earlier in the include order said yes, and the drop-in won.

    sudo sshd -T | grep -Ei 'passwordauth|kbdinteractive|permitroot|pubkeyauth|authenticationmethods'
    sudo sshd -T -C user=deploy,host=10.0.0.5,addr=10.0.0.5

    The second form evaluates Match blocks for a specific connection, which is the only sane way to test conditional rules. Put your own settings in a drop-in with a low number, like /etc/ssh/sshd_config.d/00-hardening.conf, so they load first.

    Keys, and better keys

    Passwords on an internet-facing sshd are a credential-stuffing target forever. Turn off both password paths, since keyboard-interactive will happily do password auth through PAM even when PasswordAuthentication is off:

    PasswordAuthentication no
    KbdInteractiveAuthentication no
    PubkeyAuthentication yes
    PermitRootLogin no
    MaxAuthTries 3
    LoginGraceTime 20

    (KbdInteractiveAuthentication replaced ChallengeResponseAuthentication in OpenSSH 8.7. Older configs use the old name, which is still accepted as an alias.)

    For user keys, Ed25519 is the default choice now. The bigger upgrade is hardware-backed keys. Since OpenSSH 8.2 you can generate ed25519-sk keys that live on a FIDO2 token, so the private key file on disk is useless without the physical device and a touch:

    ssh-keygen -t ed25519-sk -O resident -O verify-required -C "alice@laptop"

    verify-required (OpenSSH 8.4+) demands a PIN as well as a touch. Malware on a laptop can steal an ordinary key file in a second. With an -sk key it has to wait for the user to tap the token at the right moment, which is a much harder problem.

    Certificates instead of authorized_keys sprawl

    The real weakness of plain public keys is lifecycle. authorized_keys files accumulate keys from people who left two years ago, and nobody can say which key belongs to whom. OpenSSH has had its own certificate format since 5.4, and it’s underused. A CA signs a user’s public key with an identity, a list of principals and a validity window:

    ssh-keygen -s user_ca -I [email protected] -n alice,deploy -V +8h id_ed25519.pub

    The server trusts the CA rather than individual keys:

    TrustedUserCAKeys /etc/ssh/user_ca.pub
    AuthorizedPrincipalsFile /etc/ssh/principals/%u
    RevokedKeys /etc/ssh/revoked_keys

    /etc/ssh/principals/deploy lists which principals may log in as deploy. Now offboarding is mostly automatic, because certificates expire in hours, and the -I identity shows up in the auth log so you know which human used the shared account. Keep an emergency revocation path anyway (ssh-keygen -k builds a key revocation list), and protect the CA key like the crown jewels it is, ideally in a hardware token or a signing service.

    Host certificates solve the other half. Sign host keys with ssh-keygen -h, add one @cert-authority *.example.com line to clients’ known_hosts, and the “authenticity of host can’t be established” prompt that everyone trains themselves to click through goes away. Trust on first use is a bet that the first connection wasn’t intercepted. Certificates remove the bet.

    Prune the crypto, but don’t fight the defaults

    Modern OpenSSH defaults are good, and hand-written cipher lists copied from 2016 blog posts are often worse than the defaults they replace. OpenSSH 8.8 already disabled ssh-rsa signatures using SHA-1. OpenSSH 9.0 made the hybrid post-quantum [email protected] key exchange the default, and 10.0 moved to mlkem768x25519-sha256.

    Where explicit lists still pay off is when a specific weakness lands. Terrapin (CVE-2023-48795) was a prefix-truncation attack against the SSH transport that worked when [email protected], or a CBC cipher paired with an encrypt-then-MAC MAC, was negotiated. The fix was “strict kex” in OpenSSH 9.6, which needs both sides patched. Until clients caught up, removing the affected algorithms was a reasonable stopgap. Audit what you expose with ssh-audit rather than guessing:

    ssh-audit -p 22 server.example.com

    Decide who gets to try

    Authentication strength matters less when most accounts can’t log in at all. An allowlist is short and readable:

    AllowGroups ssh-users
    AllowTcpForwarding no
    AllowAgentForwarding no
    AllowStreamLocalForwarding no
    X11Forwarding no
    
    Match Group ssh-admins Address 10.20.0.0/16
        AllowTcpForwarding yes
    
    Match User backup
        ForceCommand /usr/local/bin/backup-receive
        PermitTTY no

    Forwarding is how an SSH foothold turns into a pivot into the rest of the network, so default it off and turn TCP forwarding back on only for admins coming from the internal range. There’s also DisableForwarding yes (OpenSSH 7.4+), which kills every kind of forwarding in one line, but it overrides the individual switches, so a Match block can’t selectively re-enable one of them without also setting DisableForwarding no there. Explicit switches are easier to reason about.

    Network placement still matters. Port knocking and moving sshd to port 2222 cut log noise, nothing more. Putting sshd behind a VPN or a bastion, or at least restricting port 22 to known address ranges at the firewall, actually shrinks who can reach the parser. And the parser has had bad days. regreSSHion (CVE-2024-6387) was a signal-handler race in sshd on glibc Linux that allowed unauthenticated remote code execution, affecting OpenSSH 8.5p1 through 9.7p1 until 9.8p1 fixed it. The temporary mitigation was LoginGraceTime 0, which trades RCE risk for an easy connection-exhaustion DoS. The xz backdoor (CVE-2024-3094) showed that sshd’s attack surface includes whatever libraries your distribution links into it. Nobody’s config file would have stopped either of those. Reachability would have.

    OpenSSH 9.8 also added PerSourcePenalties, which is on by default and makes sshd refuse connections for a while from addresses that repeatedly fail auth or crash a child. It’s a built-in, rough fail2ban.

    Log enough to answer questions later

    Set LogLevel VERBOSE. At that level sshd logs the fingerprint of the key that authenticated each session, and for certificates the key ID and serial. Without it, “which key did this login use?” has no answer. Part 3 of this series covers auditd and shipping those logs somewhere an intruder can’t edit them.

    What I’d actually do on a new server, in order: allowlist users, kill passwords, move people to certificates, keep forwarding off, limit reachability, and patch sshd quickly when the next CVE lands. Disabling root login comes for free along the way.

    Other parts of the Hardening a Linux server series: Part 2: Least privilege with systemd sandboxing, Part 3: auditd and what to log, Part 4: Patching without fear.

  • Session management: cookies, SameSite, revocation, and where JWTs go wrong

    Session management: cookies, SameSite, revocation, and where JWTs go wrong

    Part 4 of 4 in the Authentication done right series. Part 1: Password storage in 2026. Part 2: MFA that survives phishing. Part 3: OAuth 2.0 vs OpenID Connect.

    This series started with storing passwords, moved through MFA, then OAuth and OpenID Connect. All of that work produces one thing at the end: a session. From then on, every request is authenticated by whatever token the browser sends, and the password and the security key are no longer involved. If the session is weak, everything before it was ceremony.

    What a session token needs to be

    Bar chart: a server-side session stops working immediately on logout, a short JWT with rotating refresh tokens works up to 10 minutes, a 15 minute JWT up to 15 minutes, and a 24 hour JWT without a denylist up to 1,440 minutes

    Whatever form it takes, a session token has to be hard to guess, tied to one login, and revocable. For a classic server side session that means a random identifier with at least 128 bits from a CSPRNG, stored in a server side table or cache, with all the state kept on the server. OWASP’s floor is 64 bits of entropy. I don’t see a reason to stop there when 128 costs nothing.

    Two lifecycle rules catch a lot of bugs. Issue a new session identifier on every privilege change, especially login, or you’re open to session fixation, where an attacker plants a known session ID in the victim’s browser before they log in and then shares the authenticated session. And enforce both an idle timeout and an absolute lifetime, so a session that’s been stolen doesn’t stay useful for months just because something keeps it busy.

    Cookie attributes that matter

    The browser is where sessions get stolen, and cookie attributes are how you tell the browser to protect them. A good default looks like this:

    Set-Cookie: __Host-session=Xq3...; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=28800

    Secure keeps the cookie off plaintext HTTP. HttpOnly stops JavaScript from reading it, which won’t prevent an XSS payload from making requests as the user but does stop it from shipping the token off to be used from elsewhere, hours later. The __Host- prefix is underused. The browser only accepts a cookie with that name if it has Secure, Path=/ and no Domain attribute, which means a sibling subdomain can’t overwrite or shadow it. That closes off a whole family of cookie tossing attacks for free.

    What SameSite does and doesn’t do

    SameSite controls whether the browser attaches a cookie to requests that start on another site. Strict never sends it cross site, including when a user follows a link to your app from an email, which makes them look logged out. Lax sends it on top level GET navigations but not on cross site POSTs, iframes or subresource requests. None always sends it and requires Secure. Chrome began treating cookies with no SameSite attribute as Lax in 2020, and other browsers have moved in similar directions, though not identically.

    That made classic CSRF much harder, but I wouldn’t remove CSRF tokens because of it. Three reasons. Lax still sends cookies on top level GETs, so any state changing GET endpoint is still exposed. Chrome’s default-Lax behaviour includes a two minute window in which a freshly set cookie without an explicit attribute is still sent on cross site top level POSTs, to avoid breaking some login flows. And “site” means registrable domain, not origin. evil.example.com and app.example.com are the same site, so a subdomain takeover or an XSS on a marketing subdomain bypasses SameSite completely. Keep the anti-CSRF token or check the Origin header on state changing requests, and treat SameSite as defence in depth.

    Revocation is the real requirement

    Logging out has to end the session on the server. A logout that only deletes the cookie leaves the token valid for whoever copied it. Users also need “sign out of all devices”, support staff need to kill sessions for a compromised account, and a password change or MFA reset should invalidate every other session. With server side sessions all of that is a delete query.

    This is where JWTs get awkward. A signed, self contained token is valid until its exp unless the server checks something on every request. The usual answer is short lived access tokens (5 to 15 minutes) plus a refresh token that the server does track. RFC 9700 describes refresh token rotation: each refresh returns a new refresh token and invalidates the old one, and if an old one is ever presented again you assume theft and revoke the whole family.

    def refresh(presented_token):
        rec = db.refresh_tokens.find(hash(presented_token))
        if rec is None:
            raise Unauthorized()
        if rec.used:
            # a rotated token came back: someone has a copy
            db.refresh_tokens.revoke_family(rec.family_id)
            alert("refresh token reuse", rec.user_id)
            raise Unauthorized()
        rec.used = True
        new_rt = issue_refresh_token(rec.user_id, family_id=rec.family_id)
        return issue_access_token(rec.user_id, ttl=600), new_rt

    Notice that this is a database lookup on every refresh. Once you need revocation, and you will, you have server side state again. That’s fine. Just be honest that a JWT session isn’t really stateless.

    Where JWTs go wrong

    Most JWT vulnerabilities come from verification code trusting the token’s own header. The alg field says how the token claims to be signed, and early libraries used it to choose the verification method. That allowed alg: none tokens with no signature at all, and algorithm confusion, where a token signed with HS256 using the server’s RSA public key as the HMAC secret passed verification. CVE-2015-9235 in Node’s jsonwebtoken is the well known instance of the latter. The fix is to pin the expected algorithms on the server and never read them from the token:

    claims = jwt.decode(
        token,
        public_key,
        algorithms=["ES256"],
        audience="https://api.example.com",
        issuer="https://auth.example.com",
    )

    Library bugs happen even when you do this. CVE-2022-21449, nicknamed Psychic Signatures, made Java 15 through 18 accept ECDSA signatures with both values set to zero as valid for any message, which broke ES256 JWT verification on those runtimes until the April 2022 patch. Keep the JWT library and the runtime patched, and test that a garbage signature is rejected.

    RFC 8725, the JWT Best Current Practices, covers the rest: validate iss and aud, use explicit typing so one kind of token can’t be replayed as another, and don’t put secrets in the payload, since a signed JWT is encoded, not encrypted.

    Storage is the last trap. Single page apps often put tokens in localStorage, where any XSS can read them and send them elsewhere. A backend-for-frontend that holds the tokens server side and gives the browser an HttpOnly, __Host- cookie avoids that, and it fits the OAuth guidance from the previous part of this series.

    What I’d use

    For a typical web app with its own backend, I’d use opaque server side sessions in a hardened cookie. They’re simpler, revocation is trivial, and nothing about them is clever. I reach for JWTs when a token really has to be verified by services that can’t share a session store, and then I keep them short lived, pin the algorithm, and pair them with rotating refresh tokens. Choosing JWTs because they sound modern is how teams end up rebuilding server side sessions badly, one denylist at a time.

  • OAuth 2.0 vs OpenID Connect: what each one solves, PKCE, and the end of the implicit flow

    OAuth 2.0 vs OpenID Connect: what each one solves, PKCE, and the end of the implicit flow

    Part 3 of 4 in the Authentication done right series. Part 1: Password storage in 2026. Part 2: MFA that survives phishing.

    The first two parts of this series were about proving a person is who they say they are. This one is about what happens next, when one piece of software needs to act on that person’s behalf or another service needs to know who logged in. Those are two different problems, and a lot of security bugs I’ve seen come from using the tool for one to solve the other.

    OAuth 2.0 is about delegation

    Sequence diagram of the OIDC authorization code flow with PKCE between browser, client backend, authorization server and API, showing the code challenge, the back-channel token exchange with the verifier, the ID token used for login and the access token sent to the API

    OAuth 2.0 (RFC 6749) answers one question: how does a user let an application access an API without giving the application their password? The user authenticates to an authorization server, approves a set of scopes, and the application gets an access token it presents to a resource server. The resource server cares that the token is valid and grants the right scopes. It doesn’t necessarily care, or learn, who the user is.

    An access token is a capability, like a valet key for a car. It says “the holder may do these things”. It isn’t a statement about identity, and it isn’t addressed to your application in any cryptographic sense. RFC 6750 calls them bearer tokens because whoever bears one can use it.

    OpenID Connect is about authentication

    OpenID Connect (OIDC Core 1.0, finalised in 2014) is a layer on top of OAuth that adds what OAuth deliberately left out. When the client asks for the openid scope, the authorization server also returns an ID token: a signed JWT with an issuer (iss), a subject (sub), an audience (aud) that is your client ID, an expiry, and a nonce that ties it to the login request your app started.

    The ID token is addressed to your application. That’s what makes it safe to use for login, and it’s why “Sign in with” buttons should be OIDC rather than plain OAuth plus a call to some profile endpoint.

    Token confusion

    The classic mistake is using an OAuth access token as proof of login. The app receives an access token, calls the provider’s user info endpoint, sees a user ID, and logs that user in. The problem is that the app never checked that the token was issued to it. Any other application that integrates with the same provider receives access tokens for its own users. If one of those applications is malicious, or simply compromised, it can replay a user’s token against your login endpoint and become that user in your app. This pattern hit a lot of mobile apps that implemented social login in the early 2010s, and I still see it in internal tools.

    The fix is to authenticate with the ID token and validate it properly, or, if you must accept access tokens, to use the provider’s token introspection and verify the client the token was issued to. Validating an ID token means checking everything, not just the signature:

    import hmac
    import jwt
    from jwt import PyJWKClient
    
    jwks = PyJWKClient("https://idp.example.com/.well-known/jwks.json")
    
    def validate_id_token(id_token, expected_nonce):
        key = jwks.get_signing_key_from_jwt(id_token).key
        claims = jwt.decode(
            id_token,
            key,
            algorithms=["RS256"],          # never take the alg from the token
            audience=CLIENT_ID,            # issued to us, not some other app
            issuer="https://idp.example.com",
            options={"require": ["exp", "iat", "iss", "aud", "sub", "nonce"]},
        )
        if not hmac.compare_digest(claims["nonce"], expected_nonce):
            raise ValueError("nonce mismatch")
        return claims["sub"]

    Key the user record on iss plus sub, not on email. Email addresses get reassigned and some providers let users set unverified ones.

    The reverse confusion matters too. APIs should not accept ID tokens as access tokens. An ID token’s audience is the client, not the API, and treating it as an API credential throws away scopes entirely.

    The authorization code flow, with PKCE

    The standard flow redirects the user to the authorization server, which redirects back with a short lived authorization code. The client then exchanges that code for tokens over a direct back channel request. Tokens never pass through the browser’s address bar.

    The weak point was the code itself. On mobile, other apps could register the same custom URL scheme and intercept the redirect. PKCE (RFC 7636) fixes this. The client generates a random secret, sends only its hash in the authorization request, and proves it knows the original when redeeming the code:

    import base64, hashlib, secrets
    
    def b64url(b):
        return base64.urlsafe_b64encode(b).rstrip(b"=").decode()
    
    code_verifier = b64url(secrets.token_bytes(32))
    code_challenge = b64url(hashlib.sha256(code_verifier.encode()).digest())
    
    # authorization request
    #   ...&code_challenge=<code_challenge>&code_challenge_method=S256
    # token request
    #   ...&code=<code>&code_verifier=<code_verifier>

    An attacker who steals the code doesn’t have the verifier, so the code is useless. PKCE also blocks authorization code injection, where an attacker pushes a code from their own session into the victim’s client. RFC 9700, the OAuth 2.0 Security Best Current Practice published in January 2025, recommends PKCE for every client type, confidential server side apps included. Use S256, never plain.

    Why the implicit flow is gone

    The implicit flow was designed for browser apps in 2012, when cross origin requests to a token endpoint weren’t practical. The authorization server put the access token directly in the redirect URL fragment. That meant tokens ended up in browser history, could leak through referrers and open redirects, could be injected by an attacker, and couldn’t be bound to the client that requested them. There was also no safe way to issue refresh tokens.

    CORS has been universal for a long time, so none of those trade-offs are needed anymore. RFC 9700 says clients should not use the implicit grant, and OAuth 2.1 drops it entirely, along with the resource owner password credentials grant. Single page apps should use the code flow with PKCE. Better still, use a backend-for-frontend: a small server side component that performs the OAuth flow, keeps tokens out of JavaScript entirely, and gives the browser a session cookie. That’s where the next part of this series picks up.

    Smaller things that cause real incidents

    Redirect URIs must be matched exactly. Prefix or wildcard matching has led to codes being sent to attacker controlled paths more than once. The state parameter still matters for CSRF on the redirect endpoint if you’re not relying on PKCE for that. If your client talks to more than one authorization server, check the iss parameter in the authorization response (RFC 9207) to stop mix-up attacks, where one server’s code is redeemed at another. And for high value APIs, consider sender constrained tokens with DPoP (RFC 9449) or mutual TLS (RFC 8705), so a stolen access token is useless without the client’s private key.

    Keep access tokens short lived and narrowly scoped, and make sure APIs can tell them apart from ID tokens. Most OAuth problems I’ve cleaned up were not protocol flaws. They were one token type being trusted for a job it was never meant to do.

    Other parts: Part 1: Password storage in 2026, Part 2: MFA that survives phishing, Part 4: Session management.