Category: Linux

  • Horcrux dev log 6: clustering, an ISO you can install from, and real laptops

    Horcrux dev log 6: clustering, an ISO you can install from, and real laptops

    Part 6 in the Horcrux dev log series.

    Backdated: written in November 2026 from git, CI and my QA ledger. Part 5 was the audit.

    With the basics fixed I moved on to clustering. In five days the log shows an HA supervisor (failure detection, fencing, recovery, self-fencing), cluster create and join, guest save state on host shutdown (QEMU migrate-to-file for VMs, CRIU for containers), draining a node with live moves, SDN zones, Ceph management, deduplicating backups, PCI and vGPU passthrough and a Redfish fencing driver.

    On October 24 live migration got a hard block between AMD and Intel CPUs, and the drain planner never moves a running guest across vendors. Cold migration is still allowed.

    From the Forgejo API for October 22 to November 6: 725 runs, 493 success, 78 failure, 154 cancelled, about 450 commits.

    ha-failover.yml 68 success, 4 failure
    vm-migration.yml 32 success, 0 failure
    guest-power.yml 30 success, 7 failure
    live-iso.yml 21 success, 4 failure, 26 cancelled

    I haven’t checked every cancelled run, but newer pushes superseding a run in progress account for many of them.

    October 27: an ISO that boots

    The December ISO builder never got finished. This one has a different design. A Docker-only builder (Gentoo stage3 plus the official binhost, dracut with dmsquash-live, GRUB for BIOS and UEFI in a hybrid ISO), and a QEMU test that boots it by UEFI and USB and by BIOS and CD, runs an unattended install, reboots into the result and probes the web API. A daily job publishes only if the tests pass. First version: 31 files, about 1,000 lines.

    By the 28th it had a text installer, an answer-file mode, no default password, key-only SSH and aarch64 images. This is the boot menu from that week, taken from the repo’s docs folder at the time:

    The Horcrux installer boot menu in GRUB, committed October 28, 2026
    Boot menu as committed on October 28, 2026 (docs/images/boot in the repo).

    And the installer’s file system page, committed the same day:

    Horcrux text installer file system page with ext4 as the default
    File system page as committed October 28. The default is ext4, which was wrong and got changed within days.

    The default is ext4. I’d decided ZFS should be the default, with an automatic XFS fallback under 8 GB of RAM or when there’s no ZFS module for the kernel.

    T2 laptops, tablets, encrypted ZFS

    There’s a GNOME desktop edition on top of the server core and hardware profiles for an Apple T2 MacBook and a Surface tablet. The installer learned encrypted ZFS root with boot environments, and hibernate on encrypted ZFS. The initramfs has to resume from the LUKS swap before it imports the pool, because importing first and resuming into it corrupts the pool. A QEMU test does install, passphrase, hibernate, power off, boot, passphrase, same process still running, clean scrub.

    What real machines found

    QEMU didn’t catch several things. The November 3 commits are the list the laptops produced.

    The clock: a live system with a months-off RTC can’t do TLS, so Tailscale and package syncs failed. The installer now writes the hardware clock only from a verified clock, and the tests boot QEMU with -rtc base= months in the past. The size-trimming step on the desktop image kept deleting /usr/include and every static library, so installed systems had no C compiler and ZFS couldn’t rebuild its module on a kernel update. A zfs-kmod from a cached binary package had been built for a different kernel. Plain XFS and ext4 installs didn’t write /etc/fstab, so /boot wasn’t mounted and kernel updates landed on the root filesystem. GDM login started two keyrings. The disk passphrase could be reused as an account password, so the installer now refuses that. And the desktop had apps missing from the launcher because their binary didn’t exist, no sound, dead volume keys, and a Log Out button that did nothing, because the session was split across two D-Bus buses.

    Most of these got a gate in the QEMU suite afterwards (months-off clock, compiler present, module builds, fstab, keyring, password reuse, the desktop session items). On November 3 I also fixed CI reporting green after a failed step (“green-but-failed 8637” in the commit), so some earlier runs looked healthier than they were.

    Where it stands

    This week I moved on to a lab of three old laptops to prove the HA claims. The goal in the lab plan: every scenario of the HA suite green ten runs in a row, failover under 120 seconds, no data loss, never two copies of a guest. As of this writing that’s 0 of 10. The old manager took 126 seconds to fail over after a hard crash, over the limit, and a one-node partition ran two copies of a guest for several seconds. Both are logged as bugs with fixes pending. So I can’t say Horcrux has HA yet. The next post is about that.

    The aarch64 ISO build takes hours under emulation and hit the runner’s time cap on November 6, so it moved to a runner with a longer limit.

    Design decisions

    OpenRC is the only init on the ISOs. systemd wants to run the world, and that comes at a cost: feature bloat.

    ZFS by default, with an automatic XFS fallback when RAM is under 8 GB or the kernel has no ZFS module. The installer shows its reason on the disk page (“XFS instead of ZFS: only 4.0 GB RAM (ZFS wants 8 GB or more)”). I want ZFS boot environments and an automatic snapshot before every update for recovery, bit rot protection and resilience. ext4 is never the default, only a manual choice. Gentoo recommends XFS, and ZFS is just superior tech.

    Encrypted ZFS needs a static grub.cfg and a kernel per boot environment under /boot, because GRUB can’t read an encrypted ZFS pool.

    Hibernate goes to a separate LUKS2 swap partition and not a zvol, because OpenZFS doesn’t support hibernating onto its own zvol and doing it anyway risks the pool. For the Surface XFS layout the notes reject a separate LUKS swap, since it would need a second keyslot or a second prompt.

    Surface tablets get XFS plus LUKS and a light touch desktop, because the docs say a GNOME-class desktop doesn’t fit in 4 GB.

    Server first. On November 3 I wrote down the rule: “it should work on a headless server too, it shouldn’t be a desktop-only OS”. The desktop is a layer on top of a headless core and personal apps live in per-machine layers, not the public images.

    Live migration between AMD and Intel is refused and cold migration is allowed (see Oct 24 above).

    Other parts: Part 1, Part 2, Part 3, Part 4, Part 5. The series continues with the latest Horcrux dev log post.

  • Container image hardening: minimal images, non-root, read-only filesystems

    Container image hardening: minimal images, non-root, read-only filesystems

    Most container images in production are a full Linux distribution with an application on top. A Debian or Ubuntu base, a package manager, a shell, curl, and an application running as root with a writable filesystem. It works, it’s easy to debug, and it hands an attacker who gets code execution everything they need for the next step.

    Hardening an image is a handful of small changes, each removing a capability the application never needed, and that an attacker would very much like to have. None of them are new, and every one of them gets skipped all the time.

    Start with less: minimal and distroless bases

    Side-by-side comparison of a typical container image and a hardened one: distroless base, no shell, non-root UID 65532, read-only root with noexec tmpfs, all capabilities dropped, no privilege escalation, seccomp RuntimeDefault and pinned digests, with the attacker capability each change removes

    The fastest win is shipping fewer things. A typical debian or ubuntu base pulls in a hundred-odd packages, and your scanner will report CVEs against all of them, whether or not your app ever touches them. That noise has a real cost, which I covered in the vulnerability management series earlier this week. Every finding in a package you don’t use is triage time you’re not spending on one that matters.

    There are a few levels of minimal:

    • scratch is empty. Good for fully static binaries, like Go built with CGO_ENABLED=0. You have to bring your own CA certificates and timezone data if you need them.
    • Google’s distroless images contain the runtime pieces an app needs (libc, CA certs, tzdata, sometimes a language runtime) and no shell or package manager.
    • Wolfi-based images from Chainguard and minimal Alpine images sit somewhere in between, with a package manager available at build time.

    For a Go service, a multi-stage build onto distroless looks like this:

    FROM golang:1.25 AS build
    WORKDIR /src
    COPY go.mod go.sum ./
    RUN go mod download
    COPY . .
    RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app
    
    FROM gcr.io/distroless/static-debian12:nonroot
    COPY --from=build /out/app /app
    USER 65532:65532
    ENTRYPOINT ["/app"]

    The compiler, the source and the module cache stay in the build stage. The final image is your binary plus a few megabytes of support files.

    The objection I always hear is debugging. “How do I exec into it without a shell?” On Kubernetes you don’t need to: an ephemeral debug container shares the target’s process namespace and brings its own tools.

    kubectl debug -it pod/payments-api-7d9f8c --image=busybox:1.36 --target=app

    That’s better than a shell baked into every image anyway, because the tools only exist while someone is actually debugging.

    Don’t run as root

    Containers running as UID 0 are the default, and it’s a bad default. Root inside a container isn’t the same as root on the host, since namespaces and dropped capabilities limit it, but it’s much closer than people assume, and it turns every container runtime bug into a bigger problem.

    CVE-2019-5736 is the classic example. A malicious process in a container could overwrite the host’s runc binary and get root on the host the next time runc ran. The exploit needed root inside the container. A container running as an unprivileged user wasn’t exploitable that way.

    Set the user in the image, with a numeric UID so Kubernetes can verify it:

    USER 65532:65532

    Then enforce it at the platform level so an image that forgets doesn’t get scheduled. In Kubernetes that’s runAsNonRoot: true in the security context, which makes the kubelet refuse to start a container whose user resolves to 0.

    If the app needs to bind to port 80 or 443, don’t give it root. Bind to 8080 and let the Service or the load balancer map the port, or add only NET_BIND_SERVICE.

    Make the root filesystem read-only

    An attacker with code execution usually wants to write something: a downloaded tool, a modified config, a web shell, a cron entry. A read-only root filesystem takes that away, and it catches a surprising number of bugs where an app was quietly writing logs or caches into its own image.

    Most apps need somewhere to write, usually /tmp or a cache directory. Give them exactly that as a tmpfs or an emptyDir volume and nothing else. With plain Docker:

    docker run --read-only \
      --tmpfs /tmp:rw,noexec,nosuid,size=64m \
      --cap-drop=ALL \
      --security-opt=no-new-privileges:true \
      --user 65532:65532 \
      registry.example.com/payments-api@sha256:<digest>

    The noexec on the tmpfs matters. A writable directory that can also execute binaries is half of what an attacker needed from a writable root.

    The Kubernetes security context

    All of this comes together in the pod spec. This is the block I start every deployment from:

    securityContext:
      runAsNonRoot: true
      runAsUser: 65532
      runAsGroup: 65532
      seccompProfile:
        type: RuntimeDefault
    containers:
      - name: app
        image: registry.example.com/payments-api@sha256:<digest>
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]
        volumeMounts:
          - name: tmp
            mountPath: /tmp
    volumes:
      - name: tmp
        emptyDir:
          medium: Memory
          sizeLimit: 64Mi

    allowPrivilegeEscalation: false sets the kernel’s no_new_privs flag, so setuid binaries can’t raise privileges. Dropping all capabilities removes things like CAP_NET_RAW, which some runtimes still grant by default and which lets a process craft raw packets. RuntimeDefault seccomp blocks a set of syscalls that normal apps never make.

    Kubernetes Pod Security Admission’s restricted profile enforces most of this for you: non-root, no privilege escalation, all capabilities dropped and a seccomp profile. It doesn’t require a read-only root filesystem, so that one you have to enforce with your own policy (Kyverno, Gatekeeper or a ValidatingAdmissionPolicy). Turn restricted on in warn mode for a namespace first, see what breaks, and fix those workloads before switching to enforce.

    Pin what you ship, rebuild what you pin

    Reference images by digest in deployment manifests, not by tag. A tag is a mutable pointer, and :1.42.0 can be repushed. The digest is a hash of the content and can’t.

    Pinning has a cost, though. A pinned base image never gets its security fixes unless you rebuild. Pair digest pinning with an automated job that rebuilds images on a schedule against fresh bases and opens the PR to bump the digest. Pinning without rebuilding is how you end up running a two-year-old OpenSSL with perfect reproducibility.

    What this doesn’t fix

    None of this stops an exploit in your own code. A read-only, non-root, distroless container with a SQL injection bug still has a SQL injection bug, and the attacker still reaches whatever the app’s database credentials can reach. Container runtime escapes still happen too. CVE-2024-21626, the runc file descriptor leak fixed in runc 1.1.12, allowed escape through a crafted working directory, and the fix was patching runc, not anything in the image.

    What hardening does is make the second step expensive. The attacker lands with no shell, no package manager, no writable path that can execute, no root and no extra capabilities. I’ve read enough incident reports to know that “the attacker had a foothold but couldn’t do much with it” is one of the better sentences you can have in one.

  • Container Escape Vectors: A Practical Walkthrough of Namespace and Cgroup Weaknesses

    Container Escape Vectors: A Practical Walkthrough of Namespace and Cgroup Weaknesses

    Containers aren’t a security boundary. They package several Linux kernel primitives that happen to produce process isolation as a side effect, and if you treat a container like a VM you’ll eventually get burned. This post covers what enforces container isolation, three escape techniques that have shown up in production, and the mitigations worth bothering with.

    What’s actually isolating the container

    Layered diagram of container isolation: namespaces, cgroups, capabilities, seccomp

    There’s no single “container” kernel object. Docker, containerd, and friends compose four separate mechanisms.

    Namespaces partition what a process can see: pid (process tree), net (network stack), mnt (filesystem mounts), uts (hostname), ipc, and user (UID/GID mapping). A process in a new PID namespace sees itself as PID 1 and can’t see host processes, but it’s still the same kernel, same syscall table, same vulnerabilities.

    Cgroups (v1 or v2) limit and account for resource usage: CPU, memory, I/O, PIDs. They control resources, not confidentiality or integrity. Cgroup v1’s release_agent mechanism in particular is a privileged hook that has nothing to do with isolation, and as we’ll see below it can lead straight to host compromise.

    Capabilities split root’s power into roughly 40 discrete privileges (CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_SYS_PTRACE, and so on). A container running as UID 0 with a reduced capability set isn’t the same as host root, but Docker’s default set is still broad, and CAP_SYS_ADMIN alone gets you close to full root for a lot of practical purposes: mount manipulation, cgroup control, some device operations.

    Seccomp filters which syscalls a process can invoke at all, independent of capabilities. Docker ships a default profile blocking around 44 syscalls (keyctl, add_key, mount, umount2 in some paths, ptrace variants, kernel module loading, etc). Disabling it with –security-opt seccomp=unconfined is a common way to remove a real security control while chasing a build error you don’t fully understand.

    Nobody designed these four mechanisms together as a sandbox. Each closes off a slice of the host kernel’s attack surface, and the escapes live in the gaps between them.

    Escape 1: privileged containers and /dev access

    docker run –privileged disables almost everything above: full capability set, no seccomp, no AppArmor or SELinux confinement, and access to host devices. If the container can see host block devices under /dev, it can mount the host root filesystem directly:

    # inside a --privileged container
    mkdir /mnt/hostfs
    mount /dev/sda1 /mnt/hostfs
    chroot /mnt/hostfs /bin/bash
    # now you are root on the host filesystem

    This isn’t a bug, it’s documented behavior of –privileged. It still shows up constantly: CI runners, “just make it work” Kubernetes manifests, Docker-in-Docker setups that grab –privileged to avoid debugging a permissions error. The fix is to find the one or two specific capabilities or device mounts the workload actually needs (–cap-add SYS_PTRACE, –device=/dev/foo), which is almost always possible once you check.

    Escape 2: CVE-2019-5736, runc file descriptor overwrite

    This one is a real runtime race, not a misconfiguration. runc versions before 1.0-rc7 (used under the hood by Docker, containerd, Kubernetes) were vulnerable to an attack where a malicious container could overwrite the host runc binary itself.

    Here’s the mechanism: when runc exec enters a running container, it briefly opens its own binary via /proc/self/exe inside the container’s mount namespace, to re-exec itself with dropped privileges. A container process controls its own /proc/self/exe target. Replace your own binary with a malicious ELF that, when run as the container’s entrypoint, opens /proc/self/exe for writing and holds the fd open, and you can race runc into writing to that fd instead of a temp file. Since the host runc binary is mapped in during this handoff, the attacker overwrites the actual runc executable on the host with their own payload. The next container started or exec’d into on that host runs the attacker’s code as root.

    It stung because it broke the assumption that “non-privileged container, default config” was safe against a malicious image or a compromised process. The fix was having runc copy itself to a memfd (memfd_create) and exec from there instead of touching a path on disk, which removes the overwritable target. Patch your container runtime (runc, containerd, crun) on the same schedule as your kernel.

    Escape 3: cgroup v1 release_agent abuse

    Cgroup v1 supports a release_agent file: a host-side executable path the kernel invokes, as root, on the host, whenever a cgroup becomes empty. Give a container CAP_SYS_ADMIN and write access to a cgroup hierarchy’s release_agent, which happens with –privileged or with mounted cgroupfs and relaxed capabilities, and it can point that path at a script of its choosing and trigger it:

    mkdir /tmp/cgrp && mount -t cgroup -o memory cgroup /tmp/cgrp
    mkdir /tmp/cgrp/x
    echo 1 > /tmp/cgrp/x/notify_on_release
    host_path=$(sed -n 's/.*\perdir:\([^,]*\).*/\1/p' /proc/self/cgroup)
    echo "$host_path/cmd" > /tmp/cgrp/release_agent
    echo '#!/bin/sh' > /cmd
    echo 'ps aux > /output' >> /cmd   # runs as host root
    chmod +x /cmd
    sh -c "echo \$\$ > /tmp/cgrp/x/cgroup.procs"

    Researchers including Felix Wilhelm have written this one up extensively. It works because cgroups were never an isolation mechanism: release_agent is a host-wide control hook, and a sufficiently privileged container process can reach it. Cgroup v2’s unified hierarchy scopes release_agent differently and closes off this specific path, which is one reason to be on v2 by now.

    Mitigations that actually reduce risk

    Drop capabilities. Start from –cap-drop=ALL and add back only what you’ve actually measured as needed. CAP_SYS_ADMIN, CAP_SYS_PTRACE, and CAP_SYS_MODULE have no business being default-on.

    Don’t use –privileged in production. A workload that “needs” it almost always needs one specific capability or device, not the entire host.

    Run containers rootless (Podman rootless, or Docker’s rootless mode). User namespace remapping maps “root” inside the container to an unprivileged UID on the host, so a full in-container compromise doesn’t hand over host root.

    Keep the runtime patched. CVE-2019-5736-class bugs live in runc, containerd, crun, not in your application code. Track your container runtime’s CVEs with the same priority you give kernel CVEs.

    Use seccomp, and don’t disable it just to make an error go away. If a syscall is genuinely blocked and genuinely needed, add it explicitly to a custom profile instead of going unconfined.

    Prefer cgroup v2, where the release_agent host-hook vector doesn’t apply the same way.

    For genuinely untrusted workloads, reach for a sandboxing runtime. gVisor (runsc) intercepts syscalls in userspace and reimplements the kernel surface; Kata Containers runs each container in a lightweight VM with its own kernel. Both add real isolation boundaries that namespaces and cgroups alone can’t provide, and the performance hit is usually worth it for multi-tenant or untrusted-input workloads.

    Every technique above has been public for years, and every mitigation here is a config change. In my experience the gap between “container” and “isolated” is usually a missing –cap-drop, an unpatched runc, or a –privileged flag somebody added to make an error go away. I’d check those first.