Category: Software Engineering

  • Horcrux dev log 6: clustering, an ISO you can install from, and real laptops

    Horcrux dev log 6: clustering, an ISO you can install from, and real laptops

    Part 6 in the Horcrux dev log series.

    Backdated: written in November 2026 from git, CI and my QA ledger. Part 5 was the audit.

    With the basics fixed I moved on to clustering. In five days the log shows an HA supervisor (failure detection, fencing, recovery, self-fencing), cluster create and join, guest save state on host shutdown (QEMU migrate-to-file for VMs, CRIU for containers), draining a node with live moves, SDN zones, Ceph management, deduplicating backups, PCI and vGPU passthrough and a Redfish fencing driver.

    On October 24 live migration got a hard block between AMD and Intel CPUs, and the drain planner never moves a running guest across vendors. Cold migration is still allowed.

    From the Forgejo API for October 22 to November 6: 725 runs, 493 success, 78 failure, 154 cancelled, about 450 commits.

    ha-failover.yml 68 success, 4 failure
    vm-migration.yml 32 success, 0 failure
    guest-power.yml 30 success, 7 failure
    live-iso.yml 21 success, 4 failure, 26 cancelled

    I haven’t checked every cancelled run, but newer pushes superseding a run in progress account for many of them.

    October 27: an ISO that boots

    The December ISO builder never got finished. This one has a different design. A Docker-only builder (Gentoo stage3 plus the official binhost, dracut with dmsquash-live, GRUB for BIOS and UEFI in a hybrid ISO), and a QEMU test that boots it by UEFI and USB and by BIOS and CD, runs an unattended install, reboots into the result and probes the web API. A daily job publishes only if the tests pass. First version: 31 files, about 1,000 lines.

    By the 28th it had a text installer, an answer-file mode, no default password, key-only SSH and aarch64 images. This is the boot menu from that week, taken from the repo’s docs folder at the time:

    The Horcrux installer boot menu in GRUB, committed October 28, 2026
    Boot menu as committed on October 28, 2026 (docs/images/boot in the repo).

    And the installer’s file system page, committed the same day:

    Horcrux text installer file system page with ext4 as the default
    File system page as committed October 28. The default is ext4, which was wrong and got changed within days.

    The default is ext4. I’d decided ZFS should be the default, with an automatic XFS fallback under 8 GB of RAM or when there’s no ZFS module for the kernel.

    T2 laptops, tablets, encrypted ZFS

    There’s a GNOME desktop edition on top of the server core and hardware profiles for an Apple T2 MacBook and a Surface tablet. The installer learned encrypted ZFS root with boot environments, and hibernate on encrypted ZFS. The initramfs has to resume from the LUKS swap before it imports the pool, because importing first and resuming into it corrupts the pool. A QEMU test does install, passphrase, hibernate, power off, boot, passphrase, same process still running, clean scrub.

    What real machines found

    QEMU didn’t catch several things. The November 3 commits are the list the laptops produced.

    The clock: a live system with a months-off RTC can’t do TLS, so Tailscale and package syncs failed. The installer now writes the hardware clock only from a verified clock, and the tests boot QEMU with -rtc base= months in the past. The size-trimming step on the desktop image kept deleting /usr/include and every static library, so installed systems had no C compiler and ZFS couldn’t rebuild its module on a kernel update. A zfs-kmod from a cached binary package had been built for a different kernel. Plain XFS and ext4 installs didn’t write /etc/fstab, so /boot wasn’t mounted and kernel updates landed on the root filesystem. GDM login started two keyrings. The disk passphrase could be reused as an account password, so the installer now refuses that. And the desktop had apps missing from the launcher because their binary didn’t exist, no sound, dead volume keys, and a Log Out button that did nothing, because the session was split across two D-Bus buses.

    Most of these got a gate in the QEMU suite afterwards (months-off clock, compiler present, module builds, fstab, keyring, password reuse, the desktop session items). On November 3 I also fixed CI reporting green after a failed step (“green-but-failed 8637” in the commit), so some earlier runs looked healthier than they were.

    Where it stands

    This week I moved on to a lab of three old laptops to prove the HA claims. The goal in the lab plan: every scenario of the HA suite green ten runs in a row, failover under 120 seconds, no data loss, never two copies of a guest. As of this writing that’s 0 of 10. The old manager took 126 seconds to fail over after a hard crash, over the limit, and a one-node partition ran two copies of a guest for several seconds. Both are logged as bugs with fixes pending. So I can’t say Horcrux has HA yet. The next post is about that.

    The aarch64 ISO build takes hours under emulation and hit the runner’s time cap on November 6, so it moved to a runner with a longer limit.

    Design decisions

    OpenRC is the only init on the ISOs. systemd wants to run the world, and that comes at a cost: feature bloat.

    ZFS by default, with an automatic XFS fallback when RAM is under 8 GB or the kernel has no ZFS module. The installer shows its reason on the disk page (“XFS instead of ZFS: only 4.0 GB RAM (ZFS wants 8 GB or more)”). I want ZFS boot environments and an automatic snapshot before every update for recovery, bit rot protection and resilience. ext4 is never the default, only a manual choice. Gentoo recommends XFS, and ZFS is just superior tech.

    Encrypted ZFS needs a static grub.cfg and a kernel per boot environment under /boot, because GRUB can’t read an encrypted ZFS pool.

    Hibernate goes to a separate LUKS2 swap partition and not a zvol, because OpenZFS doesn’t support hibernating onto its own zvol and doing it anyway risks the pool. For the Surface XFS layout the notes reject a separate LUKS swap, since it would need a second keyslot or a second prompt.

    Surface tablets get XFS plus LUKS and a light touch desktop, because the docs say a GNOME-class desktop doesn’t fit in 4 GB.

    Server first. On November 3 I wrote down the rule: “it should work on a headless server too, it shouldn’t be a desktop-only OS”. The desktop is a layer on top of a headless core and personal apps live in per-machine layers, not the public images.

    Live migration between AMD and Intel is refused and cold migration is allowed (see Oct 24 above).

    Other parts: Part 1, Part 2, Part 3, Part 4, Part 5. The series continues with the latest Horcrux dev log post.

  • Patronus dev log 5: security holes, netlab, and the 0.2.0 release

    Patronus dev log 5: security holes, netlab, and the 0.2.0 release

    Part 5 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 4 ended with the installer test paused.

    October 16: two things at once

    First, the installer test moved off the Mac. It now runs on a KVM runner with QEMU directly: install with the ISO attached, then cold boot the disk with the media removed. A live system comes up in about 25 seconds, and install plus reboot takes about 6 minutes. The installed system comes up on rpool with zpool ONLINE. The first green run is recorded in the commit. Getting there turned up five real installer bugs, which the commit lists: a bad parted mkpart fs-type for ZFS, a free-space calculation that started partition 2 at the disk end, a root pool that kept its temporary mountpoint forever, pool features GRUB couldn’t read, and chroot bind mounts that stacked and never unmounted, so the pool export failed and the next boot refused to import it. From then to November 6, that workflow has 25 passing runs and 3 cancelled ones.

    Second, the security commits from the same day. Here’s what the commit messages say:

    patronus-web created 'admin'/'admin' and 'operator'/'operator' accounts on
    every install, so anyone reaching :8443 could take over the firewall.
    SECURITY (CRITICAL): only /api/auth/me and /api/auth/logout checked the
    session. Every other page and the whole management API -- add/delete
    firewall and NAT rules, WireGuard peers, interface up/down, backups,
    service start/stop -- answered anonymous requests

    So for a year after the “Complete Implementation” commit, the web UI had no login that mattered, and it shipped with default passwords. Both are fixed now. The admin password is per install: a random 24 character one, written to a 0600 file on first start, and the service refuses to start with a known or short one. A session is required for every page and API call, with regression tests.

    netlab

    There’s a harness called netlab: network namespaces, a router and clients and an ISP, real nftables and real daemons, scenarios as shell scripts, and power-cut and upgrade tests. The README says a ledger maps every claim to a test that proves it, and an “honesty guard” test scans the web code for old fixture values.

    Failures that only showed up in CI: sh -c with bash functions (dash can’t see them), a DNS resolver in the CI image binding port 53 before a scenario’s own, grep -q under pipefail killing nft with SIGPIPE, kernel logging that doesn’t reach a namespace, and a runner with br_netfilter loaded that made bridged traffic hit the firewall. Some scenarios couldn’t pass at all, so I parked them in a pending/ directory instead of leaving main red: UPnP port mapping (miniupnpd answered 501 on the nftables backend and I never solved it), DHCPv6 prefix delegation for a while, and VRF.

    From Forgejo, October 16 to November 6: 416 runs, 209 success, 108 failure, 82 cancelled. About 294 commits.

    Features, and what they were proven on

    Multi-WAN with failover, QoS (CAKE, fq_codel), DHCPv6 prefix delegation, OpenVPN, IPsec road warriors, VRFs, zero-touch provisioning, factory reset, LDAP login, GitOps, and per-device segmentation with DNS policy and time limits. The 0.2.0 CHANGELOG lists its own limits: unmanaged switches, IPv6 binding “next”, and simulated radios only. Most of this was proven in namespaces and one lab VM, not on a production network.

    The ISO and 0.2.0

    On October 27, Patronus got a live installer ISO built with the same builder as Horcrux, plus a text installer with no default password and key-only SSH, aarch64 images and a console menu. The v0.2.0 release on the 30th (“device segmentation (opt-in), per-device Wi-Fi, DNS rebinding, per-device DNS policy, time limits, insight history, uplink health”) needed five migrations. The first release.yml didn’t produce a release, so another commit made it do so. Fat LTO of the web crate was OOM-killed in a 6 GB runner slot, so aarch64 uses thin LTO. The stable ISO gate also failed on stale red statuses from a superseded tag.

    Design decisions

    Headless first. Horcrux’s rule was “it should work on a headless server too, it shouldn’t be a desktop-only OS”, so on November 3 I made the same call for Patronus. A box is installed and recovered from the console, and operated over SSH and the web UI. There’s no desktop edition and none is planned. CI fails the build if a desktop or personal package lands in the image inputs. There’s also deliberately no private image variant: per-box data comes in via an answer file on the install USB and via GitOps.

    Encryption and boot: a firewall should never stop at boot waiting for a console passphrase. So the installer encrypts only if an unattended unlock works (a TPM2-sealed key, self-tested). Otherwise it installs unencrypted and says so.

    ZFS by default, with an XFS fallback below 8 GB (the disk page shows the reason). I want ZFS for recovery, bit rot protection and resilience. ext4 is never the default. Gentoo recommends XFS, and ZFS is the better filesystem for this job. There’s no on-disk swap on a firewall.

    Opt-in segmentation: the release ships with it off by default, and an upgrade test asserts that an upgraded box keeps isolation off.

    State on November 6

    The UI now requires a login, the installer passes a KVM test, and v0.2.0 is out with its limits written down.

    Other parts: Part 1, Part 2, Part 3, Part 4.

  • SBOMs: what they’re actually good for

    SBOMs: what they’re actually good for

    Part 2 of 3 in the Vulnerability management that works series. Part 1: Prioritising vulnerabilities: CVSS vs EPSS vs CISA KEV.

    Software bills of materials had a hype cycle. Executive Order 14028 in May 2021 told US federal suppliers to provide them, vendors rushed out SBOM features, and for a couple of years every security conference had a talk about them. A lot of organisations now generate SBOMs because a contract or a customer questionnaire asked for one, file them somewhere, and never look at them again.

    That’s a shame, because there’s one job SBOMs do very well, and it’s the job that hurt most during Log4Shell. When a new vulnerability lands, they tell you where you’re running the affected component, in minutes instead of days.

    What an SBOM is, concretely

    Pipeline diagram: CI build, syft SBOM, cosign attestation and registry, with SBOM copies in a store that is queried when a new CVE is published, filtered through VEX, producing affected services for prioritisation

    An SBOM is a machine-readable inventory of the components inside a piece of software: package names, versions, identifiers, and ideally the relationships between them. There are two formats that matter. SPDX came out of the Linux Foundation and is published as ISO/IEC 5962:2021. CycloneDX came out of OWASP and has since been standardised by Ecma as ECMA-424. Both are fine. Pick the one your tooling handles best and stop debating it.

    The NTIA’s 2021 minimum elements document lists the fields an SBOM should carry: supplier, component name, version, other unique identifiers, dependency relationships, the author of the SBOM data, and a timestamp. In practice the identifier that does the heavy lifting is the package URL, or PURL, something like pkg:maven/org.apache.logging.log4j/[email protected] or pkg:deb/debian/[email protected]. PURLs are what let a scanner match a component against an advisory database without guessing from file names.

    Generate them at build time, from the artifact

    Where you generate the SBOM matters more than which format you use. An SBOM written from a lockfile describes what you meant to ship. An SBOM generated from the built container image describes what you actually shipped, including the OS packages in the base image that no lockfile in your repo mentions.

    Syft is the tool I reach for. Generating a CycloneDX SBOM for an image in CI is one line:

    syft registry.example.com/payments-api:1.42.0 -o cyclonedx-json=payments-api.cdx.json

    Then attach it to the image, so the inventory travels with the artifact instead of living in a CI log nobody can find:

    cosign attest --yes \
      --predicate payments-api.cdx.json \
      --type cyclonedx \
      registry.example.com/payments-api@sha256:<digest>

    Attest against the digest, not the tag. Tags move. An SBOM attached to :latest is a statement about an image that may no longer exist.

    Also push a copy into something you can query across the fleet. That can be a dedicated platform like OWASP Dependency-Track, or honestly a bucket of JSON files plus an index. Being able to ask one question across every artifact you run matters more than the format.

    The question SBOMs answer

    That question is: “which of our running things contain component X at version Y?”

    On 29 March 2024, Andres Freund disclosed the backdoor in xz-utils, CVE-2024-3094, affecting versions 5.6.0 and 5.6.1. If you had SBOMs for every deployed image, the first hour looked like this:

    for f in sboms/*.cdx.json; do
      jq -r --arg f "$f" '.components[]
        | select(.name == "xz-utils" or .name == "liblzma5" or .name == "xz-libs")
        | select(.version | test("^5\\.6\\.[01]"))
        | "\($f)  \(.name)  \(.version)"' "$f"
    done

    If you didn’t, the first hour was a Slack thread asking every team to check their base images, followed by a day of partial answers. I’ve been on both sides of that, and the gap between the two is obvious in hindsight. During Log4Shell I watched teams spend most of a week just establishing which services bundled log4j-core, because it was often a transitive dependency several levels down, or shaded inside another jar.

    The same SBOM store also feeds continuous matching. Instead of rescanning every image when the vulnerability database updates, you rescan the SBOMs:

    grype sbom:payments-api.cdx.json --fail-on high

    It’s quick, and it covers images that are already running and won’t be rebuilt this week.

    Where SBOMs are blind

    SBOMs are only as complete as the tool that generated them, and every generator has gaps.

    Statically linked and vendored code is the big one. If a C library was copied into your source tree or compiled into a binary, there’s no package manager metadata for a scanner to find. Go is better than most here because the toolchain embeds module information in the binary, which go version -m will print and Syft will read. Rust binaries carry nothing by default unless you build with cargo auditable. Copied-in JavaScript files, a single vendored .c file, or a library someone downloaded and checked in years ago will mostly be invisible.

    Version strings can lie too. Distributions backport security fixes without changing the upstream version, so a Debian package might report a version that looks vulnerable while carrying the patch. Good scanners read the distro’s security tracker data for this. Bad ones generate a stream of false positives that teach engineers to ignore the scanner.

    And an SBOM says nothing about reachability. A vulnerable component that’s present isn’t necessarily loaded, and a loaded library isn’t necessarily being called with attacker-controlled input.

    VEX is what makes SBOMs bearable

    That last gap is where VEX comes in, the Vulnerability Exploitability eXchange. A VEX document is a statement about a specific vulnerability in a specific product, with a status: not_affected, affected, fixed or under_investigation. A not_affected claim has to carry a justification, such as vulnerable_code_not_present or vulnerable_code_not_in_execute_path.

    OpenVEX, CSAF and CycloneDX can all carry VEX statements, and Grype accepts OpenVEX documents to suppress findings you’ve already analysed:

    grype sbom:payments-api.cdx.json --vex payments-api.openvex.json

    What this buys you is mostly organisational. Someone investigates a finding once, writes down why it doesn’t apply, and that decision follows the artifact. Without it, the same finding gets triaged by three different people in three different quarters, and the one who’s in a hurry marks it as a false positive with no reason recorded.

    What I’d actually do

    Generate SBOMs from built images in CI, attest them to the digest, and copy them into a queryable store. Rescan the stored SBOMs daily against current advisory data and feed the results into the prioritisation from part one, so a KEV entry on an exposed service jumps to the front. Record triage decisions as VEX next to the SBOM.

    Skip the parts that don’t pay. I wouldn’t spend time on SBOM quality scores, on publishing SBOMs to customers who haven’t asked, or on generating them for every intermediate build. The SBOM that matters is the one for the thing running in production right now.

    Part three covers the last mile: setting patch SLAs that teams can actually meet once you know what to patch and where.

    Other parts: Part 1: Prioritising vulnerabilities: CVSS vs EPSS vs CISA KEV, Part 3: Patch SLAs that teams actually meet.