CanuteTheGreat's BLOG

Tag: CI

  • Patronus dev log 4: an installer test that failed 57 times in a row

    Patronus dev log 4: an installer test that failed 57 times in a row

    Part 4 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 3 was the restart.

    The installer test

    I wanted a nightly test that installs Patronus into a real VM, with real ZFS, and reboots into it. On October 4 it went in as a workflow. It ran on a Mac build host through UTM, so an x86 guest was emulated on an ARM machine.

    It never worked. From the Forgejo record, October 1 to 15, nightly-realistic-installer-test.yml: 57 failures, 6 cancelled, 0 successes. The overall repo had 249 runs those two weeks: 87 success, 116 failure, 36 cancelled, 10 skipped.

    What the log looks like:

    • Oct 4 and 5: file handoff between the VM and the host. First assumed instant, then polled for 60 seconds, then 180.
    • Oct 5: a serial console capture finally showed where boot stuck. GRUB 2.12 showed its menu, printed “Booting `Linux lts’” and then nothing. I switched the CD-ROM from USB to IDE. The commit message calls it the most likely cause, not a proven one.
    • Oct 5: “retry the live-boot stage – confirmed non-deterministic”.
    • Oct 10: a failing screenshot step turned out to be macOS denying Screen Recording permission (TCC), not a product bug.
    • Oct 13: boot budget pushed to 2700 seconds and the job ceiling to 120 minutes.
    • Oct 14: “pause nightly installer test schedule (TCC emulation can’t finish in time)”, where TCC is a typo for TCG, QEMU’s software emulation.

    Part 5 is what replaced it.

    Bugs that mattered more

    Meanwhile, in the same two weeks:

    • The firewall validated nothing before building nft scripts, so a rule field could inject ruleset text (Oct 12). Fields are now validated first.
    • The web UI’s NAT rules changed an in-memory list and never reached nftables (Oct 12).
    • A literal interface name “any” or “all” was spliced into nft syntax (Oct 13).
    • Captive portal MAC addresses weren’t validated before use in nft commands (Oct 11).
    • Dashboard threat and alert counts were constants (42, 38 and 3) whatever the real state. A commit on the 14th derived them from real data and listed what was still a placeholder: model accuracy, health score and packet rate, “no real ML classifier or packet-rate counter exists yet to back them honestly”.
    • On Oct 13, the container image couldn’t run nft as a non-root user until it got CAP_NET_ADMIN and CAP_NET_RAW on the binary.

    Design decisions

    The commit that replaced the UTM setup says x86 under TCG never booted there.

    State on October 15

    CI on main in the fortnight: 66 successes and 43 failures.

    Other parts: Part 1, Part 2, Part 3, Part 5.

  • Horcrux dev log 3: back after nine months, 92 failed CI runs

    Horcrux dev log 3: back after nine months, 92 failed CI runs

    Part 3 in the Horcrux dev log series.

    Backdated, written in November 2026 from git and CI records. Part 2 stopped at an unfinished ISO on December 31, 2025.

    I came back on September 24, 2026 with two goals: get CI to really run, and build Horcrux the Gentoo way.

    The Gentoo goal came from a mistake. At some point the Dockerfile had turned into standard Rust boilerplate: FROM rust, cargo build, FROM debian-slim. It had drifted into boilerplate that ignored what the project is for. I fixed it on September 24 and wrote a rule into the repo notes so it can’t come back. The rule in the repo notes reads:

    A Dockerfile doing FROM rust:* -> cargo build -> FROM debian:*-slim
is wrong. It is a regression. Fixed once already on 2026-09-24.

    Now the Dockerfile builds through real Portage against the project’s own ebuild, and the ebuild is the canonical install path.

    The numbers

    Pulled from the Forgejo API, September 24 to 30:

    workflow runs: 121
  success 12
  failure 92
  cancelled 11
  skipped 6
ci.yml: 8 success, 71 failure
docs.yml: 4 success, 19 failure

    88 commits that week, a lot of them fixes for whatever the last run complained about.

    What kept breaking

    The ebuild first. It depended on virtual/rust, which doesn’t exist (the package is dev-lang/rust), so emerge said “there are no ebuilds to satisfy”. dev-util/trunk doesn’t exist either, so the UI bundler gets installed with cargo during the build. qemu’s REQUIRED_USE says vte? ( gtk ) and I’d enabled vte alone. The Xen tools ebuild in that Gentoo snapshot died in src_prepare on an unset version variable; Horcrux only does KVM, so I dropped the Xen flag instead of fighting it. A pile of qemu and spice USE flags dead-ended in a masked libglvnd, and a headless VM host has no use for audio or GL anyway.

    Then the runners. Actions that weren’t mirrored on the registry my Forgejo uses. upload-artifact v4 not supported (“GHESNotSupportedError”). A runner with a full disk. A clippy step killed by the runner. A 30 minute ephemeral timeout that killed legitimate Gentoo builds. A full emerge that hit a 3 hour limit until I gave the CI image light USE flags. Several commit messages just say “retrigger” because the last failure was the infrastructure and not the code.

    Fake stuff that came out

    Real tests started finding placeholders. The login form wasn’t wired to the auth API. Disk usage in the metrics was a hardcoded 65%; it now reads statvfs. The Kubernetes join code accepted a made-up CA cert hash and now refuses it. HA fencing for IPMI, iLO, iDRAC and SNMP PDUs, parallel volume restore and live-migration pre-flight checks were stubs and got real code on the 30th. I don’t know yet how any of that behaves on actual hardware. release.yml only built x86_64 although the README claimed three architectures, so it now cross-compiles aarch64 and riscv64.

    On the 25th and 26th the API integration suite reached 22 of 22. It’s the first green test run I can find in the history.

    State

    The Gentoo build works in CI with trimmed USE flags, the tests run, and about one run in ten was green. The README still overpromised, and I rewrote its status line on October 21 (part 5).

    Design decisions

    Gentoo first. My reason is freedom to choose. USE flags are the main part: they let an installer choose which virtualization backends, container runtimes and NAS protocols get compiled in, and the repo notes call that the main advantage over Proxmox. Compiling for specific hardware is the secondary reason. For background, I was a Debian developer from 1999 to 2006 and I’ve used Gentoo since. I’m blunt about it in the repo notes: generic Debian or Ubuntu conventions are the wrong default here.

    Dropping Xen. The commit says Horcrux is a headless KVM-only VM host and has no need for Xen guest support. The same reasoning removed audio and GL from qemu: no local display on a headless host.

    Architectures: amd64, arm64 and riscv64. The README dropped ppc64le the same day as out of scope. That’s my call, and I don’t have the reason written down. The riscv64 notes do say it needs real hardware for validation.

    Light USE flags in CI: the full emerge hit a 3 hour limit, so the CI smoke-test image builds with fewer flags. A speed trade, and it means CI doesn’t build the full-feature package.

    Other parts: Part 1, Part 2, Part 4, Part 5, Part 6. The series continues with the latest Horcrux dev log post.

  • Patronus dev log 3: back in September, with 43 red CI runs and no green ones

    Patronus dev log 3: back in September, with 43 red CI runs and no green ones

    Part 3 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 2 ended with nine months of silence.

    I picked Patronus back up the same day as Horcrux, September 24, 2026. The first job was the same: make CI run. Patronus got a Rust CI workflow that day (fmt, clippy, tests, an eBPF check, a full-workspace build). It failed. Over September 24 to 30 the Forgejo record shows 57 runs: 43 failures, 7 cancelled, 7 skipped, 0 successes. Of those, rust-ci.yml had 42 failures.

    The first failures were the dull kind: a missing libmnl-dev, a missing libelf, no linker on the runner image, and a real bug. An eBPF map-create call took its name as i8 and needed u8, which was blocking the detection and CNI crates from building at all.

    The Dockerfile again

    Same mistake as in Horcrux, found the same day. The Dockerfile had become a generic FROM rust build on a Debian slim image. The repo notes now say so, in short:

    A generic FROM rust -> cargo build -> debian slim Dockerfile is wrong.
It crept in once and was fixed on 2026-09-24.

    Patronus builds through real Portage against its own overlay. The ebuild kept breaking, in familiar ways: virtual/rust doesn’t exist (same bug as Horcrux, the commit says so), acct-user and acct-group variable names were wrong, a USE flag was called carp in the code and ucarp in IUSE, the meta USE flags defaulted on with no sub-backend, and the ebuild forwarded cargo features that don’t exist (aliases, tc, prometheus). Building Rust from source inside the Docker build took 2 hours 45 minutes, so I switched to dev-lang/rust-bin. And the Gentoo Docker build got capped at 60 minutes and made non-blocking while I sorted it out.

    On the 29th the installer picked up root-on-ZFS, preferred when the ZFS tools are present.

    Design decisions

    Gentoo and USE flags: Gentoo is the core design point. My reason is freedom to choose. USE flags come first because they let you pick which features (web UI, eBPF/XDP, SD-WAN, detection, cloud integrations) get compiled in, and the repo notes call that the main advantage over pfSense and OPNsense. Hardware specific compiles are the second reason. A container build must pass USE flags through to emerge rather than compile a fixed set.

    rust-bin over source Rust: the from-source build took 2h45m (the commit’s number).

    Architecture: the repo notes have one rule: one SQLite store as the single source of truth, one “Applier” per subsystem, one apply step at boot, and no extra in-memory list, fixture or config file that only one process knows about.

    State on September 30

    57 runs, none green, and about 60 commits. The Gentoo build was non-blocking at the end of the week and still no run was green.

    Other parts: Part 1, Part 2, Part 4, Part 5.