Patronus dev log 4: an installer test that failed 57 times in a row

Part 4 in the Patronus dev log series.

Backdated, written in November 2026 from git and CI records. Part 3 was the restart.

The installer test

I wanted a nightly test that installs Patronus into a real VM, with real ZFS, and reboots into it. On October 4 it went in as a workflow. It ran on a Mac build host through UTM, so an x86 guest was emulated on an ARM machine.

It never worked. From the Forgejo record, October 1 to 15, nightly-realistic-installer-test.yml: 57 failures, 6 cancelled, 0 successes. The overall repo had 249 runs those two weeks: 87 success, 116 failure, 36 cancelled, 10 skipped.

What the log looks like:

• Oct 4 and 5: file handoff between the VM and the host. First assumed instant, then polled for 60 seconds, then 180.
• Oct 5: a serial console capture finally showed where boot stuck. GRUB 2.12 showed its menu, printed “Booting `Linux lts’” and then nothing. I switched the CD-ROM from USB to IDE. The commit message calls it the most likely cause, not a proven one.
• Oct 5: “retry the live-boot stage – confirmed non-deterministic”.
• Oct 10: a failing screenshot step turned out to be macOS denying Screen Recording permission (TCC), not a product bug.
• Oct 13: boot budget pushed to 2700 seconds and the job ceiling to 120 minutes.
• Oct 14: “pause nightly installer test schedule (TCC emulation can’t finish in time)”, where TCC is a typo for TCG, QEMU’s software emulation.

Part 5 is what replaced it.

Bugs that mattered more

Meanwhile, in the same two weeks:

• The firewall validated nothing before building nft scripts, so a rule field could inject ruleset text (Oct 12). Fields are now validated first.
• The web UI’s NAT rules changed an in-memory list and never reached nftables (Oct 12).
• A literal interface name “any” or “all” was spliced into nft syntax (Oct 13).
• Captive portal MAC addresses weren’t validated before use in nft commands (Oct 11).
• Dashboard threat and alert counts were constants (42, 38 and 3) whatever the real state. A commit on the 14th derived them from real data and listed what was still a placeholder: model accuracy, health score and packet rate, “no real ML classifier or packet-rate counter exists yet to back them honestly”.
• On Oct 13, the container image couldn’t run nft as a non-root user until it got CAP_NET_ADMIN and CAP_NET_RAW on the binary.

Design decisions

The commit that replaced the UTM setup says x86 under TCG never booted there.

State on October 15

CI on main in the fortnight: 66 successes and 43 failures.

Other parts: Part 1, Part 2, Part 3, Part 5.