CanuteTheGreat's BLOG

Tag: Patronus dev log

  • Patronus dev log 5: security holes, netlab, and the 0.2.0 release

    Patronus dev log 5: security holes, netlab, and the 0.2.0 release

    Part 5 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 4 ended with the installer test paused.

    October 16: two things at once

    First, the installer test moved off the Mac. It now runs on a KVM runner with QEMU directly: install with the ISO attached, then a cold boot of the disk with the media removed. A live system in about 25 seconds, install plus reboot about 6 minutes, and the installed system comes up on rpool with zpool ONLINE. The first green run is recorded in the commit. Getting there found five real installer bugs, which the commit lists: a bad parted mkpart fs-type for ZFS, a free-space calculation that started partition 2 at the disk end, a root pool that kept its temporary mountpoint forever, pool features GRUB couldn’t read, and chroot bind mounts that stacked and never unmounted, so the pool export failed and the next boot refused to import it. From then to November 6 that workflow has 25 passing runs and 3 cancelled ones.

    Second, the security commits of the same day. From the commit messages:

    patronus-web created 'admin'/'admin' and 'operator'/'operator' accounts on
every install, so anyone reaching :8443 could take over the firewall.
    SECURITY (CRITICAL): only /api/auth/me and /api/auth/logout checked the
session. Every other page and the whole management API -- add/delete
firewall and NAT rules, WireGuard peers, interface up/down, backups,
service start/stop -- answered anonymous requests

    So for a year after the “Complete Implementation” commit the web UI had no login that mattered and shipped with default passwords. Both are fixed: the admin password is per install (a random 24 character one written to a 0600 file on first start, and the service refuses to start with a known or short one), and a session is required for every page and API call, with regression tests.

    netlab

    There’s a harness called netlab: network namespaces, a router and clients and an ISP, real nftables and real daemons, scenarios as shell scripts, and power-cut and upgrade tests. A README claims ledger maps every claim to a test that proves it, and a “honesty guard” test scans the web code for old fixture values.

    CI-only failures: sh -c with bash functions (dash can’t see them), a DNS resolver in the CI image binding port 53 before a scenario’s own, grep -q under pipefail killing nft with SIGPIPE, kernel logging that doesn’t reach a namespace, and a runner with br_netfilter loaded that made bridged traffic hit the firewall. Some scenarios couldn’t pass at all and were parked in a pending/ directory instead of leaving main red: UPnP port mapping (miniupnpd answered 501 on the nftables backend and I never solved it), DHCPv6 prefix delegation for a while, and VRF.

    From Forgejo, October 16 to November 6: 416 runs, 209 success, 108 failure, 82 cancelled. About 294 commits.

    Features, and what they were proven on

    Multi-WAN with failover, QoS (CAKE, fq_codel), DHCPv6 prefix delegation, OpenVPN, IPsec road warriors, VRFs, zero-touch provisioning, factory reset, LDAP login, GitOps, and per-device segmentation with DNS policy and time limits. The 0.2.0 CHANGELOG lists its own limits: unmanaged switches, IPv6 binding “next”, and simulated radios only. Most of this was proven in namespaces and one lab VM, not on a production network.

    The ISO and 0.2.0

    On October 27 Patronus got a live installer ISO using the same builder as Horcrux, then a text installer with no default password and key-only SSH, aarch64 images and a console menu. The v0.2.0 release on the 30th (“device segmentation (opt-in), per-device Wi-Fi, DNS rebinding, per-device DNS policy, time limits, insight history, uplink health”) needed five migrations. The first release.yml didn’t produce a release, so another commit made it do so, and fat LTO of the web crate was OOM-killed in a 6 GB runner slot so aarch64 uses thin LTO. The stable ISO gate also failed on stale red statuses from a superseded tag.

    Design decisions

    Headless first. After Horcrux’s rule “it should work on a headless server too, it shouldn’t be a desktop-only OS”, I made the same call for Patronus on November 3. A box is installed and recovered from the console, and operated over SSH and the web UI. There is no desktop edition and none is planned, and CI fails the build if a desktop or personal package lands in the image inputs. There’s also deliberately no private image variant: per-box data comes in via an answer file on the install USB and via GitOps.

    Encryption and boot: a firewall must never stop at boot waiting for a console passphrase. So the installer encrypts only if an unattended unlock works (a TPM2-sealed key, self-tested), and otherwise installs unencrypted and says so.

    ZFS by default with an XFS fallback below 8 GB (the disk page shows the reason). I want it for recovery, bit rot protection and resilience, and ext4 is never the default: Gentoo recommends XFS, and ZFS is just superior tech. There is no on-disk swap on a firewall.

    Opt-in segmentation: the release made it off by default, and an upgrade test asserts an upgraded box has isolation off.

    State on November 6

    The UI now requires a login, the installer passes a KVM test, and v0.2.0 is out with its limits written down.

    Other parts: Part 1, Part 2, Part 3, Part 4.

  • Patronus dev log 4: an installer test that failed 57 times in a row

    Patronus dev log 4: an installer test that failed 57 times in a row

    Part 4 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 3 was the restart.

    The installer test

    I wanted a nightly test that installs Patronus into a real VM, with real ZFS, and reboots into it. On October 4 it went in as a workflow. It ran on a Mac build host through UTM, so an x86 guest was emulated on an ARM machine.

    It never worked. From the Forgejo record, October 1 to 15, nightly-realistic-installer-test.yml: 57 failures, 6 cancelled, 0 successes. The overall repo had 249 runs those two weeks: 87 success, 116 failure, 36 cancelled, 10 skipped.

    What the log looks like:

    • Oct 4 and 5: file handoff between the VM and the host. First assumed instant, then polled for 60 seconds, then 180.
    • Oct 5: a serial console capture finally showed where boot stuck. GRUB 2.12 showed its menu, printed “Booting `Linux lts’” and then nothing. I switched the CD-ROM from USB to IDE. The commit message calls it the most likely cause, not a proven one.
    • Oct 5: “retry the live-boot stage – confirmed non-deterministic”.
    • Oct 10: a failing screenshot step turned out to be macOS denying Screen Recording permission (TCC), not a product bug.
    • Oct 13: boot budget pushed to 2700 seconds and the job ceiling to 120 minutes.
    • Oct 14: “pause nightly installer test schedule (TCC emulation can’t finish in time)”, where TCC is a typo for TCG, QEMU’s software emulation.

    Part 5 is what replaced it.

    Bugs that mattered more

    Meanwhile, in the same two weeks:

    • The firewall validated nothing before building nft scripts, so a rule field could inject ruleset text (Oct 12). Fields are now validated first.
    • The web UI’s NAT rules changed an in-memory list and never reached nftables (Oct 12).
    • A literal interface name “any” or “all” was spliced into nft syntax (Oct 13).
    • Captive portal MAC addresses weren’t validated before use in nft commands (Oct 11).
    • Dashboard threat and alert counts were constants (42, 38 and 3) whatever the real state. A commit on the 14th derived them from real data and listed what was still a placeholder: model accuracy, health score and packet rate, “no real ML classifier or packet-rate counter exists yet to back them honestly”.
    • On Oct 13, the container image couldn’t run nft as a non-root user until it got CAP_NET_ADMIN and CAP_NET_RAW on the binary.

    Design decisions

    The commit that replaced the UTM setup says x86 under TCG never booted there.

    State on October 15

    CI on main in the fortnight: 66 successes and 43 failures.

    Other parts: Part 1, Part 2, Part 3, Part 5.

  • Patronus dev log 3: back in September, with 43 red CI runs and no green ones

    Patronus dev log 3: back in September, with 43 red CI runs and no green ones

    Part 3 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 2 ended with nine months of silence.

    I picked Patronus back up the same day as Horcrux, September 24, 2026. The first job was the same: make CI run. Patronus got a Rust CI workflow that day (fmt, clippy, tests, an eBPF check, a full-workspace build). It failed. Over September 24 to 30 the Forgejo record shows 57 runs: 43 failures, 7 cancelled, 7 skipped, 0 successes. Of those, rust-ci.yml had 42 failures.

    The first failures were the dull kind: a missing libmnl-dev, a missing libelf, no linker on the runner image, and a real bug. An eBPF map-create call took its name as i8 and needed u8, which was blocking the detection and CNI crates from building at all.

    The Dockerfile again

    Same mistake as in Horcrux, found the same day. The Dockerfile had become a generic FROM rust build on a Debian slim image. The repo notes now say so, in short:

    A generic FROM rust -> cargo build -> debian slim Dockerfile is wrong.
It crept in once and was fixed on 2026-09-24.

    Patronus builds through real Portage against its own overlay. The ebuild kept breaking, in familiar ways: virtual/rust doesn’t exist (same bug as Horcrux, the commit says so), acct-user and acct-group variable names were wrong, a USE flag was called carp in the code and ucarp in IUSE, the meta USE flags defaulted on with no sub-backend, and the ebuild forwarded cargo features that don’t exist (aliases, tc, prometheus). Building Rust from source inside the Docker build took 2 hours 45 minutes, so I switched to dev-lang/rust-bin. And the Gentoo Docker build got capped at 60 minutes and made non-blocking while I sorted it out.

    On the 29th the installer picked up root-on-ZFS, preferred when the ZFS tools are present.

    Design decisions

    Gentoo and USE flags: Gentoo is the core design point. My reason is freedom to choose. USE flags come first because they let you pick which features (web UI, eBPF/XDP, SD-WAN, detection, cloud integrations) get compiled in, and the repo notes call that the main advantage over pfSense and OPNsense. Hardware specific compiles are the second reason. A container build must pass USE flags through to emerge rather than compile a fixed set.

    rust-bin over source Rust: the from-source build took 2h45m (the commit’s number).

    Architecture: the repo notes have one rule: one SQLite store as the single source of truth, one “Applier” per subsystem, one apply step at boot, and no extra in-memory list, fixture or config file that only one process knows about.

    State on September 30

    57 runs, none green, and about 60 commits. The Gentoo build was non-blocking at the end of the week and still no run was green.

    Other parts: Part 1, Part 2, Part 4, Part 5.