Tag: Gentoo

  • Patronus dev log 5: security holes, netlab, and the 0.2.0 release

    Patronus dev log 5: security holes, netlab, and the 0.2.0 release

    Part 5 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 4 ended with the installer test paused.

    October 16: two things at once

    First, the installer test moved off the Mac. It now runs on a KVM runner with QEMU directly: install with the ISO attached, then cold boot the disk with the media removed. A live system comes up in about 25 seconds, and install plus reboot takes about 6 minutes. The installed system comes up on rpool with zpool ONLINE. The first green run is recorded in the commit. Getting there turned up five real installer bugs, which the commit lists: a bad parted mkpart fs-type for ZFS, a free-space calculation that started partition 2 at the disk end, a root pool that kept its temporary mountpoint forever, pool features GRUB couldn’t read, and chroot bind mounts that stacked and never unmounted, so the pool export failed and the next boot refused to import it. From then to November 6, that workflow has 25 passing runs and 3 cancelled ones.

    Second, the security commits from the same day. Here’s what the commit messages say:

    patronus-web created 'admin'/'admin' and 'operator'/'operator' accounts on
    every install, so anyone reaching :8443 could take over the firewall.
    SECURITY (CRITICAL): only /api/auth/me and /api/auth/logout checked the
    session. Every other page and the whole management API -- add/delete
    firewall and NAT rules, WireGuard peers, interface up/down, backups,
    service start/stop -- answered anonymous requests

    So for a year after the “Complete Implementation” commit, the web UI had no login that mattered, and it shipped with default passwords. Both are fixed now. The admin password is per install: a random 24 character one, written to a 0600 file on first start, and the service refuses to start with a known or short one. A session is required for every page and API call, with regression tests.

    netlab

    There’s a harness called netlab: network namespaces, a router and clients and an ISP, real nftables and real daemons, scenarios as shell scripts, and power-cut and upgrade tests. The README says a ledger maps every claim to a test that proves it, and an “honesty guard” test scans the web code for old fixture values.

    Failures that only showed up in CI: sh -c with bash functions (dash can’t see them), a DNS resolver in the CI image binding port 53 before a scenario’s own, grep -q under pipefail killing nft with SIGPIPE, kernel logging that doesn’t reach a namespace, and a runner with br_netfilter loaded that made bridged traffic hit the firewall. Some scenarios couldn’t pass at all, so I parked them in a pending/ directory instead of leaving main red: UPnP port mapping (miniupnpd answered 501 on the nftables backend and I never solved it), DHCPv6 prefix delegation for a while, and VRF.

    From Forgejo, October 16 to November 6: 416 runs, 209 success, 108 failure, 82 cancelled. About 294 commits.

    Features, and what they were proven on

    Multi-WAN with failover, QoS (CAKE, fq_codel), DHCPv6 prefix delegation, OpenVPN, IPsec road warriors, VRFs, zero-touch provisioning, factory reset, LDAP login, GitOps, and per-device segmentation with DNS policy and time limits. The 0.2.0 CHANGELOG lists its own limits: unmanaged switches, IPv6 binding “next”, and simulated radios only. Most of this was proven in namespaces and one lab VM, not on a production network.

    The ISO and 0.2.0

    On October 27, Patronus got a live installer ISO built with the same builder as Horcrux, plus a text installer with no default password and key-only SSH, aarch64 images and a console menu. The v0.2.0 release on the 30th (“device segmentation (opt-in), per-device Wi-Fi, DNS rebinding, per-device DNS policy, time limits, insight history, uplink health”) needed five migrations. The first release.yml didn’t produce a release, so another commit made it do so. Fat LTO of the web crate was OOM-killed in a 6 GB runner slot, so aarch64 uses thin LTO. The stable ISO gate also failed on stale red statuses from a superseded tag.

    Design decisions

    Headless first. Horcrux’s rule was “it should work on a headless server too, it shouldn’t be a desktop-only OS”, so on November 3 I made the same call for Patronus. A box is installed and recovered from the console, and operated over SSH and the web UI. There’s no desktop edition and none is planned. CI fails the build if a desktop or personal package lands in the image inputs. There’s also deliberately no private image variant: per-box data comes in via an answer file on the install USB and via GitOps.

    Encryption and boot: a firewall should never stop at boot waiting for a console passphrase. So the installer encrypts only if an unattended unlock works (a TPM2-sealed key, self-tested). Otherwise it installs unencrypted and says so.

    ZFS by default, with an XFS fallback below 8 GB (the disk page shows the reason). I want ZFS for recovery, bit rot protection and resilience. ext4 is never the default. Gentoo recommends XFS, and ZFS is the better filesystem for this job. There’s no on-disk swap on a firewall.

    Opt-in segmentation: the release ships with it off by default, and an upgrade test asserts that an upgraded box keeps isolation off.

    State on November 6

    The UI now requires a login, the installer passes a KVM test, and v0.2.0 is out with its limits written down.

    Other parts: Part 1, Part 2, Part 3, Part 4.

  • Horcrux dev log 3: back after nine months, 92 failed CI runs

    Horcrux dev log 3: back after nine months, 92 failed CI runs

    Part 3 in the Horcrux dev log series.

    Backdated, written in November 2026 from git and CI records. Part 2 stopped at an unfinished ISO on December 31, 2025.

    I came back on September 24, 2026 with two goals: get CI to really run, and build Horcrux the Gentoo way.

    The Gentoo goal came from a mistake. At some point the Dockerfile had turned into standard Rust boilerplate: FROM rust, cargo build, FROM debian-slim. It had drifted into boilerplate that ignored what the project is for. I fixed it on September 24 and wrote a rule into the repo notes so it can’t come back. The rule in the repo notes reads:

    A Dockerfile doing FROM rust:* -> cargo build -> FROM debian:*-slim
    is wrong. It is a regression. Fixed once already on 2026-09-24.

    Now the Dockerfile builds through real Portage against the project’s own ebuild, and the ebuild is the canonical install path.

    The numbers

    Pulled from the Forgejo API, September 24 to 30:

    workflow runs: 121
      success 12
      failure 92
      cancelled 11
      skipped 6
    ci.yml: 8 success, 71 failure
    docs.yml: 4 success, 19 failure

    88 commits that week, a lot of them fixes for whatever the last run complained about.

    What kept breaking

    The ebuild first. It depended on virtual/rust, which doesn’t exist (the package is dev-lang/rust), so emerge said “there are no ebuilds to satisfy”. dev-util/trunk doesn’t exist either, so the UI bundler gets installed with cargo during the build. qemu’s REQUIRED_USE says vte? ( gtk ) and I’d enabled vte alone. The Xen tools ebuild in that Gentoo snapshot died in src_prepare on an unset version variable; Horcrux only does KVM, so I dropped the Xen flag instead of fighting it. A pile of qemu and spice USE flags dead-ended in a masked libglvnd, and a headless VM host has no use for audio or GL anyway.

    Then the runners. Actions that weren’t mirrored on the registry my Forgejo uses. upload-artifact v4 not supported (“GHESNotSupportedError”). A runner with a full disk. A clippy step killed by the runner. A 30 minute ephemeral timeout that killed legitimate Gentoo builds. A full emerge that hit a 3 hour limit until I gave the CI image light USE flags. Several commit messages just say “retrigger” because the last failure was the infrastructure and not the code.

    Fake stuff that came out

    Real tests started finding placeholders. The login form wasn’t wired to the auth API. Disk usage in the metrics was a hardcoded 65%; it now reads statvfs. The Kubernetes join code accepted a made-up CA cert hash and now refuses it. HA fencing for IPMI, iLO, iDRAC and SNMP PDUs, parallel volume restore and live-migration pre-flight checks were stubs and got real code on the 30th. I don’t know yet how any of that behaves on actual hardware. release.yml only built x86_64 although the README claimed three architectures, so it now cross-compiles aarch64 and riscv64.

    On the 25th and 26th the API integration suite reached 22 of 22. It’s the first green test run I can find in the history.

    State

    The Gentoo build works in CI with trimmed USE flags, the tests run, and about one run in ten was green. The README still overpromised, and I rewrote its status line on October 21 (part 5).

    Design decisions

    Gentoo first. My reason is freedom to choose. USE flags are the main part: they let an installer choose which virtualization backends, container runtimes and NAS protocols get compiled in, and the repo notes call that the main advantage over Proxmox. Compiling for specific hardware is the secondary reason. For background, I was a Debian developer from 1999 to 2006 and I’ve used Gentoo since. I’m blunt about it in the repo notes: generic Debian or Ubuntu conventions are the wrong default here.

    Dropping Xen. The commit says Horcrux is a headless KVM-only VM host and has no need for Xen guest support. The same reasoning removed audio and GL from qemu: no local display on a headless host.

    Architectures: amd64, arm64 and riscv64. The README dropped ppc64le the same day as out of scope. That’s my call, and I don’t have the reason written down. The riscv64 notes do say it needs real hardware for validation.

    Light USE flags in CI: the full emerge hit a 3 hour limit, so the CI smoke-test image builds with fewer flags. A speed trade, and it means CI doesn’t build the full-feature package.

    Other parts: Part 1, Part 2, Part 4, Part 5, Part 6. The series continues with the latest Horcrux dev log post.

  • Patronus dev log 3: back in September, with 43 red CI runs and no green ones

    Patronus dev log 3: back in September, with 43 red CI runs and no green ones

    Part 3 in the Patronus dev log series.

    Backdated, written in November 2026 from git and CI records. Part 2 ended with nine months of silence.

    I picked Patronus back up the same day as Horcrux, September 24, 2026. The first job was the same: make CI run. Patronus got a Rust CI workflow that day (fmt, clippy, tests, an eBPF check, a full-workspace build). It failed. Over September 24 to 30 the Forgejo record shows 57 runs: 43 failures, 7 cancelled, 7 skipped, 0 successes. Of those, rust-ci.yml had 42 failures.

    The first failures were the dull kind: a missing libmnl-dev, a missing libelf, no linker on the runner image, and a real bug. An eBPF map-create call took its name as i8 and needed u8, which was blocking the detection and CNI crates from building at all.

    The Dockerfile again

    Same mistake as in Horcrux, found the same day. The Dockerfile had become a generic FROM rust build on a Debian slim image. The repo notes now say so, in short:

    A generic FROM rust -> cargo build -> debian slim Dockerfile is wrong.
    It crept in once and was fixed on 2026-09-24.

    Patronus builds through real Portage against its own overlay. The ebuild kept breaking, in familiar ways: virtual/rust doesn’t exist (same bug as Horcrux, the commit says so), acct-user and acct-group variable names were wrong, a USE flag was called carp in the code and ucarp in IUSE, the meta USE flags defaulted on with no sub-backend, and the ebuild forwarded cargo features that don’t exist (aliases, tc, prometheus). Building Rust from source inside the Docker build took 2 hours 45 minutes, so I switched to dev-lang/rust-bin. And the Gentoo Docker build got capped at 60 minutes and made non-blocking while I sorted it out.

    On the 29th the installer picked up root-on-ZFS, preferred when the ZFS tools are present.

    Design decisions

    Gentoo and USE flags: Gentoo is the core design point. My reason is freedom to choose. USE flags come first because they let you pick which features (web UI, eBPF/XDP, SD-WAN, detection, cloud integrations) get compiled in, and the repo notes call that the main advantage over pfSense and OPNsense. Hardware specific compiles are the second reason. A container build must pass USE flags through to emerge rather than compile a fixed set.

    rust-bin over source Rust: the from-source build took 2h45m (the commit’s number).

    Architecture: the repo notes have one rule: one SQLite store as the single source of truth, one “Applier” per subsystem, one apply step at boot, and no extra in-memory list, fixture or config file that only one process knows about.

    State on September 30

    57 runs, none green, and about 60 commits. The Gentoo build was non-blocking at the end of the week and still no run was green.

    Other parts: Part 1, Part 2, Part 4, Part 5.