CanuteTheGreat's BLOG

Tag: rust

  • Horcrux dev log 5: auditing my own project, and what turned out to be fake

    Horcrux dev log 5: auditing my own project, and what turned out to be fake

    Part 5 in the Horcrux dev log series.

    Backdated: written in November 2026 from the repo, CI and my QA ledger. Part 4 ended with a feature list I didn’t trust.

    On October 16 I compared Horcrux against Proxmox VE. The method: take every route from the router (418 .route( calls), take every route the UI calls and match them with a script, then for each feature check whether anything actually calls the code. The result is a document with a section titled “what is fake today”.

    The first findings were bad. VMs couldn’t install an OS or get a network: the QEMU command line was memory, CPUs, one disk and -display none, with no ISO, no NIC, no boot order, no UEFI and no way to edit a VM after creating it. Nothing created a QMP socket, so snapshots, migration health checks and “suspend” backups quietly did nothing. Backups tarred a directory the code itself had just created empty and never touched the VM disk. And all configuration lived in memory, so storage pools, firewall rules, HA groups, templates, alert rules, replication jobs and backup jobs vanished on restart.

    My QA ledger from the same day has more. Firewall Apply installed no rules at all (the builder emitted tcp tcp dport, and the UI posted to a route that didn’t exist). Directory storage pools reported 0 GB free so every volume create failed. VM consoles didn’t work since QEMU ran with no display and no VNC or SPICE listener. Several UI buttons had no handler: monitoring “Configure Alerts” and “Export Data”, NAS “Create Group”, the user edit icon.

    The worst entry is marked critical. /api/auth/register was public and the RBAC middleware was never applied, so any self-registered account had full control, and the API ran over plain HTTP.

    Finding that a week after calling it feature complete is not fun. The ledger got 44 new rows on the 16th and 39 on the 17th.

    The fixing week

    Forgejo for October 16 to 21: 348 runs, 208 success, 44 failure, 34 cancelled, 62 skipped, over about 170 commits. Everything in memory moved into the database with migrations. VM backups became real (QMP pause or snapshot, flushed disks, restore, retention, verify). VNC, SPICE with audio, clipboard and USB redirection, and serial consoles got wired up, plus in-browser consoles for LXD and Incus. Auth got rewritten: users, roles, groups, ACLs, LDAP/AD/OIDC/PAM realms, 2FA, RBAC on every route, HTTPS by default, and a test that runs every built-in role against every route. VMs got NICs, an ISO library, boot order, multiple disks, UEFI, edit, reboot and a per-VM firewall. Peer-to-peer migration between nodes became real.

    What only a real run found

    On the 17th a test caught this one, which I’ll quote:

    QEMU adds a default user-mode (SLIRP) e1000 NIC when no -nic/-netdev is
given. It NATs the guest through the host's own network stack, so on an
EC2 host a guest read the host's instance metadata (CirrOS printed the
host's instance-id) and could fetch its IAM role credentials.

    The fix always passes an explicit no-NIC option, and a test boots a real VM and asks QEMU over QMP that it has none.

    Same class, next few days: disk import followed qcow2 external data files and read other files on the host (Oct 19). URL downloads for disks and ISOs could reach the node’s own services (Oct 19 and 20). Volume names could escape their storage pool by path traversal (Oct 20).

    Mutation testing

    I also added cargo-mutants runs per module group. The first storage measurement was awful. The commit says it flatly: “StorageManager – mutation baseline was 0%”. Killing survivors became routine; 17 surviving mutants in the VM snapshot code was one batch. I also split the giant API binary into crates on the 17th (core, nas, k8s, sdn, auth, vm, container, cluster, backup), with CI moving to mold and line-tables-only builds the same day.

    Still broken on the 21st

    Some features existed in the API and not the UI, or the reverse. The cluster code had been tested with containers as nodes, not on separate real machines, as far as I can tell from the CI. On the 21st I rewrote the status line to “in active development, not yet suitable for production use” and cut the stale ones.

    Design decisions

    No implicit NIC: a VM with no network configuration gets no network at all rather than QEMU’s default user-mode one, because that default NATs through the host and a guest reached the host’s cloud metadata service. URL downloads got the same rule (they can’t reach the node’s own services or metadata). HTTPS by default with a node-local CA came in with the auth rewrite.

    Snapshot-mode backups don’t pause a running VM. On the 16th it paused (QMP stop and cont); on the 19th the commit “backs up a running VM without pausing it” made it a block-level snapshot. The earlier QA entry says why: consistent but not pause-free, and Proxmox-style backup needs a copy-before-write job.

    One call about the test jobs is documented: on October 18 mutation and chaos jobs went back to every runner, because the colo-only lane’s backlog was too big.

    Other parts: Part 1, Part 2, Part 3, Part 4, Part 6. The series continues with the latest Horcrux dev log post.

  • Horcrux dev log 4: nightly tests, and a NAS login that accepted any password

    Horcrux dev log 4: nightly tests, and a NAS login that accepted any password

    Part 4 in the Horcrux dev log series.

    Backdated: written in November 2026 from git and CI records. Part 3 ended with CI finally running.

    On October 4 a nightly workflow went in that boots an actual VM (CirrOS) and runs real container and CNI network tests. On the 7th a stress suite followed, with workers logging in and hammering the API. Both run on my own runners.

    119 commits in the fortnight. From the Forgejo API for October 1 to 15:

    all workflows: 233 runs (140 success, 68 failure, 18 cancelled)
ci.yml: 53 success, 43 failure, 13 cancelled
nightly-realistic-test: 18 success, 17 failure
nightly-stress-test: 14 success, 3 failure

    About half the realistic nightlies failed. I haven’t broken those 17 failures down by cause, but the boot timeout was one of them. The VM boot deadline was 120 seconds and the runners are TCG-only (no KVM), so a commit on the 8th raised it to 240. On the 15th the commit message says “was never enough”.

    The CNI bugs

    The commit on October 4 says the nightly tests caught two real bugs in CNI and container handling that day, without naming them. My code assumed plugin responses looked one way and real plugins disagreed: cniVersion is camelCase, real plugins leave out empty optional fields, and the stdin payload needed its field names translated as well. Deleting a network also had to be refused while containers were still attached (Oct 5).

    Other fixes from those two weeks, quoted or paraphrased from the commit messages:

    Webhook signatures used Rust’s DefaultHasher, which is not a MAC. Now HMAC-SHA256 (Oct 11). TOTP and backup codes were compared with a plain ==. Now constant time (Oct 11). NAS password checking looked up the user in /etc/shadow and then returned success without comparing anything. “Any password, including an empty one, authenticated successfully for any local user.” That sat under SMB, NFS, FTP and WebDAV auth (Oct 11). Newlines and NULs could be injected into the Kerberos admin scripts and the generated smb.conf (Oct 13). Audit logs believed a client-supplied X-Forwarded-For (Oct 3). A non-ASCII Origin header panicked the CORS layer (Oct 12). NetworkPolicy enforced ports and ignored peer selectors and IP blocks (Oct 9).

    The commit messages don’t say which of these a test caught and which got spotted by reading, so I won’t claim the tests found them. On the 14th the audit job did flag four new RUSTSEC advisories against rustls, and that was a version bump.

    Smaller embarrassments

    “Docker image was building frozen v0.1.0, not HEAD” (Oct 8). Trunk never built main.css, which means the nav dropdowns had no styling at all (Oct 9). The UI hardcoded http://localhost:8006 for API and WebSocket calls, fine if you browse from the server itself and nowhere else (Oct 12 and 15). A VM CPU metric wasn’t divided by vCPU count. Poisoned locks would panic the process (Oct 15).

    What I took from it

    The NAS bypass shipped inside a feature I’d already called complete.

    State on October 15: 53 passing runs against 43 failing on the main workflow and about half the nightlies red. Part 5 is the audit I started on the 16th.

    Design decisions

    Not much is written down for this fortnight. The one clear call is that deleting a CNI network is refused while containers are attached to it (Oct 5).

    Other parts: Part 1, Part 2, Part 3, Part 5, Part 6. The series continues with the latest Horcrux dev log post.

  • Horcrux dev log 3: back after nine months, 92 failed CI runs

    Horcrux dev log 3: back after nine months, 92 failed CI runs

    Part 3 in the Horcrux dev log series.

    Backdated, written in November 2026 from git and CI records. Part 2 stopped at an unfinished ISO on December 31, 2025.

    I came back on September 24, 2026 with two goals: get CI to really run, and build Horcrux the Gentoo way.

    The Gentoo goal came from a mistake. At some point the Dockerfile had turned into standard Rust boilerplate: FROM rust, cargo build, FROM debian-slim. It had drifted into boilerplate that ignored what the project is for. I fixed it on September 24 and wrote a rule into the repo notes so it can’t come back. The rule in the repo notes reads:

    A Dockerfile doing FROM rust:* -> cargo build -> FROM debian:*-slim
is wrong. It is a regression. Fixed once already on 2026-09-24.

    Now the Dockerfile builds through real Portage against the project’s own ebuild, and the ebuild is the canonical install path.

    The numbers

    Pulled from the Forgejo API, September 24 to 30:

    workflow runs: 121
  success 12
  failure 92
  cancelled 11
  skipped 6
ci.yml: 8 success, 71 failure
docs.yml: 4 success, 19 failure

    88 commits that week, a lot of them fixes for whatever the last run complained about.

    What kept breaking

    The ebuild first. It depended on virtual/rust, which doesn’t exist (the package is dev-lang/rust), so emerge said “there are no ebuilds to satisfy”. dev-util/trunk doesn’t exist either, so the UI bundler gets installed with cargo during the build. qemu’s REQUIRED_USE says vte? ( gtk ) and I’d enabled vte alone. The Xen tools ebuild in that Gentoo snapshot died in src_prepare on an unset version variable; Horcrux only does KVM, so I dropped the Xen flag instead of fighting it. A pile of qemu and spice USE flags dead-ended in a masked libglvnd, and a headless VM host has no use for audio or GL anyway.

    Then the runners. Actions that weren’t mirrored on the registry my Forgejo uses. upload-artifact v4 not supported (“GHESNotSupportedError”). A runner with a full disk. A clippy step killed by the runner. A 30 minute ephemeral timeout that killed legitimate Gentoo builds. A full emerge that hit a 3 hour limit until I gave the CI image light USE flags. Several commit messages just say “retrigger” because the last failure was the infrastructure and not the code.

    Fake stuff that came out

    Real tests started finding placeholders. The login form wasn’t wired to the auth API. Disk usage in the metrics was a hardcoded 65%; it now reads statvfs. The Kubernetes join code accepted a made-up CA cert hash and now refuses it. HA fencing for IPMI, iLO, iDRAC and SNMP PDUs, parallel volume restore and live-migration pre-flight checks were stubs and got real code on the 30th. I don’t know yet how any of that behaves on actual hardware. release.yml only built x86_64 although the README claimed three architectures, so it now cross-compiles aarch64 and riscv64.

    On the 25th and 26th the API integration suite reached 22 of 22. It’s the first green test run I can find in the history.

    State

    The Gentoo build works in CI with trimmed USE flags, the tests run, and about one run in ten was green. The README still overpromised, and I rewrote its status line on October 21 (part 5).

    Design decisions

    Gentoo first. My reason is freedom to choose. USE flags are the main part: they let an installer choose which virtualization backends, container runtimes and NAS protocols get compiled in, and the repo notes call that the main advantage over Proxmox. Compiling for specific hardware is the secondary reason. For background, I was a Debian developer from 1999 to 2006 and I’ve used Gentoo since. I’m blunt about it in the repo notes: generic Debian or Ubuntu conventions are the wrong default here.

    Dropping Xen. The commit says Horcrux is a headless KVM-only VM host and has no need for Xen guest support. The same reasoning removed audio and GL from qemu: no local display on a headless host.

    Architectures: amd64, arm64 and riscv64. The README dropped ppc64le the same day as out of scope. That’s my call, and I don’t have the reason written down. The riscv64 notes do say it needs real hardware for validation.

    Light USE flags in CI: the full emerge hit a 3 hour limit, so the CI smoke-test image builds with fewer flags. A speed trade, and it means CI doesn’t build the full-feature package.

    Other parts: Part 1, Part 2, Part 4, Part 5, Part 6. The series continues with the latest Horcrux dev log post.