CanuteTheGreat's BLOG

Tag: Horcrux dev log

Horcrux dev log 6: clustering, an ISO you can install from, and real laptops
Part 6 in the Horcrux dev log series.

Backdated: written in November 2026 from git, CI and my QA ledger. Part 5 was the audit.

With the basics fixed I moved on to clustering. In five days the log shows an HA supervisor (failure detection, fencing, recovery, self-fencing), cluster create and join, guest save state on host shutdown (QEMU migrate-to-file for VMs, CRIU for containers), draining a node with live moves, SDN zones, Ceph management, deduplicating backups, PCI and vGPU passthrough and a Redfish fencing driver.

On October 24 live migration got a hard block between AMD and Intel CPUs, and the drain planner never moves a running guest across vendors. Cold migration is still allowed.

From the Forgejo API for October 22 to November 6: 725 runs, 493 success, 78 failure, 154 cancelled, about 450 commits.
```
ha-failover.yml 68 success, 4 failure
vm-migration.yml 32 success, 0 failure
guest-power.yml 30 success, 7 failure
live-iso.yml 21 success, 4 failure, 26 cancelled
```
I haven’t checked every cancelled run, but newer pushes superseding a run in progress account for many of them.

October 27: an ISO that boots

The December ISO builder never got finished. This one has a different design. A Docker-only builder (Gentoo stage3 plus the official binhost, dracut with dmsquash-live, GRUB for BIOS and UEFI in a hybrid ISO), and a QEMU test that boots it by UEFI and USB and by BIOS and CD, runs an unattended install, reboots into the result and probes the web API. A daily job publishes only if the tests pass. First version: 31 files, about 1,000 lines.

By the 28th it had a text installer, an answer-file mode, no default password, key-only SSH and aarch64 images. This is the boot menu from that week, taken from the repo’s docs folder at the time:

Boot menu as committed on October 28, 2026 (docs/images/boot in the repo).

And the installer’s file system page, committed the same day:

File system page as committed October 28. The default is ext4, which was wrong and got changed within days.

The default is ext4. I’d decided ZFS should be the default, with an automatic XFS fallback under 8 GB of RAM or when there’s no ZFS module for the kernel.

T2 laptops, tablets, encrypted ZFS

There’s a GNOME desktop edition on top of the server core and hardware profiles for an Apple T2 MacBook and a Surface tablet. The installer learned encrypted ZFS root with boot environments, and hibernate on encrypted ZFS. The initramfs has to resume from the LUKS swap before it imports the pool, because importing first and resuming into it corrupts the pool. A QEMU test does install, passphrase, hibernate, power off, boot, passphrase, same process still running, clean scrub.

What real machines found

QEMU didn’t catch several things. The November 3 commits are the list the laptops produced.

The clock: a live system with a months-off RTC can’t do TLS, so Tailscale and package syncs failed. The installer now writes the hardware clock only from a verified clock, and the tests boot QEMU with -rtc base= months in the past. The size-trimming step on the desktop image kept deleting /usr/include and every static library, so installed systems had no C compiler and ZFS couldn’t rebuild its module on a kernel update. A zfs-kmod from a cached binary package had been built for a different kernel. Plain XFS and ext4 installs didn’t write /etc/fstab, so /boot wasn’t mounted and kernel updates landed on the root filesystem. GDM login started two keyrings. The disk passphrase could be reused as an account password, so the installer now refuses that. And the desktop had apps missing from the launcher because their binary didn’t exist, no sound, dead volume keys, and a Log Out button that did nothing, because the session was split across two D-Bus buses.

Most of these got a gate in the QEMU suite afterwards (months-off clock, compiler present, module builds, fstab, keyring, password reuse, the desktop session items). On November 3 I also fixed CI reporting green after a failed step (“green-but-failed 8637” in the commit), so some earlier runs looked healthier than they were.

Where it stands

This week I moved on to a lab of three old laptops to prove the HA claims. The goal in the lab plan: every scenario of the HA suite green ten runs in a row, failover under 120 seconds, no data loss, never two copies of a guest. As of this writing that’s 0 of 10. The old manager took 126 seconds to fail over after a hard crash, over the limit, and a one-node partition ran two copies of a guest for several seconds. Both are logged as bugs with fixes pending. So I can’t say Horcrux has HA yet. The next post is about that.

The aarch64 ISO build takes hours under emulation and hit the runner’s time cap on November 6, so it moved to a runner with a longer limit.

Design decisions

OpenRC is the only init on the ISOs. systemd wants to run the world, and that comes at a cost: feature bloat.

ZFS by default, with an automatic XFS fallback when RAM is under 8 GB or the kernel has no ZFS module. The installer shows its reason on the disk page (“XFS instead of ZFS: only 4.0 GB RAM (ZFS wants 8 GB or more)”). I want ZFS boot environments and an automatic snapshot before every update for recovery, bit rot protection and resilience. ext4 is never the default, only a manual choice. Gentoo recommends XFS, and ZFS is just superior tech.

Encrypted ZFS needs a static grub.cfg and a kernel per boot environment under /boot, because GRUB can’t read an encrypted ZFS pool.

Hibernate goes to a separate LUKS2 swap partition and not a zvol, because OpenZFS doesn’t support hibernating onto its own zvol and doing it anyway risks the pool. For the Surface XFS layout the notes reject a separate LUKS swap, since it would need a second keyslot or a second prompt.

Surface tablets get XFS plus LUKS and a light touch desktop, because the docs say a GNOME-class desktop doesn’t fit in 4 GB.

Server first. On November 3 I wrote down the rule: “it should work on a headless server too, it shouldn’t be a desktop-only OS”. The desktop is a layer on top of a headless core and personal apps live in per-machine layers, not the public images.

Live migration between AMD and Intel is refused and cold migration is allowed (see Oct 24 above).

Other parts: Part 1, Part 2, Part 3, Part 4, Part 5. The series continues with the latest Horcrux dev log post.
November 6, 2026
Horcrux dev log 5: auditing my own project, and what turned out to be fake
Part 5 in the Horcrux dev log series.

Backdated: written in November 2026 from the repo, CI and my QA ledger. Part 4 ended with a feature list I didn’t trust.

On October 16 I compared Horcrux against Proxmox VE. The method: take every route from the router (418 .route( calls), take every route the UI calls and match them with a script, then for each feature check whether anything actually calls the code. The result is a document with a section titled “what is fake today”.

The first findings were bad. VMs couldn’t install an OS or get a network: the QEMU command line was memory, CPUs, one disk and -display none, with no ISO, no NIC, no boot order, no UEFI and no way to edit a VM after creating it. Nothing created a QMP socket, so snapshots, migration health checks and “suspend” backups quietly did nothing. Backups tarred a directory the code itself had just created empty and never touched the VM disk. And all configuration lived in memory, so storage pools, firewall rules, HA groups, templates, alert rules, replication jobs and backup jobs vanished on restart.

My QA ledger from the same day has more. Firewall Apply installed no rules at all (the builder emitted tcp tcp dport, and the UI posted to a route that didn’t exist). Directory storage pools reported 0 GB free so every volume create failed. VM consoles didn’t work since QEMU ran with no display and no VNC or SPICE listener. Several UI buttons had no handler: monitoring “Configure Alerts” and “Export Data”, NAS “Create Group”, the user edit icon.

The worst entry is marked critical. /api/auth/register was public and the RBAC middleware was never applied, so any self-registered account had full control, and the API ran over plain HTTP.

Finding that a week after calling it feature complete is not fun. The ledger got 44 new rows on the 16th and 39 on the 17th.

The fixing week

Forgejo for October 16 to 21: 348 runs, 208 success, 44 failure, 34 cancelled, 62 skipped, over about 170 commits. Everything in memory moved into the database with migrations. VM backups became real (QMP pause or snapshot, flushed disks, restore, retention, verify). VNC, SPICE with audio, clipboard and USB redirection, and serial consoles got wired up, plus in-browser consoles for LXD and Incus. Auth got rewritten: users, roles, groups, ACLs, LDAP/AD/OIDC/PAM realms, 2FA, RBAC on every route, HTTPS by default, and a test that runs every built-in role against every route. VMs got NICs, an ISO library, boot order, multiple disks, UEFI, edit, reboot and a per-VM firewall. Peer-to-peer migration between nodes became real.

What only a real run found

On the 17th a test caught this one, which I’ll quote:
```
QEMU adds a default user-mode (SLIRP) e1000 NIC when no -nic/-netdev is
given. It NATs the guest through the host's own network stack, so on an
EC2 host a guest read the host's instance metadata (CirrOS printed the
host's instance-id) and could fetch its IAM role credentials.
```
The fix always passes an explicit no-NIC option, and a test boots a real VM and asks QEMU over QMP that it has none.

Same class, next few days: disk import followed qcow2 external data files and read other files on the host (Oct 19). URL downloads for disks and ISOs could reach the node’s own services (Oct 19 and 20). Volume names could escape their storage pool by path traversal (Oct 20).

Mutation testing

I also added cargo-mutants runs per module group. The first storage measurement was awful. The commit says it flatly: “StorageManager – mutation baseline was 0%”. Killing survivors became routine; 17 surviving mutants in the VM snapshot code was one batch. I also split the giant API binary into crates on the 17th (core, nas, k8s, sdn, auth, vm, container, cluster, backup), with CI moving to mold and line-tables-only builds the same day.

Still broken on the 21st

Some features existed in the API and not the UI, or the reverse. The cluster code had been tested with containers as nodes, not on separate real machines, as far as I can tell from the CI. On the 21st I rewrote the status line to “in active development, not yet suitable for production use” and cut the stale ones.

Design decisions

No implicit NIC: a VM with no network configuration gets no network at all rather than QEMU’s default user-mode one, because that default NATs through the host and a guest reached the host’s cloud metadata service. URL downloads got the same rule (they can’t reach the node’s own services or metadata). HTTPS by default with a node-local CA came in with the auth rewrite.

Snapshot-mode backups don’t pause a running VM. On the 16th it paused (QMP stop and cont); on the 19th the commit “backs up a running VM without pausing it” made it a block-level snapshot. The earlier QA entry says why: consistent but not pause-free, and Proxmox-style backup needs a copy-before-write job.

One call about the test jobs is documented: on October 18 mutation and chaos jobs went back to every runner, because the colo-only lane’s backlog was too big.

Other parts: Part 1, Part 2, Part 3, Part 4, Part 6. The series continues with the latest Horcrux dev log post.
October 21, 2026
Horcrux dev log 4: nightly tests, and a NAS login that accepted any password
Part 4 in the Horcrux dev log series.

Backdated: written in November 2026 from git and CI records. Part 3 ended with CI finally running.

On October 4 a nightly workflow went in that boots an actual VM (CirrOS) and runs real container and CNI network tests. On the 7th a stress suite followed, with workers logging in and hammering the API. Both run on my own runners.

119 commits in the fortnight. From the Forgejo API for October 1 to 15:
```
all workflows: 233 runs (140 success, 68 failure, 18 cancelled)
ci.yml: 53 success, 43 failure, 13 cancelled
nightly-realistic-test: 18 success, 17 failure
nightly-stress-test: 14 success, 3 failure
```
About half the realistic nightlies failed. I haven’t broken those 17 failures down by cause, but the boot timeout was one of them. The VM boot deadline was 120 seconds and the runners are TCG-only (no KVM), so a commit on the 8th raised it to 240. On the 15th the commit message says “was never enough”.

The CNI bugs

The commit on October 4 says the nightly tests caught two real bugs in CNI and container handling that day, without naming them. My code assumed plugin responses looked one way and real plugins disagreed: cniVersion is camelCase, real plugins leave out empty optional fields, and the stdin payload needed its field names translated as well. Deleting a network also had to be refused while containers were still attached (Oct 5).

Other fixes from those two weeks, quoted or paraphrased from the commit messages:

Webhook signatures used Rust’s DefaultHasher, which is not a MAC. Now HMAC-SHA256 (Oct 11). TOTP and backup codes were compared with a plain ==. Now constant time (Oct 11). NAS password checking looked up the user in /etc/shadow and then returned success without comparing anything. “Any password, including an empty one, authenticated successfully for any local user.” That sat under SMB, NFS, FTP and WebDAV auth (Oct 11). Newlines and NULs could be injected into the Kerberos admin scripts and the generated smb.conf (Oct 13). Audit logs believed a client-supplied X-Forwarded-For (Oct 3). A non-ASCII Origin header panicked the CORS layer (Oct 12). NetworkPolicy enforced ports and ignored peer selectors and IP blocks (Oct 9).

The commit messages don’t say which of these a test caught and which got spotted by reading, so I won’t claim the tests found them. On the 14th the audit job did flag four new RUSTSEC advisories against rustls, and that was a version bump.

Smaller embarrassments

“Docker image was building frozen v0.1.0, not HEAD” (Oct 8). Trunk never built main.css, which means the nav dropdowns had no styling at all (Oct 9). The UI hardcoded http://localhost:8006 for API and WebSocket calls, fine if you browse from the server itself and nowhere else (Oct 12 and 15). A VM CPU metric wasn’t divided by vCPU count. Poisoned locks would panic the process (Oct 15).

What I took from it

The NAS bypass shipped inside a feature I’d already called complete.

State on October 15: 53 passing runs against 43 failing on the main workflow and about half the nightlies red. Part 5 is the audit I started on the 16th.

Design decisions

Not much is written down for this fortnight. The one clear call is that deleting a CNI network is refused while containers are attached to it (Oct 5).

Other parts: Part 1, Part 2, Part 3, Part 5, Part 6. The series continues with the latest Horcrux dev log post.
October 15, 2026