Part 5 in the Horcrux dev log series.
Backdated: written in November 2026 from the repo, CI and my QA ledger. Part 4 ended with a feature list I didn’t trust.
On October 16 I compared Horcrux against Proxmox VE. The method: take every route from the router (418 .route( calls), take every route the UI calls and match them with a script, then for each feature check whether anything actually calls the code. The result is a document with a section titled “what is fake today”.
The first findings were bad. VMs couldn’t install an OS or get a network: the QEMU command line was memory, CPUs, one disk and -display none, with no ISO, no NIC, no boot order, no UEFI and no way to edit a VM after creating it. Nothing created a QMP socket, so snapshots, migration health checks and “suspend” backups quietly did nothing. Backups tarred a directory the code itself had just created empty and never touched the VM disk. And all configuration lived in memory, so storage pools, firewall rules, HA groups, templates, alert rules, replication jobs and backup jobs vanished on restart.
My QA ledger from the same day has more. Firewall Apply installed no rules at all (the builder emitted tcp tcp dport, and the UI posted to a route that didn’t exist). Directory storage pools reported 0 GB free so every volume create failed. VM consoles didn’t work since QEMU ran with no display and no VNC or SPICE listener. Several UI buttons had no handler: monitoring “Configure Alerts” and “Export Data”, NAS “Create Group”, the user edit icon.
The worst entry is marked critical. /api/auth/register was public and the RBAC middleware was never applied, so any self-registered account had full control, and the API ran over plain HTTP.
Finding that a week after calling it feature complete is not fun. The ledger got 44 new rows on the 16th and 39 on the 17th.
The fixing week
Forgejo for October 16 to 21: 348 runs, 208 success, 44 failure, 34 cancelled, 62 skipped, over about 170 commits. Everything in memory moved into the database with migrations. VM backups became real (QMP pause or snapshot, flushed disks, restore, retention, verify). VNC, SPICE with audio, clipboard and USB redirection, and serial consoles got wired up, plus in-browser consoles for LXD and Incus. Auth got rewritten: users, roles, groups, ACLs, LDAP/AD/OIDC/PAM realms, 2FA, RBAC on every route, HTTPS by default, and a test that runs every built-in role against every route. VMs got NICs, an ISO library, boot order, multiple disks, UEFI, edit, reboot and a per-VM firewall. Peer-to-peer migration between nodes became real.
What only a real run found
On the 17th a test caught this one, which I’ll quote:
QEMU adds a default user-mode (SLIRP) e1000 NIC when no -nic/-netdev is
given. It NATs the guest through the host's own network stack, so on an
EC2 host a guest read the host's instance metadata (CirrOS printed the
host's instance-id) and could fetch its IAM role credentials.The fix always passes an explicit no-NIC option, and a test boots a real VM and asks QEMU over QMP that it has none.
Same class, next few days: disk import followed qcow2 external data files and read other files on the host (Oct 19). URL downloads for disks and ISOs could reach the node’s own services (Oct 19 and 20). Volume names could escape their storage pool by path traversal (Oct 20).
Mutation testing
I also added cargo-mutants runs per module group. The first storage measurement was awful. The commit says it flatly: “StorageManager – mutation baseline was 0%”. Killing survivors became routine; 17 surviving mutants in the VM snapshot code was one batch. I also split the giant API binary into crates on the 17th (core, nas, k8s, sdn, auth, vm, container, cluster, backup), with CI moving to mold and line-tables-only builds the same day.
Still broken on the 21st
Some features existed in the API and not the UI, or the reverse. The cluster code had been tested with containers as nodes, not on separate real machines, as far as I can tell from the CI. On the 21st I rewrote the status line to “in active development, not yet suitable for production use” and cut the stale ones.
Design decisions
No implicit NIC: a VM with no network configuration gets no network at all rather than QEMU’s default user-mode one, because that default NATs through the host and a guest reached the host’s cloud metadata service. URL downloads got the same rule (they can’t reach the node’s own services or metadata). HTTPS by default with a node-local CA came in with the auth rewrite.
Snapshot-mode backups don’t pause a running VM. On the 16th it paused (QMP stop and cont); on the 19th the commit “backs up a running VM without pausing it” made it a block-level snapshot. The earlier QA entry says why: consistent but not pause-free, and Proxmox-style backup needs a copy-before-write job.
One call about the test jobs is documented: on October 18 mutation and chaos jobs went back to every runner, because the colo-only lane’s backlog was too big.
Other parts: Part 1, Part 2, Part 3, Part 4, Part 6. The series continues with the latest Horcrux dev log post.