Part 4 in the Horcrux dev log series.
Backdated: written in November 2026 from git and CI records. Part 3 ended with CI finally running.
On October 4 a nightly workflow went in that boots an actual VM (CirrOS) and runs real container and CNI network tests. On the 7th a stress suite followed, with workers logging in and hammering the API. Both run on my own runners.
119 commits in the fortnight. From the Forgejo API for October 1 to 15:
all workflows: 233 runs (140 success, 68 failure, 18 cancelled)
ci.yml: 53 success, 43 failure, 13 cancelled
nightly-realistic-test: 18 success, 17 failure
nightly-stress-test: 14 success, 3 failureAbout half the realistic nightlies failed. I haven’t broken those 17 failures down by cause, but the boot timeout was one of them. The VM boot deadline was 120 seconds and the runners are TCG-only (no KVM), so a commit on the 8th raised it to 240. On the 15th the commit message says “was never enough”.
The CNI bugs
The commit on October 4 says the nightly tests caught two real bugs in CNI and container handling that day, without naming them. My code assumed plugin responses looked one way and real plugins disagreed: cniVersion is camelCase, real plugins leave out empty optional fields, and the stdin payload needed its field names translated as well. Deleting a network also had to be refused while containers were still attached (Oct 5).
Other fixes from those two weeks, quoted or paraphrased from the commit messages:
Webhook signatures used Rust’s DefaultHasher, which is not a MAC. Now HMAC-SHA256 (Oct 11). TOTP and backup codes were compared with a plain ==. Now constant time (Oct 11). NAS password checking looked up the user in /etc/shadow and then returned success without comparing anything. “Any password, including an empty one, authenticated successfully for any local user.” That sat under SMB, NFS, FTP and WebDAV auth (Oct 11). Newlines and NULs could be injected into the Kerberos admin scripts and the generated smb.conf (Oct 13). Audit logs believed a client-supplied X-Forwarded-For (Oct 3). A non-ASCII Origin header panicked the CORS layer (Oct 12). NetworkPolicy enforced ports and ignored peer selectors and IP blocks (Oct 9).
The commit messages don’t say which of these a test caught and which got spotted by reading, so I won’t claim the tests found them. On the 14th the audit job did flag four new RUSTSEC advisories against rustls, and that was a version bump.
Smaller embarrassments
“Docker image was building frozen v0.1.0, not HEAD” (Oct 8). Trunk never built main.css, which means the nav dropdowns had no styling at all (Oct 9). The UI hardcoded http://localhost:8006 for API and WebSocket calls, fine if you browse from the server itself and nowhere else (Oct 12 and 15). A VM CPU metric wasn’t divided by vCPU count. Poisoned locks would panic the process (Oct 15).
What I took from it
The NAS bypass shipped inside a feature I’d already called complete.
State on October 15: 53 passing runs against 43 failing on the main workflow and about half the nightlies red. Part 5 is the audit I started on the 16th.
Design decisions
Not much is written down for this fortnight. The one clear call is that deleting a CNI network is refused while containers are attached to it (Oct 5).
Other parts: Part 1, Part 2, Part 3, Part 5, Part 6. The series continues with the latest Horcrux dev log post.