{"id":592,"date":"2026-10-15T19:26:44","date_gmt":"2026-10-16T02:26:44","guid":{"rendered":"https:\/\/blog.canutethegreat.com\/?p=592"},"modified":"2026-11-07T18:11:37","modified_gmt":"2026-11-08T02:11:37","slug":"patronus-dev-log-4-an-installer-test-that-failed-57-times-in-a-row","status":"publish","type":"post","link":"https:\/\/blog.canutethegreat.com\/index.php\/2026\/10\/15\/patronus-dev-log-4-an-installer-test-that-failed-57-times-in-a-row\/","title":{"rendered":"Patronus dev log 4: an installer test that failed 57 times in a row"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><em>Part 4 in the Patronus dev log series.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Backdated, written in November 2026 from git and CI records. <a href=\"https:\/\/blog.canutethegreat.com\/index.php\/2026\/09\/29\/patronus-dev-log-3-back-in-september-with-43-red-ci-runs-and-no-green-ones\/\">Part 3<\/a> was the restart.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The installer test<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I wanted a nightly test that installs Patronus into a real VM, with real ZFS, and reboots into it. On October 4 it went in as a workflow. It ran on a Mac build host through UTM, so an x86 guest was emulated on an ARM machine.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It never worked. From the Forgejo record, October 1 to 15, <code>nightly-realistic-installer-test.yml<\/code>: 57 failures, 6 cancelled, 0 successes. The overall repo had 249 runs those two weeks: 87 success, 116 failure, 36 cancelled, 10 skipped.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What the log looks like:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Oct 4 and 5: file handoff between the VM and the host. First assumed instant, then polled for 60 seconds, then 180.<\/li><li>Oct 5: a serial console capture finally showed where boot stuck. GRUB 2.12 showed its menu, printed &#8220;Booting `Linux lts'&#8221; and then nothing. I switched the CD-ROM from USB to IDE. The commit message calls it the most likely cause, not a proven one.<\/li><li>Oct 5: &#8220;retry the live-boot stage &#8211; confirmed non-deterministic&#8221;.<\/li><li>Oct 10: a failing screenshot step turned out to be macOS denying Screen Recording permission (TCC), not a product bug.<\/li><li>Oct 13: boot budget pushed to 2700 seconds and the job ceiling to 120 minutes.<\/li><li>Oct 14: &#8220;pause nightly installer test schedule (TCC emulation can&#8217;t finish in time)&#8221;, where TCC is a typo for TCG, QEMU&#8217;s software emulation.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Part 5 is what replaced it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Bugs that mattered more<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Meanwhile, in the same two weeks:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>The firewall validated nothing before building nft scripts, so a rule field could inject ruleset text (Oct 12). Fields are now validated first.<\/li><li>The web UI&#8217;s NAT rules changed an in-memory list and never reached nftables (Oct 12). <\/li><li>A literal interface name &#8220;any&#8221; or &#8220;all&#8221; was spliced into nft syntax (Oct 13).<\/li><li>Captive portal MAC addresses weren&#8217;t validated before use in nft commands (Oct 11).<\/li><li>Dashboard threat and alert counts were constants (42, 38 and 3) whatever the real state. A commit on the 14th derived them from real data and listed what was still a placeholder: model accuracy, health score and packet rate, &#8220;no real ML classifier or packet-rate counter exists yet to back them honestly&#8221;.<\/li><li>On Oct 13, the container image couldn&#8217;t run nft as a non-root user until it got <code>CAP_NET_ADMIN<\/code> and <code>CAP_NET_RAW<\/code> on the binary.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Design decisions<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The commit that replaced the UTM setup says x86 under TCG never booted there.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">State on October 15<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">CI on main in the fortnight: 66 successes and 43 failures.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Other parts: <a href=\"https:\/\/blog.canutethegreat.com\/index.php\/2025\/10\/11\/patronus-dev-log-1-a-firewall-that-claimed-100-feature-parity-on-day-one\/\">Part 1<\/a>, <a href=\"https:\/\/blog.canutethegreat.com\/index.php\/2025\/12\/18\/patronus-dev-log-2-kubernetes-a-react-ui-and-a-rust-workspace-that-wouldnt-build\/\">Part 2<\/a>, <a href=\"https:\/\/blog.canutethegreat.com\/index.php\/2026\/09\/29\/patronus-dev-log-3-back-in-september-with-43-red-ci-runs-and-no-green-ones\/\">Part 3<\/a>, <a href=\"https:\/\/blog.canutethegreat.com\/index.php\/2026\/11\/06\/patronus-dev-log-5-security-holes-netlab-and-the-0-2-0-release\/\">Part 5<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>October 1 to 15, 2026: a nightly &#8220;real VM install&#8221; test that never got past boot, and a handful of firewall bugs that were more important than the test.<\/p>\n","protected":false},"author":1,"featured_media":616,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,26],"tags":[366,375,373,370,378],"class_list":["post-592","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-security","category-software-engineering","tag-ci","tag-nftables","tag-patronus-dev-log","tag-qemu","tag-utm"],"_links":{"self":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/592","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/comments?post=592"}],"version-history":[{"count":3,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/592\/revisions"}],"predecessor-version":[{"id":639,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/592\/revisions\/639"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/media\/616"}],"wp:attachment":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/media?parent=592"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/categories?post=592"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/tags?post=592"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}