{"id":766,"date":"2026-11-14T20:07:00","date_gmt":"2026-11-15T04:07:00","guid":{"rendered":"https:\/\/blog.canutethegreat.com\/?p=766"},"modified":"2026-11-14T11:50:28","modified_gmt":"2026-11-14T19:50:28","slug":"horcrux-i-broke-docker-on-purpose-and-rolled-it-back-from-the-web-ui","status":"publish","type":"post","link":"https:\/\/blog.canutethegreat.com\/index.php\/2026\/11\/14\/horcrux-i-broke-docker-on-purpose-and-rolled-it-back-from-the-web-ui\/","title":{"rendered":"Horcrux: I broke Docker on purpose and rolled it back from the web UI"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Full disclosure: I make this. Horcrux is my Gentoo-based hypervisor, it&#8217;s pre-1.0, and this is my notebook for the week.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Friday night I merged a Boot Environments page into the web UI. The command line tool underneath, horcrux-be, has been around for a while, but you had to know it existed. Now there&#8217;s a page that lists every root file system you can boot, the snapshots, and a yellow button that says &#8220;Roll back to previous version&#8221;. So on Saturday I installed the daily server image into a throwaway VM on one of my lab laptops, put some containers on it, and broke it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The setup<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">ZFS root, OpenRC, 2 vCPUs, 6 GB of RAM. Three containers made from the Containers page: nginx, redis, and an alpine box that only sleeps. Every row carries a warning that CRIU isn&#8217;t installed so container state can&#8217;t be saved. It&#8217;s printed on all three rows, word for word.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-containers-before-upgrade.png\" alt=\"Horcrux Containers page listing three running Docker containers, demo-cache, demo-web and demo-worker, each with a warning that CRIU is not installed\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">The three demo containers before the upgrade.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Upgrade All takes a snapshot first<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I hit Upgrade All on the Packages page. Emerge started at 20:16:54 (VM clock; the VM ended up on Berlin time, more on that below) and six seconds later there was a snapshot named horcrux@pre-upgrade-20261114-201700. Then 17 binary packages went in, a new build of Horcrux itself among them, and emerge was done at 20:18:03. 69 seconds. The snapshot was 21 MB at that point.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The web UI doesn&#8217;t take that snapshot. A hook in Portage&#8217;s bashrc does, before the first package of any emerge run, at most once every 30 minutes. Emerge from a shell gets one just like the API does.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-pre-upgrade-snapshot.png\" alt=\"Horcrux Boot Environments page showing one boot environment named horcrux and an automatic snapshot horcrux@pre-upgrade-20261114-201700\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">The snapshot Upgrade All took, on the Boot Environments page.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-snapshot-and-emerge-log.png\" alt=\"Terminal output: zfs list shows the pre-upgrade snapshot, emerge.log shows 17 of 17 packages completed, horcrux-be list shows the snapshot\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">The same snapshot from a shell, with the 17 packages emerge installed after it. The VM clock is on Berlin time.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Then I broke it<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Packages page, search for docker, Remove. The confirm dialog asks &#8220;Remove package &#8216;&#8230;&#8217;?&#8221; and that&#8217;s it. Nothing about the four containers running on it (I&#8217;d made a fourth one, demo-late, 13 seconds earlier). Six seconds of emerge &#8211;depclean and Docker was gone. Reboot, and the Containers page was empty.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-removing-docker.png\" alt=\"Horcrux Package Manager page with app-containers\/docker installed, version 29.9.0, and a red Remove button\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">Docker, one Remove button away.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-no-containers-after-break.png\" alt=\"Horcrux Containers page reading No containers found after Docker was removed and the VM rebooted\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">After the reboot: no Docker, no containers.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Rolling back<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Boot Environments page, &#8220;Roll back to this&#8221; next to the snapshot. It clones the snapshot into a new boot environment (rollback-20261114-202347), puts back the kernel that was saved with the snapshot, makes it the default in GRUB and tells you to reboot. It doesn&#8217;t reboot for you, on purpose, and the environment you&#8217;re leaving stays in the GRUB menu. If the rollback itself is bad, you pick the old entry.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video controls preload=\"none\" playsinline poster=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-rollback-from-the-web-ui-poster.jpg\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-rollback-from-the-web-ui.mp4\"><\/video><figcaption class=\"wp-element-caption\">From the empty Containers page to Roll back to this. Pauses between clicks are shortened.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The kernel came up at 20:25:33 and the API answered 16 seconds later. \/proc\/cmdline had root=ZFS=rpool\/ROOT\/rollback-20261114-202347 and dockerd was back. So was the old Horcrux build. The rollback undid the 17 package upgrade too.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The logs made it through. \/var\/log is one of the datasets that never rolls back, along with \/home, \/var\/lib\/docker, libvirt and the VM disks, so emerge.log still showed the Docker removal after the rollback.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-running-rollback-environment.png\" alt=\"Horcrux Boot Environments page: rollback-20261114-202347 is running and boots next, the original horcrux environment is still listed, the pre-upgrade snapshot is kept\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">Booted into the rollback environment. The original one is still listed and still bootable.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">What doesn&#8217;t work (yet)<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>The three containers sat at Exited (255) and nothing started them. I ran docker start by hand and nginx answered.<\/li><li>Horcrux is supposed to do that. A service called horcrux-guests resumes guests at boot, and on this install it can&#8217;t start at all: &#8220;ERROR: horcrux-guests needs service(s) horcrux&#8221;. The OpenRC script depends on a service named horcrux, which is what the source build calls the manager. The prebuilt package, the one the ISO ships, calls it horcrux-api. The systemd unit lists both names, so this only bites on OpenRC, and there&#8217;s a unit test that checks for the wrong string. No fix yet.<\/li><li>The Containers page kept saying &#8220;No containers found&#8221; with three containers up. The manager started at boot before Docker&#8217;s socket existed, logged &#8220;Falling back to CLI&#8221; and never tried again. Its boot pass handled 0 guests, so it never had a chance to start anything. Restarting horcrux-api made all four show up, but that&#8217;s a workaround, the manager should retry or wait for Docker.<\/li><li>demo-late, the container made after the snapshot, won&#8217;t start: &#8220;RWLayer of container &#8230; is unexpectedly nil&#8221;. Docker&#8217;s data is on its own dataset and wasn&#8217;t rolled back, and the Docker binary disappeared 13 seconds after that container was created. I don&#8217;t know the cause yet.<\/li><li>The Tasks page shows the first upgrade as &#8220;interrupted&#8221; with no duration. emerge.log says 17 of 17 and a clean exit.<\/li><li>After Upgrade All a green banner says the task started and points you at the Tasks page, then it&#8217;s gone in about five seconds and the page looks like nothing happened. The design doc has a banner on the Packages page that shows the snapshot it took. It isn&#8217;t built yet.<\/li><li>Times don&#8217;t line up. Snapshot names use the host clock, the UI uses your browser&#8217;s, the logs use UTC. On this VM that was Berlin, Pacific and UTC, and 20:17 in a name meant 11:17 AM on screen. A fresh install also lists its first boot environment as created on 7\/26\/2026, which is when the image was built.<\/li><li>On a phone you scroll the tables sideways to reach the buttons, and the Log out button is cut off.<\/li><li>After every reboot my browser session was gone, and the tab showed the dashboard with &#8220;HTTP 401&#8221; errors in the panels instead of the login page. Plus the self-signed certificate warning in every new browser.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-after-rollback-terminal.png\" alt=\"Terminal after rebooting into the rollback environment: docker is started, horcrux-guests is stopped, three containers are started by hand, demo-late fails with RWLayer is unexpectedly nil, nginx answers\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">After the rollback reboot: Docker started, horcrux-guests stopped, containers started by hand, demo-late refusing.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-tasks-upgrade-interrupted.png\" alt=\"Horcrux Tasks page: the second package-upgrade ok in 76 s, package-sync ok in 111 s, and the first package-upgrade marked interrupted\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">The Tasks page. The bottom row is the first upgrade, which finished fine.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-mobile.png\" alt=\"Horcrux Boot Environments page at phone width, the tables need sideways scrolling and the Log out button is cut off\" loading=\"lazy\" decoding=\"async\" \/><figcaption class=\"wp-element-caption\">Boot Environments at phone width.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Design decisions<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I wanted NixOS-style rollback in Horcrux, where a bad upgrade means you boot the system as it was before it. My notes on why say &#8220;Recovery, bit rot protection, and resilience.&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">ZFS because Gentoo recommends XFS, but ZFS is just superior tech. Horcrux never defaults to ext4, only ZFS or XFS, and boot environments need ZFS.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I picked Gentoo for the freedom to choose. USE flags are the main reason, hardware specific compiles come second. I&#8217;ve run Gentoo since 2006. Every upgrade goes through Portage anyway, so one bashrc hook is enough.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">GRUB, not ZFSBootMenu. ZFSBootMenu fits boot environments better since it reads them straight off the pool. The installer is built on GRUB though (BIOS and EFI, T2 Macs, aarch64, LUKS, mirrored \/boot), and GRUB can&#8217;t read an encrypted pool, so each boot environment&#8217;s kernel sits on a plain \/boot and the GRUB config gets regenerated. ZFSBootMenu is the candidate for a later phase.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Only the root file system rolls back. VM disks, home, logs, Docker and libvirt each get their own dataset. My own design doc says the Horcrux config database rolls back with root, while the installer gives \/var\/lib\/horcrux its own dataset. One of those is wrong, and I haven&#8217;t decided which yet.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Where it stands<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">After the rollback I ran Upgrade All again. The page listed 4 upgradeable packages (emerge&#8217;s own plan had 17 once dependencies and rebuilds were counted), the task finished in 76 seconds, and a second snapshot showed up, this time on the rollback environment.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video controls preload=\"none\" playsinline poster=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-upgrade-all-again-poster.jpg\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/horcrux-be-upgrade-all-again.mp4\"><\/video><figcaption class=\"wp-element-caption\">Upgrade All after the rollback, then the Tasks page and the Boot Environments page once it finished. The wait for emerge is cut.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">What&#8217;s next<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The guest bugs go first: the horcrux-guests dependency (and the test that checks for the wrong name) and the manager finding Docker when the socket shows up late. The task status after that. Then phase 2 from the design doc: the snapshot shown on the Packages page, an automatic fall back to the previous boot environment after a few failed boots, and moving VM disks out of the root dataset on installs that are older than this.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/horcrux.errant.solutions\/\">https:\/\/horcrux.errant.solutions\/<\/a><\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>I installed Horcrux in a throwaway VM, let Upgrade All take its automatic ZFS snapshot, removed Docker on purpose and rolled back from the new Boot Environments page. The containers did not come back on their own.<\/p>\n","protected":false},"author":1,"featured_media":772,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,26],"tags":[398,81,164,358,167,101],"class_list":["post-766","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux","category-software-engineering","tag-boot-environments","tag-docker","tag-gentoo","tag-horcrux","tag-virtualization","tag-zfs"],"_links":{"self":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/766","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/comments?post=766"}],"version-history":[{"count":2,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/766\/revisions"}],"predecessor-version":[{"id":781,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/766\/revisions\/781"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/media\/772"}],"wp:attachment":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/media?parent=766"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/categories?post=766"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/tags?post=766"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}