Full disclosure: I make this. Horcrux is my Gentoo-based hypervisor, it’s pre-1.0, and this is my notebook for the week.
Friday night I merged a Boot Environments page into the web UI. The command line tool underneath, horcrux-be, has been around for a while, but you had to know it existed. Now there’s a page that lists every root file system you can boot, the snapshots, and a yellow button that says “Roll back to previous version”. So on Saturday I installed the daily server image into a throwaway VM on one of my lab laptops, put some containers on it, and broke it.
The setup
ZFS root, OpenRC, 2 vCPUs, 6 GB of RAM. Three containers made from the Containers page: nginx, redis, and an alpine box that only sleeps. Every row carries a warning that CRIU isn’t installed so container state can’t be saved. It’s printed on all three rows, word for word.

Upgrade All takes a snapshot first
I hit Upgrade All on the Packages page. Emerge started at 20:16:54 (VM clock; the VM ended up on Berlin time, more on that below) and six seconds later there was a snapshot named horcrux@pre-upgrade-20261114-201700. Then 17 binary packages went in, a new build of Horcrux itself among them, and emerge was done at 20:18:03. 69 seconds. The snapshot was 21 MB at that point.
The web UI doesn’t take that snapshot. A hook in Portage’s bashrc does, before the first package of any emerge run, at most once every 30 minutes. Emerge from a shell gets one just like the API does.


Then I broke it
Packages page, search for docker, Remove. The confirm dialog asks “Remove package ‘…’?” and that’s it. Nothing about the four containers running on it (I’d made a fourth one, demo-late, 13 seconds earlier). Six seconds of emerge –depclean and Docker was gone. Reboot, and the Containers page was empty.


Rolling back
Boot Environments page, “Roll back to this” next to the snapshot. It clones the snapshot into a new boot environment (rollback-20261114-202347), puts back the kernel that was saved with the snapshot, makes it the default in GRUB and tells you to reboot. It doesn’t reboot for you, on purpose, and the environment you’re leaving stays in the GRUB menu. If the rollback itself is bad, you pick the old entry.
The kernel came up at 20:25:33 and the API answered 16 seconds later. /proc/cmdline had root=ZFS=rpool/ROOT/rollback-20261114-202347 and dockerd was back. So was the old Horcrux build. The rollback undid the 17 package upgrade too.
The logs made it through. /var/log is one of the datasets that never rolls back, along with /home, /var/lib/docker, libvirt and the VM disks, so emerge.log still showed the Docker removal after the rollback.

What doesn’t work (yet)
- The three containers sat at Exited (255) and nothing started them. I ran docker start by hand and nginx answered.
- Horcrux is supposed to do that. A service called horcrux-guests resumes guests at boot, and on this install it can’t start at all: “ERROR: horcrux-guests needs service(s) horcrux”. The OpenRC script depends on a service named horcrux, which is what the source build calls the manager. The prebuilt package, the one the ISO ships, calls it horcrux-api. The systemd unit lists both names, so this only bites on OpenRC, and there’s a unit test that checks for the wrong string. No fix yet.
- The Containers page kept saying “No containers found” with three containers up. The manager started at boot before Docker’s socket existed, logged “Falling back to CLI” and never tried again. Its boot pass handled 0 guests, so it never had a chance to start anything. Restarting horcrux-api made all four show up, but that’s a workaround, the manager should retry or wait for Docker.
- demo-late, the container made after the snapshot, won’t start: “RWLayer of container … is unexpectedly nil”. Docker’s data is on its own dataset and wasn’t rolled back, and the Docker binary disappeared 13 seconds after that container was created. I don’t know the cause yet.
- The Tasks page shows the first upgrade as “interrupted” with no duration. emerge.log says 17 of 17 and a clean exit.
- After Upgrade All a green banner says the task started and points you at the Tasks page, then it’s gone in about five seconds and the page looks like nothing happened. The design doc has a banner on the Packages page that shows the snapshot it took. It isn’t built yet.
- Times don’t line up. Snapshot names use the host clock, the UI uses your browser’s, the logs use UTC. On this VM that was Berlin, Pacific and UTC, and 20:17 in a name meant 11:17 AM on screen. A fresh install also lists its first boot environment as created on 7/26/2026, which is when the image was built.
- On a phone you scroll the tables sideways to reach the buttons, and the Log out button is cut off.
- After every reboot my browser session was gone, and the tab showed the dashboard with “HTTP 401” errors in the panels instead of the login page. Plus the self-signed certificate warning in every new browser.



Design decisions
I wanted NixOS-style rollback in Horcrux, where a bad upgrade means you boot the system as it was before it. My notes on why say “Recovery, bit rot protection, and resilience.”
ZFS because Gentoo recommends XFS, but ZFS is just superior tech. Horcrux never defaults to ext4, only ZFS or XFS, and boot environments need ZFS.
I picked Gentoo for the freedom to choose. USE flags are the main reason, hardware specific compiles come second. I’ve run Gentoo since 2006. Every upgrade goes through Portage anyway, so one bashrc hook is enough.
GRUB, not ZFSBootMenu. ZFSBootMenu fits boot environments better since it reads them straight off the pool. The installer is built on GRUB though (BIOS and EFI, T2 Macs, aarch64, LUKS, mirrored /boot), and GRUB can’t read an encrypted pool, so each boot environment’s kernel sits on a plain /boot and the GRUB config gets regenerated. ZFSBootMenu is the candidate for a later phase.
Only the root file system rolls back. VM disks, home, logs, Docker and libvirt each get their own dataset. My own design doc says the Horcrux config database rolls back with root, while the installer gives /var/lib/horcrux its own dataset. One of those is wrong, and I haven’t decided which yet.
Where it stands
After the rollback I ran Upgrade All again. The page listed 4 upgradeable packages (emerge’s own plan had 17 once dependencies and rebuilds were counted), the task finished in 76 seconds, and a second snapshot showed up, this time on the rollback environment.
What’s next
The guest bugs go first: the horcrux-guests dependency (and the test that checks for the wrong name) and the manager finding Docker when the socket shows up late. The task status after that. Then phase 2 from the design doc: the snapshot shown on the Packages page, an automatic fall back to the previous boot environment after a few failed boots, and moving VM disks out of the root dataset on installs that are older than this.
Leave a Reply
You must be logged in to post a comment.