Author: canutethegreat

  • Horcrux dev log 5: auditing my own project, and what turned out to be fake

    Horcrux dev log 5: auditing my own project, and what turned out to be fake

    Part 5 in the Horcrux dev log series.

    Backdated: written in November 2026 from the repo, CI and my QA ledger. Part 4 ended with a feature list I didn’t trust.

    On October 16 I compared Horcrux against Proxmox VE. The method: take every route from the router (418 .route( calls), take every route the UI calls and match them with a script, then for each feature check whether anything actually calls the code. The result is a document with a section titled “what is fake today”.

    The first findings were bad. VMs couldn’t install an OS or get a network: the QEMU command line was memory, CPUs, one disk and -display none, with no ISO, no NIC, no boot order, no UEFI and no way to edit a VM after creating it. Nothing created a QMP socket, so snapshots, migration health checks and “suspend” backups quietly did nothing. Backups tarred a directory the code itself had just created empty and never touched the VM disk. And all configuration lived in memory, so storage pools, firewall rules, HA groups, templates, alert rules, replication jobs and backup jobs vanished on restart.

    My QA ledger from the same day has more. Firewall Apply installed no rules at all (the builder emitted tcp tcp dport, and the UI posted to a route that didn’t exist). Directory storage pools reported 0 GB free so every volume create failed. VM consoles didn’t work since QEMU ran with no display and no VNC or SPICE listener. Several UI buttons had no handler: monitoring “Configure Alerts” and “Export Data”, NAS “Create Group”, the user edit icon.

    The worst entry is marked critical. /api/auth/register was public and the RBAC middleware was never applied, so any self-registered account had full control, and the API ran over plain HTTP.

    Finding that a week after calling it feature complete is not fun. The ledger got 44 new rows on the 16th and 39 on the 17th.

    The fixing week

    Forgejo for October 16 to 21: 348 runs, 208 success, 44 failure, 34 cancelled, 62 skipped, over about 170 commits. Everything in memory moved into the database with migrations. VM backups became real (QMP pause or snapshot, flushed disks, restore, retention, verify). VNC, SPICE with audio, clipboard and USB redirection, and serial consoles got wired up, plus in-browser consoles for LXD and Incus. Auth got rewritten: users, roles, groups, ACLs, LDAP/AD/OIDC/PAM realms, 2FA, RBAC on every route, HTTPS by default, and a test that runs every built-in role against every route. VMs got NICs, an ISO library, boot order, multiple disks, UEFI, edit, reboot and a per-VM firewall. Peer-to-peer migration between nodes became real.

    What only a real run found

    On the 17th a test caught this one, which I’ll quote:

    QEMU adds a default user-mode (SLIRP) e1000 NIC when no -nic/-netdev is
    given. It NATs the guest through the host's own network stack, so on an
    EC2 host a guest read the host's instance metadata (CirrOS printed the
    host's instance-id) and could fetch its IAM role credentials.

    The fix always passes an explicit no-NIC option, and a test boots a real VM and asks QEMU over QMP that it has none.

    Same class, next few days: disk import followed qcow2 external data files and read other files on the host (Oct 19). URL downloads for disks and ISOs could reach the node’s own services (Oct 19 and 20). Volume names could escape their storage pool by path traversal (Oct 20).

    Mutation testing

    I also added cargo-mutants runs per module group. The first storage measurement was awful. The commit says it flatly: “StorageManager – mutation baseline was 0%”. Killing survivors became routine; 17 surviving mutants in the VM snapshot code was one batch. I also split the giant API binary into crates on the 17th (core, nas, k8s, sdn, auth, vm, container, cluster, backup), with CI moving to mold and line-tables-only builds the same day.

    Still broken on the 21st

    Some features existed in the API and not the UI, or the reverse. The cluster code had been tested with containers as nodes, not on separate real machines, as far as I can tell from the CI. On the 21st I rewrote the status line to “in active development, not yet suitable for production use” and cut the stale ones.

    Design decisions

    No implicit NIC: a VM with no network configuration gets no network at all rather than QEMU’s default user-mode one, because that default NATs through the host and a guest reached the host’s cloud metadata service. URL downloads got the same rule (they can’t reach the node’s own services or metadata). HTTPS by default with a node-local CA came in with the auth rewrite.

    Snapshot-mode backups don’t pause a running VM. On the 16th it paused (QMP stop and cont); on the 19th the commit “backs up a running VM without pausing it” made it a block-level snapshot. The earlier QA entry says why: consistent but not pause-free, and Proxmox-style backup needs a copy-before-write job.

    One call about the test jobs is documented: on October 18 mutation and chaos jobs went back to every runner, because the colo-only lane’s backlog was too big.

    Other parts: Part 1, Part 2, Part 3, Part 4, Part 6. The series continues with the latest Horcrux dev log post.

  • Segmentation and limiting lateral movement

    Segmentation and limiting lateral movement

    Part 2 of 3 in the Surviving ransomware series. Part 1: Backups an attacker can’t delete.

    When NotPetya hit in June 2017, it didn’t need anyone to click anything after the first infection. It spread across networks on its own, using the EternalBlue exploit for SMBv1 (MS17-010) and credentials harvested from memory on machines it had already reached, then running itself remotely with legitimate Windows tools. Maersk lost tens of thousands of machines in a few hours. The White House later put the total global damage at more than $10 billion.

    What made it so destructive wasn’t novel malware engineering. It was flat networks and shared credentials. Every compromised machine could reach thousands of others, and many of them would accept the same admin password.

    Part 1 of this series covered backups an attacker can’t delete. This part is about making the attacker’s path to those backups, and to everything else, as short and noisy as possible.

    Two kinds of segmentation

    Flat network compared with a segmented network of zones and allowlisted flows

    Most people think of segmentation as VLANs and firewalls. That’s half of it. The other half is identity segmentation: which credentials exist on which machines. An attacker moves laterally by combining reachability (can I connect to that host?) with authority (do I have a credential it accepts?). Take away either one and the move fails.

    Network segmentation limits reachability. Identity segmentation limits the value of whatever credentials an attacker steals. You need both, and the identity half is usually cheaper.

    Workstations shouldn’t talk to each other

    In most offices there’s no business reason for one laptop to open an SMB, RDP or WinRM connection to another laptop. Yet in a default Windows domain with file sharing enabled, that’s exactly what’s allowed, and it’s how a single phished laptop becomes fifty.

    Block it. Inbound SMB (445), RDP (3389) and WinRM (5985 and 5986) on workstations should only be accepted from a small management subnet where your admin jump hosts and management tools live. In Group Policy you’d scope the inbound firewall rules to that subnet. The PowerShell equivalent on a single machine shows the idea:

    # Allow inbound SMB, RDP and WinRM only from the management subnet
    $mgmt = "10.20.0.0/24"
    Get-NetFirewallRule -DisplayGroup "File and Printer Sharing",
                                      "Remote Desktop",
                                      "Windows Remote Management" |
      Where-Object { $_.Direction -eq "Inbound" -and $_.Enabled -eq "True" } |
      Set-NetFirewallRule -RemoteAddress $mgmt

    Then check it from an ordinary workstation, not from the admin network:

    Test-NetConnection -ComputerName ws-0142 -Port 445
    # TcpTestSucceeded : False   <- what you want to see

    I’ve watched this single change turn a domain-wide incident into a handful of machines. It’s dull, it’s cheap, and it breaks almost nothing.

    Servers get allowlists, not blocklists

    Servers are harder because they do need to talk to each other, just not to everything. Start from observed flows. VPC flow logs, NetFlow from your switches, or a few weeks of host firewall logging will show you what actually connects to what. Then write rules that permit those flows and drop the rest.

    On a Linux database server, the host firewall alone can enforce most of it:

    # /etc/nftables.conf (excerpt)
    table inet filter {
      chain input {
        type filter hook input priority 0; policy drop;
        ct state established,related accept
        iif "lo" accept
        ip saddr 10.20.0.0/24 tcp dport 22 accept     # admin jump hosts only
        ip saddr 10.30.1.0/24 tcp dport 5432 accept   # application tier only
        icmp type echo-request limit rate 5/second accept
      }
    }

    Host firewalls and network firewalls complement each other. The network layer protects you when a host is misconfigured, and the host layer protects you from the neighbour in the same VLAN. Microsegmentation products automate this at scale, but the principle doesn’t need a product.

    Tier your admin accounts

    Microsoft’s tiering guidance, now folded into its enterprise access model, boils down to one rule: a credential that controls a high-value system must never be typed into, or cached on, a lower-value system. Domain admin accounts don’t log on to workstations. Server admin accounts don’t log on to workstations either. Help desk accounts that do manage workstations have no rights on servers.

    When an attacker dumps credentials from a compromised laptop, tiering decides what they find. With it, they get the laptop user and maybe a workstation-admin account. Without it, they often get domain admin, because someone remoted in to fix a printer driver last month.

    The supporting controls are well documented and mostly free:

    • Windows LAPS, built into Windows since the April 2023 updates, gives every machine a unique, rotated local administrator password, which kills the “same local admin password everywhere” path NotPetya-style tools rely on.
    • The Protected Users group stops its members’ credentials from being cached in ways that are easy to steal, and blocks NTLM authentication for them.
    • Credential Guard isolates secrets in virtualisation-based security so that memory scraping on the endpoint returns far less.
    • Restricting or auditing NTLM reduces relay and pass-the-hash opportunities. Start in audit mode, because something old in your environment will depend on it.

    Admins work from dedicated privileged access workstations or jump hosts in the management subnet. That’s what makes the firewall scoping above workable.

    Put the crown jewels behind their own door

    Some systems deserve their own zone with its own entry controls: domain controllers, backup infrastructure, hypervisor management, PKI, and identity providers. As covered in part 1, the backup server in particular shouldn’t be reachable from general server networks, and shouldn’t trust the production domain.

    The test I use is simple. Pick a random workstation and a random application server, assume both are fully compromised, and ask how many hops it takes to reach a domain controller or the backup console. If the answer is “one” or “it’s in the same VLAN”, that’s your next project.

    See the east-west traffic

    Segmentation also makes detection easier, because it creates boundaries where unusual traffic stands out. A workstation attempting SMB connections to forty other workstations in five minutes should alert. So should a service account authenticating to hosts it has never touched before, or remote service creation (Windows event 7045) on machines where nobody deploys software that way. Firewall drop logs between zones are a gift here: once the rules are tight, almost every drop is worth a look.

    Expect it to be incremental

    Nobody segments a flat network in a quarter. Start with the workstation peer-to-peer block and LAPS, which together cover a huge share of real-world spread. Then tier the admin accounts, then wall off the crown jewels, then work through server allowlists one application at a time.

    Even perfect segmentation won’t stop every incident. It buys time and turns a company-wide outage into a contained one, which matters a great deal when you get to part 3: actually restoring what was hit, and finding out how long that really takes.

    Other parts: Part 1: Backups an attacker can’t delete. Part 3: Restore testing and your real recovery time.

  • Backups an attacker can’t delete

    Backups an attacker can’t delete

    Part 1 of 3 in the Surviving ransomware series.

    Modern ransomware crews don’t start by encrypting things. They start by finding your backups. The playbook described in incident reports from the last few years is depressingly consistent: get domain admin, locate the backup server, delete the restore points or wipe the repository, and only then run the encryptor. If the backups survive, the ransom demand is a negotiation. If they don’t, it’s a hostage situation.

    So the question for this series isn’t “do we have backups?” Almost everyone does. It’s whether an attacker who holds your most privileged credentials can destroy them.

    3-2-1 was written for a different threat

    Backup architecture with production, backup and immutable zones and one-way data flows

    The classic 3-2-1 rule says three copies of your data, on two different types of media, with one copy offsite. It’s a great rule for fires, floods and failed RAID controllers. It says nothing about an adversary who logs in with valid credentials and deletes all three copies, because in most environments the same admin accounts can reach all of them.

    The extension you’ll often see now is 3-2-1-1-0. The extra 1 is a copy that is offline, air-gapped or immutable. The 0 is zero errors when you verify a restore. I like it because both additions target the actual failure modes: deletion by a privileged attacker, and backups that turn out to be unrestorable on the worst day of your year.

    The backup system is a target in its own right

    Backup servers are attractive because they hold everything and they usually hold credentials to reach everything. CVE-2023-27532 in Veeam Backup & Replication is a good illustration. It let an unauthenticated user who could reach the backup service port pull encrypted credentials out of the configuration database. It was exploited in the wild and added to CISA’s Known Exploited Vulnerabilities catalog in August 2023.

    The lessons I take from it are architectural rather than vendor-specific:

    • Put the backup server’s management interfaces on a network segment that ordinary servers and workstations can’t reach.
    • Don’t join backup infrastructure to the production Active Directory domain. If domain admin is compromised, the backup console shouldn’t fall with it.
    • Require MFA for the backup console and for any cloud account that holds backup copies, using accounts that exist only for that purpose.
    • Patch backup software on the same urgency as internet-facing systems, because for an intruder it effectively is one.

    Immutability, done properly

    Immutable storage means a backup can’t be modified or deleted until its retention period ends, no matter who asks. The details decide whether it holds up against a real admin-level compromise.

    On AWS, S3 Object Lock has two modes. Governance mode can be bypassed by users with the s3:BypassGovernanceRetention permission, which makes it useful against mistakes and useless against someone who has stolen an admin role. Compliance mode can’t be shortened or removed by anyone, including the account root user, until the retention date passes.

    aws s3api create-bucket --bucket example-backups-immutable \
      --region us-west-2 \
      --create-bucket-configuration LocationConstraint=us-west-2 \
      --object-lock-enabled-for-bucket
    
    aws s3api put-object-lock-configuration \
      --bucket example-backups-immutable \
      --object-lock-configuration '{
        "ObjectLockEnabled": "Enabled",
        "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 35 } }
      }'

    Test with a small bucket and short retention first. Compliance mode is unforgiving: if you set a year by mistake, you’re paying for that storage for a year.

    Put the bucket in a separate AWS account dedicated to backups, with its own tightly held credentials, so production compromise doesn’t hand over the backup account too. The same idea applies to other clouds and to on-premises object stores that support WORM retention.

    For self-hosted setups, I prefer a pull model. The backup server reaches into production to fetch data, and production holds no credentials that work on the backup server. With ZFS, the source host can delegate just enough to a dedicated user:

    # On the production host: allow the backup user to snapshot and send, nothing else
    zfs allow -u backupsvc send,snapshot,hold tank/data

    The backup server then pulls incremental snapshots over SSH and keeps its own snapshot history. An attacker with root on the production host can destroy production’s snapshots, but has no path to the copies on the backup server.

    Retention has to outlast the intrusion

    Immutability with seven days of retention doesn’t help if the attacker was inside for three weeks before encrypting anything, because every restore point you have may contain their tooling, or their changes to your data. Mandiant’s M-Trends 2024 report put median global dwell time at 10 days in 2023, and the median isn’t the case you should plan for.

    I’d treat 30 days of immutable restore points as a floor, with a longer, sparser tail (weekly or monthly) beyond that. Part 3 of this series covers how to pick a clean restore point, which depends heavily on having logs that tell you when the intrusion started.

    Offline still has a place

    Tape and removable disks rotated offsite feel old fashioned, and they’re slow. They’re also the one copy that no network-borne attacker can touch. For a smaller organisation, a weekly full backup to a disk that’s disconnected and locked away is a cheap and effective last line. Keep the backup encryption keys stored separately from the backups, including an offline copy, because encrypted backups you can’t decrypt are no backups at all.

    Watch for attacks on the backups

    Deleting backups is noisy if you’re listening. Alert on deletion or retention changes in backup jobs, on someone disabling Volume Shadow Copy or running vssadmin delete shadows, on failed or skipped jobs across many systems at once, and on new admin accounts in the backup console. These events often precede encryption by hours. That’s your window.

    The zero errors part

    The final 0 is the one most often skipped. A green “job succeeded” status means data was written somewhere. It doesn’t mean you can restore a working system from it in a reasonable time. Part 3 is about proving that, with real restore tests and measured recovery times. Before that, part 2 looks at segmentation, which decides how far an intruder gets before they reach the backup server at all.

    Other parts: Part 2: Segmentation and limiting lateral movement. Part 3: Restore testing and your real recovery time.