Category: Security

  • Patronus: who is eating my uplink, and what it does when the ISP only half works

    Patronus: who is eating my uplink, and what it does when the ISP only half works

    Full disclosure: I make Patronus, a router and firewall OS built on Gentoo. It’s young. Version 0.2.0 came out on October 30.

    Two of the 0.2.0 pages I had only ever seen as test output. One is Traffic insight, which says which device talked to which country and network. The other is Uplink health, which watches each internet connection and is supposed to notice when one goes bad in a way ping doesn’t show. On November 15 I used both the way someone running a small network would: let some devices make noise, then break the ISP in three different ways.

    What this setup is

    Another lab, not my house. One container with a handful of network namespaces: the router (the real patronus-web and the apply daemon, built from main at commit 2f72a49c, 5 minutes 48 seconds cold on a capped CI box), four LAN devices with made-up names, two fake ISPs, and a fake internet on documentation addresses. WAN1 is labelled fibre and WAN2 is the LTE backup, in a failover group with WAN1 first. The traffic is real curl, ping and dig, and the browser really reaches the admin page.

    Some of it is fake. The country and network data is not DB-IP. I couldn’t use the real files in the lab, so I generated a file in the same format with ten address ranges, invented countries and invented companies such as “Beispiel Hosting GmbH”. Which country got how much traffic means nothing. I also sped the clocks up for the video (probe tick 2 seconds, history sample 3 seconds), then ran the same faults again at the production defaults, and the timings below come from those runs. None of it ran for longer than about an hour.

    Which device is using the bandwidth

    The Insights page has a Live tab that shows flows right now, and history tabs for countries, networks and devices. The Devices tab was the one I wanted. After the lab devices had pushed traffic around for a while, laptop-demo had pulled 2.1 GB down, living-room-tv 292 MB, kids-tablet 16 MB, and nas-demo had pushed 297 MB up, which is what a backup job looks like.

    Devices tab: laptop-demo, nas-demo, living-room-tv and kids-tablet with download, upload, flows and top country
    The Devices tab, last 24 hours. The top-country column comes from my fake mapping file.

    Clicking a device gives an hourly bar and its top countries and networks.

    Per-device view of laptop-demo with an hourly bar, top countries and top networks
    laptop-demo. Everything in the bottom two tables is demo data.

    This is built from the kernel’s conntrack byte counters, not from a packet capture. A collector reads them every 15 seconds and adds the change in each flow’s counters to an hourly row for that device, direction, country and network. The design note gives the reason: the counters were already switched on by another part of the firewall and the parsing already existed. The same note admits that a flow which starts and ends between two reads loses up to one interval of bytes, and calls that acceptable.

    There is a privacy page too. History is on by default for 30 days and 200 rows per hour. You can list addresses, devices or groups that are never recorded, and the docs say adding one deletes what was already recorded for it. I put my probe client on that list. The page also has a button that deletes all recorded history, which I left alone.

    Settings tab with retention settings, a not-recorded list holding one address, and the country and ASN database status
    Settings. The database rows show the lab URL my fake files came from.

    When the ISP only half works

    The Uplinks tab shows each gateway with its state, round trip, loss, 24 hour availability and an hourly bar. WAN1 also had two extra probes, a DNS lookup against a resolver and a TCP connect, and WAN2 had a TCP probe. Those go out through the gateway’s own interface, so they test that link and not whichever one holds the default route. The SLA target was 99.5 percent, and the failover group switches on loss or on latency above 100 ms.

    Uplinks tab with both gateways online
    Both links healthy.

    First fault: 300 ms of delay on WAN1. WAN1 went to “latency” 3.3 seconds after I added it, and the router said the default route moved to WAN2 at 3.6 seconds. The events (gateway_degraded and group_failover) showed up at 8.2 seconds. When I took the delay off, WAN1 was healthy and the route back on it 7 seconds later.

    WAN1 marked latency at 300 ms round trip while WAN2 is online and carries the group
    WAN1 at 300 ms, WAN2 carrying traffic.

    Second, the ISP’s resolver stops answering while ping keeps working, which a ping-only monitor happily reports as fine. The probes did what the docs say. The DNS probe failed 10.7 seconds after the fault (it needs two misses in a row, 5 seconds apart) and the uplink_probe_failed event arrived at 17.4 seconds. Traffic stayed on WAN1, because a probe only moves traffic if you mark it critical.

    WAN1 still labelled online while its DNS probe fails, shown in amber
    WAN1 with a dead resolver. The label still says online.

    Third: I marked the DNS probe critical, then broke the resolver again. Now a failing probe counts as packet loss. Ping to the ISP still answered the whole time, yet the router said the default route moved to WAN2 at 10.7 seconds, called WAN1 down at 13.8, and wrote gateway_down at 16.4. Fix the resolver and WAN1 was healthy again 10.4 seconds later, with the gateway_up event at 16.5.

    WAN1 labelled down with a red critical DNS probe failure and a one minute outage listed
    The same resolver outage with the probe marked critical.

    The clip is the DNS fault at the accelerated clock. At the top is the Uplinks page, in the middle is a LAN client that pings, resolves and fetches about once a second, and at the bottom is what the router API reports, polled every quarter second. I flip the probe to critical about 15 seconds in, and then you can watch loss, down, recovery and failback. The captions were added afterwards. I corrected one of them after looking at the result, and the reason is under the next heading.

    DNS outage, probe made critical, failover, recovery. 56 seconds, clocks accelerated, real time otherwise.
    Event log rows including gateway_down, uplink_sla_breach and uplink_probe_recovered
    The event log after a critical DNS outage, with the SLA breach in it.

    What the docs say about the design

    Most of the design is written down in two places, so I’m quoting them. History comes from conntrack because the counters and parser already existed. DB-IP is the data source because MaxMind’s free files need an account and a licence agreement, while Patronus already shipped DB-IP files that need neither. The history tables are kept out of config revisions and restores because, in the code’s own words, they are measured data that a rollback or restore may not rewind. The extra probes exist because an ISP can answer ping with broken DNS, and a probe has to fail twice in a row so one lost UDP packet doesn’t flap a WAN.

    Nobody wrote down why it’s 30 days and 200 rows, or why probes run every 5 seconds. The same goes for conntrack over NetFlow. I won’t make a reason up.

    What doesn’t work (yet)

    The one I’d fix first is in the clip. The router reports the default route moving to WAN2 at 22 seconds. The client lines in the same recording keep showing WAN1’s address for the rest of the clip. My first caption said the client now exited through WAN2. That was wrong, so I replaced it. I don’t know yet whether the fault is in my client script, in which routing table the LAN actually uses, or in Patronus. I didn’t get to dig before the lab was gone.

    A dead resolver with a non-critical probe leaves the WAN1 label on “online”, availability at 100 percent, and the traffic on WAN1. The amber shows up in the probe chip, the chart shading and the “fully healthy” percentage. The page’s own intro says such an ISP “shows up as degraded”, and the degraded flag my script polled stayed false. I haven’t checked whether a different field carries it.

    With the probe critical, one resolver outage turned WAN1 into “down” in about 14 seconds with ping working fine, and with a single probed resolver that’s a single point of false failover. It’s awkward to set, too. The probe table only has a Delete button and the critical box is on the add form, so I changed the existing probe through the API. There may be an edit path I missed.

    Events lag the state. At production defaults the recorder reads every 15 seconds, so gateway_down arrived about 2.6 seconds after the state changed and the probe failure event about 7 seconds after the probe failed. Failback is slow too: loss decays after the fix, so the route comes back around 10 seconds later and the up event about 6 seconds after that.

    The event log shows UTC times and the Uplinks card shows browser-local time, on the same page. The SLA text counts samples as minutes, so with my 3 second samples the breach event said “2 minutes down” for an outage of well under a minute. That’s an artifact of my sped-up sample setting, and I haven’t re-measured it at one sample a minute.

    On a 390 pixel phone the device table runs off the card at the Download column, and in my event-log capture only Time, Event and About were visible. I didn’t check whether it scrolls sideways.

    Devices table on a phone, cut off at the Download column
    Devices on a 390 pixel wide phone.

    Things I didn’t try at all: the real DB-IP files and the monthly refresh, IPv6 flows, port forwards, the top-200 cutoff and 30-day pruning, webhook delivery of the new event types, a device that changes address, and a real ISP. The two netlab scenarios for these pages exist and I didn’t run them. 51 unit tests for this code pass (26 insight, 11 uplink, 7 for the network database, 7 for the probes), and nothing has touched it since October 30.

    Where it stands

    The probes do catch an ISP that answers ping and nothing else. I’d like the page to say so more plainly: right now a probe only matters if you find the critical box, and the WAN1 label keeps saying online. I can’t judge the device history, since the data under it was fake. The routing question is the one I’ll chase first.

    Next: the real GeoIP files and a few days on a real clock, and finding out where that client’s traffic really went.

    Patronus lives at patronus.errant.solutions.

  • Memory-Safe by Design: What Rust’s Borrow Checker Actually Prevents (With Real CVEs)

    Memory-Safe by Design: What Rust’s Borrow Checker Actually Prevents (With Real CVEs)

    The claim, stated precisely

    Diagram contrasting C/C++ double-free with Rust ownership and move semantics

    “Rust is memory safe” gets thrown around as marketing shorthand, and but the actual guarantee is narrow and well defined. The borrow checker enforces two rules at compile time: every value has exactly one owner (or an explicit, tracked transfer of ownership), and at any point in the program you can have either one mutable reference or any number of immutable references to a piece of data, never both. Everything the borrow checker prevents falls out of enforcing those two rules consistently, at zero runtime cost.

    That’s the whole guarantee. It says nothing about your logic. It’s a type-system claim about pointer and lifetime validity. The gap between that claim and the sales pitch is where credibility gets lost with people who’ve been burned by hype before.

    What this actually prevents

    Use-after-free. In C/C++, nothing stops you from holding a pointer after the memory it points to has been free()’d or gone out of scope. The pointer is just an address. The compiler has no concept of whether that address is still “yours.” Rust ties a reference’s lifetime to the compiler’s static analysis of how long the underlying data stays valid. Try to keep a reference alive past the point where its owner is dropped, and you get a compile error: error[E0597]: borrowed value does not live long enough. The bug class never makes it to runtime.

    fn get_ref() -> &i32 {
        let x = 5;
        &x // error: `x` does not live long enough
    }

    Double-free. This one falls out of the single-owner model almost for free (no pun intended). In C, two pointers can both think they own the same heap allocation, and both can call free() on it. In Rust, when a value’s owner goes out of scope, drop runs exactly once, because there’s exactly one owner by construction, move semantics transfer ownership instead of duplicating it. You can’t accidentally end up with two owners of a Box. The compiler tracks moves and refuses to compile code that uses a value after it’s been moved.

    Data races. The “one mutable XOR many immutable” rule, combined with the Send/Sync marker traits, gets enforced across thread boundaries too. You can’t share a mutable reference across threads without going through a synchronization primitive (Mutex, RwLock, channels, atomics) the type system knows about. That’s a narrower claim than preventing all race conditions (more on that below), but the specific, extremely common bug of two threads mutating the same memory without synchronization is a compile-time error in safe Rust, not a Heisenbug you chase with a thread sanitizer at 2am.

    Dangling pointers and iterator invalidation. Mutate a collection while holding a reference into it (push to a Vec while iterating it, say, which can trigger a reallocation and invalidate every pointer into the old buffer) and the same borrow rules catch it. Vec::push requires a mutable borrow, and the checker won’t let you take one while an immutable borrow, your iterator, is still live.

    Three real bugs from this exact class

    These aren’t hypotheticals. This bug class shows up over and over in security advisories for widely deployed C/C++ software.

    CVE-2019-5786 was a use-after-free in Chrome’s FileReader implementation, actively exploited in the wild as part of a zero-day attack chain targeting Chrome users. A FileReader object got freed while a reference to it was still reachable from another code path, and the freed memory was later reused by an attacker-controlled object, giving arbitrary code execution in the renderer process. Textbook UAF: the C++ object model has no compile-time concept of “this pointer outlived its owner.”

    CVE-2016-5195, better known as Dirty COW, was a race condition in the Linux kernel’s copy-on-write handling for memory mappings. A race between the MAP_PRIVATE write-fault handler and the madvise(MADV_DONTNEED) path let an unprivileged process trigger a write to a read-only, root-owned file mapping, leading to privilege escalation. It sat in the kernel for roughly nine years before anyone found it. This is the data-race/TOCTOU half of memory safety rather than strict use-after-free, but it’s exactly the class of concurrent-memory-access bug a type system that enforces exclusive mutable access is built to catch.

    CVE-2020-0022, BlueFrag, was a use-after-free in Android’s Bluetooth stack (the net/bluetooth subsystem), reachable over the air by a nearby attacker with no user interaction required, and capable of remote code execution within the Bluetooth daemon’s privileges. Same shape as the Chrome bug: object freed on one path, referenced on another, memory reused in between.

    These three alone span browser, kernel, and mobile-platform codebases, different teams, different review standards, different levels of sanitizer and fuzzing investment, and they all reduce to the same two or three root causes that the borrow checker structurally rules out. That’s the argument for Rust in security-sensitive code. C++ programmers aren’t careless; this bug class survives expert review, fuzzing, and static analysis at a nonzero, recurring rate, in exactly the projects with the most resources to prevent it.

    What Rust does not solve

    Logic bugs are still entirely on you. The borrow checker has no opinion about whether your authentication check is correct, whether your crypto runs in constant time, or whether your state machine has a reachable bad state. Memory safety and correctness are different axes. A Rust program can leak secrets, authorize the wrong user, or compute the wrong answer, all while being perfectly memory safe.

    unsafe exists, and it’s load-bearing. Anywhere you need raw pointers, FFI into C libraries, certain performance-critical patterns, or to build safe abstractions like Vec itself, you drop into unsafe and the compiler’s guarantees stop applying to that block. Real-world Rust codebases aren’t unsafe-free. Crates like tokio and hashbrown, plus plenty of embedded and systems code, carry meaningful amounts of it, and bugs have in fact turned up in unsafe blocks in widely used crates. Auditing unsafe is a real, ongoing discipline, not a solved problem.

    Panics are a denial-of-service vector. Rust converts a large class of memory-corruption bugs into panics: index out of bounds, unwrap on None, integer overflow in debug builds. That’s a strictly better failure mode than silent corruption, but it’s still a crash. In a network-facing service, an attacker who can trigger a reachable panic has a remote DoS primitive. Not segfaulting with corrupted memory is progress. It isn’t immunity.

    Supply chain risk doesn’t care about your type system. cargo makes pulling in transitive dependencies trivially easy, and crates.io has had real incidents: typosquatted crate names, and at least one case of a maintainer shipping a version with intentionally malicious behavior. The borrow checker does nothing to vet the hundred transitive dependencies sitting in your Cargo.lock. It’s the same supply-chain problem npm and PyPI have.

    Summary

    Rust’s ownership model closes a specific, well-understood, historically expensive category of bugs, the kind that, per Microsoft’s and Google’s own internal audits of their C/C++ codebases, accounts for roughly 70% of the high-severity security vulnerabilities they fix. That’s a real, measurable, structural improvement, and the CVEs above aren’t edge cases, they’re representative. But it buys you nothing on logic correctness, its escape hatch needs the same scrutiny unsafe C++ always did, it trades segfaults for panics instead of eliminating crashes, and it inherits the supply-chain problem of every modern package ecosystem. Pick it for what it does, not for what the slide deck implies.

  • Hardware hacking, part 7: JTAG and SWD basics, and how vendors lock the door

    Hardware hacking, part 7: JTAG and SWD basics, and how vendors lock the door

    Part 7 of 7 in the Hardware hacking series. Part 1: updating the firmware on a Pinecil V2. Part 2: the bench I work on. Part 3: using my 5V FTDI cable on 3.3V UARTs. Part 4: finding the UART console on a router I own. Part 5: reading SPI flash without wrecking it. Part 6: decoding I2C and SPI with sigrok.

    UART gives you a console, SPI flash gives you the firmware image, and JTAG or SWD gives you the CPU itself. Halt it, read registers, step through code, read and write memory, reflash, all on a part that has not been locked. That is why the debug port is the most powerful interface on a board and why vendors spend effort making it go away in production.

    This is the one part of the series where I own less than the topic needs. I have no dedicated debug probe, so this part is about understanding the interface and how defenders lock it, not a walkthrough of attacking a device. Everything here applies only to hardware you own or are authorised to test. Ownership does not switch off copyright or anti-circumvention law, so check yours before you bypass read protection on anything.

    What the two interfaces are

    Diagram I drew: STM32 read-out protection levels 0 (0xAA), 1 (0x55) and 2 (0xCC) with the allowed transitions between them

    The diagram above summarises the protection levels described below; it is a diagram I drew, not a screenshot.

    JTAG is the old one. It uses four signals, TCK (clock), TMS (mode select), TDI (data in) and TDO (data out), plus an optional active-low reset for the test logic, usually written nTRST. Devices sit in a chain, and OpenOCD’s documentation notes that JTAG also supports boundary scan testing.

    SWD is the ARM-specific alternative. The OpenOCD manual describes it as using fewer signal wires than JTAG: SWDIO and SWCLK, plus ground and usually a reset line. It has no boundary scan. Some processors support both.

    The pin counts matter for defenders because they decide how easy it is to find the port on an unmarked board. Here is the number of ways to assign the data signals if you have a row of n candidate pads and no silkscreen. These are my own permutation counts, not a vendor figure, and they ignore the ground, reset and reference-voltage pads on the same header:

    pads   JTAG (TCK,TMS,TDI,TDO)   SWD (SWDIO,SWCLK)
      6            360                    30
      8           1680                    56
     10           5040                    90
     12          11880                   132
     16          43680                   240

    Few people try them by hand, and pin-finding tools exist for exactly this. Those tools drive the pads, and an unknown pad can be a supply rail, so set the tool to the target’s measured logic level first and never use one on a mains-connected board. I treat unpopulated pads as obscurity, not as a control.

    What the software side looks like

    OpenOCD is the open source tool that sits between a debug adapter and GDB. The adapter list in my capture comes from the real 0.12.0 package on Ubuntu 24.04. One entry is ft232r, which drives an FT232R in bitbang mode, so in principle my cable could act as a crude JTAG adapter. I have not tried it and will not. The seller describes the cable as 5V logic (I confirm by measuring TXD at idle, as in part 3), and most debug ports are 3.3V or lower.

    Real terminal output from OpenOCD 0.12.0: the list of 28 built-in debug adapters, then a run with a CMSIS-DAP interface and an STM32F4 target that fails with unable to find a matching CMSIS-DAP device because no probe is attached

    Real terminal output from OpenOCD 0.12.0 in a throwaway container: the list of built-in debug adapters, then a run with no probe attached.

    The second command, run with no probe attached, shows a failed first attempt: OpenOCD opens its TCL and telnet ports, then stops with “unable to find a matching CMSIS-DAP device”. Both outputs are real, from a throwaway container, and prove nothing about any chip.

    How vendors lock it

    Most microcontroller vendors have some version of the same idea: a setting in non-volatile memory that tells the chip to refuse debug access, or to refuse it unless the flash is erased first.

    STMicroelectronics calls it read-out protection. In the HAL header for the STM32F4 family, level 0 is 0xAA (no protection), level 1 is 0x55 (read protection of the memory) and level 2 is 0xCC (full chip protection). The same header carries the warning that once level 2 is set it is no longer possible to go back to level 1 or 0. OpenOCD’s documentation for the STM32L0/L1 driver says mass erase is the only way to unlock a protected flash, unless the level is 2, which cannot be unlocked at all. On those parts, dropping from level 1 to level 0 wipes the flash, so the attacker gets the chip back but not the firmware. That is the design intent. As the next section shows, it did not always hold.

    Espressif does it with eFuses. The ESP-IDF documentation says that when secure boot is enabled, JTAG debugging is disabled via eFuse, and the bootloader sets that on first boot at the same time it enables secure boot. Flash encryption disables it by default too. The eFuse names vary by chip (DIS_PAD_JTAG and DIS_USB_JTAG on some, HARD_DIS_JTAG on the ESP32-S2). eFuses are one-time programmable, so the hard-disable ones are a one-way door too. Run your first flash-encryption experiments on a board you can afford to lose.

    Locks that fail

    A protection bit is only as good as the silicon that enforces it. Two cases I can point to with sources.

    Obermaier and Tatschner at Fraunhofer AISEC presented “Shedding too much Light on a Microcontroller’s Firmware Protection” at USENIX WOOT 2017. They looked at the STM32F0 and found three weaknesses: a common configuration that gave low protection and could be exploited with a cold-boot approach, a design weakness that let the protection level be downgraded, and a hardware race condition in the debug interface that let them extract read-protected firmware. Each attack needed only low-priced equipment, according to the abstract.

    CVE-2020-27211 is the nRF52840 case, where Nordic’s flash read-out protection is called APPROTECT. NVD describes it as being bypassable by injecting a fault during the boot phase, with a CVSS 3.1 base score of 5.7 and a physical attack vector. The score is modest for a consumer gadget. For a device holding a fleet-wide secret in flash it is not, which is the point I would make to a product team.

    My takeaway for anyone shipping hardware: assume someone with the device in hand eventually gets in, and make sure what they find is worth nothing beyond that one unit. Per-device keys, no shared credentials in firmware, no signing key near the board. The lock buys time. It does not keep a secret.

    What I check on my own devices

    On a product I am responsible for, “debug port open, same key on every unit” goes in the risk register ahead of most software findings.

    If you do connect a debug adapter to a board you own, these rules come from the earlier parts and I follow every one of them:

    • Power the target from its own supply, and only one source at a time. Never connect an adapter’s power output to a target that has its own supply, least of all the red 5V wire of a cable like mine. The reference-voltage pin, where the adapter has one, is a sense input and the only power-related pin to wire.
    • With the board powered and nothing connected, measure its supply or the header’s reference pad against ground to learn the logic level. Then power down, connect ground first and the signals second, power up, and only then start OpenOCD. Disconnect in the opposite order.
    • Power the board down, and disconnect any battery, before moving a lead or soldering a header. Let the iron tip cool before you put it away.
    • Do not probe or connect anything to a board that is plugged into mains, or whose supply is mains-connected and non-isolated. A low-voltage output does not make it safe, and bulk capacitors stay charged after you unplug.
    • Touch a grounded object first or use a wrist strap, and handle boards by the edges. Do not wear a wrist strap near mains-connected circuits.
    • Read the protection settings before you change them, and back up the firmware before you lock anything, because afterwards you cannot. Level 1 on an STM32 can only be removed by a mass erase, level 2 cannot be removed, and ESP32 eFuse burns cannot be undone and can brick the board.

    That closes the series. Measure first, read before you write, and back up the thing you cannot replace.