{"id":791,"date":"2026-11-15T11:01:12","date_gmt":"2026-11-15T19:01:12","guid":{"rendered":"https:\/\/blog.canutethegreat.com\/?p=791"},"modified":"2026-11-15T11:01:13","modified_gmt":"2026-11-15T19:01:13","slug":"patronus-who-is-eating-my-uplink-and-what-it-does-when-the-isp-only-half-works","status":"publish","type":"post","link":"https:\/\/blog.canutethegreat.com\/index.php\/2026\/11\/15\/patronus-who-is-eating-my-uplink-and-what-it-does-when-the-isp-only-half-works\/","title":{"rendered":"Patronus: who is eating my uplink, and what it does when the ISP only half works"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Full disclosure: I make Patronus, a router and firewall OS built on Gentoo. It&#8217;s young. Version 0.2.0 came out on October 30.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Two of the 0.2.0 pages I had only ever seen as test output. One is Traffic insight, which says which device talked to which country and network. The other is Uplink health, which watches each internet connection and is supposed to notice when one goes bad in a way ping doesn&#8217;t show. On November 15 I used both the way someone running a small network would: let some devices make noise, then break the ISP in three different ways.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What this setup is<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Another lab, not my house. One container with a handful of network namespaces: the router (the real <code>patronus-web<\/code> and the apply daemon, built from main at commit 2f72a49c, 5 minutes 48 seconds cold on a capped CI box), four LAN devices with made-up names, two fake ISPs, and a fake internet on documentation addresses. WAN1 is labelled fibre and WAN2 is the LTE backup, in a failover group with WAN1 first. The traffic is real curl, ping and dig, and the browser really reaches the admin page.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Some of it is fake. The country and network data is not DB-IP. I couldn&#8217;t use the real files in the lab, so I generated a file in the same format with ten address ranges, invented countries and invented companies such as &#8220;Beispiel Hosting GmbH&#8221;. Which country got how much traffic means nothing. I also sped the clocks up for the video (probe tick 2 seconds, history sample 3 seconds), then ran the same faults again at the production defaults, and the timings below come from those runs. None of it ran for longer than about an hour.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Which device is using the bandwidth<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The Insights page has a Live tab that shows flows right now, and history tabs for countries, networks and devices. The Devices tab was the one I wanted. After the lab devices had pushed traffic around for a while, laptop-demo had pulled 2.1 GB down, living-room-tv 292 MB, kids-tablet 16 MB, and nas-demo had pushed 297 MB up, which is what a backup job looks like.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-04-insights-devices-1440.webp\" alt=\"Devices tab: laptop-demo, nas-demo, living-room-tv and kids-tablet with download, upload, flows and top country\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">The Devices tab, last 24 hours. The top-country column comes from my fake mapping file.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Clicking a device gives an hourly bar and its top countries and networks.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-05-device-drilldown-laptop-demo-1440.webp\" alt=\"Per-device view of laptop-demo with an hourly bar, top countries and top networks\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">laptop-demo. Everything in the bottom two tables is demo data.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This is built from the kernel&#8217;s conntrack byte counters, not from a packet capture. A collector reads them every 15 seconds and adds the change in each flow&#8217;s counters to an hourly row for that device, direction, country and network. The design note gives the reason: the counters were already switched on by another part of the firewall and the parsing already existed. The same note admits that a flow which starts and ends between two reads loses up to one interval of bytes, and calls that acceptable.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There is a privacy page too. History is on by default for 30 days and 200 rows per hour. You can list addresses, devices or groups that are never recorded, and the docs say adding one deletes what was already recorded for it. I put my probe client on that list. The page also has a button that deletes all recorded history, which I left alone.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-07-insights-settings-privacy-1440.webp\" alt=\"Settings tab with retention settings, a not-recorded list holding one address, and the country and ASN database status\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">Settings. The database rows show the lab URL my fake files came from.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">When the ISP only half works<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The Uplinks tab shows each gateway with its state, round trip, loss, 24 hour availability and an hourly bar. WAN1 also had two extra probes, a DNS lookup against a resolver and a TCP connect, and WAN2 had a TCP probe. Those go out through the gateway&#8217;s own interface, so they test that link and not whichever one holds the default route. The SLA target was 99.5 percent, and the failover group switches on loss or on latency above 100 ms.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-08-uplinks-healthy-1440.webp\" alt=\"Uplinks tab with both gateways online\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">Both links healthy.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">First fault: 300 ms of delay on WAN1. WAN1 went to &#8220;latency&#8221; 3.3 seconds after I added it, and the router said the default route moved to WAN2 at 3.6 seconds. The events (gateway_degraded and group_failover) showed up at 8.2 seconds. When I took the delay off, WAN1 was healthy and the route back on it 7 seconds later.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-09-uplinks-latency-300ms-1440.webp\" alt=\"WAN1 marked latency at 300 ms round trip while WAN2 is online and carries the group\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">WAN1 at 300 ms, WAN2 carrying traffic.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Second, the ISP&#8217;s resolver stops answering while ping keeps working, which a ping-only monitor happily reports as fine. The probes did what the docs say. The DNS probe failed 10.7 seconds after the fault (it needs two misses in a row, 5 seconds apart) and the uplink_probe_failed event arrived at 17.4 seconds. Traffic stayed on WAN1, because a probe only moves traffic if you mark it critical.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-10-uplinks-dns-dead-ping-fine-1440.webp\" alt=\"WAN1 still labelled online while its DNS probe fails, shown in amber\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">WAN1 with a dead resolver. The label still says online.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Third: I marked the DNS probe critical, then broke the resolver again. Now a failing probe counts as packet loss. Ping to the ISP still answered the whole time, yet the router said the default route moved to WAN2 at 10.7 seconds, called WAN1 down at 13.8, and wrote gateway_down at 16.4. Fix the resolver and WAN1 was healthy again 10.4 seconds later, with the gateway_up event at 16.5.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-11-uplinks-critical-dns-failover-1440.webp\" alt=\"WAN1 labelled down with a red critical DNS probe failure and a one minute outage listed\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">The same resolver outage with the probe marked critical.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The clip is the DNS fault at the accelerated clock. At the top is the Uplinks page, in the middle is a LAN client that pings, resolves and fetches about once a second, and at the bottom is what the router API reports, polled every quarter second. I flip the probe to critical about 15 seconds in, and then you can watch loss, down, recovery and failback. The captions were added afterwards. I corrected one of them after looking at the result, and the reason is under the next heading.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video controls preload=\"none\" poster=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/patronus-insight-uplink-poster.jpg\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/patronus-insight-uplink.mp4\"><\/video><figcaption class=\"wp-element-caption\">DNS outage, probe made critical, failover, recovery. 56 seconds, clocks accelerated, real time otherwise.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-13b-event-log-sla-breach-1440.webp\" alt=\"Event log rows including gateway_down, uplink_sla_breach and uplink_probe_recovered\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">The event log after a critical DNS outage, with the SLA breach in it.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">What the docs say about the design<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Most of the design is written down in two places, so I&#8217;m quoting them. History comes from conntrack because the counters and parser already existed. DB-IP is the data source because MaxMind&#8217;s free files need an account and a licence agreement, while Patronus already shipped DB-IP files that need neither. The history tables are kept out of config revisions and restores because, in the code&#8217;s own words, they are measured data that a rollback or restore may not rewind. The extra probes exist because an ISP can answer ping with broken DNS, and a probe has to fail twice in a row so one lost UDP packet doesn&#8217;t flap a WAN.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Nobody wrote down why it&#8217;s 30 days and 200 rows, or why probes run every 5 seconds. The same goes for conntrack over NetFlow. I won&#8217;t make a reason up.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What doesn&#8217;t work (yet)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The one I&#8217;d fix first is in the clip. The router reports the default route moving to WAN2 at 22 seconds. The client lines in the same recording keep showing WAN1&#8217;s address for the rest of the clip. My first caption said the client now exited through WAN2. That was wrong, so I replaced it. I don&#8217;t know yet whether the fault is in my client script, in which routing table the LAN actually uses, or in Patronus. I didn&#8217;t get to dig before the lab was gone.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A dead resolver with a non-critical probe leaves the WAN1 label on &#8220;online&#8221;, availability at 100 percent, and the traffic on WAN1. The amber shows up in the probe chip, the chart shading and the &#8220;fully healthy&#8221; percentage. The page&#8217;s own intro says such an ISP &#8220;shows up as degraded&#8221;, and the degraded flag my script polled stayed false. I haven&#8217;t checked whether a different field carries it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With the probe critical, one resolver outage turned WAN1 into &#8220;down&#8221; in about 14 seconds with ping working fine, and with a single probed resolver that&#8217;s a single point of false failover. It&#8217;s awkward to set, too. The probe table only has a Delete button and the critical box is on the add form, so I changed the existing probe through the API. There may be an edit path I missed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Events lag the state. At production defaults the recorder reads every 15 seconds, so gateway_down arrived about 2.6 seconds after the state changed and the probe failure event about 7 seconds after the probe failed. Failback is slow too: loss decays after the fix, so the route comes back around 10 seconds later and the up event about 6 seconds after that.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The event log shows UTC times and the Uplinks card shows browser-local time, on the same page. The SLA text counts samples as minutes, so with my 3 second samples the breach event said &#8220;2 minutes down&#8221; for an outage of well under a minute. That&#8217;s an artifact of my sped-up sample setting, and I haven&#8217;t re-measured it at one sample a minute.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On a 390 pixel phone the device table runs off the card at the Download column, and in my event-log capture only Time, Event and About were visible. I didn&#8217;t check whether it scrolls sideways.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/pat-iu-14-phone-devices-390.webp\" alt=\"Devices table on a phone, cut off at the Download column\" loading=\"lazy\"\/><figcaption class=\"wp-element-caption\">Devices on a 390 pixel wide phone.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Things I didn&#8217;t try at all: the real DB-IP files and the monthly refresh, IPv6 flows, port forwards, the top-200 cutoff and 30-day pruning, webhook delivery of the new event types, a device that changes address, and a real ISP. The two netlab scenarios for these pages exist and I didn&#8217;t run them. 51 unit tests for this code pass (26 insight, 11 uplink, 7 for the network database, 7 for the probes), and nothing has touched it since October 30.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Where it stands<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The probes do catch an ISP that answers ping and nothing else. I&#8217;d like the page to say so more plainly: right now a probe only matters if you find the critical box, and the WAN1 label keeps saying online. I can&#8217;t judge the device history, since the data under it was fake. The routing question is the one I&#8217;ll chase first.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Next: the real GeoIP files and a few days on a real clock, and finding out where that client&#8217;s traffic really went.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Patronus lives at <a href=\"https:\/\/patronus.errant.solutions\/\">patronus.errant.solutions<\/a>.<\/p>\n\n\n","protected":false},"excerpt":{"rendered":"<p>Full disclosure: I make Patronus, a router and firewall OS built on Gentoo. It&#8217;s young. Version 0.2.0 came out on October 30. Two of the 0.2.0 pages I had only ever seen as test output. One is Traffic insight, which says which device talked to which country and network. The other is Uplink health, which [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":803,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,21],"tags":[252,374,403,174,404,386],"class_list":["post-791","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux","category-security","tag-dns","tag-firewall","tag-multi-wan","tag-networking","tag-network-monitoring","tag-patronus"],"_links":{"self":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/791","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/comments?post=791"}],"version-history":[{"count":2,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/791\/revisions"}],"predecessor-version":[{"id":805,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/791\/revisions\/805"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/media\/803"}],"wp:attachment":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/media?parent=791"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/categories?post=791"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/tags?post=791"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}