Tag: DNS

  • Patronus: who is eating my uplink, and what it does when the ISP only half works

    Patronus: who is eating my uplink, and what it does when the ISP only half works

    Full disclosure: I make Patronus, a router and firewall OS built on Gentoo. It’s young. Version 0.2.0 came out on October 30.

    Two of the 0.2.0 pages I had only ever seen as test output. One is Traffic insight, which says which device talked to which country and network. The other is Uplink health, which watches each internet connection and is supposed to notice when one goes bad in a way ping doesn’t show. On November 15 I used both the way someone running a small network would: let some devices make noise, then break the ISP in three different ways.

    What this setup is

    Another lab, not my house. One container with a handful of network namespaces: the router (the real patronus-web and the apply daemon, built from main at commit 2f72a49c, 5 minutes 48 seconds cold on a capped CI box), four LAN devices with made-up names, two fake ISPs, and a fake internet on documentation addresses. WAN1 is labelled fibre and WAN2 is the LTE backup, in a failover group with WAN1 first. The traffic is real curl, ping and dig, and the browser really reaches the admin page.

    Some of it is fake. The country and network data is not DB-IP. I couldn’t use the real files in the lab, so I generated a file in the same format with ten address ranges, invented countries and invented companies such as “Beispiel Hosting GmbH”. Which country got how much traffic means nothing. I also sped the clocks up for the video (probe tick 2 seconds, history sample 3 seconds), then ran the same faults again at the production defaults, and the timings below come from those runs. None of it ran for longer than about an hour.

    Which device is using the bandwidth

    The Insights page has a Live tab that shows flows right now, and history tabs for countries, networks and devices. The Devices tab was the one I wanted. After the lab devices had pushed traffic around for a while, laptop-demo had pulled 2.1 GB down, living-room-tv 292 MB, kids-tablet 16 MB, and nas-demo had pushed 297 MB up, which is what a backup job looks like.

    Devices tab: laptop-demo, nas-demo, living-room-tv and kids-tablet with download, upload, flows and top country
    The Devices tab, last 24 hours. The top-country column comes from my fake mapping file.

    Clicking a device gives an hourly bar and its top countries and networks.

    Per-device view of laptop-demo with an hourly bar, top countries and top networks
    laptop-demo. Everything in the bottom two tables is demo data.

    This is built from the kernel’s conntrack byte counters, not from a packet capture. A collector reads them every 15 seconds and adds the change in each flow’s counters to an hourly row for that device, direction, country and network. The design note gives the reason: the counters were already switched on by another part of the firewall and the parsing already existed. The same note admits that a flow which starts and ends between two reads loses up to one interval of bytes, and calls that acceptable.

    There is a privacy page too. History is on by default for 30 days and 200 rows per hour. You can list addresses, devices or groups that are never recorded, and the docs say adding one deletes what was already recorded for it. I put my probe client on that list. The page also has a button that deletes all recorded history, which I left alone.

    Settings tab with retention settings, a not-recorded list holding one address, and the country and ASN database status
    Settings. The database rows show the lab URL my fake files came from.

    When the ISP only half works

    The Uplinks tab shows each gateway with its state, round trip, loss, 24 hour availability and an hourly bar. WAN1 also had two extra probes, a DNS lookup against a resolver and a TCP connect, and WAN2 had a TCP probe. Those go out through the gateway’s own interface, so they test that link and not whichever one holds the default route. The SLA target was 99.5 percent, and the failover group switches on loss or on latency above 100 ms.

    Uplinks tab with both gateways online
    Both links healthy.

    First fault: 300 ms of delay on WAN1. WAN1 went to “latency” 3.3 seconds after I added it, and the router said the default route moved to WAN2 at 3.6 seconds. The events (gateway_degraded and group_failover) showed up at 8.2 seconds. When I took the delay off, WAN1 was healthy and the route back on it 7 seconds later.

    WAN1 marked latency at 300 ms round trip while WAN2 is online and carries the group
    WAN1 at 300 ms, WAN2 carrying traffic.

    Second, the ISP’s resolver stops answering while ping keeps working, which a ping-only monitor happily reports as fine. The probes did what the docs say. The DNS probe failed 10.7 seconds after the fault (it needs two misses in a row, 5 seconds apart) and the uplink_probe_failed event arrived at 17.4 seconds. Traffic stayed on WAN1, because a probe only moves traffic if you mark it critical.

    WAN1 still labelled online while its DNS probe fails, shown in amber
    WAN1 with a dead resolver. The label still says online.

    Third: I marked the DNS probe critical, then broke the resolver again. Now a failing probe counts as packet loss. Ping to the ISP still answered the whole time, yet the router said the default route moved to WAN2 at 10.7 seconds, called WAN1 down at 13.8, and wrote gateway_down at 16.4. Fix the resolver and WAN1 was healthy again 10.4 seconds later, with the gateway_up event at 16.5.

    WAN1 labelled down with a red critical DNS probe failure and a one minute outage listed
    The same resolver outage with the probe marked critical.

    The clip is the DNS fault at the accelerated clock. At the top is the Uplinks page, in the middle is a LAN client that pings, resolves and fetches about once a second, and at the bottom is what the router API reports, polled every quarter second. I flip the probe to critical about 15 seconds in, and then you can watch loss, down, recovery and failback. The captions were added afterwards. I corrected one of them after looking at the result, and the reason is under the next heading.

    DNS outage, probe made critical, failover, recovery. 56 seconds, clocks accelerated, real time otherwise.
    Event log rows including gateway_down, uplink_sla_breach and uplink_probe_recovered
    The event log after a critical DNS outage, with the SLA breach in it.

    What the docs say about the design

    Most of the design is written down in two places, so I’m quoting them. History comes from conntrack because the counters and parser already existed. DB-IP is the data source because MaxMind’s free files need an account and a licence agreement, while Patronus already shipped DB-IP files that need neither. The history tables are kept out of config revisions and restores because, in the code’s own words, they are measured data that a rollback or restore may not rewind. The extra probes exist because an ISP can answer ping with broken DNS, and a probe has to fail twice in a row so one lost UDP packet doesn’t flap a WAN.

    Nobody wrote down why it’s 30 days and 200 rows, or why probes run every 5 seconds. The same goes for conntrack over NetFlow. I won’t make a reason up.

    What doesn’t work (yet)

    The one I’d fix first is in the clip. The router reports the default route moving to WAN2 at 22 seconds. The client lines in the same recording keep showing WAN1’s address for the rest of the clip. My first caption said the client now exited through WAN2. That was wrong, so I replaced it. I don’t know yet whether the fault is in my client script, in which routing table the LAN actually uses, or in Patronus. I didn’t get to dig before the lab was gone.

    A dead resolver with a non-critical probe leaves the WAN1 label on “online”, availability at 100 percent, and the traffic on WAN1. The amber shows up in the probe chip, the chart shading and the “fully healthy” percentage. The page’s own intro says such an ISP “shows up as degraded”, and the degraded flag my script polled stayed false. I haven’t checked whether a different field carries it.

    With the probe critical, one resolver outage turned WAN1 into “down” in about 14 seconds with ping working fine, and with a single probed resolver that’s a single point of false failover. It’s awkward to set, too. The probe table only has a Delete button and the critical box is on the add form, so I changed the existing probe through the API. There may be an edit path I missed.

    Events lag the state. At production defaults the recorder reads every 15 seconds, so gateway_down arrived about 2.6 seconds after the state changed and the probe failure event about 7 seconds after the probe failed. Failback is slow too: loss decays after the fix, so the route comes back around 10 seconds later and the up event about 6 seconds after that.

    The event log shows UTC times and the Uplinks card shows browser-local time, on the same page. The SLA text counts samples as minutes, so with my 3 second samples the breach event said “2 minutes down” for an outage of well under a minute. That’s an artifact of my sped-up sample setting, and I haven’t re-measured it at one sample a minute.

    On a 390 pixel phone the device table runs off the card at the Download column, and in my event-log capture only Time, Event and About were visible. I didn’t check whether it scrolls sideways.

    Devices table on a phone, cut off at the Download column
    Devices on a 390 pixel wide phone.

    Things I didn’t try at all: the real DB-IP files and the monthly refresh, IPv6 flows, port forwards, the top-200 cutoff and 30-day pruning, webhook delivery of the new event types, a device that changes address, and a real ISP. The two netlab scenarios for these pages exist and I didn’t run them. 51 unit tests for this code pass (26 insight, 11 uplink, 7 for the network database, 7 for the probes), and nothing has touched it since October 30.

    Where it stands

    The probes do catch an ISP that answers ping and nothing else. I’d like the page to say so more plainly: right now a probe only matters if you find the critical box, and the WAN1 label keeps saying online. I can’t judge the device history, since the data under it was fake. The routing question is the one I’ll chase first.

    Next: the real GeoIP files and a few days on a real clock, and finding out where that client’s traffic really went.

    Patronus lives at patronus.errant.solutions.

  • Zombie infrastructure: forgotten assets and attack surface management

    Zombie infrastructure: forgotten assets and attack surface management

    Every organisation I’ve worked with had infrastructure nobody remembered. A staging environment from a project that got cancelled. A marketing microsite from a 2019 campaign, still running a CMS that hasn’t been patched since. A DNS record pointing at a cloud resource someone deleted two years ago. A test tenant with a service account whose password hasn’t changed since it was created.

    None of it shows up in the asset inventory, because if it did, someone would have turned it off. That’s what makes it dangerous. You can’t patch, monitor or decommission something you don’t know exists, and attackers are very good at finding exactly those things. It’s Halloween, so let’s talk about the undead.

    Why this keeps causing breaches

    Flow diagram: Certificate Transparency, subdomain enumeration, scans and DNS exports plus cloud inventory, IaC state and identity provider feed a reconcile step that outputs dangling DNS, unowned resources, shadow services and stale identities, each routed to assign an owner or disconnect then delete, on a scheduled loop

    Two incidents make the case better than any vendor slide.

    In the 2017 Equifax breach, attackers came in through CVE-2017-5638 in Apache Struts on an online dispute portal. The patch had been out since early March. Equifax had an internal alert about it and ran a scan, and the scan didn’t find the vulnerable system. Then, per the US House Oversight Committee’s report, an SSL certificate on a device that inspected outbound traffic had expired months earlier, so data left the network uninspected for weeks. The vulnerable app and the blind monitoring device were both things the organisation owned and had lost track of.

    In January 2024, Microsoft disclosed that the group it tracks as Midnight Blizzard had password-sprayed its way into a legacy, non-production test tenant account that didn’t have MFA. From there the attackers used a legacy test OAuth application with elevated access to reach corporate mailboxes, including senior leadership. The attack didn’t need anything clever. It needed one forgotten account in one forgotten tenant.

    The pattern is consistent. The production system everyone watches is usually fine. The breach starts in the thing that was supposed to be temporary.

    The common kinds of zombie

    Forgotten hosts and services are the obvious ones: old VMs, test environments, a jump box someone built for a migration. They’re unpatched because no patching process knows about them, and they often have weaker auth because they were “only for testing”.

    Dangling DNS is subtler and, in my experience, more common. A CNAME for promo.example.com points at example-promo.s3-website-us-east-1.amazonaws.com, or an Azure App Service hostname, or a Heroku app. The cloud resource gets deleted, but the DNS record stays. If the provider lets anyone claim that name, an attacker registers it and now serves content on your subdomain, with your cookies in scope if they’re set on the parent domain. The community-maintained “can-i-take-over-xyz” project on GitHub tracks which services are vulnerable to this and how to fingerprint them.

    Released IP addresses are the same problem one layer down. An A record still points at an Elastic IP or public IP you released. The cloud provider hands that IP to another customer, and your hostname now resolves to their server.

    Then there are identities: service accounts, API keys, OAuth apps and test tenants that outlived their project. These don’t show up in a port scan, which is why they get forgotten even by teams that do external scanning well.

    Find what you have from the outside first

    Start where an attacker starts, because that view is free of your assumptions.

    Certificate Transparency (RFC 6962) logs every publicly trusted TLS certificate, and every hostname you ever got a certificate for is in there. That’s usually the single best source of forgotten subdomains:

    curl -s 'https://crt.sh/?q=%25.example.com&output=json' \
      | jq -r '.[].name_value' | tr 'A-Z' 'a-z' | sed 's/^\*\.//' | sort -u > ct-hosts.txt

    Passive DNS and enumeration tools such as subfinder or OWASP Amass add hostnames from other sources. Then check which of them resolve and answer:

    subfinder -d example.com -silent | sort -u > enum-hosts.txt
    sort -u ct-hosts.txt enum-hosts.txt | dnsx -silent -a -cname -resp > resolved.txt
    cut -d' ' -f1 resolved.txt | sort -u | httpx -silent -status-code -title -tech-detect > live.txt

    Only run active probing against domains and IP ranges you own or are authorised to test. CT logs will happily list hostnames that belong to a vendor or a former subsidiary.

    For dangling DNS specifically, look at every CNAME pointing at a cloud provider’s namespace and check whether the target still exists. An NXDOMAIN on the target, or a provider’s “no such bucket” or “app not found” page, is the signal:

    dig +short CNAME promo.example.com
    # example-promo.s3-website-us-east-1.amazonaws.com.
    curl -s http://promo.example.com/ | grep -o 'NoSuchBucket'

    Then compare against what you think you have

    The outside view tells you what’s exposed. The inside view tells you what you’re paying for. Pull both and reconcile them.

    Every cloud provider has an inventory API. On AWS, the resource groups tagging API or AWS Config lists resources across services, and Route 53 can export every record you host. On GCP, Cloud Asset Inventory does the same with gcloud asset search-all-resources, and Azure has Resource Graph. Run them across every account and subscription, including the ones in the “sandbox” OU nobody looks at.

    The reconciliation is where the value is:

    • Hostnames in DNS or CT with no matching cloud resource are dangling records or something hosted elsewhere you didn’t know about.
    • Cloud resources with no owner tag and no recent deploys are zombie candidates.
    • Public IPs and load balancers that answer on the internet but aren’t in any service catalogue are shadow services.
    • Identities with no sign-in or API use in 90 days are accounts to disable.

    Give everything an owner, or a death date

    Discovery without ownership just produces a spreadsheet that’s wrong next month. The policy that works is simple to state and annoying to enforce: every asset has an owning team, and anything without one is scheduled for shutdown.

    Enforce it at creation time. Require an owner tag through AWS Service Control Policies or tag policies, Azure Policy or GCP organisation policy constraints, so untagged resources can’t be created in the first place. Give temporary environments an expires tag and a job that deletes them when it passes. Delete DNS records in the same change that deletes the resource they point to. Infrastructure as code helps a lot here, because removing a resource from Terraform can remove its DNS record in the same apply.

    For decommissioning, I like a two-step approach. First, disconnect the asset: remove it from DNS, block inbound traffic, disable the account. Wait a couple of weeks. If nobody screams, delete it. If someone does, you’ve just found the owner.

    Make it continuous

    A one-time sweep finds a pile of zombies and then the pile slowly grows back. Run the external discovery on a schedule, diff the results against the previous run, and alert on new hostnames, new open ports and new dangling CNAMEs. Commercial attack surface management products do this with more sources and nicer dashboards, but the core loop of discover, diff, assign an owner and decommission works fine with open-source tools and a cron job.

    The test is easy. Pick a random hostname from your CT log results and ask who owns it. If the answer takes more than a minute, you’ve got zombies.

  • SPF, DKIM and DMARC, and why p=none isn’t enough

    SPF, DKIM and DMARC, and why p=none isn’t enough

    Email was designed in an era when every host on the network trusted every other host. SMTP lets any server claim to send mail from any address, and the From: header that a person actually reads is just text the sender typed. Three standards were added later to patch that: SPF, DKIM and DMARC. Most domains I look at have all three records published, and a fair number still aren’t protected, because their DMARC policy says p=none.

    SPF: which servers may send

    Flowchart of how a receiving mail server evaluates SPF, DKIM, alignment and the DMARC policy

    Sender Policy Framework (RFC 7208) lets a domain publish, in DNS, the list of servers allowed to send mail for it. A receiving server takes the domain from the SMTP envelope sender (the MAIL FROM, also called the return path), looks up its SPF record, and checks whether the connecting IP is on the list.

    example.com.  IN TXT  "v=spf1 ip4:198.51.100.0/24 include:_spf.google.com include:sendgrid.net -all"

    SPF has two well-known weak spots. It checks the envelope sender, not the From: header the recipient sees, so on its own it does nothing against spoofing that address. It also breaks on forwarding, because the forwarding server isn’t on your list.

    There’s also a hard limit that catches growing companies: evaluating an SPF record may trigger at most 10 DNS lookups (include, a, mx, ptr, exists and redirect each count). Every SaaS tool you add with an include: eats into that budget, and when you exceed it the result is a permerror, which many receivers treat as a failure. I’ve seen a marketing team knock out a company’s outbound mail by adding one more vendor.

    DKIM: a signature on the message

    DomainKeys Identified Mail (RFC 6376) has the sending server sign selected headers and the body with a private key. The signature goes in a DKIM-Signature header naming the signing domain (d=) and a selector (s=). The receiver fetches the public key from DNS at selector._domainkey.domain and verifies.

    s1._domainkey.example.com.  IN TXT  "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA..."

    DKIM survives plain forwarding because the signature travels with the message, though it breaks if a mailing list rewrites the subject or appends a footer. Use 2048-bit RSA keys, and rotate them. In 2012 a mathematician found that Google was signing with a 512-bit key that he could factor, which led to a quick round of key upgrades across large providers.

    The catch: DKIM proves a message was signed by some domain (d=). It doesn’t, on its own, say that domain has anything to do with the From: address.

    DMARC: tying it to the From header

    DMARC (RFC 7489) is what makes the first two useful against spoofing. It adds alignment: a message passes DMARC only if SPF passes for a domain that matches the From: domain, or DKIM passes with a d= that matches the From: domain. Relaxed alignment (the default) accepts the same organisational domain, so mail.example.com aligns with example.com. Strict alignment requires an exact match.

    The DMARC record says what receivers should do with failing mail, and where to send reports:

    _dmarc.example.com.  IN TXT  "v=DMARC1; p=reject; rua=mailto:[email protected]; adkim=r; aspf=r"

    The p= tag is the policy: none, quarantine or reject.

    Why p=none isn’t protection

    A p=none policy tells receivers: evaluate DMARC, send me reports, but deliver failing mail exactly as you would have anyway. It’s monitoring only. Anyone can still send mail with your domain in the From: header to your customers, suppliers or finance team, and DMARC won’t stop them.

    Monitoring is the right first step, which is why p=none exists. The trouble is that many organisations treat publishing it as the finish line. Google and Yahoo’s bulk sender rules, in effect since February 2024, require a DMARC record for anyone sending more than 5,000 messages a day to their users, but the minimum they ask for is p=none. Plenty of domains published a record and stopped there.

    Business email compromise is where this hurts most. A convincing invoice from [email protected] is much easier to pull off when your domain doesn’t ask receivers to reject spoofs. With p=reject, major mailbox providers drop exact-domain spoofs outright. Attackers then fall back to lookalike domains. That’s still a problem, but a more detectable one.

    Getting from none to reject

    The path is boring, and it works.

    First, publish p=none with an rua address and actually read the aggregate reports. They’re XML, so use a parser or a reporting service rather than reading them by hand. Within a couple of weeks you’ll see every source sending mail as your domain: your mail platform, the CRM, the ticketing system, the HR tool somebody bought on a credit card, and the spoofers.

    Second, fix each legitimate source. Ideally each one signs with DKIM using your domain as d=, which most SaaS senders support through a CNAME to their key. Where you can only get SPF alignment, make sure the vendor uses a return path on your domain. Retire sources nobody can explain.

    Third, move to p=quarantine, then to p=reject. RFC 7489 provides the pct= tag to apply the policy to a percentage of failing mail, which lets you ramp up gradually. Note that the DMARCbis revision in the IETF drops pct in favour of a simpler testing flag, so don’t build long-term tooling around it.

    Don’t forget domains that send no mail at all. Parked domains are attractive for spoofing because nobody watches them. Lock them down completely:

    parked-example.com.          IN TXT  "v=spf1 -all"
    _dmarc.parked-example.com.   IN TXT  "v=DMARC1; p=reject;"

    Also check the sp= tag, which sets the policy for subdomains. If you leave it out, subdomains inherit p=, which is usually what you want.

    What DMARC doesn’t do

    DMARC protects the exact domain. It does nothing about examp1e.com or example-billing.com, nor display-name tricks where the friendly name says “Example Payroll” over an unrelated address. Those need receiver-side defences, user training and monitoring for newly registered lookalikes. MTA-STS (RFC 8461) and TLS reporting are separate pieces that protect mail in transit, and BIMI only becomes available once you’re at quarantine or reject.

    If you run a domain and its DMARC record still says p=none after a few months, you’ve built the monitoring and switched off the alarm. Read the reports, fix your senders, and finish the rollout.