Alert fatigue: tuning detections without going blind

Part 3 of 3 in the Detection engineering series. Part 1: Logging that actually helps an investigation. Part 2: Writing detections as code: Sigma rules tested against real logs.

In 2013, attackers inside Target’s network set off malware alerts from a detection tool the company had recently deployed. The alerts reached Target’s security team, and the US Senate Commerce Committee’s 2014 kill chain analysis of the breach concluded they weren’t acted on in time. Roughly 40 million payment cards were stolen. The tool worked. The process around it didn’t.

That pattern repeats everywhere I’ve looked. When a team gets hundreds of alerts a day and nearly all of them are benign, people learn, quite rationally, that alerts are probably nothing. Then the real one gets the same treatment.

This is part 3 of the series. Part 1 covered collecting the right events and part 2 covered writing and testing detections as code. Now we have to live with them.

Measure before you tune

You can’t tune what you don’t count. For every rule, track three numbers over a rolling 30 days: how many alerts it produced, how many were true positives after triage, and how long triage took. Precision per rule (true positives divided by total alerts) is the most useful single figure, and it’s almost never on the SIEM dashboard by default.

That means analysts have to record a disposition when they close an alert. Keep the choices small and honest: true positive, benign true positive (the rule matched what it was meant to, but the activity was authorised), false positive, and needs tuning. The “benign true positive” bucket matters, because an admin running a scanner is not the rule being wrong, and conflating the two leads to deleting good rules.

When you plot alert volume by rule, you’ll nearly always find that a handful of rules generate most of the noise. That’s good news. You don’t have to fix everything, just the top few.

Tune narrowly

The lazy fix for a noisy rule is excluding the whole host, the whole service account, or the whole subnet. Each of those is a hole an intruder can stand in. If your backup server is excluded from the PowerShell rule because its jobs are noisy, then PowerShell on the backup server is now invisible, and backup servers are exactly what ransomware operators go looking for.

Make exceptions as specific as the evidence allows: the exact parent process, the exact command line or script path, ideally the file hash and the expected account. In Sigma that’s a named filter in the rule itself:

detection:
  selection:
    Image|endswith: '\powershell.exe'
    CommandLine|contains: ' -EncodedCommand '
  filter_backup_job:
    ParentImage: 'C:\Program Files\BackupAgent\scheduler.exe'
    User: 'CORP\svc-backup'
    CommandLine|contains: 'C:\BackupAgent\scripts\prejob.ps1'
  condition: selection and not filter_backup_job

If the backup agent is compromised and starts running something else, the filter no longer matches and the alert fires.

Give every exception an owner and an expiry

Exceptions accumulate like firewall rules. Nobody remembers why half of them exist, and nobody dares remove them. Store them in the repository with metadata, and let CI enforce it:

# exceptions/powershell-encoded.yml
rule_id: 0c4b3a9e-71d2-4f5e-9b8a-6e2d1c7f3a10
filter: filter_backup_job
reason: Vendor pre-job script uses encoded commands (ticket SEC-1432)
owner: infra-backup
expires: 2027-01-31

import datetime, glob, sys, yaml

today = datetime.date.today()
expired = []
for path in glob.glob("exceptions/*.yml"):
    with open(path) as f:
        doc = yaml.safe_load(f)
    if doc["expires"] <= today:
        expired.append(f'{path}: owner {doc["owner"]}, expired {doc["expires"]}')

if expired:
    print("\n".join(expired))
    sys.exit(1)

When an exception expires, the owner either renews it with a fresh justification or it goes away. Ninety days is a reasonable default. It’s mildly annoying, and that’s what keeps the list short.

Change what happens, not just whether it fires

Not every detection should page a human. A lot of useful signals are weak on their own: a new admin tool on a workstation, a login from an unusual country, a service account authenticating interactively. Individually each one is noisy. Together, on the same host or user within a few hours, they tell a story.

That’s the idea behind risk-based alerting. Low-fidelity rules don’t create alerts; they add a risk score to the user or host they involve. A separate correlation rule fires when an entity’s accumulated score crosses a threshold, or when signals from several distinct ATT&CK tactics appear together. Splunk documents this approach as its RBA framework, but you can build the same thing in any SIEM that can aggregate by entity over a time window.

Alongside that, route by confidence. High-precision rules page someone. Medium ones land in a queue that’s worked during business hours. The rest feed risk scores and dashboards. A rule that has never produced a true positive in a year and doesn’t feed anything is a candidate for retirement, but only after you’ve confirmed it can still fire (see the next section), because a rule that never fires might be a rule that’s broken.

Prove you’re not blind

Tuning removes alerts. The danger is that you can’t easily see the alerts you removed, or the ones that stopped firing because a log source died. So you need a positive signal that coverage still exists.

Part 2 described scheduled end-to-end tests: a harmless, known action on a test host that a specific rule must catch within minutes. Run those continuously, track the pass rate per rule, and treat a failure as an incident in the detection pipeline. Combine it with a per-source volume monitor so that a domain controller going quiet raises its own alert.

I also like a quarterly review of the top 20 noisiest rules and every rule with zero alerts. The noisy ones need tuning. The silent ones need a test proving they still work.

Watch the humans

The last metric is the analysts themselves. If one person is triaging 200 alerts per shift, median triage time drops to seconds and the quality goes with it. Track alerts per analyst-hour and the share of alerts closed in under a minute. When those climb, the fix is in the rules, not in telling people to try harder.

A detection program is healthy when an alert is rare enough that people still feel a small jolt when one arrives. Getting there is mostly unglamorous bookkeeping, and it’s worth every hour.