Tag: vulnerability management

  • Patch SLAs that teams actually meet

    Patch SLAs that teams actually meet

    Part 3 of 3 in the Vulnerability management that works series. Part 1: Prioritising vulnerabilities: CVSS vs EPSS vs CISA KEV. Part 2: SBOMs: what they’re actually good for.

    Most patch SLAs I’ve reviewed were written once, by a security team, copied from a compliance framework, and then missed every month for years. Everyone knows they’re missed. The dashboard is red, the quarterly report says “remediation velocity is improving”, and the actual exposure doesn’t change much. An SLA that’s always missed stops being a control and turns into a number people have stopped reading.

    The first two parts of this series covered deciding what matters and knowing where it runs. This one is about the commitment itself: how to write an SLA that engineering teams can actually hit, and how to measure it so the number tells you something.

    Why SLAs get missed

    Table of patch SLA tiers: P1 7 days (72 hours for KEV on internet-facing assets), P2 30 days, P3 90 days, P4 untimed, with exception limits, per-tier metrics and a timeline showing the clock starting at the later of fix availability and first detection

    In my experience it’s rarely laziness. The usual causes are structural.

    The SLA is keyed to CVSS. “Criticals in 15 days, highs in 30” sounds reasonable until you see that a big share of your findings are high or critical, most of them in packages nobody can exploit. Teams triage by ignoring, and the ones that matter get lost in the pile.

    The clock starts at the wrong time. If the timer starts when the CVE is published, but your scanner only picks it up eleven days later, a team with a 15-day SLA has four days left before they’ve even heard about it.

    The fix is often outside the team’s hands. A vulnerable OpenSSL in a base image maintained by a platform team, or a library pinned by a vendor’s SDK, can’t be fixed by the team that gets the ticket. They miss the SLA and learn that the SLA is noise.

    There’s no legitimate way to say no. Without an exception process, a team facing a fix that would break a major integration has two options: silently miss, or mark the finding as a false positive. Both destroy the data.

    Tie tiers to priority, not severity

    Use the priority tiers from part one as the SLA tiers. Something like this, written as policy in a file that lives in a repo and gets reviewed like code:

    # vuln-sla.yaml
    tiers:
      P1:   # in CISA KEV, or high EPSS on an exposed asset
        remediate_within: 7d
        emergency_if: [kev, internet_facing]
        emergency_within: 72h
        max_exception: 14d
        exception_approver: security-lead
      P2:   # high EPSS, or CVSS >= 9.0 on an exposed asset
        remediate_within: 30d
        max_exception: 60d
        exception_approver: engineering-manager
      P3:   # CVSS >= 7.0
        remediate_within: 90d
        max_exception: 180d
        exception_approver: engineering-manager
      P4:
        remediate_within: null   # fixed by routine upgrades, tracked not timed
    clock_starts: max(fix_available, first_detected)

    These numbers aren’t arbitrary, though they aren’t magic either. PCI DSS v4.0 requirement 6.3.3 asks for critical patches to be installed within one month of release. FedRAMP’s continuous monitoring expects high-risk findings fixed within 30 days, moderate within 90 and low within 180. If you’re under one of those regimes, your tiers have to fit inside them. If you’re not, they’re still a sane reference point.

    P1 is deliberately short. The 72-hour emergency path exists because the gap between a fix being available and mass exploitation can be very short for internet-facing products. Equifax is the case everyone cites for a reason: the Apache Struts fix for CVE-2017-5638 was out in early March 2017, and attackers were in through an unpatched dispute portal by mid-May. That’s about two months, and the exploit was public within days of the advisory.

    Start the clock fairly

    Starting the clock at max(fix_available, first_detected) sounds generous. It’s the version that puts detection on the hook, which is where it belongs. If your SBOM store gets rescanned daily (part two), first detection is within a day of the advisory for anything you’ve inventoried, and the gap stops mattering. If it isn’t, the SLA report shows a lot of findings detected weeks after publication, and that’s a scanning problem you can fix.

    Starting at fix availability matters too. Holding a team to a 7-day SLA for a vulnerability with no patch measures nothing. For no-fix cases, the P1 action is a mitigation, such as a WAF rule, disabling the feature or pulling the service off the internet, and the SLA applies to the mitigation.

    Route the ticket to whoever can fix it

    The single most effective change I’ve seen is sending findings to the team that controls the fix instead of the team that owns the running service. A vulnerable package in a shared base image becomes one ticket for the platform team, not forty tickets for forty application teams who all have to wait on the same rebuild.

    That requires the asset inventory to know two owners per artifact: who runs it and who builds each layer. It’s worth the effort. Duplicate tickets for an unfixable root cause are the fastest way to teach people that the SLA isn’t real.

    Make the cheap path the default

    Teams meet SLAs when patching is boring. Concretely:

    • Rebuild container images on a schedule, not only when the app changes. A nightly rebuild against the latest base image clears most OS-package findings without anyone opening a ticket.
    • Let Renovate or Dependabot open dependency PRs continuously, and auto-merge patch-level updates when tests pass. The SLA work then becomes reviewing a PR that already exists.
    • Run unattended security updates on long-lived hosts, with a reboot window, so kernel and library fixes don’t wait for a human.
    • Keep a test suite good enough that people trust an automated upgrade. This is the hidden dependency of everything above.

    If a P3 finding requires a human to notice it, research it, write the fix and deploy it, a 90-day SLA will be missed in any organisation with more findings than engineers.

    Measure the distribution, not the average

    Mean time to remediate is the metric everyone reports and it’s nearly useless. A few ancient findings drag it up, a flood of auto-merged fixes drags it down, and it moves for reasons unrelated to risk. Report three things per tier instead: the percentage closed within SLA, the median and 90th percentile time to close, and the age of the oldest open finding.

    SELECT priority,
           round(100.0 * count(*) FILTER (WHERE closed_at - clock_start <= sla) / count(*), 1) AS pct_within_sla,
           percentile_cont(0.5) WITHIN GROUP (ORDER BY closed_at - clock_start) AS p50,
           percentile_cont(0.9) WITHIN GROUP (ORDER BY closed_at - clock_start) AS p90
    FROM findings
    WHERE closed_at >= now() - interval '90 days'
    GROUP BY priority
    ORDER BY priority;

    That query only sees closed findings, which flatters you. Pair it with the oldest open finding per tier, because a P1 that has been open for 40 days is the one an incident report will be about.

    Exceptions get reported too: how many are open, how many were renewed, and which teams hold most of them. An exception renewed three times is an accepted risk, and someone senior should sign it by name.

    The test of a good SLA

    An SLA is working when the percentage within SLA is high enough that a miss is surprising, and when a miss leads to a conversation about why. If your P1 compliance is 95% and every miss gets a short write-up, people will treat P1 seriously. If it’s 40%, nobody will, no matter what the policy document says.

    So start with tiers you can meet today, even if they feel loose, and tighten them as automation catches up. A 30-day P2 that’s met 95% of the time beats a 15-day P2 that’s met half the time, both for actual risk and for whether anyone believes the number.

  • SBOMs: what they’re actually good for

    SBOMs: what they’re actually good for

    Part 2 of 3 in the Vulnerability management that works series. Part 1: Prioritising vulnerabilities: CVSS vs EPSS vs CISA KEV.

    Software bills of materials had a hype cycle. Executive Order 14028 in May 2021 told US federal suppliers to provide them, vendors rushed out SBOM features, and for a couple of years every security conference had a talk about them. A lot of organisations now generate SBOMs because a contract or a customer questionnaire asked for one, file them somewhere, and never look at them again.

    That’s a shame, because there’s one job SBOMs do very well, and it’s the job that hurt most during Log4Shell. When a new vulnerability lands, they tell you where you’re running the affected component, in minutes instead of days.

    What an SBOM is, concretely

    Pipeline diagram: CI build, syft SBOM, cosign attestation and registry, with SBOM copies in a store that is queried when a new CVE is published, filtered through VEX, producing affected services for prioritisation

    An SBOM is a machine-readable inventory of the components inside a piece of software: package names, versions, identifiers, and ideally the relationships between them. There are two formats that matter. SPDX came out of the Linux Foundation and is published as ISO/IEC 5962:2021. CycloneDX came out of OWASP and has since been standardised by Ecma as ECMA-424. Both are fine. Pick the one your tooling handles best and stop debating it.

    The NTIA’s 2021 minimum elements document lists the fields an SBOM should carry: supplier, component name, version, other unique identifiers, dependency relationships, the author of the SBOM data, and a timestamp. In practice the identifier that does the heavy lifting is the package URL, or PURL, something like pkg:maven/org.apache.logging.log4j/[email protected] or pkg:deb/debian/[email protected]. PURLs are what let a scanner match a component against an advisory database without guessing from file names.

    Generate them at build time, from the artifact

    Where you generate the SBOM matters more than which format you use. An SBOM written from a lockfile describes what you meant to ship. An SBOM generated from the built container image describes what you actually shipped, including the OS packages in the base image that no lockfile in your repo mentions.

    Syft is the tool I reach for. Generating a CycloneDX SBOM for an image in CI is one line:

    syft registry.example.com/payments-api:1.42.0 -o cyclonedx-json=payments-api.cdx.json

    Then attach it to the image, so the inventory travels with the artifact instead of living in a CI log nobody can find:

    cosign attest --yes \
      --predicate payments-api.cdx.json \
      --type cyclonedx \
      registry.example.com/payments-api@sha256:<digest>

    Attest against the digest, not the tag. Tags move. An SBOM attached to :latest is a statement about an image that may no longer exist.

    Also push a copy into something you can query across the fleet. That can be a dedicated platform like OWASP Dependency-Track, or honestly a bucket of JSON files plus an index. Being able to ask one question across every artifact you run matters more than the format.

    The question SBOMs answer

    That question is: “which of our running things contain component X at version Y?”

    On 29 March 2024, Andres Freund disclosed the backdoor in xz-utils, CVE-2024-3094, affecting versions 5.6.0 and 5.6.1. If you had SBOMs for every deployed image, the first hour looked like this:

    for f in sboms/*.cdx.json; do
      jq -r --arg f "$f" '.components[]
        | select(.name == "xz-utils" or .name == "liblzma5" or .name == "xz-libs")
        | select(.version | test("^5\\.6\\.[01]"))
        | "\($f)  \(.name)  \(.version)"' "$f"
    done

    If you didn’t, the first hour was a Slack thread asking every team to check their base images, followed by a day of partial answers. I’ve been on both sides of that, and the gap between the two is obvious in hindsight. During Log4Shell I watched teams spend most of a week just establishing which services bundled log4j-core, because it was often a transitive dependency several levels down, or shaded inside another jar.

    The same SBOM store also feeds continuous matching. Instead of rescanning every image when the vulnerability database updates, you rescan the SBOMs:

    grype sbom:payments-api.cdx.json --fail-on high

    It’s quick, and it covers images that are already running and won’t be rebuilt this week.

    Where SBOMs are blind

    SBOMs are only as complete as the tool that generated them, and every generator has gaps.

    Statically linked and vendored code is the big one. If a C library was copied into your source tree or compiled into a binary, there’s no package manager metadata for a scanner to find. Go is better than most here because the toolchain embeds module information in the binary, which go version -m will print and Syft will read. Rust binaries carry nothing by default unless you build with cargo auditable. Copied-in JavaScript files, a single vendored .c file, or a library someone downloaded and checked in years ago will mostly be invisible.

    Version strings can lie too. Distributions backport security fixes without changing the upstream version, so a Debian package might report a version that looks vulnerable while carrying the patch. Good scanners read the distro’s security tracker data for this. Bad ones generate a stream of false positives that teach engineers to ignore the scanner.

    And an SBOM says nothing about reachability. A vulnerable component that’s present isn’t necessarily loaded, and a loaded library isn’t necessarily being called with attacker-controlled input.

    VEX is what makes SBOMs bearable

    That last gap is where VEX comes in, the Vulnerability Exploitability eXchange. A VEX document is a statement about a specific vulnerability in a specific product, with a status: not_affected, affected, fixed or under_investigation. A not_affected claim has to carry a justification, such as vulnerable_code_not_present or vulnerable_code_not_in_execute_path.

    OpenVEX, CSAF and CycloneDX can all carry VEX statements, and Grype accepts OpenVEX documents to suppress findings you’ve already analysed:

    grype sbom:payments-api.cdx.json --vex payments-api.openvex.json

    What this buys you is mostly organisational. Someone investigates a finding once, writes down why it doesn’t apply, and that decision follows the artifact. Without it, the same finding gets triaged by three different people in three different quarters, and the one who’s in a hurry marks it as a false positive with no reason recorded.

    What I’d actually do

    Generate SBOMs from built images in CI, attest them to the digest, and copy them into a queryable store. Rescan the stored SBOMs daily against current advisory data and feed the results into the prioritisation from part one, so a KEV entry on an exposed service jumps to the front. Record triage decisions as VEX next to the SBOM.

    Skip the parts that don’t pay. I wouldn’t spend time on SBOM quality scores, on publishing SBOMs to customers who haven’t asked, or on generating them for every intermediate build. The SBOM that matters is the one for the thing running in production right now.

    Part three covers the last mile: setting patch SLAs that teams can actually meet once you know what to patch and where.

    Other parts: Part 1: Prioritising vulnerabilities: CVSS vs EPSS vs CISA KEV, Part 3: Patch SLAs that teams actually meet.

  • Prioritising vulnerabilities: CVSS vs EPSS vs CISA KEV

    Prioritising vulnerabilities: CVSS vs EPSS vs CISA KEV

    Part 1 of 3 in the Vulnerability management that works series.

    Every vulnerability programme I’ve seen fail failed the same way. The scanner produced thousands of findings, someone decided “fix all criticals and highs within 30 days”, the backlog grew faster than anyone could patch, and within a quarter the SLA dashboard was a wall of red that everyone had learned to ignore. The problem usually isn’t effort. It’s that the team is sorting by the wrong number.

    This series is about running vulnerability management so it actually reduces risk. Part one is about the sorting, because if you get that wrong nothing downstream matters.

    CVSS measures severity, not risk

    Flowchart: a scanner finding goes to P1 if in CISA KEV or high EPSS and exposed, P2 for high EPSS or critical CVSS on exposed systems, P3 for CVSS 7 and above, otherwise P4

    The Common Vulnerability Scoring System, maintained by FIRST, is the score everyone knows. A CVSS base score describes the intrinsic characteristics of a vulnerability: attack vector, complexity, privileges required, user interaction, and the impact on confidentiality, integrity and availability. It answers “how bad is this if someone exploits it?”

    It doesn’t answer “will anyone exploit it?”, and the specification says so. The CVSS v3.1 spec states plainly that CVSS measures severity, not risk. Teams use the base score as a risk score anyway, because it’s the column the scanner shows.

    The practical problem is distribution. A large share of published CVEs land at 7.0 or above, so “fix all highs and criticals” means fixing a majority of everything. When half your backlog is priority one, you don’t have priorities.

    CVSS v4.0, released by FIRST in November 2023, tries to address this by making the extra metric groups more visible. There are now Base, Threat, Environmental and Supplemental groups, and the spec asks you to label which ones a score includes (CVSS-B, CVSS-BT, CVSS-BE, CVSS-BTE). The Threat group’s Exploit Maturity metric lets you lower a score when there’s no known exploitation. That’s an improvement in principle. In practice almost every feed you consume publishes base scores only, and the threat data has to come from somewhere else anyway. Which brings us to the other two.

    EPSS estimates likelihood

    The Exploit Prediction Scoring System, also run by a FIRST special interest group, is a model that outputs a probability between 0 and 1 that a given CVE will see exploitation activity in the wild in the next 30 days. It’s trained on observed exploitation telemetry and retrained periodically, with scores recomputed daily. Its inputs include whether public exploit code exists, mentions in security lists, the vendor and the vulnerability type.

    Each CVE gets two numbers: the probability and a percentile. The percentile tells you how it ranks against every other scored CVE. Most CVEs have a very low EPSS probability, which is the whole point. The model is telling you that the vast majority of vulnerabilities never get exploited at all, and it’s trying to tell you which few will.

    You can query it for free:

    curl -s "https://api.first.org/data/v1/epss?cve=CVE-2021-44228,CVE-2023-4863" \
      | jq -r '.data[] | "\(.cve)  epss=\(.epss)  pct=\(.percentile)  as_of=\(.date)"'

    FIRST also publishes the full daily score set as a compressed CSV, which is what you want if you’re enriching a whole scanner export rather than looking up a handful.

    EPSS has limits I’d keep in mind. It’s a prediction about the internet at large, not about your environment. A CVE with a low EPSS score in a product exposed directly to the internet on your perimeter deserves more attention than the model gives it. It also lags. A brand new CVE often scores low on day one simply because the model hasn’t seen any signal yet. And it’s a probability, so treat a score of 0.02 as “unlikely”, never as “safe”.

    KEV is evidence, not prediction

    CISA’s Known Exploited Vulnerabilities catalog launched in November 2021 alongside Binding Operational Directive 22-01. A CVE goes into KEV when three things are true: it has a CVE ID, there’s reliable evidence of active exploitation, and there’s clear remediation guidance such as a vendor patch. BOD 22-01 requires US federal civilian agencies to remediate each entry by a due date CISA sets when adding it, often two or three weeks out.

    KEV is small compared to the CVE corpus, well over a thousand entries against hundreds of thousands of CVEs, and that’s its strength. Every entry is something an attacker has actually used. It’s also available as a JSON feed:

    curl -s https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json \
      | jq -r '.vulnerabilities[] | [.cveID, .dateAdded, .dueDate, .knownRansomwareCampaignUse] | @tsv' \
      > kev.tsv

    The knownRansomwareCampaignUse field is worth carrying into your tooling. “Known” there means CISA has evidence ransomware operators used it, which in my experience is the single fastest way to get an executive to approve an emergency change window.

    What KEV doesn’t give you is coverage. Exploitation that nobody reported to CISA, or that hit products CISA doesn’t track, isn’t in there. Absence from KEV tells you very little.

    Combining them

    None of these three is a prioritisation scheme on its own. Together they’re pretty good. The way I’d layer them:

    1. In KEV: fix now, on the shortest SLA you have. Someone is using this.
    2. High EPSS and exposed: treat almost like KEV. Exposure means reachable by an attacker who doesn’t already have a foothold, so internet-facing services, VPN concentrators, email gateways.
    3. High EPSS or critical CVSS on exposed systems: the next tier, on a normal fast SLA.
    4. Everything else with a high CVSS: the standard backlog, fixed by routine patching and image rebuilds.
    5. The long tail: fixed when the component gets upgraded anyway.

    As code, roughly:

    def priority(cvss: float, epss: float, in_kev: bool, exposed: bool) -> str:
        if in_kev:
            return "P1"
        if epss >= 0.10 and exposed:
            return "P1"
        if epss >= 0.10 or (cvss >= 9.0 and exposed):
            return "P2"
        if cvss >= 7.0:
            return "P3"
        return "P4"

    The 0.10 cutoff is my choice, not a standard. Pick yours by looking at how many of your current findings land in each bucket. If P1 has more than a few dozen items in a mid-sized environment, your thresholds are too loose, or you have a real emergency and should treat it as one.

    If you want something more formal, CISA’s Stakeholder-Specific Vulnerability Categorization (SSVC) is a decision tree that takes exploitation status, exposure, automatability and mission impact and outputs one of four actions: Track, Track*, Attend or Act. It’s the same idea with more rigour, and it’s easier to defend in an audit than a Python function someone wrote on a Friday.

    The part nobody wants to hear

    The inputs that matter most for prioritisation aren’t in any public feed. Whether a vulnerable package is actually loaded at runtime, whether the vulnerable function is reachable, whether the host is exposed to the internet: that’s your asset inventory and your architecture. A Log4Shell finding in a jar that sits in an unused test directory and a Log4Shell finding in your public login service have identical CVSS, EPSS and KEV values. One of them is an incident.

    So the real first step is tagging assets with exposure and ownership. Once you have that, the public scores do most of the remaining work. Part two looks at SBOMs, which are the other half of knowing what you’re actually running.

    Other parts: Part 2: SBOMs: what they’re actually good for, Part 3: Patch SLAs that teams actually meet.