Tag: distroless

Most container images in production are a full Linux distribution wearing a trench coat. A Debian or Ubuntu base, a package manager, a shell, curl, and an application running as root with a writable filesystem. It works, it’s easy to debug, and it hands an attacker who gets code execution everything they need for the next step.

Hardening an image isn’t one big change. It’s a handful of small ones that each remove a capability the application never needed, and that an attacker would very much like to have. None of them are new, and every one of them gets skipped all the time.

Start with less: minimal and distroless bases

The fastest win is shipping fewer things. A typical debian or ubuntu base pulls in a hundred-odd packages, and your scanner will report CVEs against all of them, whether or not your app ever touches them. That noise has a real cost, which I covered in the vulnerability management series earlier this week. Every finding in a package you don’t use is triage time you’re not spending on one that matters.

There are a few levels of minimal:

• scratch is empty. Good for fully static binaries, like Go built with CGO_ENABLED=0. You have to bring your own CA certificates and timezone data if you need them.
• Google’s distroless images contain the runtime pieces an app needs (libc, CA certs, tzdata, sometimes a language runtime) and no shell or package manager.
• Wolfi-based images from Chainguard and minimal Alpine images sit somewhere in between, with a package manager available at build time.

For a Go service, a multi-stage build onto distroless looks like this:

FROM golang:1.25 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
USER 65532:65532
ENTRYPOINT ["/app"]

The compiler, the source and the module cache stay in the build stage. The final image is your binary plus a few megabytes of support files.

The objection I always hear is debugging. “How do I exec into it without a shell?” On Kubernetes you don’t need to: an ephemeral debug container shares the target’s process namespace and brings its own tools.

kubectl debug -it pod/payments-api-7d9f8c --image=busybox:1.36 --target=app

That’s better than a shell baked into every image anyway, because the tools only exist while someone is actually debugging.

Don’t run as root

Containers running as UID 0 are the default, and it’s a bad default. Root inside a container isn’t the same as root on the host, since namespaces and dropped capabilities limit it, but it’s much closer than people assume, and it turns every container runtime bug into a bigger problem.

CVE-2019-5736 is the classic example. A malicious process in a container could overwrite the host’s runc binary and get root on the host the next time runc ran. The exploit needed root inside the container. A container running as an unprivileged user wasn’t exploitable that way.

Set the user in the image, with a numeric UID so Kubernetes can verify it:

USER 65532:65532

Then enforce it at the platform level so an image that forgets doesn’t get scheduled. In Kubernetes that’s runAsNonRoot: true in the security context, which makes the kubelet refuse to start a container whose user resolves to 0.

If the app needs to bind to port 80 or 443, don’t give it root. Bind to 8080 and let the Service or the load balancer map the port, or add only NET_BIND_SERVICE.

Make the root filesystem read-only

An attacker with code execution usually wants to write something: a downloaded tool, a modified config, a web shell, a cron entry. A read-only root filesystem takes that away, and it catches a surprising number of bugs where an app was quietly writing logs or caches into its own image.

Most apps need somewhere to write, usually /tmp or a cache directory. Give them exactly that as a tmpfs or an emptyDir volume and nothing else. With plain Docker:

docker run --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --cap-drop=ALL \
  --security-opt=no-new-privileges:true \
  --user 65532:65532 \
  registry.example.com/payments-api@sha256:<digest>

The noexec on the tmpfs matters. A writable directory that can also execute binaries is half of what an attacker needed from a writable root.

The Kubernetes security context

All of this comes together in the pod spec. This is the block I start every deployment from:

securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  runAsGroup: 65532
  seccompProfile:
    type: RuntimeDefault
containers:
  - name: app
    image: registry.example.com/payments-api@sha256:<digest>
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
    volumeMounts:
      - name: tmp
        mountPath: /tmp
volumes:
  - name: tmp
    emptyDir:
      medium: Memory
      sizeLimit: 64Mi

allowPrivilegeEscalation: false sets the kernel’s no_new_privs flag, so setuid binaries can’t raise privileges. Dropping all capabilities removes things like CAP_NET_RAW, which some runtimes still grant by default and which lets a process craft raw packets. RuntimeDefault seccomp blocks a set of syscalls that normal apps never make.

Kubernetes Pod Security Admission’s restricted profile enforces most of this for you: non-root, no privilege escalation, all capabilities dropped and a seccomp profile. It doesn’t require a read-only root filesystem, so that one you have to enforce with your own policy (Kyverno, Gatekeeper or a ValidatingAdmissionPolicy). Turn restricted on in warn mode for a namespace first, see what breaks, and fix those workloads before switching to enforce.

Pin what you ship, rebuild what you pin

Reference images by digest in deployment manifests, not by tag. A tag is a mutable pointer, and :1.42.0 can be repushed. The digest is a hash of the content and can’t.

Pinning has a cost, though. A pinned base image never gets its security fixes unless you rebuild. Pair digest pinning with an automated job that rebuilds images on a schedule against fresh bases and opens the PR to bump the digest. Pinning without rebuilding is how you end up running a two-year-old OpenSSL with perfect reproducibility.

What this doesn’t fix

None of this stops an exploit in your own code. A read-only, non-root, distroless container with a SQL injection bug still has a SQL injection bug, and the attacker still reaches whatever the app’s database credentials can reach. Container runtime escapes still happen too. CVE-2024-21626, the runc file descriptor leak fixed in runc 1.1.12, allowed escape through a crafted working directory, and the fix was patching runc, not anything in the image.

What hardening does is make the second step expensive. The attacker lands with no shell, no package manager, no writable path that can execute, no root and no extra capabilities. I’ve read enough incident reports to know that “the attacker had a foothold but couldn’t do much with it” is one of the better sentences you can have in one.