Apache Struts research at scale, Part 1: Building 115 versions of Struts

When our research findings from CVE-2018-11776 prompted us to research other vulnerabilities, the first step was building 115 versions of Apache Struts.

Recently, Synopsys Cybersecurity Research Center (CyRC) coordinated with the Apache Software Foundation to publish Apache Struts Security Advisory S2-058. The advisory represents research undertaken in Belfast that focused on 64 vulnerabilities through 115 versions of Struts, identifying roughly 50 affected versions per vulnerability. We wanted to share our experiences in a series of blog posts.

This blog series is for a technical audience. It discusses insights, problems we encountered, and solutions we came up with during the project:

• Part 1: Building a decade’s worth of versions and their nuances
• Part 2: Execution environments
• Part 3: Exploitation
• Part 4: Version validation and why it’s a lot harder than expected
• Part 5: Wrapping up and some insights

We’ll publish Part 2 next month. Subscribe to the blog so you don’t miss it.

Sponsorships Available

Sponsorships Available

Sponsorships Available

Why we researched Apache Struts

Let’s begin with why we conducted this research. During August 2018, we examined a newly released Apache Struts remote code execution vulnerability (CVE-2018-11776 / S2-057). Through creating our own proof-of-concept and testing it against Apache Struts’ past releases, we discovered that the vulnerability affected more versions than were initially reported. We reported these findings in accordance with our responsible disclosure policy. But our discovery also prompted the question, what about all the previous Apache Struts vulnerabilities? We set out to create a system where we could conduct vulnerability research at scale.

Struts 2 is an old project, going back over a decade. It’s never changed build systems and has used Maven since its beginnings. But that doesn’t mean that building a large range of Struts wouldn’t have its challenges. We encountered a variety of issues, which we’ll explore in this post.

Acquiring officially distributed binaries for exploit testing

First we had to gather all the officially distributed binaries of Apache Struts. This seemed simple enough; public Maven repositories are usually indexed and searchable on search.maven.org, which allowed us to discover that Apache Struts 2 binaries are hosted in the repo.maven.apache.org repository. However, something was amiss: The first available version is 2.0.5, but we were expecting 2.0 from the versions we are aware of operating in the wild.

Our customers and potential circumstances we analyze for

Our customers use solutions such as Black Duck, which identifies licensing conflicts and sends alerts when new vulnerabilities are discovered for an open source component they’re using. This puts us in a good place to understand how open source components are used. In the Java world, it’s common for developers to use dependencies as built. However, we’ve identified circumstances where this is not the case.

Sometimes a component requires customizations to fulfill the specific needs of the developer. The developer maintains these customizations internally as a separate fork. Further, rebasing on new releases can cost a lot in time and effort. Some developers backport fixes when vulnerabilities discovered in later versions are known to affect the versions they’re using. Under these circumstances, it’s possible that developers are using Struts as far back as 2.0 and running a maintained fork of it today. They might also adopt an in-development version when it contains a vital feature or bug fix necessary for their software stack.

Because our customers are using older versions of Struts in the above scenarios, we were determined to conduct vulnerability testing on those versions, even though some of the binaries had never been published or were no longer available. It was clear that we needed to produce our own builds of Struts, as no repository or project was maintaining an extensive set of Struts version builds across different build toolchains.

Deciding on the source of truth

Because we’d decided to build the source code, particularly a long range of versions, we had to decide where to acquire the sources from and use that as the source of truth to map version releases against.

Maven is a very effective toolchain in Java. It combines dependency management, repository synchronization, build, source code management (also known as version control), and a test execution tool. In fact, it’s useful even in retrieving sources for dependencies. However, repo.maven.apache.org doesn’t contain source code releases for Apache Struts versions where binaries are not available. Even for the binary releases that exist, not all versions provide a source jar on the repository or source included in the jar file. For example, version 2.0.5 doesn’t contain any sources.

We looked at the official web archives from Apache Struts, which didn’t provide a complete version listing either. However, at least it had sources for more versions than Maven did (such as 2.0.5). Further, when we trial-built versions of Struts to make some headway, a variety of issues in missing files caused the builds to fail on Maven.

We identified the Git repository for Struts as the best choice, since we could identify each release version based on the tags. From here we accurately identified 115 versions of Struts, and we no longer ran into the problem of missing files causing Maven to fail builds.

Newer Maven cannot be used with older runtimes due to compatibility

We wanted to try to build the project against the original Java Development Kits (JDK). The simplest way would have been to run Maven via older versions of Java, but there are some limitations to this.

It seemed simple enough to attempt to build Struts using a variety of combinations of the JDK and Maven and collect the artifacts produced if it compiled successfully. We’d used this method successfully before.

Older runtimes and Maven repositories

Maven is a very extensible platform and splits its own functionality into plugins, such as compilation, deployment, signing JARs, and so on. To use older runtimes, we needed to download Maven plugins from an HTTPS site. HTTPS has seen many changes since it began: deprecating technologies, introducing new ones, and so on. HTTPS security is typically handed off to the runtime to handle, meaning it is runtime dependent on what security is supported.

As a result, Maven was unable to programmatically retrieve its own dependencies. To work around this, we looked at connecting over HTTP. At the time, the unencrypted site was returning errors (although that seems to be resolved now). So we configured a local reverse proxy for Maven that offered an unencrypted HTTP endpoint and forwarded requests to the HTTPS server. Fortunately, we were able to configure it easily using Apache HTTP Server and Maven.

ProxyPass /apache "https://repo.maven.apache.org/maven2"
ProxyPassReverse /apache "https://repo.maven.apache.org/maven2"

<profiles>
  <profile>
    <id>apache</id>
      <pluginRepositories>
        <pluginRepository>
          <id>Apache</id>
          <name>Maven Repository</name>
          <url>http://localhost/apache/</url>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
        </pluginRepository>
      </pluginRepositories>
  </profile>
</profiles>

Older Maven cannot run plugins

The older Maven releases could not use plugins on the repository, as the latest plugin versions had been compiled against newer Java runtimes, and we were using older runtimes to run older Maven releases. Upon deeper inspection of the repository, we identified older versions of the plugins we needed, including binaries that hadn’t been touched in over a decade. We tried manually injecting these into Maven to use.

mvn install:install-file -Dfile=maven-compiler-plugin-2.0.2.jar 
-DgroupId=org.apache.maven.plugins -DartifactId=maven-compiler-plugin 
-Dversion=2.0.2 -Dpackaging=jar

We had limited success and many complications. We had to disable the previously configured repository to prevent Maven from being “too smart” and updating or pulling other plugin versions that would not run. However, we needed other dependencies from this repository for the Apache Struts project, and it would have added a significant amount of effort to do this for 115 versions of Struts.

We considered creating our own “in-house” series of Maven repositories that would contain the contents of repo.maven.apache.org for each Struts release. We looked at various approaches we could use to generate these repositories.

Apache Struts POMs use repositories that do not exist

Older versions of Struts, in particular, use a few repositories that don’t exist today, such as maven2.opensymphony.com and snapshots.maven.codehaus.org. Without these dependencies, we couldn’t build these older versions. Fortunately, many of the stable releases of these repositories have made their way into repo.maven.apache.org. So we configured these as mirrors in Maven’s global settings.xml file.

<mirror>
  <id>Codehaus snapshots</id>
  <mirrorOf>snapshots.maven.codehaus.org</mirrorOf>
  <name>Codehaus.org repo is now in Central.</name>
  <url>http://repo.maven.apache.org/maven2</url>
</mirror>
<mirror>
  <id>opensymphony</id>
  <mirrorOf>opensymphony.com</mirrorOf>
  <name>Open Symphony repo is now in Central.</name>
  <url>http://repo.maven.apache.org/maven2</url>
</mirror>
<mirror>
  <id>opensymphony maven2</id>
  <mirrorOf>maven2.opensymphony.com</mirrorOf>
  <name>Open Symphony repo maven2 is now in Central.</name>
  <url>http://repo.maven.apache.org/maven2</url>
</mirror>

Dependencies that simply don’t exist anymore

Despite these successes, we continued to discover dependencies that were not available online anymore. Examples of missing dependencies:

• opensymphony:xwork:jar:2.0-SNAPSHOT
• org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-10-SNAPSHOT
• com.opensymphony.xwork:xwork:jar:2.1-SNAPSHOT
• velocity-tools:velocity-tools:jar:1.3

We attempted to match indicated versions with the nearest versions publicly available. For example, in the case of velocity-tools, we found it under a different Maven groupId, org.apache.velocity. We also looked to see if the dependency source repositories contained the specific tags/branches matching the missing versions. When this failed, we tried defaulting to the nearest version number.

Even when we used the closest versions, the project would often refuse to build owing to differences in the APIs. Fortunately, we operate one of the largest and oldest open source knowledge bases (Black Duck KnowledgeBase™), built originally to identify license compliance issues and code snippets. We were able to find missing components, retrieve them from our KnowledgeBase, and use them without error.

The repository problem

After spending some time looking at the Maven repository dependency issue, we thought creating separate in-house Maven repositories based on time periods of Maven releases was a fun idea. But it seemed excessive and probably worth spinning off into its own project for future vulnerability research. Instead, we decided to see if we could configure the current Maven release to build projects using old JDKs, rather than running Maven with an older JRE/JDK.

maven-compiler-plugin comes with features to set the source and target features of the Java compiler. This means you can use language features from a specific version of Java but target a different version of the JRE for compatibility. Even better, you can set the path to the specific Java compiler you want to use, which suited our purposes.

We considered not using a different JDK, but on our initial tests, we found that Java 8 and higher removed the annotation processing tool from tools.jar, which breaks the build for a large chunk of Struts versions. To build a solution that would operate longer than just our initial look at Struts at scale, we needed to make the system more versatile.

Unfortunately, you can’t simply define the Java source, target, and compiler on a global scale. You have to modify the project POM file to compile with these settings. The attributes that need to be set in the POM file to source, target, and compile using the JDK 1.5 compiler are described below.

<project>
  <properties>
    <maven.compiler.source>1.5</maven.compiler.source>
    <maven.compiler.target>1.5</maven.compiler.target>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
          <verbose>true</verbose>
          <fork>true</fork>
          <executable>/opt/jdk/1.5/bin/javac</executable>
          <compilerVersion>1.3</compilerVersion>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

We tested this on a simple “hello world” program created as a Maven project and confirmed that compilation to our target environment was successful by looking at the major version.

$ javap -verbose HelloWorldApp.class

Classfile /home/Struts/test/helloworld/HelloWorldApp.class
  Compiled from "HelloWorldApp.java"
class HelloWorldApp
  SourceFile: "HelloWorldApp.java"
  minor version: 0
  major version: 49

POM hierarchy

We initially tried to configure these options in the POM file located in the root of sources (we’ll refer to this as “master” from now on). However, when we ran builds, Maven appeared to completely ignore the configuration and continued to fail with the same failures as before. Maven’s own documentation wasn’t very helpful. When we looked closer at the POM file we were modifying, we found it was not actually handling compilation but passing the buck to other POM files through module directives.

<modules>
  <module>bom</module>
  <module>core</module>
  <module>apps</module>
  <module>plugins</module>
  <module>bundles</module>
</modules>

We found that even the modules had modules, which is awkward when these settings are not inherited. We investigated a solution, and the community consensus was to modify all the POM files, as our use case was considered uncommon. This left us with a lot of POMs to work with.

Master POM
├── apps POM
│   ├── rest-showcase POM
│   └── showcase POM
├── assembly POM
├── bom POM
├── bundles POM
│   ├── admin POM
│   └── demo POM
├── core POM
└── plugins POM
    ├── bean-validation POM
    ├── cdi POM
    ├── config-browser POM
    ├── convention POM
    ├── dwr POM
    ├── embeddedjsp POM
    ├── gxp POM
    ├── jasperreports POM
    ├── javatemplates POM
    ├── jfreechart POM
    ├── json POM
    ├── junit POM
    ├── osgi POM
    ├── oval POM
    ├── pell-multipart POM
    ├── plexus POM
    ├── portlet POM
    ├── portlet-tiles POM
    ├── rest POM
    ├── sitegraph POM
    ├── sitemesh POM
    ├── spring POM
    ├── testng POM
    └── tiles POM

XML parsing POMs

A scan of the repository revealed 5,203 POM files across 115 tagged versions of Apache Struts. Further, some POMs had attributes we needed, some didn’t, and they used a variety of DTDs. A quick search-replace wasn’t going to cut it.

Surprisingly, a lot of XML parsers and generators we tried for Python had a variety of problems. Some of the problems we encountered:

• Some generators didn’t support setting a DTD and would create our elements outside the defined DTD.
• Some parsers would drop seeing the DTD altogether. When we generated an XML from that parse, Maven would refuse to accept it.
• Some generators, when asked to create some attribute in a sub-element, would just duplicate an entire element tree, which Maven’s XML parser seemed to ignore.

We ended up having to build what could be considered an “overengineered” tool that would take in a POM XML file, figure out what the default DTD schema was, explore the DTD to find if it had the attributes/elements we wanted (and create them if they didn’t exist), then change them to the settings we wanted. It turns out that parsing and modifying over a decade’s worth of changing XML files is not that trivial.

Simultaneous PoC compilation

Since Maven POM files can define module dependencies, we created in our build system the option to build proofs-of-concept. Much like the master POM files for the Struts project, all we had to do was create a new POM file in the parent directory that would call it as a module, then call the PoC that we wanted to build against it. For example, for a PoC for the Apache Struts S2-015 vulnerability, we could do something like the following:

<modules>
  <module>struts</module>
  <module>S2-015</module>
</modules>

With the build system we created, we could build Struts and proofs-of-concept against a variety of Java versions and validate whether different JRE and JDK versions have an effect on a given vulnerability.

What we learned

• Some releases of Struts are missing repositories, requiring us to build 115 versions of Struts at scale. We ended up using Git as our source of truth, as the official source code releases skip versions and are incomplete.
• HTTPS does not work well for online repositories if you want to use the old toolchain, owing to its continuous backward-incompatible evolution.
• Online repositories change; they might not exist tomorrow, or they might have different content that breaks your build toolchain.
• Modern JDKs, while capable of targeting older versions of Java, may have problems compiling code that the original development kit could compile without issue.
• Sometimes a little overengineering is required to deal with complicated problems and future-proof against breakages.

Researchers

• Omer Demirok
• Stephen Mort
• Padraig Donnelly
• Ashley Stone