User-Friendly Fuzzing with Sienna Locomotive

by Eric Hennenfent on April 8, 2019

Fuzzing is a great way to find bugs in software, but many developers don’t use it. We hope to change that today with the release of Sienna Locomotive, a new open-source fuzzer for Windows that emphasizes usability. Sienna Locomotive aims to make fuzzing accessible to developers with limited security expertise. Its user-oriented features make it easy to configure, easy to run, and easy to interpret the results.

Fuzzing is Underused

At Trail of Bits, we use state-of-the-art program analysis tools every day, but even those can’t replace the bug-finding potential of random fuzz testing.

Explicitly testing software for bugs is incredibly important. Many open-source developers once believed in the “many eyes” hypothesis–-that open-source projects would be less susceptible to security issues because anyone could search the code for bugs. The past few years have shown that this only applies to high-value targets, and even then, only in a limited capacity. For example, OpenSSL was running on at least 2/3rds of web servers in 2014, but it still took over two years for researchers to discover the Heartbleed flaw. Open-source software makes up so much of the internet that we can’t afford to count on that kind of luck again. As Google poignantly put it when they announced OSS-Fuzz: “It is important that the open source foundation be stable, secure, and reliable, as cracks and weaknesses impact all who build on it.”

We asked ourselves: why don’t developers fuzz their own software, and what can we do to change that?

Barrier to Entry

One likely reason for fuzzers’ disuse is that they can be relatively difficult to use, especially on Windows. WinAFL (the de-facto standard), in particular, places fairly strict requirements on the functions it can target. If your code doesn’t meet these constraints, WinAFL won’t work correctly. It doesn’t help that most fuzzers are designed for Unix platforms, despite Windows’ 75+% market share. It’s easy to understand why developers often do not bother setting up a fuzzer to test their Windows software.

Sponsorships Available

What to Do?

We believe that security tools only succeed if they’re actually used. Fuzzing techniques that improve code coverage or increase executions per second are the subject of new research at almost every security conference, but these improvements are moot if the code gathers dust on a shelf. We saw this as an opportunity to improve the state of security in a different vein: instead of building a smarter or faster fuzzer, we would build a fuzzer with a lower barrier to entry.

Introducing Sienna Locomotive

To address these problems, we engineered Sienna Locomotive with three features that make fuzzing as painless as possible.

Easy Configuration

We want Sienna Locomotive to be usable for testing a wide variety of software, so we made it easy for developers to tailor the fuzzer to their specific application. New targets can be configured with just the path to the executable and a command line string. Other than setting timeouts for applications that don’t exit on their own, there are very few settings the user needs to configure.

Configuring the Fuzzgoat application shown in the demo video

Powerful Function Targeting

Developers don’t have to be picky about which functions they can target, nor do they have to modify the binary to target a function for fuzzing. Sienna Locomotive runs the target application once and scans for functions that take user input (like ReadFile or fread) and allows the developer to select individual function calls to target. This is especially useful when fuzzing a program that makes incremental reads because it allows the user to fuzz only one specific portion of the file.

A sample function targeting window for a built-in Windows utility

Helpful Crash Triage

Fuzzers produce a myriad of crashes with varying severity. You want to debug the most critical crashes first, but how can you tell where to start? Tools like Breakpad and !exploitable analyze a crashing program to estimate the severity of the crash, which helps developers decide which crashes to debug first. In September, we open-sourced Winchecksec, a component of Sienna Locomotive that helps power our triaging system. Sienna Locomotive combines a reimplementation of the heuristics used by Breakpad and !exploitable, augmented with information from Winchecksec and a custom taint tracer, to estimate the severity of each crash. This helps developers to prioritize which crashes to debug and fix first.

An export directory containing triaged crash information

Will Sienna Locomotive Work for You?

If you describe yourself as more of a Windows developer than a security expert, Sienna Locomotive may be the right tool for you. With relatively minimal effort, it can help you test your code against a much larger space of mutated inputs than you could ever write unit tests for. Depending on how you’ve structured your program, you may also be able to make testing during development more efficient by only fuzzing newly implemented features.

Sienna Locomotive makes some performance tradeoffs for the sake of usability. If you’re more interested in test case throughput than usability, or you’re looking for bugs in Chrome and need to perform thousands of iterations per second, Sienna Locomotive isn’t for you.

Try it Out!

We think that Sienna Locomotive will improve the state of Windows software security by making it easier for developers to test their code via fuzzing. To try out Sienna Locomotive for yourself, download a prebuilt binary from the releases page on our GitHub Repo, or follow the instructions in the readme to build it yourself. If you’d like to help make Sienna Locomotive better, visit our issues page.