Where Mayhem’s Automated Security Testing Fits Best into Your DevOps Pipeline

by Josh Thorngren on January 31, 2023

For developers and DevOps engineers, running tests against applications is a vital part of the development process. But running tests manually can be time-consuming and tedious. That’s where Mayhem comes in.

Mayhem is an application security platform that uses machine learning (ML) techniques like fuzzing and symbolic execution to automatically create and run thousands of tests.

This post explains how Mayhem fits into the development lifecycle, continually analyzing the main branch of your repo and generating regression tests for developers to run against feature branches.

Sponsorships Available

The Advantages of Using an Automated Security Testing Tool Like Mayhem

Using an automated system like Mayhem has many advantages over manually running tests against code. First, it saves time by eliminating the need to manually generate and run multiple regression tests against each feature branch. Second, since all regressions are generated by an AI system rather than manually written by humans, there is less chance for errors or missed bugs to slip through the cracks.

Finally, since this process is automated and constantly running in the background, there is no need for developers to remember or keep track of which test scripts need to be generated or updated—it’s all taken care of by Mayhem!

Testing Your Main Branch

When new code is pushed into the main branch of your repository, Mayhem uses machine learning and techniques like fuzzing to execute your application with thousands of different inputs and conditions, then learns from each result to generate new executions. This process automatically generates thousands of test cases, which are then de-duplicated and consolidated into easily runnable tests.

Mayhem’s integrations into ticketing systems ensure that bugs or vulnerabilities found during this process are opened up for follow-up and fix. At the end of each run, a consolidated set of test cases is provided that can be easily run throughout your development pipeline.

Running Regressions

Each test case is a simple script and requires little to no compute resources. This makes it easy to run the entire suite of tests at any point in your process. Developers can set up a regression-only run of Mayhem against their feature (or local—see below) branch that executes these regressions and identifies issues still open.

By shifting the bulk of testing to the earliest commit possible, Mayhem is making it easier to fix issues faster, before they make it down the pipeline. Instead of manual test case creation, developers can rely on hundreds of Mayhem-generated tests and Mayhem’s detailed reproduction and triage instructions for every single defect found.

End to End Test: Main vs. Feature

Above, we’re talking about Mayhem running on your main branch, with regression runs occurring when new code is pushed into individual feature branches. However, this concept can be extended to circumstances where feature branches are being leveraged by multiple teams, or have longer lifespans.

In this case, instead of running Mayhem’s behavioral testing on the main branch, new merges into a feature branch would kick off an end-to-end Mayhem run, identifying defects and creating test cases. The consolidated Mayhem test suite would then be run as regression tests against a local development branch when new code is committed.

In scenarios like this, many of our customers continue to run Mayhem against their main branch to identify vulnerabilities in production code, but leverage the feature branch runs to ensure that the test suites are performant for individual developers or groups.

Save Time and Reduce Risk With Mayhem

In summary, using an automated security testing platform like Mayhem can make life much easier for developers and DevOps engineers who are responsible for ensuring that their applications remain secure at all times.

Mayhem provides peace of mind that bugs have been identified early on in the development process. By using Mayhem to continually monitor repositories for changes while generating and running regression tests automatically against feature branches before they are merged into the main branch, you can save time while reducing risk.