Breaking Down the Product Benefits
This is the second post in the Fuzz Testing ROI Framework series.
The sections below outline the intangible values each solution delivers as cited by customers. Product justifications often focus on qualitative data. However, we find quantitative data to be equally critical for ensuring a full 360 degree examination of a selected technology’s impact across an entire organization.
Vulnerability analysis rarely ends with a single assessment. When defects are uncovered and fixed the same set of security testing must be performed, once again, to validate fixes — also known as regression testing. Ownership over application test suites is a driving purchasing requirement for some organizations, especially for those who are maturing their application security processes.
Manual Penetration Testing. While manual pentesting services offload the work of conducting security in-house, any test suites generated as a part of the service becomes the consulting organization’s proprietary information. Therefore, clients are required to book additional assessments for validating fixes.
Protocol Fuzzing. Because protocol fuzzers license per test suite, users have access to the test suite for regression testing as well. The test suites are pre-built by the vendor, therefore they often encompass common or known attack patterns that assume a one-size-fits-all approach. These test suites are not custom to your application
Bootstrapped Continuous Fuzzing and ForAllSecure Mayhem. Both bootstrapped continuous fuzzing and Mayhem autonomously generate test cases on-the-fly, based on feedback from the SUT. Test cases can be saved for future regression testing, and in Mayhem’s case, done continuously as a part of CI/CD pipeline. Users may observe a noticeable difference in quality between continuous fuzzing and Mayhem’s test suites. The quality of results — defects found as well as test suite — from open source fuzzers is largely dependent on implementation. More often than not, fluency behind the technical workings of fuzzing is required for a fruitful outcome from these open source solutions.
The quality of analysis has thus far been overlooked. Code coverage is a critical factor in results quality. However, because DAST does not require access to source code, these solutions have little understanding of their coverage. Below is a typical graph of new defects found over time:
Most DASTs fail to offer continuous ROI due to the pesticide paradox. Pesticide Paradox states that if the same tests are repeated over and over again, eventually the same test cases will no longer find new bugs. It is a misconception that no reported bugs indicates the software under test is secure. More often than not, it indicates defects have clustered in limited sections of the software, creating hotspots. Below is an pesticide-immune graph of new defects over time:
Protocol Fuzzing. Protocol fuzzers take a systematic approach to delivering test cases — meaning they can be thorough. However, this approach is the greatest victim of the pesticide paradox. They automate testing to the same areas of code, centralizing defects throughout an application. Users cite that the ROI of these solutions decrease over time, forcing them to consider other complementary fuzzers for testing variety.
Manual Penetration Testing and Bootstrapped Continuous Fuzzing. Manual penetration testing and continuous fuzzing offer the testing variety users of protocol fuzzing seek. However, their randomized approach has its benefits and drawbacks. Without the methodical approach of protocol fuzzers, they easily miss subtler defects.
ForAllSecure Mayhem. Mayhem leverages strengths from each of the aforementioned methods. Mayhem unifies the randomized approach of guided fuzzing with the systematic approach of symbolic execution. Symbolic execution ensures thorough analysis, finding deep defects other solutions miss. It continuously identifies and breaks through new areas of software, maximizing its code coverage and preventing hotspots for defects to cluster.
[Want to learn more about the value of code coverage within the fuzzing cycle? Read this blog on, “Beginning Fuzz Cycle Automation: Improving Testing and Fuzz Development with Coverage Analysis”.]
The challenge with negative testing is that it aims to tackle the “infinite space” problem. There are an infinite number of ways software can be misused. While negative testing is vital, it is tedious and boring, requiring extensive time, resources, and cost. Thus, automation is a significant feature that deeply influences the effectiveness and scalability of a solution.
Manual Penetration Testing. A human-in-the-loop approach limits scale. Humans are imperfect beings with emotional, mental, and physical needs. Overwork and boredom lead to inconsistencies, oversight, and demoralization, impacting result quality – another limit to scalability. As organizations mature in their application security program, they opt to discontinue their penetration testing services for a solution they can run in-house.
Bootstrapped Continuous Fuzzing. Standing up a MVP solution is manageable. However, as application security programs mature, organizations require greater automation for scale. Requirements become exponentially complex and difficult to manage. Security engineers of the ClusterFuzz and OSS-Fuzz team have disclosed that even with their padded budgets and world-class experts, it took Google years to achieve full automation. For a long-term solution that grows with your organization and application security program, interviewees recommended a vendor solution.
Protocol Fuzzers. Protocol suite licenses are for consecutive use only. Concurrent use will require the purchase of additional test suite licenses – not to mention hardware and real estate to house the hardware. Protocol fuzzers run against systems, not software. This presents challenges when scaling horizontally. There is no easy or economic way to replicate systems. In order to horizontally scale, they must buy a number of the same system, exponentially adding to costs.
ForAllSecure Mayhem. Google considers “sufficient” fuzzing to be 1 CPU years. Mayhem saves test cases, allowing users to not only continuously run regression testing quickly and effortlessly, but pick up exactly where they left off in their last run. Depending on the number of cores utilized, Mayhem can scale up or down based on an organization’s testing needs. Features such as concurrency also allow organizations to test multiple applications at once.
Fuzzing is most effective when it is integrated as a part of the developer pipeline. However, traditional fuzzers, although they have a quicker time to fuzz, are notorious for their inability to integrate into DevOps pipelines — their largest limitation. As software testing gets pushed out further right of the SDLC, remediation becomes increasingly expensive and time-to-market delayed. In the long run, this can affect an organization’s productivity and overall appsec cost.
Manual Penetration Testing and Protocol Fuzzers. Manual pentesting and protocol fuzzing typically occur in post-development phases, such as QA. These solutions are excellent for right-of-ship testing. However, when they are forced into CI/CD pipelines, they can be costly and even impossible to incorporate in the developer workflow.
Bootstrapped Continuous Fuzzing. Modern open source fuzzers can be integrated as a part of the development lifecycle. However, they weren’t built with enterprise use cases in mind. They require customization and specialization from security experts and academics. Mike Walker, Senior Director of Microsoft Research NExT Special Projects, observes, “typically the future of technology is already here, it’s just unevenly distributed.” He shares that the fuzzing technique perfectly fits this generalization. He recounts how many view fuzzing as “black magic”, leaving many organizations scratching their heads about how they’ll ever be able to bring this advanced technique into their organization. Many of the complexities around bootstrapping continuous fuzzing have unfortunately fed into this very myth.
ForAllSecure Mayhem. Commercialized modern fuzzers, such as Mayhem, seek to make continuous fuzzing widely available and lower the technical barrier to entry. Mayhem places a framework around the entire fuzzing process with features around automation, triaging, scriptability, and integrations.
Want to learn more? Download the Fuzz Testing ROI Framework white paper.