The Evolution of Security Testing

The Evolution of Security Testing

A benchmarking study by the NSA Center for Assured Software found that the average SAST tool covers only 8 out of 13 weakness classes and finds only 22 percent of flaws in each weakness class. Based on these numbers, the average SAST tool is likely to find only 14 percent of the vulnerabilities in an application’s code.

Security needs to be part of the development experience. This has given rise to the application security space. And, like the internet itself, it needs to evolve. AppSec 1.0 was built with waterfall methodologies in mind. These include static analysis software testing and penetration testing and it assumes that security is binary. You are either secure or insecure, there is no grey area. Yet it is in the grey areas, the zero days, where the attackers lurk. It then becomes a question of code coverage – is your application security solution providing protect your organization?

Verification and validation testing is conducted in two different ways:

• Positive Testing, or functional testing, is a testing process where an application is sent a valid set of inputs. The purpose of positive testing is to ensure the application behaves as expected. While this type of testing is typically conducted by QA teams, modern collaborate closely with security or development teams.
• Negative Testing, or non-functional testing, is a testing process where an application is sent an

invalid set of inputs. The purpose of negative testing is to ensure the application remains stable in unexpected use cases. While this type of testing is typically conducted by security teams, modern DevOps shops may collaborate closely with QA or development teams.

Positive testing is easier to conduct. There is a finite number of features and flows introduced per release. Therefore, there is a finite combination of valid inputs to test. Thus, there is a clear definition of what “done” means. Security is, however, a game of cat and mouse. Organizations are always trying to stay one step ahead of the attacker. And when you are trying to stay ahead of something, speed (and agility) is everything. So negative testing is necessary.

Instead of looking at code as static, as either secure or insecure, AppSec 2.0 security testing understands the developer and attacker mindsets by identifying coding

mistakes early and often. Not only does this reduce costs but it results in faster time to market. Previously, with waterfall, developers were only given information as they needed it, when they needed it. The goal was to keep them focused and to only do what they do best. Don’t pull them away if you don’t have to. And, certainly, never come to them at the 11th hour market is demanding a change in the product or that there’s a problem with something they weeks ago. Yet that is what happens in the modern world. Instead, organizations, in order to stay ahead, be innovative, yet they still need to think like an attacker. They need to know offense in order to implement defense. This is where fuzz testing plays a vital role.

What is Fuzz Testing?

Fuzz testing, or fuzzing, is a dynamic application security testing (DAST) technique for negative testing. Fuzzing aims to detect known, unknown, and zero-day vulnerabilities. Fuzzers send malformed inputs to targets. Their objective is to trigger bad behaviors, such as crashes, infinite loops, and/or memory leaks. These anomalous behaviors are vulnerabilities. Fuzzing helps organizations verify that the application works as expected, even in unexpected situations. This is key as ecosystems get complex. It’s not just about people mistreating applications, it’s also about how an application will react if an integrated app misbehaves. In other words, if a system connected to your app acts up, can the app still function? Or will it crash. And if it crashes, does that allow for malicious code to run instead?

At a high level, fuzzing provides predictability. If testing is done continuously during the development cycle, this decreases time to market and should reduce the costs associated with the application over its lifetime. There will be a lower number of in-field issues when properly tested first. Fuzzing also contributes to productivity. It’s a security test solution that protects developer productivity with zero false positives. Zero. The crashes that are reported are indeed reproducible vulnerabilities, allowing developers to address them quickly.

As the issues are shared and fixed, regression testing remembers the previous test crashes and verifies the remediations.

Fuzz testing is a heavy-weight yet versatile DAST solution that is able to conduct multiple types of testing across the SDLC. It runs quietly, continuously, and synchronously in the background as a part of the build process. It’s also proven technology. Google, for example, identifies 80% of bugs with fuzz testing while the other remaining 20% is found through other means (SCA) or in production

Continuous Testing with Advanced Fuzz Testing

There are different types of fuzzers available. Random fuzzers are just that: random. It’s hard to tell how much code coverage is tested. Template takes known good input and mutates it. Again, it’s hard to tell how much code coverage there is. Guided fuzzing provides a roadmap, it’s more targeted. Here, the code coverage increases.  to autonomously map out the usable parts of the

application for testing. Symbolic execution takes binaries and mathematically reasons through various logic and functions, so it can break into new areas of the program for further testing. Advanced fuzz testing is particularly effective for continuous testing because it aims to protect developer productivity at all costs. The key is autonomous test case management. Once Advanced Fuzz Testing identifies how, it breaks through functions, using guided fuzzing to craft test cases to test each branch. It uses the application’s behavioral feedback to inform what test cases to generate next. If it identifies minor anomalous behavior, it will continue to probe by generating alternative variations.

This means better and more thorough testing and coverage. This is a significant breakthrough from conducting coverage-guided fuzzing alone. Guided fuzzers can be powerful with guidance from a technical security expert to help inform the fuzzer how to traverse through the code. However, without manual intervention, it will continue blindly, guessing inputs to break through

functions, which can result in breaking the code or the fuzzer being stuck for a while. It also deduplicates test cases to give developers the minimum set of test cases to thoroughly test the code.

The Value of AFT to the Developer

For individual developers, there’s component testing. As AFT identifies issues, developers are notified that there are issues. AFT has a zero false-positive rate and to prove it, it shares a test case also known as a witness to reproduce the issue. They’re also provided system level

information that will help developers understand the consequence (from a technical standpoint) to the application. AFT runs deduplication. There may be a single defect. running all of them. AFT reduces them to the one test case. The point of this is to reduce testing efficiency.

The Value of AFT to the Development Team For development teams, there’s regression testing. each owning a component of the larger application. Ideally, when the components are assembled, they’ll work together seamlessly. This is hardly the reality. Google cites that 45% of bugs that they identify are through regression testing. AFT compiles the test case from each component for regression testing. This is where the dedup effort at the component level pays off in dividends because the regression testing time will be reduced through efficiency.

Time is of the essence in release cycles, especially in agile software development. Regression testing picks up where it left off, meaning it doesn’t rerun regression tests reduces regression test cycles and ensures you continue to get the most out of your limited 

The Value of Advanced Fuzz Testing

• AFT provides software predictability

• AFT provides better and more thorough testing and coverage

• AFT provides a lower number of in-field issues when properly tested first

• AFT provides regression testing, so you know that something is fixed

• AFT demonstrates an application can withstand unexpected situations

• AFT uncovers more zero days than other AST

The Value of AFT to the Security Team

Fuzzing provides a proactive approach to security testing. It is the negative or non-functional testing. It shows whether or not an application can withstand unexpected situations, and it helps uncover zero days. One way to think about (and justify) Advanced Fuzz Testing is that it is penetration testing in a machine. Like pen testing, Advanced Fuzz Testing thinks box. However, there are benefits to Advanced Fuzz Testing not found with pen testing. Unlike pen testing, Advanced Fuzz Testing is continuous, not just a point in time. It can be done at human) speed. It can be performed at machine scale, and with machine accuracy. This coverage than what a human is capable of doing.

Learn more by downloading our Guide To Automated Continuous Security Testing