The Forrester SAST Wave: Not a True Reflection of the Market

Checkmarx leads the SAST market and displays the most significant and impressive growth in the industry. In this blog post, we demonstrate our growth and 2017 accomplishments in light of the latest Forrester Wave, which we feel fails to reflect the SAST market as it is.

 

Checkmarx – A Continuous Success Story:

 

  • Sales in 2017 have more than quadrupled compared to sales in 2014
  • Our Application Security Testing market share grew from 1% in 2014 to 9% in 2017
  • Our SAST market share is nearing 20%
  • We tripled our customer base to 1,500+ customers, with many Fortune 1,000 companies including half of top 10 software companies and banks
  • We grew from a 100+ employee company to a 400+ employee company
  • We are the number one fastest growing cybersecurity company in Israel, five years in a row, according to Deloitte

 

As demonstrated above, Checkmarx cemented its leadership position in the market over the past 3 years and accomplished its best growth to date. We had unparalleled business achievements in the industry, yet the latest Forrester wave undermines that.

 

The Role of an Analyst Is to Reflect the Market’s Reality, Not Paint It.

When analysts neglect to consider clear business and product facts, they provide highly misleading analysis. Such is the case with the Forrester Wave for SAST 2017.

All the facts point at growth of our offering, business, and market footprint.

 

Moreover, the numbers and facts clearly demonstrate our superiority over Forrester’s perceived market leaders. This can also be evaluated by looking at the technical reasons that drove customers to choose us either as a new product or often as replacement.

 

Lack of Market and Product Understanding

The following are a few examples of Forrester’s lack of market understanding. These are a selection of many examples we experienced throughout the wave and escalation process.

 

1) The Forrester Wave Is Based Solely On Online Documentation.

Forrester’s evaluation was based on online documentation and included no customer references or other types of proof. To roughly quote the analyst behind this Wave, “we only accept publicly documented blogs or online documentation materials, we won’t accept any other proof”.

We presented many features during the demo stage which answer Forrester’s product criteria, but Forrester never included these in their final scoring. In hindsight we believe there was no need to invest much thought or energy into the product demo as it seems Forrester disregarded it.

 

2) Problematic Product Evaluation

 

Accuracy

 

In their evaluation of our SAST offering, Forrester asked “how does the product reduce false positives while increasing the ability to find exploitable vulnerabilities?” Grading was based solely on the presence of the term Machine Learning in the online documentation. An engine’s accuracy and ability to reduce false positives does not and cannot depend solely on machine learning capabilities as this would be a very narrow view of the industry and product capabilities.

 

We told the team at Forrester that product accuracy is go-no-go in any POC and along with that provided significant materials showing how Checkmarx addresses the False positive issue not only with machine learning. Checkmarx provides the best accuracy in the market and hundreds of customer POCs have proven that over the years.

 

The OWASP 2017 benchmark makes it clear that Checkmarx’s accuracy comes out on top:

 

Checkmarx CxSAST V8.4.2 scored 54.27% out of the box compared to industry-wide averages, of between 17% and 33%.

With a 97.11% True Positive Rate, we believe that Forrester’s representation of their accuracy findings (Based on product documentation) is somewhat misleading to Wave readers.

 

OWASP is an industry-respected organization focused on providing independent and impartial information about application security and provides a tool that verifies the accuracy of software vulnerability detection solutions.

 

Based on the OWASP benchmark, Checkmarx significantly outperforms the industry standard. Our customers and prospects use the OWASP benchmark as an independent and impartial means to measure and compare the quality of the Checkmarx Application Security solution.

 

A second example under the Wave’s accuracy section is Forrester’s question regarding how the product scores a vulnerability. In order to obtain a score Forrester required severity scoring based on “standards such as CVE”.  This shows a very clear lack of market understanding, as CVE has nothing to do with SAST, SAST looks for CWEs, and not CVEs. At first, we thought it was a typo, but no – it appeared 3 times. To top that, CVE is not a “standard” to begin with.

 

SDLC integration

 

Another example is Forrester’s scoring regarding the Checkmarx software development lifecycle (SDLC) integration:

Forrester say: “Checkmarx CxSAST offers very strong remediation advice and breadth of source code language support along with sound risk reporting. By comparison, however, the product offers weak support of SDLC integrations.

 

Apart from Gartner stating the exact opposite in regards to Checkmarx’s SDLC integration capabilities in their 2017 Magic Quadrant, here is a testimony by Sam Guckenheimer, Product Owner Microsoft VSTS (Visual Studio Team Services):

 

Part of the problem is that most security tools are too slow to work in a Continuous Integration model,” said Guckenheimer. “Checkmarx is probably the tool that’s cracked that first. Ideally, you want to be able to have your code scanned as part of the pull request in the Continuous Integration flow, and that’s just not practical with most tools that exist.”(Link to full article: https://sdtimes.com/necessity-mother-rugged-devops-movement/)

 

3) Overlooking What Customers Say

The market growth we demonstrated could not be achieved with a weak offering. CISOs in our respected client list, some of which are renowned thought leaders, chose us due to our company size, product merit and vision. Forrester never asked for customer testimonials or considered the fact that our 1500+ customers thoroughly researched the market before choosing us over others.

 

We feel this approach fails to reflect objective commercial success. In this modern world where people provide free reviews to share with the public, Forrester could have just looked at Gartner Peer Insights on Application security products. They could have easily drawn a vivid comparison of the leading products and find that Checkmarx is rated the best in overall AST offering, strong first on SAST as a tool, tied for first in SAST as a service and second in integration.

agartner

https://www.gartner.com/reviews/market/application-security-testing/compare/checkmarx-vs-synopsys-vs-veracode-vs-micro-focus?utm_source=gartner&utm_medium=email&utm_campaign=share&utm_content=mail_icon

 

Why We Published This Blog

We decided to share this experience with Forrester’s clients, other vendors and our 1,500+ loyal customers, as well as the many more customers to come, in order to help them see through this problematic report.

We are always open to analysts’ examination but ask that all aspects and public domain proof is accepted in order to truly reflect the market and product positioning.

The following two tabs change content below.

Moshe is SVP Product Strategy & Corporate Development at Checkmarx. He served over 20 years as an executive and professional in various positions in the software industry spanned over international locations, including USA and Europe. Moshe gained vast experience developing products and projects for Fortune 500 customers like International Paper, UBS and Siemens in cooperation with large solution providers like IBM and Accenture.

Latest posts by Moshe Lerner (see all)

Stay Connected

Sign up today & never miss an update from the Checkmarx blog

This is a Security Bloggers Network syndicated blog post authored by Moshe Lerner. Read the original post at: Blog – Checkmarx