Separating The Wheat From The Chaff

OOne of the most significant challenges organizations face in selecting a Next Generation Firewall (NGFW) technology is wading through all of the vendor-produced information in order to come up with an accurate picture of your choices. It can be even harder to come up with actual data to back up that view. Vendor datasheets, anecdotal stories told by sales teams, and theoretical calculations often fail to reflect how devices will perform in the real world.

The best and most reliable way to make such a comparison is by conducting a performance, functionality, and security effectiveness bake-off between the different vendors you are considering. This allows you to run the same tests using equivalent configurations across multiple vendors to get results you can directly compare. The only problem with that approach is you may not have the time or resources to get it done.

Fortunately, NSS Labs, a leading provider of third-party testing, has recently completed a comprehensive set of tests for NGFW solutions and made those results available. And for the 5th consecutive year, NSS Labs has awarded Fortinet’s FortiGate with a “Recommended” rating. It has also been placed in the coveted upper right quadrant on their Security Value Map (SVM). The accompanying comparative reports for security effectiveness, performance, and TCO provide detailed results on each of the ten vendors that participated in this test.

Security Effectiveness

One of the three main factors a customer looks at when trying to select a Next Generation Firewall is the ability of the device to protect their network from threats. In their Next-Generation Firewall Comparative Report on Security Effectiveness, NSS Labs used three components to calculate a Security Effectiveness rating: Block Rate, Evasions, and Stability and Reliability. The Block Rate is the percentage of attacks that were successfully blocked by each NGFW solution. Evasions refers to the percentage of tricks/techniques designed to allow exploits through that the NGFW device was still able to block. Finally, the Stability and Reliability portion of the results shows if a device was incapacitated in its ability to do its job during any of the tests.

Only two of the ten vendors achieved above 99% security effectiveness, including Fortinet. The next two vendors fell into the 98%-99% range for security effectiveness, and then there is a significant gap before the next two vendors show up in the 90%-98% range. The security effectiveness of the remaining four vendors ranged from 25%-89%. While Check Point and Sophos both tied with the lowest security effectiveness scores of 25%, Check Point also had substantial deficiencies in the Stability and Reliability category. By comparison, Fortinet demonstrated a potent blend of security effectiveness, resistance to evasions, and stability and reliability.

A significant component of the Block Rate test is the Live Exploit section. In this testing segment, NSS used exploits that are currently being detected in the wild, or that have been actively employed on their honeypot systems. This test shows how well the devices being evaluated were able to react to today’s live threats, as opposed to defending against older threats with well-known signatures.

Only two vendors were able to block 100% of those live exploits: Fortinet, and Check Point. Fortinet’s excellent block rate, combined with catching 100% of evasions, and 100% reliability kept Fortinet’s overall security effectiveness in the Top Two. However, Check Point’s problems with evasions and reliability significantly dropped their total Security Effectiveness score.

While having a high block rate is very important, equally important is the ability to overcome evasion tactics. Attackers commonly try to hide their attacks so that security devices can’t tell that something malicious is happening. So one of the new evasions that NSS added this year was to launch attacks on non-standard ports. These are normal attacks any organization might experience, but using applications running on different TCP ports than what might be expected. Detecting this sort of evasive strategy should be table stakes for any NGFW that claims to support application identification.

Interestingly, half of the vendors failed to detect attacks on non-standard points, including well-known NGFW players Palo Alto Networks and Cisco. Because this non-standard port test is new in the 8.0 methodology, we cannot tell if their failure to detect attacks on non-standard ports has always existed or only occurs in recent versions, but it should be a red flag for any organization evaluating solutions. While Check Point managed to detect attacks on non-standard ports, it failed to detect some fundamental RPC fragmentation issues, which caused their overall evasion detection percentage to drop to 25%. Fortinet was one of only four vendors to pass every one of the evasion tests NSS threw at it, including not being tripped up by fundamental table stakes tests merely because they were new.


The performance section of the NSS Next-Generation Firewall Comparative Report contains a lot of data, but the first, and most-often discussed subject is throughput. Each vendor submitted a different sized box, which makes comparisons difficult. How do you compare an $18K appliance from WatchGuard with a $385K device from Cisco? Well, one way is to do that is to compare their datasheet claims against the results of the NSS Labs throughput test results.

As can be seen in these results, most vendors did not manage to meet their own datasheet numbers. In fact, NSS-tested throughput was less than the published claims of seven out of ten vendors. One reason for this is that the numbers posted by most vendors is often based on results from their own customized tests used in their own test labs designed to produce the most favorable results. Naturally, the test each vendor uses for its datasheet numbers is unlikely to be the same as those conducted by NSS Labs.

However, it is reasonable to expect some similarity between published numbers and results based on real-world conditions. NSS Labs throughput results are based on an average of several tests to more accurately reflect the real world traffic that organizations might actually experience on their network. So while the NSS tests may be harder or easier than the tests the vendors use to generate their datasheet numbers, the results give you better insight as to how ‘real world’ each of those vendor’s tests turns out to be.

For the seven vendors who did not meet their datasheet claims (Barracuda, Cisco, Palo Alto Networks, SonicWall, Sophos, Versa Networks, and WatchGuard), we can infer that their datasheet numbers are based on tests that are easier than the ones used by NSS Labs—and therefore also less like what organizations might experience against real-world traffic. And in the case of Cisco (-65%) and Versa (-82%), far less.

For those three vendors who met or exceeded their datasheet claims (Check Point, Forcepoint, and Fortinet), it seems fair to claim that their internal tests are a much closer representation of real-world traffic. However, while the other two vendors just barely managed to exceed their published datasheet numbers, Fortinet blew past our datasheet value of 5.2 Gbps with an NSS-tested throughput of 6.7 Gbps—an astonishing additional 30% above datasheet values.

Performance alone, while a critical consideration, is only part of the equation. Considering the comparative costs of different devices also requires correlating high performance with cost, and this continues to put Fortinet at a considerable advantage as well, which we will see in the TCO section below.

Due to the nature of some enterprise traffic, some protocols are sensitive to high or variable latency. Protocols that have a real-time component, such as streaming VOIP, audio, or video can be adversely affected by this type of slowdown, so it’s always a good idea to keep latency to a minimum. A Next Generation Firewall cannot introduce so much latency that it interferes with the normal operation of those flows.

Fortinet is the clear leader in this area as well, with a latency of 6.84 µs, which is half the latency of the next closest competitor, and more than twenty times better than the least effective solution. This demonstrates one of the real advantages of Fortinet’s SPU custom silicon hardware, as well as its advanced Parallel Path Processing architecture that enables devices to provide maximum performance with minimal overhead.


In the end, a customer’s buying decision is still heavily influenced by cost. That’s why NSS Lab’s Next Generation Firewall Comparative Report on Total Cost of Ownership is so valuable, as it enables customers to make informed decisions about cost and value while still keeping the categories of security effectiveness and performance in mind.

Because each vendor submitted devices in different price and performance categories, NSS attempted to normalize these values by presenting TCO results in terms of the cost per megabit of protected traffic. This approach blends NSS-tested performance and security effectiveness with a three-year TCO score.

The TCO per Protected Mbps results produced a wide range of results. Fortinet leads the pack with a TCO price of $2 per Mbps, which is half the price of the next two vendors on the list—and even then you still get higher security effectiveness (99.3% vs. 90.4%). At the other end of the spectrum, you have Check Point, Cisco, and Sophos, whose high cost per Mbps is the result of their already high device price and the TCO of their hardware combined with their poor security effectiveness ratings. As can be seen, Fortinet’s combination of $2 per protected Mbps and 99.3% security effectiveness represents exceptionally high value for customers.

One of the biggest highlights of the NSS 2018 Next Generation Firewall tests was the testing that was done with SSL inspection enabled. In 2016, NSS Labs found that HTTPS (SSL/TLS-encrypted) traffic was growing 90% year over year and that 50% of enterprise traffic was encrypted. According to Google’s Transparency Report, the percentage of pages loaded over HTTPS in Chrome in the US during December 2017 was nearly 80 percent. By 2019, Gartner estimates that more than 80 percent of enterprise web traffic will be encrypted—and that some encryption will also be used in more than half of new malware campaigns. They also forecast that number to grow to be more than 70 percent in 2020. Clearly, HTTPS inspection in Next Generation Firewalls is and will continue to be increasingly critical to stopping threats and providing value to customers. 

In spite of these trends, end users have been wary about turning on SSL inspection because of the tremendous performance hit that occurs when enabling this feature. Once again, Fortinet led all of the competition in this area. Its baseline inspection performance was 6753 Mbps, and after turning on SSL inspection, its performance only dropped 14.5% to 5773 Mbps, which is only a minor drop in performance in exchange for being able to fully inspect all SSL-encrypted traffic.

When looking at the cost of turning on SSL inspection, Fortinet starts at $1.68 per Mbps and only rises to $1.97 per Mbps when enabling SSL. By comparison, Palo Alto Networks starts with a price per Mbps of $6.66 (over 3x the cost of Fortinet) and then quintuples to $30.90 when enabling SSL—while only providing a maximum throughput of 1699 Mbps. For other vendors such as Check Point and Cisco, the cost of turning on SSL gets even higher. Given the amount of SSL traffic on the network that needs to be inspected, this is an area of comparison that should be impossible to ignore.


The results are in, and Fortinet clearly provides best-in-class security effectiveness, performance, and TCO. Our values and commitment to our customers are also reflected in the fact that we have a datasheet you can believe in, not just because we say so, but because it’s been independently verified by a 3rd party. And not just this year. This is our 5th consecutive year of having a “Recommended” rating for NGFW.

In addition, because Fortinet has a commitment to customers to do what we claim, there are many other Fortinet products tested by NSS that have also received Recommended ratings. True independent third-party testing (and not pay-to-play) enables us to improve our products continually, and we pass those results along to our the customers.

Download the NSS Labs Next Generation Firewall Test Report and Security Value Map here.

All of the NSS NGFW 2018 reports are available for Fortinet, as well as the related comparative reports.

Read more about the Fortinet Security Fabric and the Third Generation of Network Security

Visit Fortinet’s FortiGate Next-Generation Firewall homepage to learn more about this advanced security solution. 

*** This is a Security Bloggers Network syndicated blog from Fortinet All Blogs authored by Fortinet All Blogs. Read the original post at: