Why POCs Fail and Why You Must POC Anyway!?

A lot of people in the industry assume that we Gartner analysts walk on water …

… and we do. We do walk on the churning waters of vendor propaganda, misdirection and “messaging.”

However, sometimes when clients ask us a tough question about how well some technology will work in their environment, we tell them “TEST IT FOR YOURSELF.” In other words, do a proof of concept deployment (PoC / POC). This post is my brief lament about the glamors and horrors of today’s security technology POC.

Now, most if not all of our recent papers (on SIEM, SaaS SIEM, UEBA, SOAR, deception, etc) say something like “conduct a 14-30 day POC deployment.” For example, our recent UEBA piece says “although a common POC duration is 30 days, much longer POC deployments are not uncommon [for UEBA].”

Why is that? Here are some ideas:

• For some technologies, such as those that use various non-deterministic approaches (today this is a wide range from EPP to EDR to UEBA), your choice is “POC-based or luck-based” and no other (due to this stuff, mostly). In the 1990s, you can count the signatures of 2 IDS devices, but with today’s “ML-rich” security tech, you just have to POC.
• Make POC as close to production as possible. We did see sad stories of “POC win, production fail” (in particular, for UEBA) where the test relied on assumptions that were just plain incorrect in customer’s own production environment, or the POC was “hand-tweaked” by the field engineers who then left and the tool fell apart.
• Appreciate “PoC gems” (amazing, but likely totally random finds during the POC, like being hacked by the elite APT at the very moment of a new tech test), but don’t rely solely on them to buy. This may never happen again! Understand why they were detected and what else can be – in your environment.
• You may not have enough people or time to do a POC. What is the solution in this case? Do a POC anyway – or waste money on tools that will not work for you. Sorry, but this is the truth.

For additional advice, read the section “Running a UEBA POC” of our “A Comparison of UEBA Technologies and Solutions” (Gartner GTP access required) or an excellent “Use the Gartner Playbook for a Successful SIEM Proof of Concept” (Gartner access required).

*** This is a Security Bloggers Network syndicated blog from Anton Chuvakin authored by Anton Chuvakin. Read the original post at: https://blogs.gartner.com/anton-chuvakin/2018/05/01/why-pocs-fail-and-why-you-must-poc-anyway/