Home » Security Bloggers Network » 10 Questions to Ask a Bot-Mitigation Vendor

10 Questions to Ask a Bot-Mitigation Vendor

by David Holmes on June 9, 2019

You figured out that you have a bot problem. Maybe you have a high account takeover (ATO) rate, or someone’s cracking all your gift cards, or scraping your site. You tried to handle it yourself with IP blacklists, geo-fencing, and dreaded CAPTCHAs, but it became an endless battle as the attacker retooled and retooled and you’re sick of it.

So now you’ve decided that you’re going to call in professionals to stop the problem, and get some of your time back. You’ve narrowed it down to a couple or three, and you’re going to get them in and ask them some questions. But what questions? Here are some good ones that can give you an idea if the vendor’s solution is a fit for your environment.

1. How does the vendor handle attacker retooling?

This is your most important question. When a countermeasure is put in place, persistent attackers will retool around it—using another countermeasure and another retool. Victims of credential stuffing say that fighting bot automation by themselves is like playing whack-a-mole. You are paying a service to play this game for you, so ask how they handle it, because attackers always retool.

Attackers always retool. How does the vendor respond?

2. Does the vendor dramatically increase user friction?

CAPTCHAs and 2FA dramatically increase user friction. Human failure rates for the former range from 15% to 50% (depending on the CAPTCHA), and lead to high cart-abandonment and decreased user satisfaction. Honestly, think carefully about vendors who rely on these countermeasures. Your goal should be to keep CAPTCHA off your site, not pay someone to annoy your users.

3. How does the service deal with false positives and false negatives?

A false positive for an anti-automation vendor is when they mark a real human as a bot. A false negative is when they mark a bot as human and let it through (this is by far the most common case, but sometimes the less important one). Bot mitigation will have some of both; be suspicious of any vendor who claims otherwise. But a vendor should be very responsive to the problem of false positives; that is, you should be able to contact them, complain, and have the false positive determination addressed.

Ideally, games of whack-a-mole should be 100% outsourced

4. When an attacker bypasses detection, how does the service adapt?

There will be advanced attackers who manage to bypass detection, becoming a false negative. When it happens, you may not know about it until you see the side effect (fraud, account takeovers, etc.). Then you’ll need to contact your vendor and work with them on how to remediate. How do they handle this process?

5. How does the vendor handle manual fraud (actual human farms)?

If your vendor is particularly adept at keeping out automation (bots), a very, very determined attacker will hire a manual fraud team to input credentials by hand in real browsers. Many services do not detect this (since technically, human farms are not bots) Can their service detect malicious intent from even real humans? Shape can.

How does the service handle manual fraud farms?

6. If one customer gets bypassed, how does the vendor protect that bypass from affecting all other customers?

Ideally, the vendor should have custom detection and mitigation policies for every customer. That way, if an attacker retools enough to get around the countermeasures at one site, they can’t automatically use that config to get into your site. Each customer should be insulated from a retool against a different customer.

7. If an attacker bypasses countermeasures, does the service still have visibility on attacks?

It is very common for a service to be blind after an attacker bypasses defenses. If the vendor mitigates on the data they use to detect, then when an attacker bypasses mitigation, you lose the ability to detect. For example, if they block on the IP, when the attacker bypasses the block (distributes globally) the vendor may lose visibility and doesn’t know how bad you are getting hammered.

An example of a system that is working correctly is when 10,000 logins come through and they all look okay initially because they have behavioral analytics within the proper range for humans. But later it is determined that all 10,000 had identical behaviors, which means the logins were automated. A good vendor will be able to detect this for you, even after the fact.

8. Is there a client-side or browser agent?

If yes, how large is the integration and how expensive is the execution? Does the user or administrator have to install custom endpoint software, or is it automatic? If there is no endpoint presence how does the vendor detect rooted devices on mobile and how does it detect attacks using latest web browsers on residential IPs?

For example, one of our competitors takes pride in having no endpoint presence – not even a browser-agent. A common customer of ours used both their solution and ours simultaneously and found that the competitor missed 95% more automation (ask for details and we can provide them).

9. Does the vendor rely on IP-Blacklisting or IP-Reputation?

Our own research shows that automation attackers re-use an IP address an average of only 2.2 times. Often they are only used once per day or per week! This makes IP-Blacklisting useless. There are over a hundred client signals besides the IP address; a good service will make better use of dozens of those rather than relying on crude IP blacklisting.

10. How quickly can the vendor make a change?

When the attacker retools to get around current countermeasures, how quickly will the vendor retool? Is it hours, or is it days? Does the vendor charge extra if there is a sophisticated persistent attacker?

There are other questions that are table stakes for any SaaS vendor. Things like deployment models (is there a cloud option) and cost model (clean traffic or charge by hour). And, of course, you should compare the service level agreement (SLA) of each vendor. But you were probably going to ask those questions anyway (right?).

Yes, this article is slightly biased, as Shape Security is the premiere automation mitigation service. But consider the hundreds of customers we’ve talked to who chose us; these are the questions they asked, and we hope that they help you, even if you end up choosing a different bot-mitigation vendor.