Is it Time for Some Threat Hunting?

The discovery of the massive SolarWinds supply chain hack has many companies wondering if they were affected. Not using the affected SolarWinds software isn’t a guarantee, as a hack could come via an affected third-party vendor. More generally, many CIOs and CISOs are losing sleep over the larger implications: the thought that they might have been hacked in the past and wouldn’t know it.

To answer that question, some organizations are turning to threat hunting, an information security discipline that searches for signs of a past hack, or current hacks, by looking for “indications of compromise” (IoC) and other evidence. There are telltale files, log entries and IP access markers that could indicate a network or system has been compromised by a particular piece of malware. With the meteoric rise in ransomware infections and the SolarWinds incident, the practice is rising in popularity after long being a high-end specialty mainly used by those companies who had already been compromised. 

But the time to do threat hunting is before a breach becomes fully public, or the bad guys spring their ransomware trap or steal your customer’s data. In a recent study, 53% of breaches were found by outsiders, not internal company staff. This is because not enough companies are willing to ask (and spend resources to answer), “Have we already been hacked?”

This is vital, because hackers often wait long periods of time before launching attacks so they can find additional vulnerable servers and caches of data, and embed themselves further in the network. The average “dwell time” of hackers in a network can be months, or even years, before they launch their final attack or successfully breach data. If you can discover a hack during this dwell time, you have a decent chance of kicking out the hackers and repairing vulnerabilities before real damage can be done. If the large federal agencies hacked by the SolarWinds breach had done more threat hunting, they might have seen signs of the exploitation and remediated the hacks much earlier. As it was, they had to wait for FireEye, a security company, to be hacked and perform an exhaustive investigation before they were notified that they were victims, as well.

Defining Threat Hunting

But what is threat hunting? The name itself is somewhat of a misnomer; it’s not a safari where you stalk active hacker predators roaming through your network. While you may uncover signs of that activity, it is more similar to the subtle tracking of the prey; making the hunters become the hunted. Threat hunters mainly look for those IoCs. These are well-documented for most major forms of malware; in fact, there is already a list of IoCs for the SolarWinds attack. It includes certain files that will exist on your network, as well as permitted access to specific command-and-control servers that operate the back end of the malware operation. Threat hunting also involves combing through firewall logs, IDS/IPS reports, server logs, application logs and other audit sources to look for anomalous activity. The best threat hunting teams will compile these sources into massive data lakes and analyze them programmatically using AI and other machine learning tools to identify patterns and trends human eyes can’t.

Threat hunters are looking for large migrations of certain types of data, or unusual access times or source IPs. By themselves, these might not be problematic, but when put together they can indicate something is wrong. Then, it’s time to go through the systematic elimination of each of the findings as either false positive (which they usually are) or true signs of a hack. Rinse and repeat. Having very granular logs makes it more likely you’ll get results. The more data you have, the most likely you are to see any malicious activity happening on your network. 

There should also be a predictive aspect to threat hunting. Start with the hypothesis that some entity, say, a Russian advanced persistent threat actor, would be interested in a specific asset of yours. Then, you work backward, relying on open source intelligence (OS-INT) sources, published reports and other intel, to see if they’ve tried, or perhaps, even been successful. This can be as simple as combing through your firewall and router logs for the known IP ranges of these actors, or, if you want to get more complex, looking at known techniques and tools used. Obviously, these kinds of expeditions have to be limited in scope and subject, but by dealing with the most likely and largest attackers, you are eliminating the biggest and most likely threats to your network, rather than spending time and resources to deal with the ankle biters and script kiddies. 

The Importance of Threat Hunting

Even if threat hunting uncovers nothing, it is still a valuable exercise to perform regularly. The practice can assure you that you didn’t miss anything, and that your controls and defenses are working as expected. Even finding evidence of a long-ago hack that is no longer active is highly useful. Often, data breaches take years to come to light, when hackers finally try to monetize a stolen data cache or your data shows up on a public repository. Finding it now allows you to take positive action, and get your ducks in a row for the storm that is sure to come. 

Threat hunting is a tedious, technical exercise that takes a lot of time and specific talent. Few organizations have the expertise and knowledge internally, which is why hiring an outside threat hunting firm often makes sense. This also avoids the biases and conflicts of interest that internal teams might have. So whether you do the work in-house or hire out, it may be time to consider doing some threat hunting yourself. You’ll sleep better at night, knowing you are ahead of the breach curve.

Avatar photo

Tony Howlett

Tony Howlett is a published author and speaker on various security, compliance, and technology topics. He serves as President of (ISC)2 Austin Chapter and is an Advisory Board Member of GIAC/SANS. He is a certified AWS Solutions Architect and holds the CISSP, GNSA certifications, and a B.B.A in Management Information Systems. Tony is currently the CISO of SecureLink, a vendor privileged access management company based in Austin.

tony-howlett has 14 posts and counting.See all posts by tony-howlett