Instrumenting Splunk with Verodin SIP

SIEM solutions like Splunk are often advertised as being foundational for most organizations as it relates to security incident detection, analysis, and response. Unfortunately, while the SIEM technology is solid and the people managing SIEM are highly skilled, SIEMs virtually always underperform due to a lack of instrumentation, validation, and configuration assurance. 

In fact, Verodin discovered in the 2017 Security Effectiveness Report, that, on average, only a small percentage of SIEM correlations perform as intended. The SIEM may have performed well during the POC and perhaps even over the first couple months. But over time, “Defensive Regression” such as segmentation, tap, span, NTP, and proxy changes, as well as patches, configuration changes, and other asset alterations, negatively impacts the SIEMs value to the organization.

It’s hard to get SIEMs working right and even harder to keep them working. 

Splunk and other SIEMs aren’t the only solutions that suffer from Defensive Regression. The same issues can be attributed to virtually all network, endpoint, email and cloud security controls. A couple of months ago, I wrote a blog about Instrumenting Palo Alto Next-Generation Firewalls with Verodin. Like PAN, I believe Splunk is a solid SIEM, and I know it’s an extremely popular solution which is why I chose to write about it. Based on my experience, there’s untapped value that can be realized by leveraging the Verodin Security Instrumentation Platform (SIP) to help you instrument and improve Splunk.

Verodin SIP prescriptively helps you create searches. It enables you to validate that Splunk is functioning the way you want. And Verodin SIP continuously measures Splunk’s effectiveness to ensure it is operating as desired in the face of Splunk and non-Splunk environmental changes.

This piece will focus on leveraging Verodin SIP to instrument Splunk ES.

Shown below is the Verodin SIP Director. It’s illustrating a rather simple network with three zones represented by white icons:  Desktop, DMZ, and Internet. It also has a Verodin Actor in each zone represented by the black circles.

‍

‍

In the next image, we’re selecting an attack to execute between two of the Verodin Actors. We’ve chosen Vawtrak to keep this use case consistent with some previous pieces we’ve written.

By executing the attack between the Verodin Actors, we can determine if the security controls in-between, such as firewalls and IPS solutions were successfully able to block the attack. Because Verodin Network Actors only attack each other, thus controlling state, you’ll know with 100 percent certainty if the attack was successful or not.

Based on an easy and fast API integration between the Verodin Director and the SIEM (in this case Splunk), we’ll know if the attack generated event(s) that are sent to Splunk. And further, we’ll know if the received event(s) resulted in a Splunk Notable Event that bubbled up an alert, which should notify the security team to respond in the flood of potentially thousands of raw events getting generated every second within Splunk.

‍

‍

Next, we simply specify the Verodin Actors we want to use in Vawtrak attack. As shown below, the source is the Desktop Actor and the destination is the Internet Actor. 

‍

‍

Now we see the results of the attack below. Note that the Vawtrak attack was not blocked. Also, note that nine events were created. In this case, the nine events represent events that made it to Splunk from the reporting security controls that detected the Vawtrak attack.

‍

‍

Below, we can drill into those nine events to see precisely what showed up in Splunk. On the top, we see the metadata generated by Splunk and below that, we see the raw log data. Note that nothing reads “Vawtrak.” While there were events, there were no notable events. This means that it is unlikely that an analyst would notice the events derived from the Vawtrak attack. And even if they did, there is nothing that points to it being a Vawtrak attack.

This gap that exists between what is actually happening in your environment, and what your SIEM reports, can be incredibly frustrating. It’s important to note that it’s not all on Splunk because Splunk is dependent on the information that it receives. Unlike IPS signatures, it’s challenging to share correlation rules across organizations because based on your network architecture, your security control configurations, etc. Vawtrak on your network will look different than someone else’s.

For this reason, it is so critical to validate your actual production security against real attack behaviors. By doing this, you can measure, manage and improve your security effectiveness. You are working with specifics by leveraging Verodin SIP, not general best practices.

Let’s now switch from the Verodin SIP Director to Splunk ES below and build a Correlation Search that will generate a Notable Event within Splunk. The great part about creating these searches after validating with Verodin SIP is that now we have the exact, prescriptive details for the Splunk Correlation Search from Verodin SIP.

The messages displayed above show exactly how Vawtrak reacts in your network, based on all your settings, and further, how it is then reported and interpreted by Splunk. This prescriptive capability saves a ton of time and effort when building out Correlation Searches. Anyone that works with any SIEM knows what a nightmare validating rules and searches can be. Removing the guesswork yields more exacting results, more quickly.

‍

‍

In Splunk below, you can see the details that provided for you by Verodin SIP now represented in the Splunk “Search” inside the red box. This one thing, this prescriptive detail that is copied from Verodin SIP and pasted into Splunk to build the correlation search, is a game changer for SIEM administrators. It means no more guesswork. This search is exactly what Vawtrak looks like on your network as reported to Splunk by your security tools. Period.

‍

‍

After you update Splunk, go back to the Verodin SIP Director to repeat the attack. As you see below, the attack still isn’t blocked. This is to be expected because we didn’t make any incident prevention changes on firewalls, IPS, etc. While instrumenting firewalls, DLPs, WAFs, IPS, endpoint security controls, email security controls, proxies, etc. is entirely within the scope of what Verodin SIP does, it’s outside the scope of this exercise. However, check out the blog on Palo Alto Networks Next-Generation Firewalls for more details on how Verodin SIP can help instrument other security tools like firewalls.

Because we only adjusted our incident detection capabilities within Splunk, we expect to see the events change, and they do. Instead of seeing nine events like we saw the first time, we are now seeing 10 events.

‍

‍

Drilling into these events as we did previously, we see something different. We see a Notable Event was generated. This is something a SIEM administrator would take note of or perhaps fire off an incident response ticket.

That Notable Event is called “Malware Download” which is what we named it when defining the Search Name in Splunk. Further, we can see the exact events that make up that Notable Event directly below it.

‍

• Using Verodin SIP to provide instrumentation, validation and configuration assurance for Splunk, we’ve successfully validated that Splunk was not creating a Notable Event for Vawtrak.
• We then used the prescriptive output from Verodin SIP to easily build the Splunk Correlation Search.
• We validated that our configuration within Splunk was implemented correctly by re-running the Vawtrak attack and analyzing the results.
• Once this process is complete, this Verodin SIP instrumentation test can be executed automatically and continuously, with potentially thousands of other tests, to ensure that as Splunk and other infrastructure variables change, this Splunk Correlation Search continues to operate as desired.
• If it stops working, you’ll be notified. Verodin’s Continuous Monitoring capability empowers organizations to manage by exception and get notified when solutions like Splunk stop correlating, firewalls stop blocking, endpoints stop preventing, etc.

There are many solid security solutions like Splunk in the industry. Having a Security Instrumentation platform like Verodin SIP ensures that organizations get the most out of these solutions, retire the ones that no longer provide value, and invest in the right ones with evidence-based evaluations. Verodin SIP delivers the automated and continuous capabilities to measures, manage, and improve so you can realize full value, more quickly, and at a lower cost. Thanks for following along.