Using Stats in Splunk Part 2: Seasonality

by Tim Strawbridge on September 29, 2020

Seasonality, which states predictable variations in data will occur over specific time periods, is one the most important concepts in statistical analysis of time series data in Splunk. For example, it’s expected that you’d see more data logged during business hours, and less during off-hour times. These variations can throw a wrench into typical anomaly detection techniques–as outlined in part 1–if not taken into account.

This article will offer an explanation of seasonality as well as techniques for taking it into account in your searches; we will also provide you with a practical example of how to account for this type of behavior in your anomaly detection searches.

Real world example

To help explain seasonality, we’ll work through a real world example in detecting unexpected dips in indexed data. Many sources of machine data generate more logs during normal business hours (when they’re being actively used), so this is a situation where taking seasonality into account is appropriate.

Before we start, here is the full example that we’ll break down:

Copy to Clipboard

index=_internal source=*license_usage.log* type=”Usage” | bin _time span=15m | stats sum(b) AS byte_sum by idx, _time | xyseries utctime, idx, byte_sum | makecontinuous _time span=15m | fillnull value=0 | untable _time idx byte_sum | eval date_wday=strftime(_time,”%w”) | eval date_hour=strftime(_time,”%H”) | eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend") | eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday==”weekend”)), “no”, “yes”) | stats avg(byte_sum) as average, stdev(byte_sum) as std by idx, weekday_weekend, biz_hours | outputlookup avg_index_bytes_15m.csv

Once you get the stats generated in the lookup–and have a search that populates it every so often–you can implement the following:

Sponsorships Available

Copy to Clipboard

index=_internal host=*.splunkcloud.com source=*license_usage.log* type=”Usage” | bin _time span=15m | eval date_wday=strftime(_time,”%w”) | eval date_hour=strftime(_time,”%H”) | eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend") | eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday==”weekend”)), “no”, “yes”) | stats sum(b) AS byte_sum by idx, weekday_weekend, biz_hours, _time | join type=inner idx,weekday_weekend, biz_hours [| inputlookup avg_index_bytes_15m.csv] | eval devs=(byte_sum – average)/std | eval currentGB=round((byte_sum/1024/1024/1024), 3) | eval averageGB=round((average/1024/1024/1024), 3) | eval stdGB=round((std/1024/1024/1024), 3) | table _time, idx, currentGB, averageGB, devs, stdGB | where devs < -3 AND averageGB > 0.75 | sort 0 – devs | rename idx AS Index, currentGB AS “GB Indexed over Past 15 Minutes”, averageGB AS “Average GB Indexed per 15 Minutes”, stdGB AS “Standard Deviation”, devs AS “Z-Score”

There are two spans that ensure data is accurate. The first takes into account the fact that data may not come in during certain time periods. The following will fill in data during those spans where no logs are generated.

Copy to Clipboard

| xyseries _time, idx, byte_sum | makecontinuous _time span=15m | fillnull value=0 | untable _time idx byte_sum

This packs data into a specific format, makes it continuous, fills in null values with a value, and then unpacks the data. Note that the xyseries command takes exactly three arguments. If you have more than three you’ll need to do something like the following:

Copy to Clipboard

| stats list(field1) as field1, list(field2) as field2, list(field3) as field3 by idx, byte_sum, _time | eval zipped=mvzip(mvzip(field1,field2,”!!!!!field2=”),field3,”!!!!!field3=”) | mvexpand zipped | fields _time, idx, byte_sum, zipped | mvexpand zipped | rex field=zipped “^(?<field1>.*)!!!!!field2=(?<field2>.*)!!!!!field3=(?<field3>.*)$”

Using !!!!! is arbitrary; you only need to have a separator string that won’t appear in your data normally.

The second seasonality piece is the following:

Copy to Clipboard

| eval date_wday=strftime(_time,”%w”) | eval date_hour=strftime(_time,”%H”) | eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend") | eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday==”weekend”)), “no”, “yes”)

This should be more self explanatory. We get the day of the week and the hour from the timestamp, and we evaluate when it occurs. From this, we can calculate statistics based on which category the event fits into.

Conclusion

With these techniques, you can now incorporate seasonality in your searches. It is a powerful technique which can really help you cut down on the noise in your alerts. Be sure to keep an eye out for Part 3 of this series–I’ll be taking a look at some less commonly-used commands and how they may (or may not) be useful in your investigations.

The post Using Stats in Splunk Part 2: Seasonality appeared first on Hurricane Labs.