Optimizing Your Splunk Cloud Scheduler for Enterprise Security

Introduction

When deploying Splunk Enterprise Security, there are several configuration optimizations which can be used to improve the performance of the environment. A notable example is the scheduler configuration, which allows for more scheduled and summarization searches to run simultaneously. The default scheduler settings in Splunk often do not allow for enough resources to be used for these searches. 

How this works with Splunk Enterprise

For on-premise Splunk Enterprise, it is recommended to make the following changes in limits.conf to adjust the behavior of the Splunk scheduler:

This configuration means that 75% of the available search slots are used for scheduled searches, and 100% of the scheduled search slots can be used for summarization searches. 

Without any customization, a search head that meets the minimum number of CPU cores for Splunk Enterprise Security (currently 16 CPU cores) would have 22 available search slots ( <max_searches_per_cpu: 1 > x <number of CPU cores SH: 16> + <base_max_searches: 6>). The default scheduler configuration would then allow for 11 concurrent scheduled searches and 5 concurrent acceleration searches.

The configuration above increases these settings to allow for 16 concurrent scheduled searches to run, all of which can be used for acceleration searches. This increases the number of acceleration searches that can be run simultaneously by more than three times–which can make the difference between these searches skipping and falling behind or not. 

What to do on Splunk Cloud?

While Splunk Cloud does allow for the self-installation of apps, there are a number of configurations that cannot be included in an app in order for it to pass the app vetting process. This includes settings in limits.conf, such as the scheduler stanza noted above. 

In the past, it was difficult to get this sort of change made on a Splunk Cloud Enterprise Security search head, since it required intervention from the support team in order to make a manual configuration change. However, recent versions of Splunk Cloud introduced an option in the GUI to make adjusting this setting easy. 

I first learned about this from a coworker when troubleshooting a scheduler error, so I’m sure the existence of this feature is new information to many more of you (or I’m just living under a rock). 

Changing Scheduler Behavior

First, Navigate to Settings → Server Settings → Search preferences. The screen that appears will have two options for adjusting search concurrency, one for scheduled searches and one for summarization searches.