First Look: Splunk 9.0 Configuration Change Logging
One of the most requested features in Splunk has been better audit logging for changes. With the introduction of Splunk Enterprise 9.0, a new feature has been introduced for configuration change tracking. Let’s take a look at how this new feature works!
Overview
To implement the change logging feature, Splunk 9.0 introduces a new log file, configuration_change.log, and a new index, _configtracker. While this feature is enabled by default, you may need to update your indexes configuration to ensure that the _configtracker index is stored on the correct volume when you upgrade.
The configuration_change.log file is stored in the default Splunk log directory, $SPLUNK_HOME/var/log/splunk. Indexing for this log is enabled by default in Splunk 9.0.
Let’s explore a few changes and how they’re logged in this file and in Splunk.
Configuration Change in SplunkWeb
Let’s start by making a simple change in SplunkWeb to observe how it is logged. I’ll start by navigating Settings -> Fields -> Field Aliases to make a new field alias. For this example, everything I’m creating is very fabricated, just so it’s easy to identify in the logs.
Review the change log
Once this change is made, it will be logged to the configuration_change.log file:
Since you have Splunk, rather than looking at change on the command line, let’s just search for it!
This search will show you everything in the configuration_change.log file in your environment:
Since we know that the change contained demo_alias, we can add this term to the search:
The search results will show the change details (expand some of the sections to see all of the relevant information):
From this example, we can see the stanza that was modified as well as the new_value and old_value of the configuration in question.
Here’s a video demonstrating this logging in action:
What about command line changes?
Good news! This same configuration logging is also available for changes made to the filesystem (they will be recorded when Splunk is restarted). I can see this being enormously helpful when troubleshooting Splunk issues and trying to identify when a problem occurred and why.
Since we already have the change for our first example, let’s directly edit the configuration file (/opt/splunk/etc/users/tom/launcher/local/props.conf) on the command line:
Upon restarting Splunk, we’ll see that a new entry is logged to configuration_change.log:
Re-running our Splunk search, we’ll also see another event for this updated stanza:
Here’s a video demo showing how this works in practice for filesystem changes:
Conclusion
This configuration change logging is a feature that I’m really looking forward to seeing widely deployed across every Splunk environment. It’s been long overdue for inclusion in the product and will be a great tool for troubleshooting Splunk deployments.
The post First Look: Splunk 9.0 Configuration Change Logging appeared first on Hurricane Labs.
*** This is a Security Bloggers Network syndicated blog from Hurricane Labs authored by Tom Kopchak. Read the original post at: https://hurricanelabs.com/splunk-tutorials/first-look-splunk-9-0-configuration-change-logging/?utm_source=rss&utm_medium=rss&utm_campaign=first-look-splunk-9-0-configuration-change-logging
