Gathering LDAP Identity Data with Splunk Cloud

When organizations think about moving their Splunk implementation to the Cloud, there are a couple of pain points that they encounter. For starters: LDAP Authentication. In this article, we’re not going to address that issue, as there aren’t current workarounds for it – other than opening a firewall rule between Splunk Cloud and one of your domain controllers. The second sticking point, that will be addressed in this post, is pulling identity data into the Splunk App for Enterprise Security.

Organizations typically want to set up an automated way to pull identity data from their LDAP infrastructure. In the past, this has involved setting up the previously mentioned firewall rule and then configuring the Splunk Support for Active Directory in the Cloud. Fortunately, I recently came across a way to bypass that need and instead ship identity information up to the Cloud.

Of course, you can always set up a Read Only Domain Controller and that will put some of your Windows Admins at ease. You can also use firewalls to lock down connections only coming from a single IP on a single port towards that Domain Controller. These are great strategies, but usually it’s a lot of extra work if all you need is a list of identities.

You can get the same level of identity information by using something in Splunk called summary indexing and a Splunk Heavy Forwarder in your environment. This would involve setting up Splunk Support for Active Directory locally and eliminating the need for any connections inbound to your domain controllers. All data would be sent out to Splunk Cloud via the same port as the rest of your data. Let’s get started!

Step 1: Create an index in Splunk Cloud

To create the index in Splunk Cloud:

  1. Login to your Splunk Cloud installation
  2. Navigate to Settings > Indexes
  3. Create a New Index titled “summary_ldap”

Step 2: Create an index in your Heavy Forwarder

Next you’ll want to create an index on your Heavy Forwarder:

  1. Login to your Heavy Forwarder Splunk installation
  2. Navigate to Settings > Indexes
  3. Create a New Index titled “summary_ldap”

Step 3: Schedule a search to send LDAP data to Splunk Cloud

You’ll want to schedule a search with the following settings on your Heavy Forwarder

Search Box

| ldapsearch domain=yourdomain search="(&(objectclass=user)(!(objectClass=computer)))" attrs="sAMAccountName,personalTitle,displayName,givenName,sn,mail,telephoneNumber,mobile,manager,department,whenCreated,userAccountControl" |makemv userAccountControl
|search userAccountControl="NORMAL_ACCOUNT"
|eval suffix=""
|eval priority="medium"
|eval category="normal"
|eval watchlist="false"
|eval endDate=""
|rename sAMAccountName as identity, personalTitle as prefix, displayName as nick, givenName as first, sn as last, mail as email, telephoneNumber as phone, mobile as phone2, manager as managedBy, department as bunit, whenCreated as startDate


Select a schedule that is appropriate for how often you’d like to see your identities get updated. The key here is to make sure to select “Summary Indexing” and to choose the “summary_ldap” index you recently created.

Step 4: Create the lookup table file in Splunk Cloud


The base lookup table file can be created by uploading a skeleton file through SplunkWeb. You will want to make sure that the destination app is the SplunkEnterpriseSecuritySuite and that the sharing permissions are set to Global (object should appear in all Apps).

Creating the lookup table file (uploading a skeleton file)

Add New

The header of the CSV should be as follows:


Setting the appropriate permissions


File saved successfully

Lookup Table Files

Step 5: Create the lookup table definition in Splunk Cloud

The lookup table definition will be used in a saved search to generate the lookup table. The destination app for this lookup table definition should again be the SplunkEnterpriseSecuritySuite. The name can be whatever you would like the lookup definition to be called (ad_identities generally works fine for Active Directory). The type should be file-based, and the lookup file should be set to the CSV you uploaded in step 3.

Second link used for configuring lookup definitions


Adding a new lookup definition

Add New

Lookup definition created, permissions set to private

Lookup Definitions

You will want to make sure that the sharing permissions are set to Global on this as well.

Setting permissions


Correct permissions displayed

Display Successful

Step 6: Create the lookup table and verify that it is updated on the filesystem in Splunk Cloud

Run the following search:

index=summary_ldap | table identity prefix nick first last suffix email phone phone2 managedBy priority bunit category watchlist startDate endDate

But add the

| outputlookup <lookup_name>


| outputlookup <lookup_name>

  command to the search. This will output the results to the lookup table specified.

Verify that this is functioning properly by running the following search 

|inputlookup <lookup_name>

 and verifying the number and format of the results.

Step 7: Save the search as a report

This is assuming that the lookup table is being generated properly. Also, this can be scheduled at a later point in time to automatically update the lookup table.

Step 8: Configure Identities in ES: Navigate to App: Enterprise Security -> Configuration -> Identity Management

Create a new identity, being careful to identify the type as “identity” and the source to the lookup definition you created earlier, which will be lookup://<lookup_name>.  This should be enabled by default once it’s created, but verify that this is the case in the web interface.

Identity Management Configuration

Identity Management

Adding a new identities to the identity manager

Identity Manager Settings

Step 9: Create the merged identity file

Before you create the merged identity file, wait approximately 5 minutes for Splunk to automatically detect the change in the identity configuration.

The following search will show you the status:  

index=_internal source=*python_modular_input.log *identit*

Step 10: Schedule the report you created earlier to run on a semi-regular basis

This is being done assuming all is working as expected. Ensure that set the time frame so you don’t get duplicate results. If you scheduled the search on your Heavy Forwarder for once per day, this search should also look only for the past 24 hours and not any more. Also, as a final note, don’t forget to disable the sample identities that are enabled by default in Enterprise Security.

This is a Security Bloggers Network syndicated blog post authored by Ryan O'Connor. Read the original post at: Hurricane Labs