Automating Certificate Generation for Distributed Splunk

Splunk is a distributed computing environment once a certain level of data ingestion is achieved, so having multiple systems communicating with each other is quite common. Even if the amount of data being ingested is small and there is only one Splunk server in the environment, you still have users and administrators connecting via browser to Splunkweb. Plus, you have the communication of your Universal Forwarders with your deployment server and your indexers. Having this communication secured is important, and one of the primary ways this is accomplished in Splunk is through the use of SSL certificates.

The process of managing your certificates is one that can easily become tedious. In this blog post, I will be discussing a solution to make the creation and management of certificates for many systems at once a bit less painful.

Quick note

This article won’t be about why you should use SSL certificates or even a comprehensive guide on how to implement certificates in Splunk. If you’re looking for that guide, I highly recommend jumping over to Adam’s blog post for those details. Instead, I want to focus on how tedious it is to create certificates for many systems at once.

How the need for automation came about

If your organization has a certificate management system that handles creating everything from scratch, this won’t be as useful. Most customers I have experience with, however, have an internal process that involves sending the internal PKI administrator a set of certificate signing requests (CSR), one for each system.

Managing this process isn’t difficult if you have only one server, like I previously mentioned, but when building a new distributed Splunk infrastructure from the ground up, it often requires a number of individual components, such as search heads, indexers, deployment servers, cluster masters, and hopefully also a monitoring console.

If you’re working from a Windows workstation, you won’t necessarily have the tools installed locally to do this, so certificates tend to be created on the Splunk system itself–likely running on Linux.

If you’re trying to satisfy all of the recent requirements in browsers for modern certificates, an important one from a few years ago is that both Chrome and Firefox no longer pay attention to the commonName (CN) of the certificate for hostname matching. Now they use the X.509 version 3 Subject Alternative Name (SAN). The difficulty here is that until OpenSSL 1.1.1–which is not available on any recent RHEL distro, but which does show up in Ubuntu starting with version 18.04–there is no good way to specify the SAN on the command line. As a result, it is common to update the OpenSSL configuration file (openssl.cnf) on each system to contain the correct details.

When you’re talking about dozens of systems, we’ve now come back around to how tedious this can be to do.

My simple solution for quick and accurate CSRs

Eliminating tedium and bringing consistency to any kind of process is a good thing, so I set out to make this easier. I wanted this to work with as many versions of OpenSSL as possible, including the “openssl” command from LibreSSL on macOS. This requirement unfortunately makes using the new “addext” feature in OpenSSL 1.1.1 not possible.

After some Googling and much trial and error, I finally found that when you supply the configuration file to the openssl binary, instead of just giving a filename, you can redirect the output of a subshell as a file descriptor rather than having the OpenSSL binary open the file and read it itself. This allows us to inject extra configuration on the CLI without ever modifying the original OpenSSL configuration file.

Now that we can create keys and CSRs with only the CLI, my thoughts turned quickly toward scripting this to avoid having to write out all of these commands. Additionally, there is no need to create each CSR separately on each system. We are no longer relying on configuration files, so we can create all of the signing requests on one system.

To make this as easy and portable as possible, the script–which you can also find on GitHub–has a few variables at the top to configure things rather than relying on a configuration file or some other separate input.

Using the example hosts in the table above and sticking an echo in front of openssl, we can see how the script turns that table of hosts into the commands:

Notice when echo is used with our subshell redirection, we end up with a reference to the dev filename of the file descriptor, which is file descriptor 63 (/dev/fd/63). OpenSSL reads from this file descriptor rather than creating a new one attached to the configuration file that would normally be passed into the option, e.g. /etc/ssl/openssl.cnf.

Conclusion

Using this method, the process of securing Splunk’s web communication becomes a much less daunting task. If you haven’t taken the time to set up your own certificates, I hope this script is the final push you need to get it completed. Happy Secure Splunking!