Today’s security information and event management (SIEM) solutions are inundated with incoming events and tracking potential threats to network infrastructure. Significant events must be identified and correlated to detect lateral movement and kill chains to signal when an attack has occurred or is in progress.

Given the huge volume of data that must be processed, most SIEM solutions employ “big data” techniques to tackle this challenge. However, using big data to store events in data lakes and process them offline can lead to delays of minutes or hours, giving attackers a key advantage. Is there a way to rethink this software architecture, enhance current techniques and obtain insights fast enough to help interrupt ongoing attacks?

Accelerating SIEM Capablities

The answer may lie in the use of a software technology called in-memory computing, which has evolved rapidly over the last few years. This technology stores fast-changing data as software “objects” in memory and enables extremely fast, scalable, data-parallel computing using clusters of commodity servers (or virtual servers in the cloud). It has the potential to rapidly accelerate the execution of SIEM algorithms in correlating events, detecting malicious attacks and even intervening in time to stop them in their tracks.

Most data analytics platforms that employ in-memory computing on live data just use it to cache incoming messages and events in memory for fast retrieval by analysis algorithms. While this approach can help accelerate existing software architectures by avoiding database lookups, it does not reap the full benefits of in-memory computing for scalable, data-parallel computing in SIEM software.

The unique characteristics of in-memory computing software (namely, its abilities to ingest, store and analyze large volumes of incoming data within milliseconds) create new opportunities for SIEM software. Instead of just storing incoming events, an in-memory computing platform can both correlate them by data source and analyze them as they arrive. By doing this, it gives SIEM software the ability to maintain a real-time assessment of the threat probability for each network entry point or internal choke point (network node) that is sending events to the system for analysis. Instead of requiring security analysts to analyze logged events, they can now view the results of continuous analysis for every data source within a network infrastructure.

A New Approach to Real-Time Event Monitoring Across the Kill Chain

To process events in real-time, the in-memory computing platform can create a software object, called a digital twin, to correspond to every network node that sends events to the system. When a new event arrives, the platform associates the event message with its corresponding object within a millisecond and immediately runs a threat assessment algorithm. This algorithm, which can incorporate machine learning techniques, combines the incoming event, time-series history of previous events and other state information (such as the characteristics of the node) to update its threat assessment. It also provides analysts with an up-to-the-second view of threat probability, considering both recent events and other available information in a fully integrated manner. Its ability to dynamically learn may also reduce false positive alerts.

This new approach to tracking emerging cybersecurity threats can be further extended to alert neighboring nodes in real-time and look for lateral movement across a kill chain. For example, when the digital twin for a network node elevates its threat probability, it can send a message to other digital twins corresponding to key nodes (such as routers on the same subnet). This additional information can assist those nodes’ threat assessment algorithm in looking for suspicious behavior, enabling them to signal their corresponding physical nodes fast enough to isolate a high probability threat.

Sometimes finding the needle in a haystack just requires a new strategy. Because massive network infrastructure generates huge volumes of events, there may not be enough time to correlate and analyze event logs fast enough to stop a rapidly evolving cyberattack. New software techniques, such as in-memory computing, can bring both desperately needed real-time computing power and a fresh perspective to SIEM. With ever more sophisticated attacks on our critical systems, it may be time to add these capabilities to our arsenal.