Network traffic analysis for IR: Statistical analysis

Introduction to statistical analysis

Statistical analysis is one of the three main categories of analysis that can be performed on network traffic data. It provides a much more detailed analysis than simple connection analysis and takes a different approach to identifying potential indicators of compromise than event-based analysis.

Statistical analysis is typically geared toward performing anomaly detection. Based on the wealth of information available to the analysis algorithm, it can make educated guesses about what should be considered “normal” versus what is “abnormal” or “anomalous.” Any deviations from the norm may be an indicator that something is going on, making statistical analysis ideally suited to helping an incident responder determine where their investigative efforts can be focused to maximize their probability of success.

Performing statistical analysis

In order to successfully and rapidly respond to a potential incident, cyberanalysts first need to know where to look for potential indicators of attack. Data science is extremely good at identifying patterns and correlations from large amounts of data.

Statistical analysis uses the tools and techniques of data science. Data science is a very large field, and most incident responders don’t have the background to be a data scientist.

However, even simple statistical analysis techniques can be extremely useful for incident response. Techniques like clustering and stack analysis can be easily performed by anyone and can be extremely helpful in drawing attention to data that may warrant further investigation.

Clustering

Clustering is an application of unsupervised machine learning where the developer does not provide any input to the algorithm to point it toward a certain solution. Instead, the developer provides the desired number of clusters that they believe should exist in the dataset and the algorithm generates what it thinks is the best allocation of data points to clusters.

Several different clustering algorithms (Read more...)