Preparing Linux Hosts for Unexpected Threats 

Capsule8 Co-Founder and Chief Scientist, Brandon Edwards, recently joined Adrian Sanabria and Paul Asadoorian on a Security Weekly webcast to discuss how to effectively prepare Linux hosts for unexpected threats. Specifically, what comprises the current Linux threat landscape – the malware, vulnerabilities, and exploits that are currently impacting the industry – and how companies can address them with the right combination of best practices and monitoring/detection solutions. 

Linux is everywhere. It’s a part of almost every enterprise and really snuck up on infrastructure as a whole in the last decade. What’s more, many security professionals don’t realize just how integrated Linux is in their systems. While it may not have captured a significant share of the desktop market, it’s making a big impact in enterprise systems, and that means a whole new suite of security solutions is needed to gain the transparency to protect them. In this recap, we’ll take a closer look at many of these factors, how to address them, and what those conversations ultimately look like. 

The State of Linux in 2021

From hobbyists and homebrew solutions in the ’90s, Linux has become the backbone of most modern enterprise systems. In fact, 96% of the top one million websites are running on Linux-based servers and 70% of all known websites are powered by some Unix-based operating system. Amazon, the largest cloud provider in the world, runs 94% of its EC2 cloud computing platform in Linux. Even Microsoft, the biggest brand name in operating systems, has embraced Linux, allowing users to run Linux within Windows. While Microsoft has attempted to update and release modernized server options with a headless implementation, it didn’t catch on because Linux has so thoroughly cornered that market for lean, server solutions without a GUI. 

But all this development and growth has created a truly massive system. The Linux kernel alone has 27.8 million lines of code, an increase of 1.6 million from just one year ago. Bugs are still being discovered in 2021, and that means, along with the new lines of code added each year, that there are substantial security issues that need to be addressed as proactively as possible. 

Malware in Linux 

As Brandon notes in the discussion, the threat landscape is interesting because many of the threats that are most prevalent are not particularly sophisticated. “People don’t have to be sophisticated because the bar is so low.” Which is true to the nature of these threats in general. Cybercriminals will only ever be as sophisticated as they need to be to succeed in their attacks. For every Mirai botnet that is designed to infect IoT devices by cross-compiling the binary for a dozen CPU architectures, there are many others that are using tools that are nearly twenty years old to infiltrate systems that are not being protected properly.

One of the major issues that has developed and that has accelerated the frequency of malware attacks is that the tools available for developers are the same ones malware developers can leverage to build new software with which to attack. They’re able to live off the land, using what’s already there to get into the system. It’s possible to constrain access and shrink the attack surface with containers to a significant degree, but the result is a significantly more complicated system that relies heavily on infrastructure to run and support containers, and the complexity that comes with that. 

Additional Linux Security Issues 

One of the major issues that the panel discussed on the webcast was the reliance the industry as a whole has on the developers at Debian or Canonical to merge code together to ensure it is secure when new fixes are made available. When running an apt-update  or apt-upgrade, is that code safe and if it is not, how will you know? Brandon notes the chaos that would unfold if “someone slipped something into the APT package system that then fully adds or modifies APT repository source files.” Supply chain risk is one of the major reasons that transparency is so important to effectively monitor for issues. 

Another issue Brandon identified is the risk of a bug or backdoor introduced in a place that is less likely reviewed, or less likely to receive the same level of security-conscious review as other more obvious targets. He cites LibGMP, a math library specialized for computation on large numbers, as an example of a library often imported as one component among many comprising broader cryptographic toolchains. It’s not generally seen as posing the same kind of security risk as a direct modification to the SSH code base, and it’s often the type of code no one wants to look at. “With all that fancy math code, who’s going to spot something?” Similarly, any code or algorithm which is itself derived by computers (such as the Remy TCP congestion implementation) present an enticing option for backdoors, as the meaning or intent of the code is often unclear to human eyes. 

In general, the explosion in the use of Linux and the complications this has introduced to the code base cause it to become less appealing than when it was possible to understand every element and modify it directly. In the past, monitoring Linux would have been as simple as keeping an eye on things like resolv.conf changes or NSS switch changes. Today, it’s more complicated and if monitoring hasn’t been adjusted to match, it can create significant problems. 

“You might ask “What’s the reason for this being here?” and the answer is so that we can have DNSSEC, which then also introduces more code-level vulnerabilities, and so you’ve introduced complexity to how you manage DNS. And the reason you’re doing this is to support new code that you’ve added that includes new attack surface that you don’t understand, not that you fully understood the old attack surface.” Every action can lead to increased risk. It’s important to be aware of the impact these decisions can have. 

Challenges in Monitoring For Security Threats in Linux

One of the fundamental challenges in monitoring Linux is the number of distributions that need to be catered to. How can a security product effectively update and manage every release? 

Brandon speaks to a number of different environments that Capsule8 addresses. “We’ve tried to take the approach of spotting behaviors that can be generalized as much as possible, but there are still distribution-specific elements to consider. Turtles upon more turtles. It’s not even a turtle beneath a turtle. At every level, you uncover a turtle and it’s actually five turtles dressed as one turtle, and the pain for administrators is real, especially when they realize that they don’t even know where to begin taking catalog of this stuff.”

Another issue that came up from a viewer question was the importance of having runtime telemetry on a Linux system. Many people rely on logs, but if logs are being deleted and that behavior cannot be monitored, it’s nearly impossible to know that an issue has developed. At this stage, security is stuck looking for signatures after the fact. It becomes a cat and mouse game and the mouse in this case is very fast. Telemetry is vital because there’s no way to know for sure if even the configuration is correct because of the complexity of the configuration, but if you can observe the behavior, it’s possible to get a sense if there is an issue. 

Complexity in Linux Only Increases Attack Surfaces

The core of Brandon’s discussion is that complexity creates challenges. At a certain stage, between seccomp, capabilities (as in Linux privileges), and AppArmor, there is a significant amount of complexity for the level of control, and resulting security it provides. And with new kernel vulnerabilities appearing seemingly every week, sometimes accompanied by exploits which continue to circumvent kernel security measures, how much advantage does this provide? 

The bottom line is that any additional code, any updates to distributions, those are all increases in attack surface – new mechanisms and dependencies need to be added with great care because they can directly affect how vulnerable your systems are. 

Watch the full presentation with Brandon and Security Weekly to see just how much of an impact the right security can have and the types of threats that are out there and most people are unaware of. Or click here to connect with a member of our team and discuss how Capsule8 provides the monitoring and visibility needed to know when logs are deleted, detect exploits as they surface, identify when security mechanisms are being disabled, and more to keep your Linux environment safe.

*** This is a Security Bloggers Network syndicated blog from Capsule8 authored by Capsule8. Read the original post at: https://capsule8.com/blog/preparing-linux-hosts-for-unexpected-threats/