How to Choose Linux Kernel Live Patching Software

In 1991, two unrelated events occurred, each the promise of two very different kinds of freedom: the death of the Cold War and the birth of Linux.

Kernel live patching arrived in 2008 during Linux’s teenage years. Today, with the Linux kernel approaching 30 years old, live patching has matured, ready to ditch its reputation as an optional extra—a “nice to have.”

There are two reasons for this. The first is the predominance of Linux as the platform of choice for cost-effective, versatile web hosting—more than half of all known websites now run on Linux. The second is the recognition that live patching is not just a convenience; it’s also an effective, low-impact way to augment Linux system security.

Ksplice, the first Linux kernel live patching solution, graduated effortlessly to profitable status upon launch, cementing its lead when technology behemoth Oracle bought it in 2011. Today, it’s the most famous of the five organizations offering automatic security patching services for Linux. In alphabetical order, they are:

• Canonical Livepatch Service for Ubuntu.
• KernelCare for Ubuntu, Red Hat Enterprise Linux, Oracle Linux, Amazon Linux and others.
• Kpatch for Red Hat Enterprise Linux.
• Ksplice for Oracle Linux.
• SUSE Linux Enterprise Live Patching for SUSE Linux Enterprise.

Each company promotes their service with the same clichéd keywords, the same stock phrases, recited ad infinitum. I’ll give you the most popular ones and how you’re expected to read them.

• Rebootless: Updates a Linux kernel without rebooting.
• Automated: Checks for and installs kernel security updates without supervision.
• Easy install: Setup and configuration is simple or unnecessary.
• Fully supported: You don’t have to write patches yourself. (Look at Kpatch’s Patch Author Guide to see why this is important.)

Look past these blindingly obvious selling points and you’ll find other features deserving attention, other factors you can use to differentiate one live patching service from another. This article looks at some of them, lightly peppered with validation and edification. But first, a reminder of what live patching is.

What is Linux Kernel Live Patching?

The Linux kernel has more than 20 million lines of code, written mostly (one assumes) by human programmers. As with all software, it has bugs. In this age of heightened cybersecurity awareness, bugs mean vulnerabilities. To address them, Linux vendors try to deliver updates for their kernels as quickly as possible. Linux system administrators also try to respond quickly installing these updates but scheduling the work is difficult. This is because there is no pattern to the discovery of vulnerabilities. Here are some reasons why:

• At any one time, there are many different versions of the Linux kernel in use. Some will need patching; some won’t.
• One patch can fix many kernel problems, or, one problem might need a series of patches.
• Linux kernel maintainers may choose to group low-severity vulnerabilities, releasing a single, combined patch at a later date.
• Linux vendors may prefer to use their own vulnerability-severity grading schemes. So, one vendor patches a particular vulnerability, another doesn’t.

There is one other point, rarely mentioned: It’s a system administrator’s instinctive abhorrence of stopping a machine, the stigma of having to say, “Server down for maintenance.”

Why? Because a kernel update isn’t applied until you reboot. If you don’t reboot—if you put it on ice—you’ll find yourself in hot water, because an unpatched server is an unsafe server. There’s more on this mantra later, but the essence of it goes like this:

When vulnerabilities go public, so do the exploits for it.

Or, to put it another way, you have a vulnerability you didn’t know about but now you do. And so does everyone else.

The savior in this scenario is live patching. This is a way of updating not the entire Linux kernel but just the broken part of it. And it does it without stopping processes or interrupting users.

However, there are limitations. Live patching can’t fix all kinds of kernel bugs. The mind-boggling complexity of kernel code and the technical challenges inherent in modifying code on the fly means it’s only used for fixing critical issues, usually security-related ones.

Despite this, a system with live patching software is safer than one without. Here are some questions worth asking if you’re in the market for a live patching solution:

1. What’s in a patch (and who’s behind it)?

When you subscribe to a live patching service, it is the patches you are paying for more than the software itself. This is understandable when you start to dig into the Linux kernel source code and when you appreciate how a poorly written patch can cripple a Linux system.

I mentioned earlier that the kernel is a huge body of work that seems to be growing in both activity and volume. Take a look at the growth in the number of lines of code in the master branch of the Linux kernel repository. (The counts are as of Jan. 1 for each year. They include only C/C++ files and headers, Python and Perl files.)

With this much code, it isn’t easy writing Linux kernel patches. It takes a deep familiarity with the kernel’s architecture and data structures. It takes experience with the Linux community’s coding conventions and quality standards. And it takes a special kind of talent to invent solutions that don’t alter functionality, performance or stability.

There are two opposing methods live patching vendors use to design and deliver kernel patches. I’ll call them incremental and monolithic.

Incremental patches, like a bubblegum ball, build upon one another, each modifying the behavior of the one before. You, the customer, must keep track of which patches you install and which you don’t, and the order you install them in. Not paying attention to these matters risks the behavior of your system changing in unpredictable ways. This patch-stacking technique lets vendors quickly release small, targeted fixes. But, over time, a system can become unstable to the point where only a full kernel upgrade can restore it.

A monolithic patch is one that includes all earlier patches. In essence, there is only one master patch that replaces the active one instead of adding to it. This approach makes for a more stable platform, yielding significantly longer server uptimes, often measured in the thousands of days.

2. How long before the next patch?

There is an unavoidable delay between a vulnerability report and a fix for it. It takes time to analyze kernel bugs and time to measure their impact. It takes inventiveness and ingenuity to figure out a solution, and a ton of testing to check that it works. Then there’s the necessary check-in and approval of the Linux development process.

Avoidable delays happen when a Linux vendor overrides the community severity assessment with one of its own. It means the vendor can tackle several vulnerabilities with fewer patches, but, on average, a customer waits longer for fixes to specific problems.

Linux administrators who understand the reasons behind the erratic nature of patch releases are no better off than those who don’t. Neither finds solace in knowing that while they wait for patches, their systems are even more vulnerable to an exploit than they were before its discovery.

Here’s why: Members of the cybersecurity research community like to announce a vulnerability along with an illustrative use case. These commonly take the form of a proof-of-concept, a detailed technical description of how to reproduce the fault or exercise the exploit. These descriptions have honest intentions: to assist kernel developers in finding and closing the bug. But, they also save hackers a lot of time and effort, giving them a shortcut to a literal recipe for disaster in their race to weaponize a vulnerability.

3. How bad must a bug be?

Almost all newly uncovered vulnerabilities get a CVE identifier. Later, when security researchers look more closely, a vulnerability gets a severity rating, a measure of its impact.

An important rating scheme is the Common Vulnerability Scoring System (CVSS). This represents a vulnerability as a set of numbers, each an estimate of a characteristic, such as:

• How easy it is to reproduce (exploit).
• How easy it is to remedy.
• The scale of the impact on the availability of servers and services.
• The importance or confidentiality of any exposed data.

The full set comprises many more such measures. An algorithm consolidates them into a base score, a neat single figure expressing the severity of the vulnerability in the range 0 (low) to 10 (high). CVSS version 2 maps ranges of these numbers onto the keywords LOW, MEDIUM and HIGH. The newer CVSS version 3 adds two more, NONE and CRITICAL.

So, not only do the number of vulnerabilities vary by month and year, but they also vary by severity.

Vulnerability rating schemes such as CVSS gives Linux vendors a way to judge how to react to kernel vulnerabilities. For example, they may choose to write patches for vulnerabilities with a base score of 7 and above. This means HIGH in CVSS v2, HIGH or CRITICAL in CVSS v3. A vendor claiming to target critical vulnerabilities may only mean those with severity ratings 9 and 10 if using CVSS v3.

4. Can I unpatch a patch?

Installing patches automatically without any supervision is a scary notion for many system administrators. They know that even carefully tested patches can change a system’s behavior, affecting its performance or functionality in subtle and not easily recognizable ways. When this happens, when a server seems different after patching, the quickest way to deal with it is to uninstall the patch.

Some live patching services have the ability to uninstall patches. This makes it easier to see whether a recent patch update is the cause of changes in system behavior.

5. Can I host my own patch server?

A live patching software agent checks with remote patch servers for available patches. It does this at regular, configurable intervals, usually with the ability to run ad hoc checks. If a patch is available, the agent software will download and install it. But if the patching agent can’t reach the patch server, where the vendor’s patching service stores the patches, live patching won’t happen.

The way around this is to set up an in-house patch server. Such an onsite server feeds patches to all your company’s computers from within your firewall. A secure copy facility downloads patches and then moves them across the firewall after verifying the integrity of patch files. You can strictly manage and audit the process, and keep your company’s compliance board off your back.

There are other benefits, too:

• You have greater control over what patches your servers get and when.
• You can lock large numbers of servers to known patch levels.
• It is easier to separate clusters of servers for development, testing, staging and production.

Live patching needs a local agent and a remote patch server. Control of both is important for enterprise-scale deployments.

Conclusion

Before summing up, here are three bonus questions to ask a prospective live patch solution provider:

• Can I override automatic updating? There are times when you don’t want to automatically update kernels.
• Which platforms does it run on? Although your prevailing server environment may preclude any flexibility when choosing a live patching solution, it’s better if the live patching subscription doesn’t restrict which Linux flavor you run.
• Can I run custom kernels? Not everyone runs the default, vanilla kernels offered by mainstream Linux vendors. Some are specialists who develop and maintain custom kernels for bespoke hardware or performance-optimized environments. They need a live patching service that can take on the responsibility of providing patches for custom kernels.

Being less prominent on the lists of the live patching vendors’ product descriptions doesn’t make the features covered in this article less important. Any of them might make or break your decision to go with a particular vendor. Each of them can directly affect the effectiveness of the solution and the relevance to your environment. With per-server subscription fees ranging from a few tens per month to thousands of U.S. dollars per year, it’s good to be thorough when choosing a live Linux kernel patching solution.