Linux Kernel Live Patching on Ubuntu 20.04 LTS

What happened to the promise of live patching Linux kernels? This article looks at its history, its problems, and the cheapest and easiest ways to do it on Ubuntu Focal Fossa (20.04 LTS).

Introduction

‘Live patching’ is the act of updating a program without stopping the system on which it’s running. It was first done with solder and wire, later, with scissors and glue—it’s nothing new. Today, live patching Linux kernels is far less sticky.

In this article, I’ll explain what it is, how it works (in non-technical terms), and where it comes from. I’ll finish off by showing how to automate kernel security updates on Ubuntu 20.04 LTS (Focal Fossa) with Canonical Livepatch Service and KernelCare.

What is Live Patching?

In software, a patch is a small piece of rectifying code. Patching is repairing or improving a small part of a program without disrupting the workings or specifications of the whole. Live (or hot) patching means changing a running program without stopping it.

Imagine you’re trapped in a moving car and you need to fix a light bulb. Not too bad, you might say, if it’s on the inside, a bit trickier outside. Now imagine repairing a camshaft, replacing a piston rod, or sealing a cracked block.

This is akin to what live patching is trying to do on the Linux kernel. It’s trying to repair the parts of something in motion, without changing or breaking it, but most importantly, without stopping it. The kernel is the only part of a Linux system that needs a shutdown and restart to apply an update. When a vendor releases a kernel update, system administrators have no choice but to schedule a reboot of a server.

What’s so bad about rebooting?

A reboot means different things to different people, depending on whether they’re on the system or in charge of it. Many sysadmins regard regular reboots as a sign of good health, like regular bowels. Just as many don’t, resisting any disruption to applications they’re invested in and making money from, applications like these, for example.

  • web servers, with busy, active users across many time-zones
  • online multiplayer games
  • pay-per-view live or recorded video streaming
  • cryptocurrency mining
  • remote, 24/7 video surveillance and recording services

For me, the most relatable reason is the fear that the system won’t be the same afterwards, and that critical (money-making) applications won’t start up. I think it is this, and not visions of rampaging users, that makes many system administrators put off kernel updates, even the most important kind, security updates.

(Here, I’m only talking about updates. Kernel upgrades are something else. Upgrades are completely new kernels. Patches are updates to parts of the kernel, usually bug fixes that can’t wait because they have security or other wide-ranging implications.)

When a Linux vendor releases a kernel patch, it’s usually to fix a security problem. The associated advisory note will say something like, ‘Install at your earliest convenience.’ On the same page will be a version of, ‘If you don’t, you’ll be vulnerable to the exploits we’ve patched that everyone now knows about. Have a nice day.’

Such callously written notes highlight the dilemma driving the advent of live patching: should you keep users ‘sore but safe’, or ‘composed but exposed’? Live patching promises to be the Paradise Island between Rock and Hard Place. Live patching promises to help keep your servers safe and at the latest security levels, without affecting downtime, and without affecting services.

Paradise Island? What’s the catch?

While the source code for live patching software is freely available, the patches are not. The sweet promise of live patching sours when you have to write your own patches. Your blood pressure eases with less administration but climbs again handling complex kernel code.

Although it’s technically possible to make your own patches, it takes a lot of work and a lot of specialized knowledge. And it’s risky–a badly written patch can crash a system. (This message on the kpatch github page reads like something from a steam hammer operator’s manual: ‘WARNING: Use with caution! Kernel crashes, spontaneous reboots, and data loss may occur!’)

Linux vendors know how hard it is to do live patching properly, and have developed profitable service offerings to take advantage of this fact. Each major Linux distribution has a live patching approach that is free to install, but not to use. The patches, without which the whole idea is useless, must be paid for. 

Still, the benefits of live kernel patching are so compelling that these business models thrive in Linux’s predominantly free and open-source software ecosystem. To me, this is a sign that the technology has significance and an important role in the future of Linux-based systems.

How Live Patching Works

Still trapped in that imaginary car, now thundering downhill toward an imaginary pile of (hopefully) empty cardboard boxes, how would you fix the brakes? Build a temporary rolling jack on which to do the work? Lean over on three wheels, wait for one to stop, take it off, repeat until done?

Linux kernel programmers must have used the same kind of thinking in tackling the problem of live patching. I sense the same kind of conceptual apparatus (suspending, swapping, the use of temporary supporting structures) at work in their solutions. My crude brake-change analogy above illustrates two opposing strategies implemented by live patching software vendors.

  • Stop everything and do all the fixes in one go.
  • Wait for individual components to stop, then swap them out. Repeat until done.

This division explains why each vendor has come up with different approaches to solving the same problem. What they do share, however, is the use of the Linux’s loadable kernel module framework. The software that orchestrates and implements the patching process is a Linux kernel module. That means it’s easy to add patching functionality to compatible kernels, and just as easy to remove it.

Before we get carried away, I must mention the caveats.

There is a limit to the scope and scale of software patches you can apply to running systems. For one thing, customized, optimized, or non-standard kernels make it difficult or impossible to live patch a kernel. For another, live patching can’t remap data or memory across patches; it can only alter function definitions.

These shortcomings didn’t stop Ksplice from becoming the first in the field of Linux live patching. It works by comparing old and new source code from which it creates a binary patch. It freezes the system, figures out which functions need changing, and edits their headers. When called, functions divert to the new versions. If the patch is well-written, control proceeds as if nothing has happened.

A seismic event (described in the next section) saw Red Hat’s Kpatch and SUSE’s Kgraft join the scene with their own interpretations on live patching’s core problems.

Kpatch compares old and new source code to make patches. Like Ksplice, it works by redirecting function calls using the kernel ftrace framework to switch changed functions in one go.

Kgraft works differently. It uses two concurrent sets of functions, old and new, with an orchestrator module deciding when to redirect functions depending on where execution has reached. It’s impossible to predict where in a function a program pointer is at any one time, so the transition is gradual, not instantaneous. 

The Origins of Live Patching

How did the idea of fixing software without anyone noticing get into that ever-changing monolith of human effort, the Linux kernel?

Although the concept traces back to the earliest days of programmable computation, for our purposes the trail starts in 2001 when Hewlett Packard patents a way of dynamically updating software to compensate for missing hardware functionality. A year later, Microsoft presents an idea for updating a system (Windows) without interruption.

Neither mentions Linux, but the applications are broad, each describing how to remedy software or hardware problems on a computer without disruption to the processes running on it. (If the idea doesn’t strike you as especially useful, perhaps two words will make you think again: Meltdown and Spectre.)

In 2008, Jeff Arnold announces Ksplice, software to patch Linux kernels without restarting them. It’s the first of its kind for Linux. And we have an unknown, unnamed hacker to thank for it.

To see why, let me take you back to Jeff’s MIT student days. He’s a member of a volunteer group administering servers for the student community. One of the servers needs a security patch. He doesn’t want to kick off his users, so he lets it slide for a few days.

In those few days, before he can update it, somebody hacks the system. The only way to get it back online is by completely reinstalling it from scratch. Let’s assume his colleagues notice. Even supposing they don’t suffer, I imagine Jeff spending the rest of the semester slow-roasting on cinders of student derision.

If the story is apocryphal, then consider it a fable, a reminder that live patching is the child of security, not of convenience. And like all good fables, there’s a happy ending.

The incident inspires Jeff to study how to install Linux kernel patches without delay and without interruption. He makes this problem the topic of his 2006 master’s thesis. The solution comes in the form of software called Ksplice. With a colleague, he writes a paper describing it, titled ‘Ksplice: Automatic Rebootless Kernel Updates.’

Jeff and three of his student colleagues form a company, and in May 2009, they win the MIT $100K Entrepreneurship Competition prize. They launch a commercial service in 2010. Another year later, in the kind of karmic closure every software developer dreams of, Oracle buys Ksplice Inc.

Karma is far from the minds of the users and customers of this amazingly useful new utility. For them, it is the start of a long and exhausting string of nightmares. Oracle assimilates Ksplice completely, making the tool-free only for customers of their own, support-fee-funded versions of Linux.

This seismic shock spurs SUSE and Red Hat into developing their own solutions, neither telling the other of their intentions or solution architectures. They work in isolation from 2011 to 2014, releasing their own distinct approaches within weeks of each other. And in May 2014, CloudLinux, producers of a Linux variant well-known in web hosting spheres, releases a commercial Linux kernel live patching service under the name of KernelCare.

At the same time, a live patching ABI is working its way into the kernel mainline, seeing the light of day in the 4.0 release under the name of Livepatch. In October 2016, Canonical, Ubuntu’s creators, use it as the basis for commercial service under the unashamedly appropriated name of ‘Canonical Livepatch Service’. Canonical release it first for 16.04 LTS, then 14.04 LTS, both requiring setup on the command line. In 18.04 LTS, it has become more visible, easier to use, and better integrated into the Ubuntu desktop experience and now it is available for 20.04 LTS.

How to do it: Automated kernel security updates on Ubuntu 20.04 LTS

Now it’s time to see it in action. There are two live patching solutions for Ubuntu, covered in the next two sections.

Installing Canonical Livepatch Service (CLS)

CLS is free for non-commercial individuals, for up to three machines. It requires registration, but you can use an Ubuntu One account if you have one. If you want to install it on more than three machines, you’ll have to buy an enterprise-style support service agreement.

Before you start

  • Make sure your system is up to date and backed up.
  • Register for a Livepatch or Ubuntu One account.
  • For 20.04 LTS Server, get a key and make a note of it. (This is not needed on the Desktop edition.)

Installing Livepatch on Ubuntu 20.04 LTS Desktop

Ubuntu 20.04 LTS Desktop has a GUI setting to activate the CLS. You can do it either during the post-install setup, or later, by opening the Software & Updates and going to Livepatch tab.

On a fresh Ubuntu installation

After the first reboot of a fresh install, watch out for the second screen of the ‘What’s new in Ubuntu’ dialogue window. This lets you set up Livepatch.

  1. Click ‘Set Up Livepatch…’
  2. At the ‘Ubuntu Single Sign-On Account’ screen, log in with your Livepatch or Ubuntu One account.
  3. If you’re using the text-based installation GUI, under ‘Featured Server Snaps’, select canonical-livepatch.
On an existing Ubuntu installation
  1. Open ‘Software & Updates’ and go to ‘Livepatch’ tab.
  2. Sign in.
  3. After signed in, click Continue when the popup shows up confirming you are logged in.
  4. That's it. The livepatch is set up on your Ubuntu 20.04 Desktop.


Errors In Ubuntu 20.04 with Livepatch

You may encounter the following error when enabling Livepatch on Ubuntu 20.04 Focal Fossa:

Failed to enable Livepatch: cannot enable machine: this machine ID is already enabled with a different key or is non-unique. Either "sudo canonical-livepatch disable" on the other machine, or regenerate a unique /etc/machine-id on this machine with "sudo rm /etc/machine-id /var/lib/dbus/machine-id && sudo systemd-machine-id-setup" server response: Conflicting machine-id

To fix the error, type the following commands in the terminal:

cp /etc/machine-id /etc/machine-id.original 
cp /var/lib/dbus/machine-id /var/lib/dbus/machine-id.original
nano /etc/machine-id (to remove the existing value)
systemd-machine-id-setup
> Initializing machine ID from D-Bus machine ID.
cat /etc/machine-id

Installing Livepatch on Ubuntu 20.04 LTS Server

This is the command-line approach for standard server versions without a windowing system installed. It also works on 16.04 LTS, 14.04 LTS and 18.04 LTS versions.

Open a terminal and enter these two commands:

sudo snap install canonical-livepatch
sudo canonical-livepatch enable <your key>
Tips
  • The second command returns a machine token. It serves no purpose and there’s no need to record it.
  • Keep track of the machines you are registering. If you lose track, or dispose of a machine or VM before deregistering, you are effectively throwing away one of your three free licenses. (There is help here.)
  • To deregister a server, use this command:
sudo canonical-livepatch disable <your key>
  • To check the status of the service, use this command:
sudo canonical-livepatch status --verbose

Installing KernelCare

KernelCare uses a command-line setup; there is no GUI. It supports a wider range of operating systems, including CentOS, RHEL, Oracle Linux, Debian, Ubuntu and others. It also supports the older 2.6.32 kernel line.

In contrast to CLS, it’s fully automated and will check for patch releases every four hours, installing them without supervision if any are available. If you need to override that ability or tie servers to fixed-date patches, there is a command-line utility (kcarectl) that lets you do this and other things.

KernelCare is free for non-profit organizations, or there’s a free, 30-day trial for the rest of us. (If you’re a Ksplice user, you may want to try out the two-step Ksplice-to-KernelCare migration script.)

Before you start

  • Make sure your system is up to date and backed up.
  • Get a free trial key from here.

Installing KernelCare on Ubuntu 20.04 LTS Desktop and Server

Open a terminal and enter these two commands:

sudo wget -qq -O - https://repo.cloudlinux.com/kernelcare/kernelcare_install.sh | bash
sudo /usr/bin/kcarectl --register KEY

These commands also work on 16.04 LTS, 14.04 LTS and 18.04 LTS versions.

Tips

  • To deregister a server, use this command:
sudo kcarectl --unregister
  • To check the status of the service, use this command:
sudo kcarectl --info

Conclusion

Live patching on Linux was much too useful to stay free for long.

With Ubuntu’s 20.04 LTS release, it’s easier to enjoy the advantages of live security patching of Linux kernels, and it’s free for up to three hosts. After that, yearly, per-server fees apply.

If you run many flavors of Linux, but don’t want to learn different products or subscribe to different support contracts, take a look at KernelCare. It’s about five times cheaper when subscribing yearly, and they offer flexible monthly subscriptions.

Share this page:

0 Comment(s)