Customise the Linux kernel
Neil Bothwick shows you how to boost your geek cred and build your own Linux kernel.
When Linus Torvalds released the Linux kernel into the wild 25 years ago, it was as source code. To use it, you had to configure and compile it. Things have moved on in the intervening years and distributions (distros) now supply a kernel already configured and compiled for the vast majority of hardware, yet still there are people that compile their own kernels. Why would you do this? There are actually a number of reasons for doing this that we will cover briefly before moving on to how to configure, compile and install a kernel and associated drivers.
Distro kernels are general purpose, they include almost everything that almost everyone needs, which means that you don’t need most of what they come with. They are also built for a generic CPU, which means, by compiling it yourself, you get a tighter kernel that is tailored to your hardware and usage. Distro kernels are generally well tested, but that takes time so you don’t get the latest — for instance, Ubuntu 16.04 came with kernel 4.4.0 and it was already three months old at that point. A further three months later, it’s using a slightly later revision of 4.4.0. The latest stable release from www.kernel.org is 4.6.4. This isn’t a criticism of Ubuntu or any other distro, if you install their software, you expect it to be well tested. However, you may want or need features in a more recent kernel, such as drivers for recent hardware, such as the AMDGPU.
It may also be that your hardware requires a patch to the kernel to be fully supported, in which case, a recompile of the existing kernel may suffice. If you want the maximum performance from your hardware — particularly if it’s low-end or embedded — a custom kernel may make quite a difference. And, of course, there are the geek points to be gained when you tell people that you roll your own kernel.
TO THE SOURCE
So where do we start? That depends on whether you want the latest release or want to work with your distro’s kernel. Most distros patch their kernels, for performance or compatibility, so starting with theirs may be a good idea. However, the patching is probably not as prevalent as it used to be — for example, distros no longer need to patch for their init system as they almost all use Systemd. If you want to start with your distro’s kernel source, install it from your package manager
in the usual way. In Ubuntu and friends, the package you want is called linux-source. After installing it, you should find a tarball of the source code at /usr/src/linux-source-x.y. z. If you want to use the latest source from www.kernel.org, download the tarball. Whichever option you chose, it is now time to open a terminal, switch to root with su or sudo -i , cd to /usr/src and unpack the tarball with: $ tar xf
linux-source-4.6.4.tar.xz . Before we go any further, we must make sure we have a compiler and associated build tools installed. With Ubuntu, this is done by installing the
build-essential package. You also need to install libncurses5-dev to be able to use the kernel configuration program. Now we can cd into the directory we’ve just unpacked and start configuring the kernel. The kernel is configured in a text file called .config in the main source code directory, but don’t think about editing this by hand, it’s well over 4,000 lines long. There’s a program to help you with this, so maximise your terminal window and run $ make
menuconfig . This presents a hierarchy of settings sections; move between them with the cursor up and down keys, descend into sections with Enter and toggle options with the space bar. Toggling on an option category will usually present more options. The left and right keys move between the actions at the bottom, Exit should really be labelled back, as that’s what it does. It only exits the configuration program when pressed at the start screen. Help is self-explanatory. Hopefully, the same description also applies to what you see when you press it, further information on the highlighted option.
Despite previous comments about tailoring a kernel to your hardware, the first objective is to get a kernel that just works. That’s not too difficult as most of the options have sane defaults. Once you have a working kernel, you can decide how much time you want to devote to optimising and tweaking it.
BUILT-IN VS MODULES
In the early days of Linux, the kernel was truly monolithic. It was a single file that included all the drivers, filesystems, network code and other options selected when compiling it. This became unwieldy as the range of supported hardware grew, so modules were introduced. These are extra bits of the kernel that exist as separate files (in /lib/modules/kernel-version) and are loaded into the main kernel when needed. When you press the space bar to toggle an option, some switch between showing a * for ‘on’ and nothing for ‘off’. Others have a third option, ‘M’, which means the option will be built as a module. This is how distro kernels support so much hardware — they have modules for just about everything and the hardware detection magic loads the correct ones.
When building your own kernel, you don’t need to include everything. Not only do they occupy disk space but building everything takes a lot longer than compiling just what you need. How do you decide whether to build something as a module or into the kernel? When you are compiling for a specific machine, a good rule is to build in drivers that you’ll always use but create modules for things that are only used some of the time, such as on a desktop or server system, it makes sense to compile the driver for the Ethernet card into the kernel as you will always need it. On the other hand, a laptop will sometimes use wired networking, sometimes wireless and sometimes neither, so modules make more sense. Similarly, whichever filesystem you use for your root partition and the driver for your hard disk controller (usually the generic AHCI driver) should be compiled in.
FAT filesystems, as used on USB sticks, and the USB mass storage driver itself could be modules. If you don’t build in essential items, such as the root filesystem, you won’t be able to boot your kernel without an initramfs, which we will look at shortly.
The ncurses interface of menuconfig is extremely functional but pretty basic in interface terms. There’s also a nice GUI version, which require Qt. If you’re using KDE, you already have this, otherwise install libqt4-dev.
THE POINTY-CLICKY ALTERNATIVE
Then run $ make xconfig . There’s also a GTK version with
make gconfig , although it doesn’t have all the features of the Qt version. Whichever interface you use, the procedure is the same. Let’s start with some general features. Go to General Setup and enable ‘Kernel .config support’ and ‘Enable access to .config through /proc’. The first stores the current kernel config in the kernel and the second makes it available through
/proc/config.gz. This can be incredibly useful when you’re experimenting with different kernels. Go to ‘Processor type > Features > Processor family’. This will be set to a generic x86_64 CPU by default. Unsetting the 64-bit kernel option will then give you a choice of 32-bit CPUs as well. This is the default when using a 32-bit OS.
Next, you need to make sure the modules for your hardware are configured. This is generally quite easy, thanks to the lspci command. This command lists all the hardware on your motherboard, onboard chips, as well as PCI cards, and with the -k option, it shows the module used by each device. If you don’t already have a working kernel on your target machine, you can run this from a live CD. $ lspci -k 07:00.0 Ethernet controller: Realtek Semiconductor Co...
Subsystem: Gigabyte Technology Co., Ltd Motherboard
Kernel driver in use: r8169
So I need the r8169 option for my network card, but where do I find that? Simple, if you’re using the Qt config program, press Ctrl-F and type in the name of the module or anything else you need to search for. It will show you a list of matches, from where you can enable the one you want. Clicking on the option description shows information about it in the pane below. If you’re using the ncurses config program in a terminal, press ‘/’ to search. This shows a list of search results, and whether each one is enabled or not. Press the number alongside the one you want to go straight to it. From there, you can also view the help text for the option.
Sometimes, an option will show up in the search results but not be available as an option to select. This is because it depends on some other option being set first. The help screen for an option will often have a ‘Depends on:’ line that lists the options it needs and their current setting. Make sure they are all set to ‘y’ or ‘m’ (built-in or module). Usually there are only one or two to consider but it can sometimes be a treasure hunt tracking down all the options you need.
MAKING THE KERNEL
So you’ve been through the configuration and got something you think is what you want, now it’s time to build and install it. There are three commands to do this: $ make all $ make modules_install $ make install You can string them together in a single line like this:
$ make all modules_install install
But it’s best to run each separately while you’re getting the hang of things. If something goes wrong, you will know which stage failed. You shouldn’t normally get errors during compilation unless you’re using a patched kernel and the patches conflict. If you do get a build failure, paste the error message into your favourite web search engine. The first command builds the kernel
and all the modules you asked for, the second installs the modules into
/lib/modules/[kernel-version] and the last one copies the kernel itself, and a copy of its config, to /boot.
If you built everything needed for the first part of booting, like the root filesystem and your disk interface controllers, into the kernel and not as modules, you are almost ready to try it. One final step is needed, to add an entry for your new kernel to the boot menu with one of update-grub or grub
mkconfig , depending on your distro: $ update-grub $ grub-mkconfig -o /boot/ grub/grub.cfg
NEED AN INITRAMFS?
If you look in the /boot directory of just about any Linux distro, you will see at least one vmlinuz-version file, these are the kernels. You will usually see a matching initrd or initramfs file. These are initial ramdisks and are a way to have a kernel boot on all sorts of hardware without building everything into the kernel.
The initial ramdisk is a filesystem that the kernel mounts as soon as it starts up. This filesystem mainly consists of a collection of modules, a few essential programs and a script — usually called init — that the kernel runs. This script runs the magic of detecting your hardware, loading the relevant modules, mounting your root filesystem and then passing control of the boot process to the real filesystem. The initramfs is unmounted when it is no longer needed, freeing up and memory it used.
There are a number of ways to create an initramfs, most distros have their own method, but there’s now a generic solution that makes the whole thing much easier. Dracut leverages the hardware detection abilities of udev to produce an initramfs that will then boot your kernel on any hardware for which it has support. The best part is that creating an initramfs this way is so easy, just run: $ dracut --kver=[kernelversion] .
This creates a suitably named initramfs file in /boot for the kernel you specify (if you don’t specify a version, it’s built for the current running kernel). The run update-grub/
grub-mkconfig and the initramfs will be added to your boot menu. Dracut can do more than this, such as assembling RAID arrays or unlocking encrypted partitions. So-called ‘out of tree’ drivers are kernel modules that are not included with the kernel. The most common example of these are the Nvidia graphics drivers, closely followed by various Wi-Fi drivers, excluded from the kernel because they contain proprietary code.
After compiling a new kernel, you will need to rebuild these drivers. You can wait until after you have rebooted, but that may mean you have no network connection, so make sure you have the driver packages available. You can usually reinstall them before you reboot, but they have to know which kernel to build for. The de facto standard way of doing this is to look for a symlink from /usr/src/linux to your kernel sources and build for that version, so it is a good habit to create this symlink whenever you unpack a new kernel. If you have to do some of these, it may be easier to use a short shell script to handle the various steps in compiling and building a kernel, something like: #!/bin/sh date -Iminutes >localversion make all make modules_install make_install dracut --kver=$(cat include/ config/kernel.release) update-grub run nvidia installer reinstall wireless drivers This takes care of everything from giving each kernel a unique version to updating the bootloader and reinstalling the out of tree drivers.
We have mainly looked at using either the vanilla sources from www.kernel.org or our distro’s sources, but there are some other kernel versions out there. These are usually supplied as patchsets that you apply to the vanilla kernel source. One of the most well known is CK patchset from users.tpg.com.au/ckolivas/kernel but you should concentrate on learning to build a stock kernel before getting adventurous with these. If you want to get really adventurous, you can download the sources for the latest release candidate from www.kernel.org or even get the very latest sources with git from github.com/torvalds/linux.