The human effort
A considerable army is required to maintain the kernel, and that army has got to be managed. Sort of.
The kernel started with just one software engineer, Linus Torvalds, who’s still in charge of the whole operation. But as he’s said many times before, if he were to be hit by a bus the kernel would still keep being developed. It’s worth checking out an analysis of this statement with the fine example of Y2k humour at www.crummy.com/ writing/segfault.org/Bus.html ,
Torvalds’ day-to-day work is more signing out patches than writing critical functions, and the cadre of subsystem maintainers would be able to manage things if he were to suddenly stop contributing to Linux. So the so-called Bus Factor isn’t really a problem for the kernel so much as it is for smaller projects.
The first version of Linux, 0.01, comprised 10,239 lines and 88 files. Today the kernel 5.13 sprawls some 30 million lines and this release alone included contributions from over 2,000 developers (more than 300 of which were first-time contributors). Over the whole of 2020, some 4,000 people contributed. It’s hard to get a contributor count across the whole history of Linux, since prior to 2002 there was no versioning system in place. But somewhere around 20,000 seems reasonable. At any rate, it’s absurd to think one person could keep track, let alone be in control, of all this development. And indeed Linus is not – he just happens to have the final say in matters of process.
Interestingly, a number of functions remain unchanged since that first release. The file vsprintf.c is the only file from v0.01 not to have been entirely created by Torvalds. As he said in the Usenet announcement, it was “written mostly by yours truly except for the vsprintf routine which was co-written with Lars Wirzenius”. That function is found unchanged in today’s kernel. See for yourself at https://github.com/torvalds/ linux/blob/master/lib/vsprintf.c.
Looking back in time
Lots of functions from the early days are still around, but since others in the same file will change you need to do a bit of work to see what they are. Fortunately people, in particular Dr Daniel German through his cregit work, have tokenised all the many definitions, structures and more, and it turns out that out of the hundreds of millions in today’s kernel, just under 3,000 are from that first release. And many more came from not long after. Who says good code doesn’t last?
In 1996 Linux introduced the MAINTAINERS file, to show who was responsible for what. At first there were only three maintainers: Torvalds, Alan Cox and Jon Naylor. Today, the kernel is divided into several subsystems, including storage, device drivers, USB and Bluetooth, each with their own maintainers (there are about 150 in total). But within each subsystem, another layer of maintainers are responsible for different components. Today’s kernel lists over 1,500
maintainers, so responsibility is shared across many shoulders. Since 2004, the kernel has adopted a standardised Developer Certificate of Origin (DCO) for each commit. This shows who wrote the code and, together with the move to Git in 2005 means that the provenance of contributions can be tracked much more readily. By signing a commit thusly (and it won’t be accepted without this attestation) contributors agree to the Linux Foundation’s DCO, which can be read at https://developercertificate.org.
The decision to mandate DCO signoffs stems from the SCO vs IBM, Red Hat, Novell et al (there’s a lot of them) legal wrangles that began in 2003, with a baseless accusation that code from System V Unix had been co-opted into Linux. Things took a turn when it was discovered SCO didn’t even have rights to most of that System V code, took another turn when Microsoft contributed a cool hundred million dollars into SCO coffers and ended with, ah well technically they haven’t ended yet. Even though the SCO Group filed for bankruptcy in 2007 and no longer exists, its remaining intellectual property having been bought by UnixWare and OpenServer Vendor UnXis (now Xinuos) in 2011, after it lost a second appeal against Novell. In April 2021 Xinuos rekindled the 18-year-old claims against IBM, and rails at its acquisition of Red Hat to boot. The protracted zombie case continues.
A great deal of kernel work is now done by robots. They might not be writing patches yet, but they are reporting bugs. Cast your eye over the 5.12 development statistics report at https://lwn.net/Articles/853039
and you’ll see that the top three bug reporters are all bots. And so are positions 4 and 10 by the way. These “fuzzers” come up with fiendish sequences of kernel calls that may or may not be realistic in real life, but nonetheless manage to crash it. Syzbot (the number two reporter) is built around Syzkaller, Google’s open source kernel fuzzer. Syzkaller is now used to test Mac, Windows and Fuschia kernels too.
You’ve got mail… lots of mail
One thing that hasn’t changed in a long time is that new kernel contributions are still submitted through the Linux Kernel Mailing List. Email might be a relic for many, but if you still use your inbox and want to see it (very) full, sign up to LKML by sending a subscribe email to majordomo@vger.kernel.org. A more sane approach is to just browse the LKML archives at https://lkml.org. Of course, maintainers (or anyone) are unlikely to read every single message, so most people will filter messages to the subsystems of their interest. One of the most disheartening things for first-time contributors is having their post ignored, but this generally means no one has seen it rather than it not being worth responding to. To reduce the noise, a couple of subsystems have their own mailing lists (for example, SCSI, Bluetooth and Media). The kernel comes with a Perl script, /scripts/get_maintainer.pl, that will tell you which mailing lists and maintainers your patch should be submitted to. It also comes with another script, check_format.pl, which will check your patch for various formatting violations and optionally correct them.
Linus wrote Git in 10 days following Linux’s departure from the proprietary BitKeeper VCS (version control system) in 2005. Apparently, he named it after himself, showing his appreciation for pejorative terms of the British Isles. After going to all the effort of inventing (the world’s most popular) VCS, one would be forgiven for thinking that it seems a little silly to be still handling all these patches over email. Well, it turns out using Git this way, and allowing any old maintainer commit access to the main tree, is a bad idea. Linus does maintain his own Git tree, hosted at GitHub (https://github.com/ torvalds/linux) as well as on kernel.org, and that’s what all patches submitted to the mailing list should be based on. If you look in the Pull Requests section of that
GitHub page, you’ll see hundreds of people trying to submit PRs and hundreds of automated responses saying that’s not how we do things here.
The Kernelnewbies site is a great resource for any budding kernel developers. It has a guide to submitting your first patch at https://kernelnewbies.org/ FirstKernelPatch, which covers setting up vim, Mutt, esmtp and Git, and then diving into the testing branch of the staging tree. Things aren’t really expected to work here so you’ll be sure to find something to fix. Even if you don’t have coding experience, if you familiarise yourself with the Kernel coding style at www.kernel.org/doc/ html/v5.13/process/coding-style.html then you can get started fixing indentation, spellings or other trivial things.
As Linux reaches middle age, there’s increasing pressure to find new maintainers. Eventually, barring advances life longevity, the old guard will retire and there’ll be full-time shoes to fill. To those brave souls aspiring to such works, we salute you!
THE POWER OF TEAMWORK
“Today’s kernel lists over 1,500 maintainers, so responsibility is shared across many shoulders.”