On the DVD
Get your Repair & Rescue Toolkit on the DVD!
Serendipity comes in many forms, and we were putting this feature together it came in the form of a Microsoft security bulletin. These don’t usually contain good news and this one was no exception. A flaw in the Windows print spooler service was discovered that could lead to local privilege escalation and, worse, remote code execution. The bad news continued: details of the flaw (dubbed PrintNightmare) were accidentally publicised (on Microsoft’s own GitHub ironically) leading to widespread exploitation.
Okay, so you might be reading this thinking, “I don’t care. I use Linux”, and that’s reasonable. We do too. But we also dual-boot a lot of Windows machines, and much as it pains us we like to keep those machines up to date. We’d wager that a fair few readers are in the same boat, and we’d imagine that most of them have been in the situation where, post-Micorsoft sticking plasters being applied, one’s convenient boot menu is gone and the machine goes straight to Windows.
So, we felt justified in starting this feature about repairing Linux with our own experiences post Print Nightmare, because we lost our boot menu too. Actually we lost about half an hour or so, because we hadn’t booted into Windows for ages and there were all manner of other (nightmarish) updates to apply. So you can understand our disappointment when after a reboot our nice Linux Mint-themed GRUB screen didn’t appear and we were left scratching our heads.
Recovering the bootloader
The bootloader is one of the first and most common places errors can strike. If your machine can’t find a bootloader (or EFI partition), then it’ll act as if there are no operating systems. Maybe this can be fixed in the BIOS or UEFI) settings, or perhaps data was corrupted by a power failure or hard system crash (which should never happen on Linux).
Booting a rescue distro is your best bet here, so it’s just as well there’s a whole bunch of them on our DVD. Technically any distro (be it another one you have installed or one that’s laying around on a USB stick or stray Linux Format DVD) can be a rescue distro, as long as it boots (and rescues you or your Linux). But sometimes you need specialist tools.
SystemRescue is a utility-packed toolkit for fixing all kinds of system woes. It’s also based on Arch Linux 32 which we like BTW. It boots to the terminal, which is handy if your graphics card has packed in, but you can launch (or try to launch) a friendly GUI by typing startx . We used it to restore our GRUB menu and if you find yourself in a similar predicament then it can probably do the same for you. Key to this operation is the
grub-install command, which writes GRUB to the boot sector or makes an appropriate
EFI entry (although you’ll probably want to download the 64-bit version in this case).
We can’t just point the installer to our disk and hope for the best (you can see in the screenshot that it results in a cryptic error) because GRUB needs to write configuration inside the filesystem, too. So the first step is to mount the partition where GRUB was previously installed. You might have separate boot, home and (several) Windows partitions, or you might just have a single partition with your favourite distro. At any rate, don’t identify it incorrectly in the subsequent commands because that could make matters worse. We’ll use the fictitious /dev/sdx1 here:
# mount /dev/sdx1 /mnt # grub-install /dev/sdx
We’ve used the # prompt to denote that these commands are run as root, because SystemRescue uses only the root account. If you use a different distro
you might need to use sudo . At any rate, after performing this little incantation you will hopefully see Installation finished. No error reported.
Hopefully no news will be good news, so when you reboot you’ll find everything is once again hunky dory.
The boot process might get a little further, as far as a boot menu, but then dump you at the GRUB (or systemd-boot) rescue prompt when you try and boot Linux. This can happen after a Windows Update, and might also come with an error: unknown filesystem
message. The rescue prompt appears whenever GRUB is unable to find its files on the disk. This could be because Windows moved some partitions around, or because a Linux install went wrong.
On classic BIOS systems, the early stage of GRUB gets installed to the Master Boot Record of a disk, right before the partition table. This stage 1 image tells the system where, physically, to find the stage 2 image on the disk. There’s sometimes a stage 1.5 too, but let’s not worry about it. On UEFI systems, an EFI stub is used to hand control over to the bootloader on disk. In both cases the handover can be destabilised by files or partitions moving around on disk.
One thing to try at the rescue prompt is to manually tell GRUB where its bits are, and see if that breathes life back into your beloved distro. You might be familiar with how Linux labels devices as filesystem nodes, for example /dev/hda (old disks), /dev/sda (newer disks) and /dev/nvme0n1p1 (partitions on shiny NVMe controllers). If not, it doesn’t matter, because GRUB has its own drive indexing system. Drives are named (hd0) then (hd1) and so on. If a drive has an old MBR-style partition table, then the partitions are named (hd0,msdos1) and so on. Newer-style GPT partitions are denoted (hd0,gpt1) etc. In both cases note the inconsistency: the first device is number 0 but the first partition is number 1.
At the rescue prompt, enter ls to see the list of devices. Now use trial and error to find which one contains your Linux partition. Auto completion works at the rescue prompt in the same way as your favourite shell, so use the Tab key to save yourself some keystrokes as you work your way through all the ls (hd0,msdos1)/ and so on. When you’ve found the magic numbers, substitute them in the following: grub rescue> set prefix=(hd0,msdosx)/boot/grub grub rescue> insmod normal grub rescue> normal
The last command tells GRUB to boot as it would normally, and hopefully this time you don’t end up summarily dumped back at the rescue prompt. If that happens, don’t lose heart because there are plenty of other things to try. If it boots as normal, then the change still needs to be made permanent, which should just be a matter of running sudo update-grub
once you’re logged in. It’s never a good idea to try and edit GRUB’s configuration file (/boot/grub/grub.cfg)
by hand – there’s even a warning at the top of the file. The previous command will generate a new, hopefully working, one and you won’t have to worry about its confusing and sprawling syntactics. If the new configuration file doesn’t work then you can re-install GRUB as described above.
Bring a hard drive back to life
We will see in the DVD pages that Zorin OS might need some extra help installing GRUB. And there we’ll use an Inception-like process of chrooting into the almostcorrectly installed Zorin and shoehorning Grub into place. Yet there’s more to rescuing and repairing than just fixing bootloaders. If you care about your data, then you should worry about drive failure. Backing up important data is obligatory (see the earlier boxout on Avoiding Problems), but if you’re really serious (or have a penchant for data forensics) then backing up entire partitions is worth looking at.
Rescuezilla describes itself as the Swiss Army Knife of System Recovery and you’ll find it on the Linux Format DVD. Just follow the three-step guide (right) to see how simple the process can be. You can restore a partition to a whole new drive (for example, after hardware failure), so long as there’s enough space on that drive. If it’s a partition that boots Linux, then you’ll probably need to
CAN GRUB SAVE THE DAY? “One thing to try at the rescue prompt is to manually tell GRUB where its bits are, and see if that breathes life back into your beloved distro.”
re-install GRUB (as mentioned previously) to get it working again. But enough about bootloaders already.
You might have heard about modern filesystems such as Btrfs and ZFS, and in particular their slightly mystical sounding ‘self-healing capabilities’. You might then be forgiven for thinking that by using these technologies you’ll never have to worry about data loss again. Alas, this is not the case. For one thing, no chunk of code is smart enough to protect against your own actions, and for another all hardware fails eventually. Self-healing is a significant advance though: it mitigates against silent corruption of files using checksums, hopefully catching failing hardware before any real damage is done. If you do run into a disk error, then do remember that in all likelihood it’s probably not an isolated incident and now would be a good time to consider new hardware.
What not to do, if you’re using these next generation filesystems, is to try and repair them yourself. The btrfs check command does have a –repair option, which is meant to be used offline. In fact it comes with all kinds of options for rebuilding checksum and extent trees. But they almost certainly won’t help you and in the wrong hands will probably do more bad than good. The Btrfs man page even tells us, “Do not use --repair unless you are advised to do so by a developer or an experienced user.” If the filesystem can be easily fixed, Btrfs will fixed it transparently. If not, don’t make the situation worse. Mount the device read-only, get what data off it you can, and then find a new home for it.
There’s plenty more to repairing Linux, and plenty more repair distros to try. We wanted to include the venerable Rescatux on the disc for example, but once again we were scuppered by DVD capacity. Rescatux is great because you can solve a lot of common problems straight from its GUI front-end. Check it out at www.supergrubdisk.org/rescatux and let us know your rescue horror (or glory) stories.