Disk management
Stuart Burns isn’t boring, he’s essential – just like looking after your disks
Recently, I have spent a not insignificant amount of time remediating Linux systems with some poor design choices regards disk layout and management on Linux. I thought of it as something worthwhile sharing with the community at large.
While not geeky or new, it’s important to understand the strong correlation between well-designed and well-managed systems and uptime. There isn’t space to cover it all, so I hope to show some of the bad decisions that can be made with disk layouts.
The items below can’t be done retrospectively but if used at build time, will pay the admin back in spades over time.
Let’s start at the beginning. Using LVM (Logical Volume Management) makes disk management easier as it abstracts the underlying physical disks from the upper layer of logical volumes. It makes adding additional physical disks into the system and expanding the filesystems simple.
As an example, if you have an Ubuntu server with several disks (RAID volumes/ ADADM volumes), the first disk is set up as a default LVM disk configuration. Partitioning a fixed disk and formatting directly is just lazy management.
Several volume groups (VGs) can run on a system and putting the /var/logs on a separate volume is good practice for many reasons. If you are running production Docker systems, you would be crazy not to do this. At the same time, if /var/log gets to 100% full, it can, in certain scenarios, prevent swapping to root, effectively preventing proper management.
To give a real-world example of this, last week I was upgrading a fleet of machines that had been set up with meagre disk space for the LVM that had /var/log hosted on it. Unfortunately, because there wasn’t enough disk space, the upgrades errored out. As a quick fix, I was able to allocate a few GB from that unused space, grow the filesystem and then the upgrades worked as expected.
When setting up the logs, set up a partition that keeps 20% as emergency slack space. Instead of the standard lvcreate , use the following, substituting values as needed:
$ lvcreate -n logs -l 80%FREE vg01
The administrator can then add a /var/log
entry to fstab and done properly, on boot it isolates the logs to not be able to cause as many problems.
If you do this, in the bad times, adding an extra 2GB of space becomes as simple as:
$ lvextend -L+2G /dev/mapper/apps $ xfs_growfs /apps
This approach works as far as you want to take it. Separating out the app’s volume is also a great idea, with the same percentage in reserve.
The reason for this is that sometimes volumes can run short of space and adding space when you have spare becomes as simple as running lvextend and then growing the filesystem.
This is doubly complex if the underlying disk is cloud-based, and expanding the filesystem is a significant undertaking when multiples of the existing disk are required, even if a small amount of space is needed.
If nothing else, that spare space buys you time to fix the issue properly, rather than potentially running out of space and risking corruption.
While LVM may be new to some, it is without doubt the best way to manage disks coherently. It allows so much flexibility compared to any other way. It may take some learning, but understanding and using LVM is one of those skills that keeps on giving.