Setting up big storage
Make room for your large files, where they’ll be protected from drive failure and general tomfoolery.
Readers of a certain vintage may recall various BIOS and platform limitations to hard drive capacity over the years. MS DOS couldn’t handle partitions larger than 2GB, old BIOSes didn’t understand drives larger than 8GB, the FAT32 filesystem can’t handle files larger than 4GB. The latest such limit which we’ve surpassed now is the 2.2TB limit for disks partitioned using a traditional (sometimes called MBR or MS-DOS, both of which are misleading) partition table. If you have a drive (or drives) larger than this, then you can use a newer GUID Partition Table (GPT) in order to access it in its entirety.
However, we don’t need partitions where we’re going. The next-gen btrfs filesystem supports installation directly to a device. It also has support for snapshots, which are quite handy. Btrfs also has its own native RAID support for striping and mirroring data across multiple drives. This obviates the classical ways of doing this, namely using hardware RAID or the Mdadm layer. Of course, you don’t have to use these features, but we’re going to.
We’re going to use two drives in a RAID1 configuration, which is the easiest to understand: data and metadata are just mirrored identically across both devices. Before we go further, RAID is not backup–if you mistakenly delete a file on a RAID it’s still gone, unless it’s backed up. RAID protects against drive failure. If a drive fails then the array will continue to function, albeit with some loud warnings and in a degraded state. The faulty drive should be replaced and the array rebalanced as a matter of priority.
None of this should interfere with a solid back-up regimen. What if further devices failed? What if someone got their terminals confused and accidentally deleted everything? What if some rare kernel bug suddenly strikes? RAID doesn’t protect against these things at all. So keep backups. Please. It’s generally easy enough to recreate the system side of things for a home machine. If the worst happens then recover what you can, re-install and add all the packages that you added. But all your personal settings and documents, unless you backed them up, may be lost forever.
A Mint-flavoured solution
Mint includes the simple BackupTool, which enables you to back up personal data as well as installed software packages. Running this from time to time (store the data on a USB stick or in the Cloud if you’re okay with that) could save you significant hassle later. There are more advanced, multi-tiered back-up strategies, and for serious data you really should look into these. Btrfs has yet another nifty feature called CoW snapshots, which you can read about in the box ( below).
Setting up our RAID1 array is easy (there are many options), but we shan’t use them since there are also sane defaults. We do need to know which devices are which though, so use the lsblk command to find your two storage drives. We’ll assume they’re /dev/sdb and /dev/sdc, but don’t just copy this blindly: if you get it wrong then bad things will happen. We just need to tell mkfs.btrfs that we want our data mirrored (rather than striped) across both disks: $ sudo mkfs.btrfs -d raid1 /dev/sdb /dev/sdc
The program should give a successful report such as:
UUID: ce6bcaf9-cd41-4bb1-b531-87901485714c Node size: 16384 Sector size: 4096 Filesystem size: 3.91TiB Block group profiles: Data: RAID1 1.01GiB Metadata: RAID1 1.01GiB System: RAID1 12.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 2 Devices: ID SIZE PATH 1 1.95TiB /dev/sdb 2 1.95TiB /dev/sdc
Now we can make a mountpoint for our new storage, so let’s do that with:
$ sudo mkdir /mnt/lxfraid
Now check it works with
$ sudo mount /dev/sdb /mnt/lxfraid df
and have a look at the output of to see the huge amount of space at your disposal. Btrfs is smart enough to know that the drive we specified is part of an array, so it’s not like we’re only mounting half of it, as the command may suggest. We’d like our new device to be mounted on each boot so, as we did early with our LVM, use the blkid command to get the UUID of one of the drives. Then add a line of the form: UUID="abcd…” /mnt/lxfraid btrfs defaults 0 0 to /etc/fstab.
Bad COWs
Btrfs uses a technique called Copy on Write for storing bits to the drive. In essence, this doesn’t overwrite old data when it’s modified, either redirecting it or copying it elsewhere. This can save resources in a copy-heavy environment and provides some resilience if the host machine crashes, since there would be no partial committing of in-flight data, which could cause corruption.
This redirection or duplication does incur a slight performance penalty and in certain situations – namely where lots of data changes in a large file – this becomes more visible, culminating in files becoming unduly fragmented over time. Specifically, databases, virtual machines and possibly your steam library would prefer not to be CoWed. Fortunately, we can use file attributes to disable CoW for specific files or directories. For example:
$ chattr +C /path/to/VMs
will ensure that new files in the given directory will have CoW disabled. Note that this won’t affect already existing files. Using this command on a non-empty file won’t quite have the desired effect either: some blocks will still be treated as CoW and will take a few transactions before they stabilise. To forcefully disable CoW for existing files or directories, we have to (somewhat inelegantly) rename and copy them. For example, if your /path/to/VMs directory is already full of virtual disks and you want them to go faster:
$ mv /path/to/VMs /path/to/VMs.old $ mkdir /path/to/VMs $ chattr +C /path/to/VMs $ cp -a /path/to/VMs.old/* /path/to/VMs $ rm -rf /path/to/VMs.old
We used the tried and tested ext4 filesystem for our root LV, but this can be converted by rebooting with any live disc (you can’t do this while it’s mounted as a root filesystem, but non-root filesystems can be converted on the fly). It’s better to do this before any important data is stored on there. Just reboot and run:
$ sudo btrfs-convert /dev/mapper/mint--vg-root
You’ll then need to mount the filesystem and update /etc/fstab (at the very least, you’ll need to change ext4 to btrfs and change the last field from 1 to 0 (btrfs consistency checking takes place automatically). Our default Mint install doesn’t reference our root VG by UUIDs, which is good because this will change (get the new one with blkid ).
Finally, you’ll need to chroot and update GRUB, or you’ll be greeted with an unbootable system. Don’t worry, though – dealing with unbootable systems is all good practice.
“Mint’s Backup Tool enables you to back up personal data as well as installed software”