Red Hat GlusterFS
With a pile of old servers he had in the garage, Jolyon builds a distributed filesystem capable of handling the largest of workloads.
Many years ago now, I recall a colleague emailing me the output of a command showing details of a newly built filesystem. Not just any filesystem mind you— this was the first one we’d ever created which broke the magical 1TB ceiling. Even more impressive than that (you had to be there, I guess), it was mirrored between data centres using the proprietary software we used at the time, via a dedicated cross-site link which could handle a (then) huge amount of bandwidth and a SAN which connected our ridiculously expensive servers to equally expensive disk arrays. Happy days indeed. For hardware vendor sales people anyway. It’s a different ball game nowadays with hardware costs at a fraction of what they were (if you actually need your own hardware), our favourite operating system and more open source software than we could ever have hoped for giving us so many choices when it comes to building systems. Some things haven’t changed though. Those pesky users still generate data in the form of files, expect to be able to do so forever without ever running out of space and equally expect the data to still be there when they next look for it. They’d really like it all to be cheaper than it used to be as well, if that wouldn’t be too much of a problem? Thanks!
I don’t seriously expect any readers to get quite so excited by this month’s column as I admittedly was when seeing that glorious 1TB monster back in the day. But I am going to talk about a filesystem this month called GlusterFS, which I’ve been using for quite a while and which I’ve been quietly impressed with over that period. If you have a need to provide a lot of storage to a lot of endpoints and users, I think it’s definitely worth taking a look at.
First off the bat, I should point out that this isn’t that new or shiny. GlusterFS has been around in various iterations for several years. I don’t know about you, but this is a quality I look for when it comes to storing data (especially my data). No 0.1a-pre type versioning here thank you very much. Back in 2007 saw Gluster Inc produce the first version of its filesystem and in 2011 the whole kit and caboodle was swallowed up by Red Hat for somewhere north of 100 million dollars. The Red Hat Gluster Storage product can be purchased with a support contract for those that require that kind of thing. But, of course, GlusterFS remains wholly open source and available for anyone to use should they wish.
The basics
GlusterFS bills itself ( www.gluster.org) as a “scalable network filesystem”. It can be built using off the shelf (commodity) hardware. It’s distributed, scaling into the (several) petabyte range for those lucky enough to have access to that much storage. It can handle thousands of clients and those clients can access it in many ways. For example, NFS, SMB/CIFS or via a small dedicated driver for GlusterFS itself. Failover can be handled automatically as can recovery. There’s very little to worry about if a node drops offline for a while. It runs on top of familiar Unix filesystems like ext4, xfs (or even zfs). There are no ‘central’ servers to worry about. Readers of this column are probably familiar with these. They’re the little boxes that sit off to the side of the slide in vendor presentations that the whole system depends on. They always end up being standalone with no high availability solution out of the box and “definitely in the next version” the friendly pre-sales person will tell you (do I sound bitter here? I am, believe me).
But I digress. No such worries with GlusterFS. Now, before charging off into a demo it’s best to understand what Gluster is and isn’t good at. Unstructured data is its speciality. You have backup dumps, spreadsheets and a huge archive of cat pictures? GlusterFS is your friend. Structured data? Not so much. Don’t expect to run your database on it. This would be
a pretty bad idea all round. At the most fundamental level, GlusterFS is simply a collection of regular filesystems (or in Gluster parlance, ‘bricks’) shared/exported out by multiple systems (nodes) and merged into a Gluster ‘volume’ so that writes and reads of data get shared out among those systems. There’s a lot more to it than that, of course, and multiple options exist around how everything is organised, but the basic premise is correct.
In organisations like Facebook (which uses it pretty heavily, there are some interesting presentations on the setup it uses) a node might consist of 100TB of usable storage spread across a few dozen hard disks. Hardware or software RAID will still feature on these locally. The bricks will be presented via xfs (or whatever Linux usable filesystem is the preference). Handling these nodes brings with them the usual baggage. They will need to be provisioned and maintained. Disks will fail and need to be replaced. Patches will need to be applied. At this scale, a ‘trusted pool’ of Gluster nodes in a particular data centre might spread across multiple racks and have volumes made up of many very large bricks. I’m aiming here for something a little less ambitious, of course.
By now you’ve probably realised that the words ’Gluster’ and ‘GlusterFS’ are used some what interchangeably when talking about this software (even the project website is guilty of this). At the command line, gluster is the word used.
My example here runs on a trio of local VMs (Ubuntu 16.04). Each of them has a small additional disk attached which I’m going to use for my Gluster volume. I’ve set some entries up in /etc/hosts in lieu of any kind of DNS setup. I could equally have used physical kit or one of the many cloud providers to set this up. The hosts know each other as gluster1, gluster2 and gluster3.
The first step is installing GlusterFS itself. In Ubuntu, the package to go for is glusterfs-server (running all of the following commands on all three nodes) $ sudo apt-get install glusterfs-server
Now to install a filesystem and mount it. My spare disk is located at /dev/sdb. XFS seems to be the most common underlying filesystem in use in production that I’ve seen, but this is only a test, so I’ll just use ext4 for now. $ sudo mkfs.ext4 /dev/sdb $ sudo mkdir -p /export/brick1 I then need to add the following line to the bottom of the /etc/fstab file: /dev/sdb /export/brick1 ext4 errors=remount-ro 0 1 Now on each server I can make my new brick available with the $ sudo mount -a command. Now to run some Gluster commands properly. From my first server I am able to set up my trusted pool via the simple peer probe syntax. $ sudo gluster peer probe gluster2 $ sudo gluster peer probe gluster3 Rewarded with a peer probe: success message, I can run sudo gluster peer status on either of the other nodes and have them report that they have two peers, with assigned uuids for each and a status of Peer in Cluster (connected) . All well and good so far.
A replicated volume
On each server I want now to create another directory, /export/brick1/gv0. Then, once this is done I can create and start a Gluster volume from the first node: $ sudo mkdir -p /export/brick1/gv0 # do this on each node $ sudo gluster volume create gv0 replica 3 gluster1:/export/ brick1/gv0 gluster2:/export/brick1/gv0 gluster3:/export/ brick1/gv0 $ sudo gluster volume start gv0 $ sudo gluster volume status
Now, $ df -h from any of the nodes doesn’t really show anything. The volume might report that it’s up and running but how can we access it? For now I’ll just use one of the servers (gluster3) as a client: $ sudo mount -t glusterfs gluster1:/gv0 /mnt
Now /gv0 can be seen in all its (empty) glory on my third node. But is it working?