Want to add RAID storage to your homebrew NAS storage box? It isn’t as straightforward as it seems, Steve explains, but he does deliver at the end.
Want to add RAID storage to your homebrew NAS storage box? It isn’t as simple as it seems, Steve explains, but he does deliver at the end
As sudden as a light bulb, the reader mailbox burst into life in mid-February. One particular question stood out: what’s the best add-on disk controller for a domestic, small NAS project?
Seems simple on first pass, right? So I dearly wish, but getting the right bits to run in the right way and actually work as protectors of your data is not the walk through a daisylined park that you might imagine. Contrary to the impression I may have given in previous columns, the entire storage business is a nightmarish labyrinth of not-really-compatible, but apparently perfectly standard, components and software.
Take a look at the storage market in devices for hosting centres, where replacement or expansion drives for heavyweight rack-mount storage expansion cages are very machinespecific. Many a support forum thread peters out because some hapless storage guy eventually admits that he thought the manufacturer label on an enterprise NAS drive was an entry point into a whole genealogy of related (and hopefully bigger) other drives that could be plugged and played without let or hindrance.
The simple rule is that the further up the price and feature scale you go, the more constrained the design becomes. Sometimes you’ll come across top-end enterprise storage devices that work with one sole model of drive, of a single revision with a single capacity. Custom firmware is the method of ensuring this situation: vendors commonly justify this style of entrapment by pointing out that their firmware is an inescapable component of the collection of tools and techniques for going faster. ( See Jon’s column on p108 for just such an example.)
I have a few readers up in the megascale storage business to whom this kind of knowledge is beyond value, but I also have a lot of readers who are trying to make sense of what’s possible at the other end of the market. And they can’t figure out why all these megascale woes have any business intruding into their little home network. The reasons are mostly simple: even the humblest NAS server can present files over
NFS, which is a protocol from the enormous machines of the 1970s.
The amount of baggage from those early protocols, from intellectual concepts right through to dodgy cut-and-pastes of source code, that’s still hanging about in desktop NAS machines is extraordinary.
My go-to examples here are the excellent little machines made by Iomega, which then sold out to EMC, which then divested to Lenovo.
These offered capable, well-cooled, excellent iSCSI implementation… and
were almost completely impossible to upgrade. First, the downloadable OS image became hard to find, then available only with the direct help of the official service technicians. Effectively, a death sentence.
It should not be surprising, then, to find that the more general-purpose boxes we have kicking around under our desks are much more accepting of various common upgrades and drive sizes, in the main, than their specialpurpose distant relatives. But! As my email questioner realised, way back in the assumptions within his deceptively short query, the universality of a basic everyday disk controller chipset on an arbitrary motherboard is gained at some cost. To be frank, that chip is dumb as a rock. So dumb, indeed, that a lot of DIY NAS operating systems assume the worst, taking direct control of each disk at the lowest possible level.
Accusations fly around as to why this decision was taken, but only amongst those who have never tried to actually live with the real consequences of a disk or machine failure built on this “level” of tech. In practice, it’s not smart to believe that you can rely on mass-market SATA technology to store stuff in configurations that will survive an attempt to replace a member disk when the bad thing happens.
Taking control
Quite a few attempts at storage management in homebrews don’t work at all once the hardware is misbehaving; this is also true of the motherboard chipsets that say they can be flipped into a RAID mode. Of course, there are exceptions, but these kick in around the scale of the HP Z Series workstation I describe in the boxout overleaf. As a rough guide, if you can see more than four drive connectors on a motherboard then it has more than one storage controller: quite possibly the second one is SAS, not SATA, and that’s where things start to get interesting.
So, why do NAS vendors want to take such low-level control? One answer you may hear is that the OS for your NAS homebrew will likely want to ignore any apparently sophisticated hardware RAID support it finds. Buying anything clever is, in this situation, a waste of money. I have to say, I hear this advice a lot, and I can’t find a single thing in it that I agree with: I can point to a half-dozen projects I put together in the early 2000s, probably numbering about 50 servers in all, in which I spent more on smart hardware RAID cards than I did on any other part of the machine. Some of those servers are still running now. I fervently wish they weren’t for the sake of my own income stream, but the number of times that one of the far-flung global support team tells me how easy it was for them to swap out a dead drive in that deployment proves that putting the money into the controller smarts definitely works.
So we’re looking for a balanced device, something that doesn’t carry too many features you won’t need, and yet which isn’t so completely empty of anything above the bare minimum hardware that it runs like a dog. There are manufacturers I look to when these jobs come up, and their names don’t always ring a bell with the homebuilders. Areca, anybody? Sonnet? Nvidia? Highpoint?
Nvidia is in there because you might come across machines with Nvidia storage controller chips. I have three of those from way back sitting around here looking forlorn, because I didn’t specify a tolerant enough RAID drive layout for the chipset to be able to recover from a dead drive in the array. When they run right, they’re great, but I wouldn’t use this level of support in anything less than a matched-pair of mirrored NAS devices, because the recovery isn’t that simple. I’d be enormously tempted to spend the money on a complete Supermicro motherboard instead: Supermicro motherboard services are often much better then what you can do with add-on cards at the same cost levels.
But that’s not really where we’re operating here. We want to do a bit better on a home NAS box – or, as I once did, we want to take the stress off an overloaded server by standing up some fast (but not too large) auxiliary servers as iSCSI targets.
Dead zones and limits
In this field, there are some natural limits to how much disk you can stand up. There’s a dead zone from eight drives (as is common in medium-size business NAS deployments) up to 12 drives, which is where the lower end of the enterprise kit starts to appear. Perhaps my perfect NAS chassis is a Supermicro bundle, with 12 or even 24 disk bays matched by the same number of SATA headers on the motherboard. Not something you’d expect to find at the dumpster-diving level we love so much when it comes to homebrew NAS. Or is that just me? Even when you have an eight-drive NAS design, or a rack-mount 12-banger plumbed together in a nest of SATA cables, you aren’t out of the woods when it comes to disk controllers. Most of the performance limits of a standard PC disk controller chip come from how many devices it has to herd together and exchange data with. At a certain point in the
“In this field, there are some natural limits to how much disk you can stand up”
basic construction of that chip, there are single-tasking pipelines. The more drives you put through those “smart motorways”, the slower the overall architecture of your NAS will run.
This often frustrates the bigger box owners, because they don’t want to be bothered with completely emptying the machine, changing the split between volumes and arrays so that smaller groups of drives are active at the same time, thereby reducing load on that low-spec disk controller chip lurking on the motherboard.
Of the add-on RAID card vendors I’ve mentioned, Areca is the longest lived and most popular in professional circles. Sonnet has a long history of quirky hardware on Macs, including drive controllers, and quite a lot of its devices are actually platformagnostic. Highpoint is the most interesting for the original, homebrew brief, because it makes some very low-level devices – and the key to keeping a NAS user happy is to give them as much contiguous storage space as the various separate layers of disk size, controller capability, NAS firmware limitations and userfriendly setup conspire to permit.
It’s easy to make a single logical volume if you have a nice matched set of shiny new drives. However, the really hardcore home hacker just has whatever they could carry home from the server room, which means massively different geometries, sizes and speeds. Bringing these together in a smart RAID card’s menus is not where those cards do well: you should be fiddling with the highest possible level of volume construction in the NAS software configuration… and be ready for it to collapse totally at the first sign of trouble from a rag-bag of drives.
Lastly, the one thing people do with homebrew NAS that makes no sense is they don’t spend money on RAM. Look at the architecture again. If your software is ignoring any hardware smarts because it can put individual drives into a softwarepowered simulation of a RAID then the least you can do is give it some caching and working storage space to ensure there’s enough room for these laboriously recoded low-level services to do their thing.
Putting together a homebrew with an Areca controller with multiple channels, its own battery and cache RAM, 12 eager drives and then not increasing the machine’s 4GB of system memory is my very definition of a false economy.