“Spending money on storing your stuff isn’t an investment in tech; it’s an investment in a good night’s sleep”
Steve tries to find and restore a VM that’s disappeared from a NAS device, and gets some security tips from the Israeli military and US security forces
Nobody really understands catnip. Why should there be a plant that has such a powerful hold over the entire genus felidae? It’s irrational – just like the relationship people have with their data and things on which they store it. Why do people passively accept the idea that because a favoured, cheap tech vendor says it’s “great for backups!”, they should go ahead and believe them? When I get drawn into someone’s selection process – or worse still, a recovery – most of my time isn’t spent leafing through product catalogues. It’s dealing with the catnip problem.
Those buying a backup device seem quite sure that what they really need isn’t the most capable option, but the cheapest. It must be foolish to spend over £39.99 on a couple of terabytes of storage, they say.
Of course, as hardened data people, we understand that it’s the opposite of foolish. Spending money on storing your stuff isn’t an investment in technology; it’s an investment in a good night’s sleep. Eventually, having bought as cheap as possible, you’ll be lying awake on a hot and uncomfortable night, wondering if those noises you can hear are the bearings in your “backup disk”, locked away in a fanless USBconnected hell at 50 oC
I much prefer to spend a bit more money, with the absolute requirement not one of droppability or density of terabytes in your pocket. I want my backups to be on an OS that isn’t the same as that of my main PC; that has its own separated list of users and passwords; that can do a lot of the fundamentally boring stuff – such as syncing with cloud stores – without me having to buy that device its own separate malware scanner, email address and social media account.
I also like nice big cooling fans, because I know that when I’m in the depths of a recovery process, the disk that’s housing my backup is going to be boiling its little backside off, returning my data. My data-recovery kit has one of those semi-hypnotic USB fans on a flexible metal neck, with LEDs that flash as the little blades spin, because if I have to recover something from the wrong type of drive then I can pretty much guarantee the victim won’t have a cooler handy.
At the point of purchase, however, the idea that you might need to spend extra on a device such as the Synology DS216j is greeted with derision. Please guys: when normal people try to take control of a buying process such as that, fight back. We’ve liked Synology NASes for some time at PC Pro, and this isn’t only down to the software. On the back of the box is a big lazy turning fan, and if high temperatures are a known risk for you then you can monitor the state of the disks from the phone in your pocket.
My last major pain from a recovery process is, I suspect, an exercise in precisely the kind of humility that those self-defeating buyers of sub-par devices want to see from their experts. In this case, I had decided to push the boat out on a machine recovery: not only did I drag the VM files over to a bottom-end £120 NAS box with an iSCSI connection, I actually left the VMs running that way. This is identifiably slower than being directly connected – but only during certain rare operations during startup and shutdown.
This was tolerable enough for this client that they never rang back, for almost ten months – but when they did, it was a corker. That VM was, they said, invisible. Not just to its normal connecting PCs, but also to the VMware management utility. Out came the bat-bike, and about 20 minutes later, glowing gently from the ride, I was looking at the vSphere management screen that said there was no NAS there, right on the same screen as a browser window that said there was.
The browser was showing the web management interface. Sometimes, the VMware box would “re-see”
“If I have to recover something from the wrong type of drive then I can guarantee the victim won’t have a cooler handy”
the iSCSI target, which wouldn’t change anything on the NAS box management screen one tiny bit. I tried a few simple moves – change the patch lead; change which switch port it used – to no avail. I thought about deleting and re-creating the various iSCSI structures that relate the data on disk to the target name presented to the network, but I’m not crazy enough to try that on a live system with a current problem.
So I went home and had a go with a very similar NAS from the same vendor on my own network – and found that any deleting or editing of those structures seemed to trigger a vast outbreak of disk activity, with no progress bar or update announcement, much less any ETA for completion. So that NAS went on the “disk scrub and start over” pile and I went back to the client the next day with a set of to-NOT-dos as well as the more usual to-dos.
Thinking about whether I could use the relatively short uptime windows to move or copy the ensnared VM to another device, or to an unused bit of that NAS, I spent some more time staring at the several places where disk use is reported, between the disk volume manager, the VM on-disk file sizes, the iSCSI target configuration and the snapshot monitor.
Wait. What? I’d gone past this little config statement several times without really reading the display: this particular NAS management interface has rediscovered the charms of minimalism, and many features hide below either a left- or a right-click on bits of the GUI. Upon a closer look, the display summarised what I already knew: the VM took up about 112GB of a storage object that had been set up to take all 4TB of the available space, but which was thinprovisioned, expanding on demand. Alongside the 114GB of real data was 220GB of snapshots.
But neither I nor any of the client’s staff had been asking for snapshots. We dug deeper: there was a long list of rather irregularly dated snapshots, of many different sizes. It looked as if the NAS box would trigger a snapshot once it thought enough changes had happened in the storage volume, although the size and timestamps didn’t explain this very well. In any case, the real change had kicked in on 1 September, when the snapshotting went from random to daily. No, the client said, there was no change of use or sudden influx of nerdy staff that day. Neither had the snapshots got that much bigger.
A snapshot is a useful concept, sure enough, but you won’t find many advocates recommending that you keep almost three years of the things on the same disk as the volume whose snap has been shot. Half a dozen is a sensible maximum, especially on thin-provisioned volumes and lower-end NAS devices, where the cycle of “grow a bit/take a snapshot” activity will guarantee that the volume becomes heavily fragmented. In this case, it was so severe that VMware was making attempts to retrieve bits of VM disk that simply timed out, causing the VMware volume to vanish off the server. Of course, deleting the snapshots took over a day. No progress bar was available to show how this invisible but very machine-hungry process was going. That also applied to the equally obtuse post-deletion disk-tidying activity: clearly, things were happening according to the disk access lights and the system monitor, but there was no status information or expected completion time. So, we left it over a holiday weekend, with the client instructed to give me a ring once they were back and had seen that the disk light had gone out. Fortunately, this client isn’t into the blame game. I’m pretty sure I didn’t turn on snapshotting, and a review of a very similar device at home (different-sized disks but same NAS OS revision) showed me that snapshots aren’t turned on by default for iSCSI volumes in this setup. But as obscure consequences of what should be agreed industry-standard behaviour goes, this one was a corker!
Hack attack: tools of the military
It turns out that there are more disclosures of nasty hacker tools than just the negligent deposit of weapons-grade tools perpetrated by US intelligence agencies in 2017. I bumped into Javelin Networks at this year’s Silicon Valley tour, organised by NetEvents, and it had probably the most incendiary leaflet I’ve seen in 30-plus years of IT.
Javelin is in part a retirement home for data-infiltration teams from the Israeli military, and it’s figured out that some of the tools its employees used to wield are both effectively
“Half a dozen snapshots is sensible, especially on thin-provisioned volumes and lower-end NAS devices”