STE VE CASSIDY
In praise of firmware upgrades and the miracles they can confer, but why the secrecy? Plus a data security howler and how not to deal with it
In praise of firmware upgrades and the miracles they can confer, but why the secrecy? Plus a data security howler and how not to deal with it.
The thrill never goes away: being published every month in PC Pro ought to be enough, you might say, but it was every bit as exciting to get my car on the front cover of Jaguar World magazine. And the connections that triggered this bit of good fortune convinced me that the world is increasingly dependent on firmware.
My car has a well-known failure mode, in which the supposedly lifelong transmission fluid isn’t lifelong at all. Replacing it, however, is only the start of the process. The reinvigorated transmission only works properly if it’s given a flash upgrade – in our terms, as if someone upgraded the device driver, except that the computer in question is inside a ZF six-speed gearbox.
I’m used to seeing substantial differences in behaviour after a firmware upgrade, but zippy performance and no blue screens on your server are nothing compared to whatever ZF transmissions have re-coded in this upgrade. The car has gone from a reluctant, thumpyshifting embarrassment to a suave and silent limousine.
I tried to pull that “computers are my forte” thing with the ZF agent, but ran headlong into a brick wall that we computer people just aren’t used to seeing. For these guys, that firmware upgrade was their best-kept secret. Partly because it solved a lot of problems without having to take the car apart – but partly because people just don’t want to hear about this low-level, irritating and puzzling lump of code.
Long-term readers may remember how unpopular I made myself at Google over a firmware issue. Again, the technical problem itself wasn’t overly interesting, but the
company didn’t want anyone to know how the fix was applied: namely, the right firmware in the Wi-Fi routers of a paperless school. This wasn’t something the school’s IT incumbent could find out by browsing public support forums but a deep-set secret, for release solely to internal staff and selected partners. One of those partners had in effect stonewalled access to the upgrade for almost six months, in a drawn-out game of problem-diagnosis poker that did none of the participants any favours at all. Just like my car transmission experts, it seemed Google and its representatives had an instinctive need to keep firmware as a dirty secret, away from the prying eyes of that Cassidy fella. So I published.
Starting out from a networking angle, I’m most often aware of firmware when it comes to network cards and switches. The one that sticks most clearly in my mind was a humble-looking LAN card driver from Intel that, about ten dialogs into the firmware, let you run line diagnostics, including whether the line was broken and, if so, how many metres from the card’s port the break was located. That was as impressive as
it was well-hidden: despite the parlous state of almost everybody’s network cables then, and now, I have yet to find anyone who has made use of a firmware feature that is functionally equivalent to an oscilloscope, just to track down a crushed or kinked Cat5e lead.
But that’s how far you can go with a good combination of driver, firmware and hardware. The true tragedy in a modern network these days is the opposite: a device with an orphan device driver. That is, a machine that could do the job you need perfectly well, but that’s stopped from doing so by a conscious and deliberate choice by the manufacturer and designer. A few months back, I ranted extensively about a NAS sent along for evaluation, running Windows Storage Server. At first, this sounds like a great bit of design – taking out the expensive server-grade PC, while keeping all the tweakability and software catalogue of Windows. However, I couldn’t get it to work on my home test network satisfactorily. Copy speeds of 3MB/sec were the result, rather than my norm of between 80 and 120MB/sec.
The reason didn’t emerge until I spent a fun-filled day checking firmware. The NAS has 10GbE capability and a couple of older Gigabit ports, too. The chipset for the Gigabit connections is from Intel and still relies on a driver included in the Windows install. Which can’t, it seems, set the network tweaks I like (mostly Jumbo Frames) to get the copy speeds up to an acceptable level. This box had one job and the manufacturer seems to have cynically limited its capability by deliberate use of unsupported hardware. Most of the benefit of combining NAS architecture with Windows software is to dip a toe into some complicated fields, such as network function virtualisation, hyperconvergence and iSCSI. These are the exact areas where driver and firmware tweaks are the main causes of success or failure – so an unsupported or non-updated chipset suddenly occupies centre stage.
Perhaps my favourite firmware story has been something that only came to my attention by way of a trip outside my comfort zone: I went to see the launch of a new smartphone. I guess it’s my own silly fault for expressing interest in things like 5G and smart cities over the past couple of years, as the companies involved appear to believe that anything 5G-related is also phone-related. As a result, I received a mismatched invitation, which turned out to be mostly about camera firmware.
Nonetheless, I was intrigued to see this new phone with enhanced low-light capabilities. As camera hardware has shrunk to fit inside a mobile phone chassis, so the borders between clearly separated components have started to blur. It’s no longer true that a sensor sits behind a lens and a CPU pulls values off that sensor and puts them in memory. The relationship between the sensor and the processor is arbitrated by – you guessed it – firmware. The CPU can look at the first few pixels from the chip, and tell the hardware to behave differently according to the results. This was what these chirpy, consumer-sector promoters wanted to highlight to the assembled journalists – a software download that improves the camera hardware in low-light situations.
The assumption of these shindigs is that in writing up the name and model of the device, they get some publicity, you get to play with some gadgets and everybody wins. However, for the reasons I describe below, I can’t identify the device or the PR company.
I was hoping to lay out some clear moral boundaries here, based around the idea that transparency is the right approach with bad software and poorly kept firmware – and here I am, about to keep secrets in order to tell a bigger story.
Feel bad about a data breach? Read this
I should apologise in advance for making things a bit vague here. It isn’t my intention to get anyone fired, so I’m going to skirt around exactly which mobile device I was looking at. All you need to know is that I returned home with a new toy, somewhat bemused by the consumerish-ness of the whole exercise, wondering how I was going to explain this one away in a PC Pro column that’s mainly about business IT.
My salvation, and the downfall of the PR campaign, started with me finding that this device featured a mobile Excel sheet viewer app. Cool, I thought, about time for a neatly portable Excel viewer! So I opened the app, and the online file store associated with the Google login for the device.
I was then presented with a few files, the biggest of which had the abbreviation “PASWD” in
“The NAS manufacturer seems to have cynically limited its capability”
“This is the kind of sequence of events that lands up in a classic data breach story”
the name. It was a good half-megabyte too. One double-tap later, I was in and was presented with over 1,000 social media accounts and identities. The people preparing these devices had left the master password file in the cloud storage for all the devices being handed out to the press.
This is a howler; even our unflappable editor hissed sharply through his teeth on hearing about it. I am sure that if you have 1,000 social media accounts then it makes eminent sense to centralise their management, and the tool most people turn to straight away would be Excel, but surely you would protect it with a master password to match?
In fact, the longer I thought about what I had found, the more difficult my response became. It seemed likely that the file had been put in the cloud store by someone unrelated, not knowing that the cloud store was about to be co-opted in to a role with a bunch of media types, most of whom wouldn’t replicate my old fogey’s rush to the Excel file viewer. So blaming the direct contacts I had wasn’t a 100% safe bet – they may have no idea at all the file is there.
On the other hand, at the higher levels of the business, someone ought to have taken a view of the way that resources such as master password lists are handled, updated and deleted from less appropriate file stores. Talking to the junior guys would have next-to-no impact, since their natural impulse is to cover it up, especially if the mistake came from their team.
Yet I didn’t want anyone to get fired. So I cold-called the company, asking to speak to the managing director. She doesn’t answer cold calls, it seemed. Several juniors flatly refused to put me through, without knowing the subject of the conversation. After a few days of this shuttling, I happened to be in a call with one of the clients whose Facebook, Twitter and Instagram accounts were on the list, so I made their team leader fully aware of just what I was looking at.
About 20 minutes later, the PR company rang me. Just circling back, they said, on the variety of messages you’ve been leaving for us over the past few days… what was the problem? And about 20 minutes after that, the mobile device that had started all the fuss simply dropped off the Google Cloud. Password error, it said.
As whistleblower experiences go, it wasn’t very satisfying. At one level, I was trying to figure out the company culture from the outside, to make sure that they came out of this staggering security failure a bit wiser rather than just waking up one day with no job. On the other hand, by choosing not to receive this type of heads-up, they open themselves up to a journalist exposing the story, warts, names and all. But where are the learned lessons out of an approach like that?
So, I had a nice chat with the lady, who eventually realised I should be called back. I managed to pick a client rep who didn’t see any gain in being overly nasty either, so I think in the end we achieved a balance between panic at the situation and some sense of proportion over what was at risk here. I see no benefit in identifying any of the players – the whole matter strikes me as being something you might expect to happen when interns are left to put an event together, rather than being a company-wide cultural failure or similar. Whoever left the file in the wrong cloud silo gets a bit of a shouty appraisal and comes out better equipped.
Yet not all of this went quite perfectly. Changing the password of the cloud account is not the same as changing all the customer account passwords in the list. It looks like taking action, and it will have some effect visible to all those given the device at the event, but if any of them are wily birds, used to copying files off the Google Cloud down to the machine and on to one of those jolly handy little dual-interface USB cards, then things are likely to get a lot more murky. This is exactly the kind of sequence of events – where no one person has made a specific move or a planned series of actions – that lands up in a classic data breach story, where a particular big organisation hands over their passwords to a subcontractor, and keeps on answering all those security audits with a nice firm “yes” – but in reality has no idea where their data has gone or who has been entrusted with it.
So, when you feel you’re fighting the good fight on data breaches, but it’s difficult to know whether you’re doing the right things or attending to the most urgent issues, just keep this little incident in mind and remember that all the people involved are in the technology sector. They’re not just casual users with a proper job to do. This stuff is their daily bread, and they ought to be expected to do the right thing when it comes to not just arcane government regulations, but also the common-sense data security measures to keep people like me from writing pieces like this. The fact is that the whole data breach problem isn’t pseudo-macho nonsense about being vigilant or appeals to absurdly complex systems auditing tools: it’s keeping track of which human needs to do what, with which pieces of data, and making everyday decisions on that basis without being blinded by hype.