How did an Amazon glitch leave people in the dark?
Outage at cloud provider Amazon Web Services resulted in websites and smart homes failing
Is this the future of our internet-connected lives? Here’s a cautionary tale about the future of the internet: an over three-hour outage in an obscure, if tremendously profitable, wing of online retailer Amazon resulted not only in websites such as Medium and Business Insider failing, but also in people unable to turn on their lights.
This outage affected Amazon Web Services (AWS), an Amazon subsidiary that provides cloud computing services to other businesses. If you’ve ever been told something is stored or run “in the cloud”, the likelihood is that it was in servers owned by Amazon — or by similar services provided by its two main competitors, Microsoft and Google.
When that model works, it works brilliantly, providing low barrier to entry for small firms needing an online presence, economies of scale for larger companies warning worldclass hosting — and huge profits for Amazon itself.
In the last year, the company has pulled in profit of $2.4 billion (Dh8.8 billion) overall, but AWS alone made a profit of $3.1 billion (the missing billions are a result of how much money the company continues to lose in international sales).
All fall down
When it doesn’t work, everything comes crashing down. Subscribers, and the wider web, first got a hint that something was up around 6pm UK time, as glitches began to hit websites and apps that use Amazon’s cloud storage and computing.
It took slightly longer than might be expected when a huge tranche of the net collapses simultaneously for the common cause to be diagnosed, though.
In part, that’s because Amazon’s data centres are distributed around the world. Only its “US East” region, based in north Virginia, was hit, sparing customers — like the Guardian — who use their cloud services elsewhere in the world. (The majority of the Guardian’s use of AWS, for distributed publishing tools, is based in Amazon’s Dublin centre; some interactive visuals were hosted from Virginia, however, leading to downtime and delays).
Scale of outage
The US-East region does contain some of the company’s most visible customers, however, in part because it’s the natural region for America’s largely east coast-based publishing industry. It also contains a couple of other customers which helped hide the scale of the outage for many: downtimemonitoring service downforeveryoneorjustme.com, which lets users see whether a particular site is offline or not, was down for everyone.
Worse still, Amazon’s own monitoring tools were broken. The company’s Service Health Dashboard reported everything fine across the board, until engineers managed to fix the first problem of the day and update the system to display the many, many other problems ongoing.
A short internet outage might sound like a trifling matter — don’t we all need a little bit of time offline now and then? — but for many, the pervasiveness of the so-called “internet of things” meant downtime at Amazon affected their physical life as well as their digital one.
Ifttt, a website which allows users to link together services from multiple providers into one seamless operation, was one of the Amazon customers knocked offline by the outage. For some, that simply meant that their workflows were disrupted, with emails not being forwarded or weather summaries not being automatically emailed.
For others, that meant slightly more drastic problems: “smart home” owners reported losing control of their houses after the jury-rigged system they used to control internetconnected locks and light bulbs failed.