Errant keystrokes blamed for Amazon’s cloud outage
Amazon was back with an apology and an explanation Thursday for a high-profile malfunction that caused websites across the internet to grind to a halt for hours on Tuesday.
The online retail giant, which runs a widely used cloud computing platform for sites such as Airbnb, Netflix, Reddit and Quora, is blaming the outage on a simple employee mistake.
A team member was doing a bit of maintenance on Amazon Web Services Tuesday, trying to speed up the billing system, when he or she tapped in the wrong codes — and inadvertently took a few more servers offline than the procedure was supposed to, Amazon said in a statement Thursday. With a few mistaken keystrokes, the employee wound up knocking out systems that supported other systems that help AWS work properly.
The cascading failure meant that many websites could no longer make changes to the information stored on Amazon’s cloud platform. For everyday users, that meant being unable to load pages, transfer files or take other actions on some of the sites they regularly use.
“In this instance, the tool used allowed too much capacity to be removed too quickly,” Amazon said. “We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level.”
Translation: Employees will no longer be able to unplug whole parts of the internet by mistake.
Amazon said it was sorry for the outage’s effect on its customers and vowed to learn from the incident. One immediate next step? The company said it will subdivide its servers even more than before “to reduce blast radius and improve recovery” should something like this happen again.
Although Amazon may be known mainly for retail shipping, cloud computing is where it actually makes money. The company owns big data centers around the world and rents out computing power and storage on demand. This global cloud infrastructure service is dominated by just four players, and Amazon’s market share is greater than that of the next three combined.