Friday, 03 March 2017 06:31

Human error caused Amazon Web Services outage Featured

By

A wrong command entered by a member of its technical staff was responsible for the outage experienced by Amazon Web Services simple storage service this week.

In a detailed explanation, the company said the S3 team was attempting to debug an issue that caused a slowdown in its billing system when, at 9.37am PST on Tuesday (4.30am Wednesday AEST), one of its technical staff ran a command that was intended to remove a few servers from one of the subsystems used by the S3 billing process.

The worker entered one wrong input for the command and ended up removing a much larger number of servers than intended, some of which supported two other S3 subsystems.

Among these were the index subsystem that manages the metadata and location information of all S3 objects in the Northern Virginia (US-EAST-1) Region and is needed to serve all GET, LIST, PUT, and DELETE requests.

The outage took down a large number of websites and services. AWS services something close to 1% of the top million websites.

The placement subsystem that allocated new storage was also disconnected; it depends on the index subsystem to be working for its own system to function.

"The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart," the company said.

"While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable."

The company also said it had been unable to update the AWS service health dashboard for two hours after the issue began, because the administration console has a dependency on Amazon S3.

"Instead, we used the AWS Twitter feed (@AWSCloud) and SHD banner text to communicate status until we were able to update the individual services’ status on the SHD. We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions."

The company outlined several changes it would be making as a result of the outage and apologised for the breakdown. "While we are proud of our long track record of availability with Amazon S3, we know how critical this service is to our customers, their applications and end users, and their businesses. We will do everything we can to learn from this event and use it to improve our availability even further," it said.

LEARN HOW TO REDUCE YOUR RISK OF A CYBER ATTACK

Australia is a cyber espionage hot spot.

As we automate, script and move to the cloud, more and more businesses are reliant on infrastructure that has high potential to be exposed to risk.

It only takes one awry email to expose an accounts payable process, and for cyber attackers to cost a business thousands of dollars.

In the free white paper ‘6 steps to improve your Business Cyber Security’ you will learn some simple steps you should be taking to prevent devastating malicious cyber attacks from destroying your business.

Cyber security can no longer be ignored, in this white paper you will learn:

· How does business security get breached?
· What can it cost to get it wrong?
· 6 actionable tips

DOWNLOAD NOW!

ADVERTISE ON ITWIRE NEWS SITE & NEWSLETTER

iTWire can help you promote your company, services, and products.

Get more LEADS & MORE SALES

Advertise on the iTWire News Site / Website

Advertise in the iTWire UPDATE / Newsletter

Promote your message via iTWire Sponsored Content/News

Guest Opinion for Home Page exposure

Contact Andrew on 0412 390 000 or email [email protected]

OR CLICK HERE!

Sam Varghese

website statistics

Sam Varghese has been writing for iTWire since 2006, a year after the site came into existence. For nearly a decade thereafter, he wrote mostly about free and open source software, based on his own use of this genre of software. Since May 2016, he has been writing across many areas of technology. He has been a journalist for nearly 40 years in India (Indian Express and Deccan Herald), the UAE (Khaleej Times) and Australia (Daily Commercial News (now defunct) and The Age). His personal blog is titled Irregular Expression.

VENDOR NEWS & EVENTS

REVIEWS

Recent Comments