Category Sponsorship Banner Left

Category Sponsorship Banner Right

Category Sponsorship Banner Middle

Thursday, 10 June 2021 03:29

The internet outage - Fastly: A customer did it (kinda) Featured

By
The internet outage - Fastly: A customer did it (kinda) Image by Christelle Olivier from Pixabay

 

Fastly's latest response to yesterday's outage puts the blame indirectly on an unnamed customer.

iTWire has already reported extensively on the outage here, here and here.

According to Nick Rockwell, Fastly's Senior Vice President of Engineering and Infrastructure, in a blog post a few hours ago, "We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change. We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration. Within 49 minutes, 95% of our network was operating as normal.

"This outage was broad and severe, and we're truly sorry for the impact to our customers and everyone who relies on them."

Rockwell continues, "On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances.

"Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors."

Unfortunately, when you're one of the Internet's leading CDN providers, 49 minutes is a VERY long time.

 

 

Rockwell's blog post offers a simplified timeline.

fastly timeline

The only part of this timeline that bothers us is in the final few lines. Over seven hours between recovery and the deployment of a bug fix.

Rockwell concludes the company's mea culpa with this:

"Where do we go from here?

"We're deploying the bug fix across our network as quickly and safely as possible.

"We are conducting a complete post mortem of the processes and practices we followed during this incident.

"We'll figure out why we didn't detect the bug during our software quality assurance and testing processes.

"We'll evaluate ways to improve our remediation time."

Of course, the internet is a complex beast, but outages like this should never happen. With luck, everyone will learn from this.  And a note to CDN customers - don't put all your delivery eggs in one basket.


Subscribe to ITWIRE UPDATE Newsletter here

GRAND OPENING OF THE ITWIRE SHOP

The much awaited iTWire Shop is now open to our readers.

Visit the iTWire Shop, a leading destination for stylish accessories, gear & gadgets, lifestyle products and everyday portable office essentials, drones, zoom lenses for smartphones, software and online training.

PLUS Big Brands include: Apple, Lenovo, LG, Samsung, Sennheiser and many more.

Products available for any country.

We hope you enjoy and find value in the much anticipated iTWire Shop.

ENTER THE SHOP NOW!

INTRODUCING ITWIRE TV

iTWire TV offers a unique value to the Tech Sector by providing a range of video interviews, news, views and reviews, and also provides the opportunity for vendors to promote your company and your marketing messages.

We work with you to develop the message and conduct the interview or product review in a safe and collaborative way. Unlike other Tech YouTube channels, we create a story around your message and post that on the homepage of ITWire, linking to your message.

In addition, your interview post message can be displayed in up to 7 different post displays on our the iTWire.com site to drive traffic and readers to your video content and downloads. This can be a significant Lead Generation opportunity for your business.

We also provide 3 videos in one recording/sitting if you require so that you have a series of videos to promote to your customers. Your sales team can add your emails to sales collateral and to the footer of their sales and marketing emails.

See the latest in Tech News, Views, Interviews, Reviews, Product Promos and Events. Plus funny videos from our readers and customers.

SEE WHAT'S ON ITWIRE TV NOW!

BACK TO HOME PAGE
David Heath

David Heath has had a long and varied career in the IT industry having worked as a Pre-sales Network Engineer (remember Novell NetWare?), General Manager of IT&T for the TV Shopping Network, as a Technical manager in the Biometrics industry, and as a Technical Trainer and Instructional Designer in the industrial control sector. In all aspects, security has been a driving focus. Throughout his career, David has sought to inform and educate people and has done that through his writings and in more formal educational environments.

Share News tips for the iTWire Journalists? Your tip will be anonymous

WEBINARS ONLINE & ON-DEMAND

GUEST ARTICLES

VENDOR NEWS

Guest Opinion

Guest Interviews

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News

Comments