20220509 ITWire 160x1200 may22 1

20220509 ITWire 160x1200 may22 1

20220509 ITWire 705x108 may22 1

Thursday, 10 June 2021 03:29

The internet outage - Fastly: A customer did it (kinda) Featured

By
The internet outage - Fastly: A customer did it (kinda) Image by Christelle Olivier from Pixabay

 

Fastly's latest response to yesterday's outage puts the blame indirectly on an unnamed customer.

iTWire has already reported extensively on the outage here, here and here.

According to Nick Rockwell, Fastly's Senior Vice President of Engineering and Infrastructure, in a blog post a few hours ago, "We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change. We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration. Within 49 minutes, 95% of our network was operating as normal.

"This outage was broad and severe, and we're truly sorry for the impact to our customers and everyone who relies on them."

Rockwell continues, "On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances.

"Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors."

Unfortunately, when you're one of the Internet's leading CDN providers, 49 minutes is a VERY long time.

 

 

Rockwell's blog post offers a simplified timeline.

fastly timeline

The only part of this timeline that bothers us is in the final few lines. Over seven hours between recovery and the deployment of a bug fix.

Rockwell concludes the company's mea culpa with this:

"Where do we go from here?

"We're deploying the bug fix across our network as quickly and safely as possible.

"We are conducting a complete post mortem of the processes and practices we followed during this incident.

"We'll figure out why we didn't detect the bug during our software quality assurance and testing processes.

"We'll evaluate ways to improve our remediation time."

Of course, the internet is a complex beast, but outages like this should never happen. With luck, everyone will learn from this.  And a note to CDN customers - don't put all your delivery eggs in one basket.

Read 2146 times

Please join our community here and become a VIP.

Subscribe to ITWIRE UPDATE Newsletter here
JOIN our iTWireTV our YouTube Community here
BACK TO LATEST NEWS here

SONICWALL 2022 CYBER THREAT REPORT

The past year has seen a meteoric rise in ransomware incidents worldwide.

Over the past 12 months, SonicWall Capture Labs threat researchers have diligently tracked the meteoric rise in cyberattacks, as well as trends and activity across all threat vectors, including:

Ransomware
Cryptojacking
Encrypted threats
IoT malware
Zero-day attacks and more

These exclusive findings are now available via the 2022 SonicWall Cyber Threat Report, which ensures SMBs, government agencies, enterprises and other organizations have the actionable threat intelligence needed to combat the rising tide of cybercrime.

Click the button below to get the report.

GET REPORT!

PROMOTE YOUR WEBINAR ON ITWIRE

It's all about Webinars.

Marketing budgets are now focused on Webinars combined with Lead Generation.

If you wish to promote a Webinar we recommend at least a 3 to 4 week campaign prior to your event.

The iTWire campaign will include extensive adverts on our News Site itwire.com and prominent Newsletter promotion https://itwire.com/itwire-update.html and Promotional News & Editorial. Plus a video interview of the key speaker on iTWire TV https://www.youtube.com/c/iTWireTV/videos which will be used in Promotional Posts on the iTWire Home Page.

Now we are coming out of Lockdown iTWire will be focussed to assisting with your webinars and campaigns and assistance via part payments and extended terms, a Webinar Business Booster Pack and other supportive programs. We can also create your adverts and written content plus coordinate your video interview.

We look forward to discussing your campaign goals with you. Please click the button below.

MORE INFO HERE!

BACK TO HOME PAGE
David Heath

David Heath has had a long and varied career in the IT industry having worked as a Pre-sales Network Engineer (remember Novell NetWare?), General Manager of IT&T for the TV Shopping Network, as a Technical manager in the Biometrics industry, and as a Technical Trainer and Instructional Designer in the industrial control sector. In all aspects, security has been a driving focus. Throughout his career, David has sought to inform and educate people and has done that through his writings and in more formal educational environments.

Share News tips for the iTWire Journalists? Your tip will be anonymous

WEBINARS ONLINE & ON-DEMAND

GUEST ARTICLES

VENDOR NEWS

Guest Opinion

Guest Interviews

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News

Comments