Security Market Segment LS
Tuesday, 25 June 2019 08:15

Verizon route leak stuffs up many sites worldwide Featured

Verizon route leak stuffs up many sites worldwide Pixabay

Content delivery services provider Cloudflare has blamed US telco Verizon for a large-scale border gateway protocol leak on Monday night (Australian time) that caused many large sites to be inaccessible.

Many of these routes instead transited through DQE Communications, a small company in Pennsylvania.

Cloudflare's Tom Strickx said in a blog post that the problem had been magnified by the involvement of a so-called BGP optimiser product from a company known as Noction. The problems began at about 10:30 UTC (8.30pm Monday night AEST) and were sorted two hours later.

"This was the equivalent of Waze routing an entire freeway down a neighbourhood street — resulting in many websites on Cloudflare, and many other providers, to be unavailable from large parts of the Internet," Strickx said. "This should never have happened because Verizon should never have forwarded those routes to the rest of the Internet."

Explaining what this optimiser did, Strickx wrote: "This product has a feature that splits up received IP prefixes into smaller, contributing parts (called more-specifics).

"For example, our own IPv4 route was turned into and It’s as if the road sign directing traffic to 'Pennsylvania' was replaced by two road signs, one for 'Pittsburgh, PA' and one for 'Philadelphia, PA'.

"By splitting these major IP blocks into smaller parts, a network has a mechanism to steer traffic within their network, but that split should never have been announced to the world at large. When it was, it caused today’s outage."

DQE was using a BGP optimiser and announced such specific routes to its customer Allegheny Technologies; this information was sent to its other transit provider, which happened to be Verizon, Strickx said.

Verizon advertised these "better" routes to the entire Internet; they were "better" because they were more granular and more specific.

"The leak should have stopped at Verizon. However, against numerous best practices outlined below, Verizon’s lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Fastly, Linode and Cloudflare," Strickx said.

More than eight hours after the incident, Strickx said Cloudflare had not heard back from Verizon despite trying to make contact both through phone calls and email.

Launtel chief executive Damian Ivereigh told iTWire that Australian sites were not affected a great deal by the incident.

"From what I can tell we were not affected very much, I was not personally online at the time and I haven't seen any posts in our Facebook Users Group complaining of slow speeds etc," the head of the small Launceston-based ISP said in response to an inquiry.

He said: "Given the anatomy of the leak - namely a route in the US being badly optimised by a small provider in the US - it would make sense that we would not be affected much.

"Cloudflare uses different addresses for different areas of the world. The optimiser would have only been manipulating routes for destinations close to it (because it only wants to optimise routes to high-traffic areas - people in the US don't send much traffic to Cloudflare in Oz).

"Thus it would have only been the US cloudflare servers whose routes would have been affected. Given that Australian users don't often talk to US cloudflare servers (they talk to local ones), it would not have affected us much."

iTWire has contacted Verizon for comment.

Disclosure: iTWire uses Cloudflare's services.


You cannot afford to miss this Dell Webinar.

With Windows 7 support ending 14th January 2020, its time to start looking at your options.

This can have significant impacts on your organisation but also presents organisations with an opportunity to fundamentally rethink the way users work.

The Details

When: Thursday, September 26, 2019
Presenter: Dell Technologies
Location: Your Computer


QLD, VIC, NSW, ACT & TAS: 11:00 am
SA, NT: 10:30 am
WA: 9:00 am NZ: 1:00 pm

Register and find out all the details you need to know below.



iTWire can help you promote your company, services, and products.


Advertise on the iTWire News Site / Website

Advertise in the iTWire UPDATE / Newsletter

Promote your message via iTWire Sponsored Content/News

Guest Opinion for Home Page exposure

Contact Andrew on 0412 390 000 or email [email protected]


Sam Varghese

website statistics

Sam Varghese has been writing for iTWire since 2006, a year after the site came into existence. For nearly a decade thereafter, he wrote mostly about free and open source software, based on his own use of this genre of software. Since May 2016, he has been writing across many areas of technology. He has been a journalist for nearly 40 years in India (Indian Express and Deccan Herald), the UAE (Khaleej Times) and Australia (Daily Commercial News (now defunct) and The Age). His personal blog is titled Irregular Expression.



Recent Comments