In a message posted about an hour ago [4.37am AEDT], a Twitter user mentioned the nuking of the records, adding sarcastically, "And suddenly millions of retired boomers are having to speak to their spouses for the first time in years."
Unable to post a message on its own media statements site due to the outage, Facebook said in a tweet: "We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience." Update: Facebook begins to come back
Now, here's the fun part. @Cloudflare runs a free DNS resolver, 1.1.1.1, and lots of people use it. So Facebook etc. are down... guess what happens? People keep retrying. Software keeps retrying. We get hit by a massive flood of DNS traffic asking for https://t.co/qq6U47Tjc6
— John Graham-Cumming (@jgrahamc) October 4, 2021
About two hours ago, Cloudflare vice-president Dane Knecht posted a tweet about the outage, saying: " @Facebook DNS and other services are down. It appears their BGP [border gateway protocol] routes have been withdrawn from the Internet. @ Cloudflare 1.1.1.1 started seeing high failure in last 20mins."
|
John Graham-Cumming, the chief technology officer of Cloudflare, said: "About five minutes before Facebook's DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook's ASN."
Every ISP advertises the routes for which it is authorised to carry traffic. When it comes to inter-carrier routing, carriers often need to send traffic to each other.
BGP is the protocol through which this is managed, allowing each carrier to broadcast what IP address ranges or prefixes should be sent to them.
Between 15:50 UTC and 15:52 UTC Facebook and related properties disappeared from the Internet in a flurry of BGP updates. This is what it looked like to @Cloudflare. pic.twitter.com/PFw5FR2W5j
— John Graham-Cumming (@jgrahamc) October 4, 2021
But since the security of the protocol is not very good, one carrier can announce incorrect prefixes; this effectively means taking over the address ranges of another provider and taking them down.
It means that a carrier can switch off a number of other providers if they wished to do so.
Graham-Cumming added: "Between 15:50 UTC [2.30am AEDT] and 15:52 UTC Facebook and related properties disappeared from the Internet in a flurry of BGP updates."
Scenes at Twitter HQ seeing that Facebook, Instagram and WhatsApp are all down. pic.twitter.com/uUYNHbiLmh
— Sam Inkersole (@Sam_InkersoleTM) October 4, 2021
And he added: "And so, Facebook etc. are down, and teams @Cloudflare have to get spun up to make sure things keep running smoothly during the onslaught.
"Good reminder that the Internet is a network of networks that works through standards and co-operation."
British security expert Kevin Beaumont said in a tweet: "Also heard this. Facebook have lost their LAN/WAN due to networking woes so there are a ton of knock on impacts.
This Monday today as we are not glued to a network that tracks us and stuffs us with ads, we can choose to be social instead.
— Mårten Mickos (@martenmickos) October 4, 2021
"It’s basically a core network failure for them (which I’ve been through at other companies, not pretty, will take hours for recovery)."
The last time there were issues caused by BGP mistakes was in April last year, when Russian Internet service provider Rostelecom advertised routes through BGP which belong to big Internet players like Google, Facebook, Akamai, Cloudflare, Hetzner, Digital Ocean and Amazon AWS.
So, someone deleted large sections of the routing....that doesn't mean Facebook is just down, from the looks of it....that means Facebook is GONE. pic.twitter.com/OCZWPD2okw
— The Academy (@BenjaminEnfield) October 4, 2021
The advertising of these routes went on for almost an hour, according to a blog post at the time by Andree Toonk, the founder of BGPmon, a service that is owned by Cisco and monitors BGP routing information in real-time. It affected 8870 network prefixes belonging to almost 200 autonomous systems.
This sounds plausible... Operators' worst nightmare...https://t.co/BqdpZ5xw8N pic.twitter.com/y6LjegHZpp
— Andree Toonk, Adelante! (@atoonk) October 4, 2021
Beaumont said the BGP configuration snafu appeared to be an "epic" one. "This one looks like a pretty epic configuration error, Facebook basically don't exist on the internet right now," he wrote in a tweet.
"Even their authoritative name server ranges have been BGP withdrawn."
This is a good time to reconsider giving yet more usage information to data aggregators like Facebook by using their sign in with feature.
— Chris Wysopal (@WeldPond) October 4, 2021