Russia Hijacks Traffic of Huge Cloud and CDN Services
Google, Amazon, Facebook and 200 other services had their internet traffic routed through Russia on Wednesday, April 1. Yet again, the internet got bitten by a bogus BGP routing announcement.
It was no April fool, but was it malicious? Some say yes (others disagree).
It looks like a fat-finger error, but perhaps that’s what they want you to think. In today’s SB Blogwatch, we give Vlad the benefit of the doubt.
Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: scambaiting again.
BGP FAIL
What’s the craic, Catalin? Domnule Cimpanu reports—“Russian telco hijacks internet traffic”:
Traffic meant for more than 200 of the world’s largest content delivery networks (CDNs) and cloud hosting providers was suspiciously redirected through Rostelecom, Russia’s state-owned telecommunications provider. … Impacted companies [include] Google, Amazon, Facebook, Akamai, Cloudflare, GoDaddy, Digital Ocean, Joyent, LeaseWeb, Hetzner, and Linode.
…
BGP hijacks have been an issue for the internet backbone since the mid-90s, and efforts to bolster the BGP protocol’s security have been underway for years, with projects like ROV, RPKI, and — more recently — MANRS. … Yet progress on adopting these new protocols has been slow.
…
The entire system is extremely brittle because any of the participant networks can simply “lie” and publish an announcement (BGP route) claiming that “Facebook’s servers” are on their network, and all internet entities will take it as legitimate and send all the Facebook traffic to the hijacker’s servers.
…
Not all BGP hijacks are malicious [but] some entities continue to be behind BGP hijacks on a regular basis … incidents that many experts are labeling as suspicious. … The last major Rostelecom hijack … happened in 2017 when the telco hijacked BGP routes for some of the world’s largest financial entities, including Visa, Mastercard, HSBC. … It appeared to impact only financial services, rather than random ASNs.
Pass the vodka. Cohen Coberly adds—“Russian telecom accused of ‘hijacking’ … traffic”:
Russia has been blamed for plenty of cyberattacks in the past, and we’re seeing another example of that now. … BGP hijacking is the “malicious rerouting” of internet traffic that exploits the “trusting nature” of the internet’s Border Gateway Protocol.
…
Some skepticism is certainly warranted here, and it’s no secret that Russia has stepped up its cyber warfare game over the past few years.
Who spotted it? Andree Toonk—“This is how you deal with route leaks”:
For approximately an hour, starting at 19:28 UTC on April 1, 2020, the largest Russian ISP — Rostelecom (AS12389) — was announcing prefixes belonging to … Akamai, Cloudflare, Hetzner, Digital Ocean, Amazon AWS, and other famous names. … Paths between the largest cloud networks were somewhat disrupted — the Internet blinked. … It saturated the route decision-making process for a few Tier-1 ISPs.
…
[It] illustrates how fragile the IETF-standardized BGP routing is, and especially — during such stressful times in terms of traffic growth. [But] Rostelecom got a warning from [our] real-time feed and reached out for help with the incident troubleshooting [so] the incident came to an end rather quickly, and the proper routing was restored.
…
We strongly encourage other ISPs … to start monitoring their BGP announcements to prevent incidents. … RPKI Origin Validation is something everyone should … implement.
Wait, what? You mean it wasn’t malicious? Sam Varghese calls it a “BGP routing error”:
Given the existing climate in the US as far as Russia is concerned, [Cimpanu] called the redirection of traffic “suspicious”, even though Toonk himself was unable to decide whether the hijack was deliberate.
Sharing the blame around, Aftab Siddiqui says it was “Not just another BGP Hijack”:
BGP hijacks are sadly common, but most are very short-lived and don’t create service disruptions on a global level. Most … routing incidents happen because of configuration mistakes, but … strict filtering drastically reduces the chance that these mistakes will propagate further into the network and cause additional disruption.
…
This week’s hijack unfortunately did create service disruption … as it was shared by many prominent members of the community … (4569 unique announcements). … Out of those 4569 prefixes, 4255 belong to Amazon(AS16509 and AS14618), 85 belong to Akamai (AS20940, AS16625), and the rest belong to … Level3, Alibaba, Digital Ocean, Linode, and others.
…
All of this would have been prevented if AS20764 (Rascom) implemented strict filtering. … AS174 (Cogent Co) and AS3356 (Level3) should have done a better job by filtering at all levels. … Mistakes happen, but we all have to learn from these mistakes.
…
I can’t emphasise this enough – this can happen again at any time. Network operators have a responsibility to ensure a globally robust and secure routing infrastructure.
So what went wrong at Level3 and Cogent? jlgaddis explains:
Rostelecom (AS12389) … announced the hijacked routes … to Rascom (AS20764), who accepted them and sent them on to Cogent (AS174), who accepted them and propagated them on to Level3 (AS3356), who, unfortunately, then accepted them from Cogent.
Ick. A suspicious Job Snijders performs a “Post-mortem”:
If we look at the list of ASNs which were most impacted, the top ten seems mostly anchored to the US (thus under the ARIN TAL), and almost all of them seem heavyweights in the cloud / CDN space. … Were these prefixes just unlucky? [Or] was this the result of sophisticated planning?
But what can IT people do? Mark Dargin offers a way to “Protect Your Enterprise”:
BGP was created well before security became a major concern. … Anybody can advertise or announce a route to a network without any verification.
…
The best solution requires a community approach. … In 2014, the Internet Society launched the Mutually Agreed Norms for Routing Security (MANRS) initiative with the purpose of eliminating common routing threats including BGP hijacking.
…
BGP hijacking across the globe would be significantly reduced if all ISPs followed the policies of MANRS. … Without their participation, MANRS cannot be successful.
…
If providers have enough clients requesting them to join MANRS, then more will do so. … The more enterprises insist their providers do so or at least follow similar requirements, then the fewer incidents there will be and the less impact BGP hijacking will have.
Meanwhile, this Anonymous Coward offers a “simple” solution:
Can’t we just disconnect Russia from the rest of the internet? They seem to be more trouble than they are worth. Plus, they’ll probably like it since they’d be restricted to the Russian net where the only thing to do is to exchange Borscht recipes, discuss Vodka prices and adore Putin.
And Finally:
Laryy Elison, Jeffb Bezos and Jude Law named in hilarious scam spam
You have been reading SB Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites… so you don’t have to. Hate mail may be directed to @RiCHi or [email protected]. Ask your doctor before reading. Your mileage may vary. E&OE.
Image source: DonkeyHotey (cc:by)
