GitHub Hit by Record DDoS Attack Through Exposed Memcached Servers

In what might be the largest ever recorded distributed denial-of-service (DDoS) attack, GitHub was hit this week with more than 1TB of malicious traffic per second generated by hijacked Memcached servers.

DDoS mitigation providers had warned recently that a large number of Memcached servers are exposed to the internet and are being abused to launch DDoS attacks using traffic reflection and amplification techniques.

“Between 17:21 and 17:30 UTC on February 28th we identified and mitigated a significant volumetric DDoS attack,” GitHub said Thursday in a report on its engineering blog. “The attack originated from over a thousand different autonomous systems (ASNs) across tens of thousands of unique endpoints. It was an amplification attack using the memcached-based approach described above that peaked at 1.35Tbps via 126.9 million packets per second.”

Attackers can send spoofed requests to Memcached servers to trick them to send unsolicited data to a victim’s IP address. The problem is that the triggered responses can be 51,000 times larger than the requests that generate them—for every byte sent by an attacker to a Memcached server, the server could direct up to 51KB toward the end target.

According to past reports, there are over 88,000 Memcached servers exposed to the internet and accessible without authentication. These are used to cache database and API responses for web applications, but attackers can also inject their own payloads into their data stores.

“If you are using memcached, please disable UDP support if you are not using it,” researchers from Cloudflare said in a blog post. “On memcached startup you can specify –listen to listen only to localhost and -U 0 to disable UDP completely. ”

DevOps Unbound Podcast

Systems administrators should also make sure that their Memcached servers are not directly accessible from the internet. First off, they can hold sensitive data from cached queries, so they pose a privacy risk.

Fortunately, some large hosting companies and ISPs are reacting. Digital Ocean, OVH and Linode told Cloudflare that they’ve taken action to prevent Memcached servers hosted on their networks by customers from being abused.

23,000 SSL Certificates Revoked after Keys Exposed

DigiCert was forced to revoke more than 23,000 SSL certificates belonging to individuals and businesses after the reseller through which the certificates were acquired sent the company its corresponding private keys via email.

The reseller, a company called Trustico, was unhappy with its business relationship with Symantec, whose certificate authority (CA) business was recently taken over by DigiCert. As a result, Trustico tried to get its customers’ certificates revoked and reissued under Comodo, a different CA.

The decision was also driven by the fact that Google Chrome is expected to untrust all certificates issued by Symantec because of repeated certificate mis-issuance incidents at Symantec and its resellers over the years.

Trustico contacted DigiCert in early February and asked for its customers’ 50,000 certificates to be revoked, according to Jeremy Rowley, executive VP of product at DigiCert. DigiCert refused because the industry rules for CAs, known as the Baseline Requirements (BRs), only allow for a certificate to be revoked if the request came from the subscriber—the certificate owner—or if there is evidence the certificate’s private key has been compromised.

Trustico then sent the private keys for more than 23,000 certificates to DigiCert via email, effectively forcing the CA to revoke the certificates within 24 hours, as required under the BRs. Once the keys were received via email, DigiCert had to consider them compromised because the BRs state that only the certificate owner should have access to the private keys.

In a statement posted on its website, Trustico said that it recovered the private keys from cold storage as a result of DigiCert asking for them to initiate revocation. However, the question on everyone’s mind now is, why did Trustico have those keys in the first place?

“Trustico allows customers to generate a Certificate Signing Request and Private Key during the ordering process,” the company said in its statement. “These Private Keys are stored in cold storage, for the purpose of revocation.”

Many security experts and people from the industry don’t agree that’s an acceptable practice for a certificate reseller, giving that not even CAs—which issue certificates in the first place—are not allowed to retain customers’ private keys.

The incident is currently being discussed on the Mozilla security policy mailing list and on the CA/Browser Forum’s mailing list. The CA/B Forum is the organization that maintains the BRs and people are now calling for more clarity on how to handle such cases in the future.

“With the private key you can decrypt the traffic by MITM proxy the traffic (by presenting in middle with the private key) so this looks like it fundamentally broke security for a lot of orgs,” security researcher Kevin Beaumont said on Twitter. “I think there’s a can of worms here.”

“The first certificate I looked at, where DigiCert now have the private key (ie it’s compromised) was a government banking email server, the certificate protecting the encryption of their email,” Beaumont said in a different tweet. “So… Trustico had that private key, w/o even a password, for XYZ years.”

There are other situations where third parties other than the certificate owners have access to certificate private keys. For example, because the private key needs to be installed on a web server, hosting providers technically can have access to their customers’ private keys. Whether this is an acceptable practice needs to be clarified as well.

Lucian Constantin

Lucian Constantin

Lucian has been covering computer security and the hacker culture for almost a decade, his work appearing in many technology publications including PCWorld, Computerworld, Network World, CIO, CSO, Forbes and The Inquirer. He has a bachelor's degree in political science, but has been passionate about computers and cybersecurity from an early age. Before he chose a career in journalism, Lucian worked as a system and network administrator. He enjoys attending security conferences and delving into interesting research papers. You can reach him at [email protected] or @lconstantin on Twitter. For encrypted email, his PGP key's fingerprint is: 7A66 4901 5CDA 844E 8C6D 04D5 2BB4 6332 FC52 6D42

lucian-constantin has 298 posts and counting.See all posts by lucian-constantin

2 thoughts on “GitHub Hit by Record DDoS Attack Through Exposed Memcached Servers

Comments are closed.

Cloud Workload Resilience PulseMeter

Step 1 of 8

How do you define cloud resiliency for cloud workloads? (Select 3)(Required)