Home » Security Bloggers Network » Let’s Encrypt to Revoke 3 Million SSL Certificates on March 4

Let’s Encrypt to Revoke 3 Million SSL Certificates on March 4

by Casey Crane on March 4, 2020

The world’s leading free SSL provider announces that millions of
certificates are being revoked due to a bug they discovered days ago — giving subscribers
potentially only hours to respond

Let’s Encrypt, the world’s biggest free SSL certificate authority (CA), announced to subscribers today (March 3) that they discovered a bug that’s causing them to revoke more than 3 million SSL/TLS certificates by tomorrow, March 4 (at 00:00 UTC at the earliest). The trouble? Their announcement barely gives their users time to react.

Due to the short revocation timeline that’s stipulated by
the CA/B Forum’s baseline requirements, it means that Let’s Encrypt had to rush
to inform users about the revocation that’ll be completed in less than 24
hours. That means, unfortunately for LE certificate subscribers — people like
you, possibly — that your certificates may be affected and you may not know it.

But why do they need to revoke these certificates at all? What
does this mean for Let’s Encrypt SSL subscribers? And what should you do if
you’re one of those whose certificates are affected?

Let’s hash it out.

What we’re hashing out…

Sponsorships Available

What’s the Situation with Let’s Encrypt’s Mass Revocation?

Graphic: security issue errors for Let's Encrypt certificates in cPanel

On Feb. 29, Let’s Encrypt discovered a bug in their code was
allowing the issuance of SSL/TLS certificates that didn’t go through proper
domain record checks. This is resulting in a mass revocation of 3,048,289 valid
SSL certificates out of their 116 million. That’s 2.6% of all of their existing
certificates!

So, if you don’t renew your affected SSL/TLS certificate(s)
before the March 4 revocation deadline, you’re going to find yourself in hot
water. If Let’s Encrypt revokes your cert before you have a chance to renew,
your website users will see ugly security warnings that may drive them away
from your website. These messages will continue to display until you renew your
certificate!

Jacob Hoffman-Andrews, lead developer for Let’s Encrypt, posted on Mozilla’s Bugzilla web forum to explain the issue more in depth:

On 2020-02-29 UTC, Let’s Encrypt found a bug in our CAA code. Our CA software, Boulder, checks for CAA records at the same time it validates a subscriber’s control of a domain name. Most subscribers issue a certificate immediately after domain control validation, but we consider a validation good for 30 days. That means in some cases we need to check CAA records a second time, just before issuance. Specifically, we have to check CAA within 8 hours prior to issuance (per BRs §3.2.2.8), so any domain name that was validated more than 8 hours ago requires rechecking.
The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt.”

So, what does all of this mean? What we’re talking about here
relates to checks of certificate authority authorization (CAA) records.

Why CAA Records Matter in the Certificate Issuance Process

A CAA record is a resource that was created to help prevent fraudulent SSL/TLS certificates from being issued for any domain and to help strengthen the PKI ecosystem. It’s a DNS record on the domain that every issuing CA is required to check. It’s an easy way for a CA to know whether they’re authorized to issue a certificate for a domain:

If a CAA record exists, it means that only the CA(s)
listed can issue certificates for that specific domain.
If a CAA record doesn’t exist, it means that any
CA can issue a certificate for the domain in question.

So, whenever a certificate authority issues an SSL/TLS certificate, they’re required to follow specific steps outlined in the CA/Browser Forum’s baseline requirements (BR) documentation. The CA/B Forum is an industry body that consists of CAs, browsers, and device manufacturers. They’re responsible for outlining and enforcing requirements relating to the industry. BR §3.2.2.8 (CAA Records) stipulates the following:

As part of the issuance process, the CA MUST check for CAA records and follow the processing instructions found, for each dNSName in the subjectAltName extension of the certificate to be issued, as specified in RFC 6844 as amended by Errata 5065 (Appendix A). If the CA issues, they MUST do so within the TTL of the CAA record, or 8 hours, whichever is greater.
This stipulation does not prevent the CA from checking CAA records at any other time.”

This essentially means that Let’s Encrypt was required to
check the CAA records within 8 hours prior to the certificate being issued. If
they don’t meet these requirements, whether because of the bug or for any other
reason, they must mass revoke any certificates that weren’t issued with proper
CAA checks.

Let’s Encrypt Appeals for Exemption

Bearing all of this in mind, Let’s Encrypt has found itself in the unenviable position of needing to revoke millions of certificates. So, they informed users but first tried to request a certificate revocation exemption with one of the major web browsers, Mozilla Firefox.

However, Mozilla pointed them to their docs for guidance for what actions CAs, like Let’s Encrypt, should take with regard to certificate mis-issuances:

The CA should cease issuance from the affected
portion of the public key infrastructure (PKI) to properly diagnose the cause
of the issue; or
The CA should explain why they have chosen to
not do so.

Furthermore, any affected CA that decides to restart
reissuance once the cause of their problems is diagnosed must have a process in
place that prevents additional mis-issuances.

Mozilla also offers guidance for situations requiring certificate
revocation:

Mozilla recognizes that in some exceptional circumstances, revoking misissued certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web. However, Mozilla does not grant exceptions to the BR revocation requirements. It is our position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of BR section 4.9.1 outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement.”

It may cause significant harm? Yeah, that’s quite the
understatement. However, revoking certificates that aren’t properly issued is
an absolute must for any CA. So, Let’s Encrypt is doing the right thing by
revoking the certificates. But that doesn’t mean this response isn’t causing
issues for subscribers or indicative of larger issues.

Manage Digital Certificates like a Boss

14 Certificate Management Best Practices to keep your organization running, secure and fully-compliant.

Download the Checklist

Why a Let’s Encrypt Revocation is Such a Big Deal

Okay, so virtually every CA has found themselves having to
revoke SSL/TLS certificates at one time or another. But why is this particular
situation causing such a stir?

Not Everyone Knows That They May Be Affected

Let’s Encrypt says that they emailed “affected
subscribers for whom we have contact information.” However, because
Let’s Encrypt doesn’t have contact information for all of their customers (you
know, seeing as how they’re basically just a domain validation SSL certificate
provider), it means that they may not reach people in time for them to update
their certificates before the revocation becomes effective.

They did, however, post on the Let’s Encrypt forum a link to a tool you can use to see if you’re using an affected certificate. But this doesn’t help if you don’t receive a notification and you don’t check their forums daily for news on the off chance there might be a sudden revocation announced.

This is indicative of a larger issue of visibility for Let’s Encrypt users — one that can be very costly issue for affected organizations. Considering that KeyFactor and the Ponemon Institute estimate that unplanned certificate outages due to expired certificates can cost more than $11 million, imagine how costly it can be for organizations that don’t know that they’re experiencing outages for reasons beyond their control!

LE Rate Limits Hamper Users

There was a lot of chatter on the Let’s Encrypt forum about rate limits. But what does that mean? According to their documentation:

Let’s Encrypt provides rate limits to ensure fair usage by as many people as possible. We believe these rate limits are high enough to work for most people by default. We’ve also designed them so renewing a certificate almost never hits a rate limit, and so that large organizations can gradually increase the number of certificates they can issue without requiring intervention from Let’s Encrypt.”

Basically, Let’s Encrypt has several types of rate limits:

New Certificate Orders (for ACME clients):
300 per account, per 3 hours.
Pending Authorizations: 300 per account.
Certificates Per Registered Domain: 50
per week.
Names Per Certificate: 100.
Duplicate Certificate: 5 per week.
Failed Validation: 5 failures per
account, per hostname, per hour.

Surely the rate limits reset in instances of revocation,
though, right? Apparently not. The document goes on to state that “Revoking
certificates does not reset rate limits, because the resources used to
issue those certificates have already been consumed.”

So, basically, this means that LE certificate holders who
need to renew their certificates found themselves facing rate limitations of
300 new orders per account every three hours. A silver lining, though, is that Let’s
Encrypt has agreed to temporarily:

double the duplicate certificate limit;
override the global rate limit from 300 per three
hours to 10,000 per three hours; and
increase the invalid authorizations per account
rate limit from 5 to 10.

LE Subscribers Have Limited Visibility and Support

If you’re going to issue a mass revocation, you’d ideally
want to give your customers adequate time to respond and give them plenty of
resources to help them do so. But seeing how Let’s Encrypt is a free CA, all
people really have to turn to for help is a bunch of digital documents and web
forum conversations. Not very helpful — particularly when you’re on such a
tight deadline!

What You Should Do If You’re Using Any Let’s Encrypt Certificates

So, if you’re one of the unlucky people who are now on the
brink of having your Let’s Encrypt certificate(s) revoked, we’re here to help.
We’re not just going to tell you to read an insanely long series of web forum
posts, nor are we going to force you to search a community forum… We’ll
actually walk you through the steps here.

Step One: Check Your Certificates

Not sure if your certificates are affected — and if so, which ones are specifically? You can check your certificate serial numbers to see if any match Let’s Encrypt’s list of affected certificates. You’ll want to download the list of affected certificates from the link above and search for lines that start with account IDs.

Otherwise, you can use this handy online tool or run a command in your interface (if you’re using Linux or a BSD-like system):

openssl s_client -connect example.com:443 -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

Renew Your Certificates

Once you’ve determined that your certificates are, in fact,
affected, the next step is to renew your certificates. If you were using
commercial SSL/TLS certificates with a longer lifespan than 90 days, it would
make more sense to reissue your certificate. However, since this is affecting Let’s
Encrypt free SSL/TLS certificates with significantly shorter validity, then it
makes sense to simply renew your certificates.

If you’re using an ACME client to renew your certs, you’ll
need to refer to its specific documentation relating to the SSL/TLS certificate
renewal process.

If you’re using Certbot, try using the following command:

certbot renew --force-renewal

Lastly, if you’re using cPanel to manage your Let’s Encrypt
certificate(s), you can renew yours there as well.

When we tested, we were able to issue a new SSL certificate.

Final Thoughts

Every major CA has had instances where they’ve needed to
revoke certificates. And it’s never a fun process. But as this incident
highlights, revocations can cause significantly more problems when they involve
a free CA. The lack of contact details and support capabilities make it harder
to ensure customers can re-issue certificates quickly to avoid downtime.