The Day the Music Died: Certificate Expiration Takes Down Spotify
“Something expired deep inside
The day the music died”
Earlier this week, the music streaming service Spotify went down for about an hour. (We “heard it from a friend who heard it from another”…on Twitter.) All signs point towards a certificate expiration being the root cause of the downtime. So, what happened to take Spotify offline? And what does an hour of downtime add up to for Spotify?
Let’s hash it out.
Spotify Fans Cry A River on Twitter
As usually happens when a popular internet services goes down, many people “Heard It Through the Grapevine” on Twitter. Starting around 8AM EST on August 19th Spotify users started posting on Twitter saying they couldn’t access the service:
As always, many of the Tweets were pretty funny. (Hey, if there’s no music…you may as well enjoy some Twitter comedians!)
Of course, a few fans of competing services took the opportunity to throw some shade at Spotify:
In short, Twitter lived up to its reputation as the best downtime monitor in the world – up to the minute status information and as an added bonus, you might get a laugh from all the salty posts.
About an hour and a half later, Spotify announced that everything was back up and running normally:
What Caused Spotify to Go Down?
Spotify doesn’t appear to have made an official announcement explaining the technical details of what happened. But once again, Twitter comes to the rescue with the details. Louis Poinsignon, a Network Engineer at Cloudflare, seems to have sleuthed out the issue:
The certificate expired at 12 PM GMT, which was 8AM US Eastern Time, just a few minutes before the tweets from music-deprived users started rolling in.
How Big of a Deal was this Certificate Expiration?
Here at Hashed Out, we’ve occasionally highlighted the consequences of letting a certificate expire. Here are a few of the incidents we’ve examined before:
- US Government shutdown causes dozens of sites to go down due to SSL certificate expirations
- Ericsson lets certificate expire, 32 million people lose cellular service
- Equifax misses a breach for 76 days because of an expired certificate
- Users experience VPN issues after Cisco lets one of its SSL certificates expire
- Pokemon Go goes down after a certificate expires
Check out our article “What happens when your SSL certificate expires?” for the full stories.
Now, let me be clear—we’re not bashing these organizations. Not at all. The reality is that large organizations have 10,000’s of certificates, and keeping track of all of those expiration dates is a gargantuan task.
But here’s the harsh reality: letting even a single certificate expire can have a huge impact.
How much trouble can a single certificate expiration cause? One easy way to very roughly estimate the cost of a certificate expiration is to look at how much revenue the company would typically make in that time period.
Spotify’s revenue in 2019 was $7.44 billion, which equates to $20,383,561 per day, or about $1,273,9726 in an hour-and-a-half.
Now, most of Spotify’s revenue comes from subscriptions, so their revenue didn’t literally drop to $0 for that hour and-a-half. But direct revenue losses are only part of the cost—certificate-related downtime can be costly in a variety of ways:
- Direct revenue losses
- Reduced new customer acquisition
- Increased customer support time/costs
- Customer churn (existing customers go elsewhere)
- Damage to brand reputation
- Potential compliance issues
Another way to calculate the cost of certificate-related downtime is to ask: how many users were impacted?
According to BusinessofApps, Spotify has about 286 million monthly active users, and they listen for an average of 25 hours per month. That means that any given time, there are approximately 9,781,200 users “all in the mood for a melody” and listening on Spotify.
It doesn’t appear that Spotify was down for all users in all locations (some users reported they were able to listen with the desktop app but not the mobile app, while others reported the opposite) but it definitely looks like there were millions of Spotify users who were NOT “feelin’ alright” while the service was down.
Implementing the Right Certificate Management Practices is Key
We don’t know how Spotify’s certificate expired without being caught. Was the certificate not being monitored? Did some wires get crossed? Either way, the bottom line is: certificate management is challenging.
As organizations require more and more digital certificates for various types of use cases (SSL/TLS, device, code signing, S/MIME, etc.) implementing effective certificate management practices has become absolutely critical.
Don’t make the mistake of managing your certificates manually. Certificate management methods such as spreadsheets and calendar reminders might be OK if you just have one or two certificates, but they’re far too error prone for organizations with many certificates.
The first step to minimize the chance of certificate-related downtime is to automate the most critical certificate management functions. Specifically, organizations should implement a certificate management tool with the following features:
- Automated certificate discovery (with public and private scanners)
- Automated expiration notifications to responsible parties and organizational admins
- Automated notification escalations for imminent expirations
- Where possible, automated certificate renewal and installation
- Automated checks and notifications for security vulnerabilities (e.g. POODLE)
- Automated approval flows for staff to request certificates through official channels (to discourage shadow certificates installed without the organization’s knowledge)
This screenshot from DigiCert’s CertCentral management platform is a great example showing how thousands of certificates can be summarized in a single screen, making it easy for IT admins to identify and update any expiring certificates:
Of course, a dashboard like this is only useful if it actually includes all of the certificates in your organization. And that’s why an automated certificate discovery feature is also very important—it finds all your certificates and loads them into the dashboard so you can see and manage them in one convenient place.
“Stayin’ Alive”…With a Little Preparation
But these certificate management features only work if your organization has correctly implemented them. The best certificate management tool in the world won’t save you if the discovery feature isn’t setup correctly or the notifications are sent to the wrong person.
The old ways that used to work aren’t good enough anymore. To succeed in the age of “always on” encryption, certificate management needs to be prioritized and “baked in” to your existing IT and cybersecurity workflows. For many organizations, the certificates themselves aren’t the most important part—the critical part is the management of the certificates. That’s why DigiCert now ships CertCentral with their certificates—but, it’s still up to your organization to implement the certificate management features and processes to avoid situations like this.
Summary
Certificate-related downtime, like the incident Spotify experienced earlier this week, has unfortunately become more and more common these days. Organizations are accumulating more certificates than ever, and the task of managing them all while staying ahead of expirations has become a significant challenge. Fortunately, effective certificate management practices can be a huge help in easing this headache, making it easier for your company to avoid costly downtime. The result? Happier customers and more revenue.
*** This is a Security Bloggers Network syndicated blog from Hashed Out by The SSL Store™ authored by Adam Thompson. Read the original post at: https://www.thesslstore.com/blog/the-day-the-music-died-certificate-expiration-takes-down-spotify/
