Security Bloggers Network 
SBN

The Secret’s Out: How Stolen Okta Auth Tokens Led to Cloudflare Breach

Avatar photo by Thomas Segura on February 2, 2024

What Happened?

The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare Breach

Cloudflare’s internal Atlassian systems were breached using tokens and service accounts compromised from a previous Okta breach. The attackers gained access to the Confluence wiki, Jira database, and Bitbucket source code system. The incident illustrates the damaging domino effect of secrets sprawl and the importance of maintaining rigorous secrets security across the supply chain.

Cloudflare Security Breach Recap

1. On November 14th, Cloudflare’s self-hosted Atlassian server was breached by suspected nation-state hackers.

2. The hackers utilized one access token and three service account credentials stolen from Okta’s previous compromise. Cloudflare had failed to revoke these credentials after the Okta breach.

3. The breached systems included Cloudflare’s Confluence, Jira, and Bitbucket systems.

4. Attempts by the threat actors to hack into Cloudflare’s São Paulo data center – not yet in operation – were unsuccessful.

5. Threat actors established persistent access and attempted to gain deeper access to Cloudflare's global network.

6. Cloudflare detected the breach on November 23rd and cut off the attackers' access the following morning.

7. Remediation efforts included rotating all 5,000 production credentials, segmenting test and staging systems, extensive forensic triage, and rebooting of all the company's systems.

Cloudflare Hacked: The Long Story

On November 14th, reputedly a 'nation-state attacker' infiltrated Cloudflare's Atlassian server, compromising its Confluence wiki, Jira database, and Bitbucket source code management system. From the breach's onset, the hackers established relentless access to Cloudflare's Atlassian server using the ScriptRunner tool for Jira.Interestingly, the attackers used one access token and three service account credentials obtained during Okta's earlier breach in October 2023.

💡
In October 2023, Okta disclosed its support system was breached, and customer-uploaded HTTP Archive (HAR) files were accessed, including session tokens and user cookies. Okta revoked the session tokens and advised customers to sanitize these files. Both BeyondTrust and Cloudflare detected malicious activity related to this breach and were able to respond quickly. Only to realize later some access tokens had not been properly rotated.

Cloudflare's failure to rotate these compromised details was a significant oversight and a primary enabling factor of the attack.
While attempts by threat actors to infiltrate Cloudflare’s data center in São Paulo were unsuccessful, extensive damages occurred.

A thorough examination of accessed wiki pages, bug database issues, and source code repositories suggested the hackers were seeking comprehensive data on the architecture, security, and management of Cloudflare's global network.
Although Cloudflare concluded that customer data and global network systems were unaffected, the implications are severe.

"Even though we understand the operational impact of the incident to be extremely limited, we took this incident very seriously because a threat actor had used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code,"

Prince, Graham-Cumming, and Bourzikas.

This breach underscores the formidable security challenges businesses face, including the urgent need to manage secrets sprawl and detect intrusions as soon as possible to prevent similar impacts.

Lesson Learned

The Cloudflare breach vividly illustrates the actual threat of secrets sprawl – the spreading of sensitive data across various platforms in an uncontrolled manner. It shows that even businesses reputed for high security, such as Cloudflare, are not impervious to attacks if secrets are not adequately secured.
In the case of Cloudflare, threat actors navigated from one leaked secret to another, reflecting how secrets serve as pivot points in hackers' attack chains. From gaining an initial foothold into a third-party auth provider (Okta) to infiltrating a downstream target (Cloudflare), the hackers collected more secrets during reconnaissance stages.

Knowing What Secrets You Have

If you are not aware you have credentials that need rotating, they will never get remediated. GitGuardian Secret Detection Platform is built to help enterprises solve this exact issue. With over 400 enterprise-tuned detectors, you can quickly identify where any secrets reside in your code or in your Slack and (soon) Jira instances. Once you know a secret has been exposed, then you can rotate it, hopefully before a breach occurs. The platform also helps you track the remediation efforts, giving you a clear timeline of when the credential was added to your codebase or first mentioned in a Slack channel and track who is working on the fix. The platform also gives you a way to quickly recheck if a token is valid and if it has been exposed publicly on GitHub yet, allowing you to assign the proper urgency to your queue. After the incident is resolved, you are left with a record of all the steps taken and can rest assured that the threat has been mitigated.

Early Warning is Key

We commend the Cloudflare team for being transparent here and for working quickly to protect their customers. While a nine-day dwell time is shorter than the average, we would always like to know sooner than later. The best defense here, we believe, is to mislead attackers into giving themselves away in a matter of minutes after establishing a foothold through GitGuardian Honeytokens. Honeytokens are decoy credentials that can be deployed at scale throughout your environments. There is no legitimate use for them, but when someone tries to use one, alarms go off, and you get the IP address, user agent, and actions the user, or more likely the scripted tool, was attempting to perform. In this case, as the threat actors moved through various systems, there is a very high likelihood they would have attempted to exploit any secrets found, making themselves known back in step 3 instead of step 6, using the stages listed in the summary.

Be Safe Out There

This incident has underlined the severe dangers of underestimating the threats secrets sprawl can pose.
It serves as a stark reminder that cybersecurity is a shared responsibility stretching across the supply chain. In order to truly secure our digital assets, there needs to be stringent control and management of secrets and prompt updates or rotations whenever a breach occurs.

More Breach Explained

Sumo Logic Breach Shows Leaked Credentials Still a Persistent Threat
Sumo Logic reported a security breach on November 3, 2023, due to a compromised credential that allowed unauthorized AWS account access.
The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare BreachGitGuardian Blog – Automated Secrets DetectionMackenzie Jackson
The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare Breach

Three Recent Examples of Why You Need to Know How Vulnerable Your Secrets Are
In today’s digital landscape, the issue of compromised credentials has become a major concern. Discover how renowned companies like Microsoft, VMware, and Sourcegraph were recently confronted with the threats of secrets sprawling.
The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare BreachGitGuardian Blog – Automated Secrets DetectionCarole Winqwist
The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare Breach

Toyota Suffered a Data Breach by Accidentally Exposing A Secret Key Publicly On GitHub
On October 7th, Toyota revealed a partial copy of their T-Connect source code had been accidentally exposed for 5 years, including access to data for over 290,000 customers.
The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare BreachGitGuardian Blog – Automated Secrets DetectionDwayne McDaniel
The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare Breach

Microsoft AI involuntarily exposed a secret giving access to 38TB of confidential data for 3 years
Discover how an overprovisioned SAS token exposed a massive 38TB trove of private data on GitHub for nearly three years. Learn about the misconfiguration, security risks, and mitigation strategies to protect your sensitive assets.
The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare BreachGitGuardian Blog – Automated Secrets DetectionThomas Segura
The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare Breach

*** This is a Security Bloggers Network syndicated blog from GitGuardian Blog - Automated Secrets Detection authored by Thomas Segura. Read the original post at: https://blog.gitguardian.com/the-secrets-out-how-stolen-auth-tokens-led-to-cloudflare-breach/

Thomas Segura
Avatar photo

Thomas Segura

What You Need to Scale AppSec Thomas Segura - Content Writer @ GitGuardian Author Bio Thomas has worked both as an analyst and as a software engineer consultant for various big French companies. His passion for tech and open source led him to join GitGuardian as technical content writer. He focuses now on clarifying the transformative changes that cybersecurity and software are going through. Website:https://www.gitguardian.com/ Twitter handle: https://twitter.com/GitGuardian Linkedin: https://www.linkedin.com/company/gitguardian Introduction Security is a dilemma for many leaders. On the one hand, it is largely recognized as an essential feature. On the other hand, it does not drive business. Of course, as we mature, security can become a business enabler. But the roadmap is unclear. With the rise of agile practices, DevOps and the cloud, development timeframes have been considerably compressed, but application security remains essentially the same. DevSecOps emerged as an answer to this dilemma. Its promise consists literally in inserting security principles, practices, and tools into the DevOps activity stream, reducing risk without compromising deliverability. Therefore there is a question that many are asking: why isn't DevSecOps already the norm? As we analyzed in our latest report DevSecOps: Protecting the Modern Software Factory, the answer can be summarized as follows: only by enabling new capacities across Dev, Sec and Ops teams can the culture be changed. This post will help provide a high-level overview of the prerequisite steps needed to scale up application security across departments and enable such capabilities. From requirements to expectations Scaling application security is a company-wide project that requires thorough thinking before an y decision is made. A first-hand requirement is to talk to product and engineering teams to understand the current global AppSec maturity. The objective at this point is to be sure to have a comprehensive understanding of how your products are made (the processes, tools, components, and stacks involved). Mapping development tools and practices will require time to have the best visibility possible. They should include product development practices and the perceived risk awareness/appetite from managers. One of your objectives would be to nudge them so they take into account security in every decision they make for their products, and maybe end up thinking like adversaries. You should be able to derive security requirements from the different perceptual risks you are going to encounter. Your job is to consolidate these into a common set for all applications, setting goals to align the different teams collaborating to build your product(s). Communicating transparently with all relevant stakeholders (CISO, technical security, product owner, and development leads) about goals and expectations is essential to create a common ground for improvement. It will be absolutely necessary to ensure alignment throughout the implementation too. Open and accessible guardrails Guardrails are the cornerstone of security requirements. Their nature and implementation are completely up to the needs of your organization and can be potentially very different from one company to the other (if starting from scratch, look no further than the OWASP Top10). What is most important, however, is that these guardrails are open to the ones that need them. A good example of this would be to centralize a common, security-approved library of open-source components that can be pulled from by any team. Keep users' accessibility and useability as a priority. Designing an AppSec program at scale requires asking “how can we build confidence and visibility with trusted tools in our ecosystem?”. For instance, control gates should never be implemented without considering a break-glass option (“what happens if the control is blocking in an emergency situation?”). State-of-the-art security is to have off-the-shelf secure solutions chosen by the developers, approved by security, and maintained by ops. This will be a big leap forward in preventing vulnerabilities from creeping into source code. It will bring security to the masses at a very low cost (low friction). But to truly scale application security, it would be silly not to use the software engineer's best ally: the continuous integration pipeline. Embed controls in the CI/CD AppSec testing across all development pipelines is the implementation step. If your organization has multiple development teams, it is very likely that different CI/CD pipelines configurations exist in parallel. They may use different tools, or simply define different steps in the build process. This is not a problem per se, but to scale application security, centralization and harmonization are needed. As illustrated in the following example CI/CD pipeline, you can have a lot of security control steps: secrets detection, SAST, artifact signing, access controls, but also container or Infrastructure as Code scanning (not shown in the example) (taken from the DevSecOps whitepaper) The idea is that you can progressively activate more and more control steps, fine-tune the existing ones and scale both horizontally and vertically your “AppSec infrastructure”, at one condition: you need to centralize metrics and controls in a stand-alone platform able to handle the load corresponding to your organization’s size. Security processes can only be automated when you have metrics and proper visibility across your development targets, otherwise, it is just more burden on the AppSec team's shoulders. In turn, metrics and visibility help drive change and provide the spark to ignite a cultural change within your organization. Security ownership shifts to every engineer involved in the delivery process, and each one is able to leverage its own deep (yet partial) knowledge of the system to support the effort. This unlocks a world of possibilities: most security flaws can be treated like regular tickets, rule sets can be optimized for each pipeline based on criticality, capabilities or regulatory compliance, and progress can be tracked (saved time, avoided vulnerabilities etc.). In simpler terms, security can finally move at the DevOps speed. Conclusion Security can’t scale if it’s siloed, and slowing down the development process is no longer an option in a world led by DevOps innovation. The design and implementation of security controls are bound to evolve. In this article, we’ve depicted a high-level overview of the steps to be considered to scale AppSec. This starts with establishing a set of security requirements that involve all the departments, in particular product-related ones. From there it becomes possible to design guardrails to make security truly accessible with a mix of hard and soft gates. By carefully selecting automated detection and remediation that provide visibility and control, you will be laying a solid foundation for a real model of shared responsibility for security. Finally, embedding checks in the CI/CD system can be rolled out in multiple phases to progressively scale your security operations. With automated feedback in place, you can start incrementally adjusting your policies. A centralized platform creates a common interface to facilitate the exchange between application security and developer teams while enforcing processes. It is a huge opportunity to automate and propagate best practices across teams. Developers are empowered to develop faster with more ownership. When security is rethought as a partnership between software-building stakeholders, a flywheel effect can take place: reduced friction leads to better communication and visibility, automating of more best practices, easing the work of each other while improving security with fewer defects. This is how application security will finally be able to scale through continuous improvement.

thomas-segura has 62 posts and counting.See all posts by thomas-segura