CircleCI Rotates GitHub OAuth Tokens After Security Incident
Following a security incident, CircleCI has completed the process of rotating GitHub OAuth tokens for their customers.
CircleCI said Saturday that while customers could still rotate their own tokens, it has “confidence in the security of the CircleCI platform, and customers can continue to build.”
The platform first sent out an alert on January 4, 2023, saying it was investigating an incident—without providing further details. While the company said it was “confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well,” CircleCI urged customers to “immediately rotate any and all secrets stored in CircleCI … in project environment variables or in contexts” and recommended they “review internal logs for their systems for any unauthorized access” from December 21, 2022, to January 4, 2023, or once secrets rotation had been completed.
“Organizations should take this seriously. Separation of privilege is ‘future work’ for a lot of platform teams, and executives determining the blast radius of this breach will likely find it includes production access,” said John Steven, CTO at ThreatModeler. “When combined with access to assets like configuration and source code, this kind of breach equates to ‘access to prod and a map of what to do when you get there.’”
Attackers are increasingly targeting CI/CD and software development tool vendors. CircleCI is the latest vendor to make the news after LastPass and Slack, to name just two. Why? Because these tools hold the secrets that enable a wide range of abuse scenarios, said Aakash Shah, co-founder and CTO at oak9.
“They have tokens that give access to code repositories. We have recently seen both Okta and Slack fall victim when attackers got access to their code repositories. We have seen IDE compromises for tools like Visual Studio Code in the wild,” said Shah. “If the attacker can make changes to the code, this allows them to introduce vulnerabilities in widely used enterprise software.”
Of course, compromise of a CI/CD platform “is extremely concerning, as it can have far-reaching impacts. First, there is concern about the potential loss of secrets and keys, which is a singularly serious concern,” said Patrick Tiquet, vice president, security and architecture, at Keeper Security.
“While the investigation is ongoing and full details of this breach are still forthcoming, there is also the possibility an intruder could have obtained write access or been able to make code or configuration changes,” said Tiquet. “If this were the case, this could have a grave and lasting impact on customers.”
Nick Rago, field CTO at Salt Security, expressed concern about the lack of information about the nature of the incident. “For CircleCI to not just recommend that customers update their tokens but proactively expire and remove API tokens from their systems on behalf of customers leads me to believe that something substantial occurred,” he said.
“It is a little unsettling for customers to blindly take action without knowing details about what the security incident was, the extent of it, and what CirleCI has done to mitigate any future risks,” said Rago. “Certainly, this type of disruption and the added work of hunting down secrets stored in CircleCI and resetting API keys to restore critical automated processes is not how CircleCI customers envisioned they would be starting off their new year.”
But, despite the vagueness surrounding the incident, Scott Gerlach, co-founder and CSO at StackHawk, praised CircleCI for its communications with customers.
“When it comes to security, there are two sides to the coin: Prevention/detection and response. How you react to a breach is just as crucial as putting tooling and processes in place to prevent and detect one,” said Gerlach. “Kudos and #HugOps to the team working to remediate the issue and help customers identify what they can do to keep their infrastructure and secrets safe. Good customer communication post-incident goes far, and the timely response from CircleCI to its users is exactly what you want to see.”
There are a number of steps Circle CI customers should take. Shah recommends:
- Rotate your keys/secrets/tokens as per the company’s guidance
- Ensure that you have a well-designed secrets management solution to protect your secrets and help you react quickly in a scenario like this
- Consider the security design of your entire pipeline and development workflow and model the threats–where do secrets live, what access do they have, what can an attacker do if a secret is compromised? Build off of industry best practice patterns (e.g., security reference architectures)
- Mature your access management capabilities to ensure minimum necessary just-in-time access
- Require MFA for all user accounts and privileged activities
- Ensure that you have visibility into all privileged activities in the pipeline so that you can detect and react to such incidents
