Security Bloggers Network 
SBN

Zombie Leaks: Unrevoked Secrets Lurking on GitHub

Avatar photo by Thomas Segura on April 29, 2024

Zombie Leaks: Unrevoked Secrets Lurking on GitHub

In our latest State of Secrets Sprawl, we shed light on some interesting findings about secrets accidentally published to public repositories over the past year.

While the sheer number of secrets we discovered on GitHub was certainly noteworthy, what really caught our attention this year was what happened after we identified the exposed secrets. In this blog, we are going to explore the concept of zombie leaks and why they deserve more attention.

What's a Zombie Leak?

We coined this term after a startling (yet somewhat unsurprising) observation: repository owners often react to a sensitive leak by either deleting the repository or making it private, with the idea of cutting off public access to the problematic information.

The problem is that in doing so, they can create one major security gap for themselves or the organization they work for: they can create a “zombie leak.”

A zombie leak occurs when a secret is exposed but not revoked, remaining a potential attack vector. The commit author may believe that deleting the commit or repository is sufficient, overlooking the crucial revocation step.

Let's be clear from the onset: the only way to mitigate the risk of a secret being leaked is to remove any permissions associated with it. That's exactly what revoking means in that context. By doing so, we render the secret as harmless as any character string.

Deleting that leak, on the other hand, has absolutely no impact on your security posture since you do not have control over the "sprawling" of that secret (yes, even if the secret was leaked within a "private" perimeter). Worse, it can eventually make the work of security specialists looking to 'patch' this kind of vulnerability harder, if not impossible, because the leak is now invisible.

Imagine this scenario: a software engineer accidentally commits the .env file containing production secrets of their company to their personal GitHub repository. Quick to realize their mistake, they make the repository private. Feeling guilty and shameful about this mistake, they are reluctant to disclose the incident to their security team and reckon that their reaction was fast enough to mitigate any risk of a compromise. Who would be able to find this needle in the haystack that is GitHub anyway?

Months later, the company's security team is conducting an audit to evaluate its exposure on GitHub. To that end, they send a security questionnaire to all the software engineers, requiring them to declare whether they have a GitHub account. Based on that, the security team then proceeds to scan all public repositories for leaked credentials. Here is the thing: they are going to miss the leaked .env since that file has now disappeared from the public part of the platform. This vulnerability has now escaped their radar unless they are using a tool able to query the global GitHub history.

On the other hand, a malicious actor could've detected the .env file and the credentials hard-coded in it (our stats show there's a more than 50% chance this file extension contains valid secrets). They have now plenty of options to exploit them to achieve their objectives.

This is a textbook example of a zombie leak: the secret is out there, but nobody (wishing you good) is actively monitoring it or taking steps to revoke it.

Zombie Leaks are Probably Very Common

GitGuardian's pro-bono service alerts repository owners when we discover a leaked secret in their public Git commits. Over the next five days, we'll perform repeated checks to determine whether those secrets remain valid.

The State of Secrets Sprawl highlighted two concerning trends: first, we found that over 91% of leaks remained valid after 5 days. Second, many repository owners deleted or privatized their repositories after notification but did not revoke the leaked secret. Randomly selecting 5,000 repositories where the secret was gone within 5 days showed that 28.2% of the repositories were still public while 71.8% had been deleted or made private.

While we can only speculate about the prevalence of zombie leaks, there is certainly evidence suggesting they're fairly widespread.

The Bottom Line: Leaked Secrets are High-Risk Vulnerabilities

When discussing secrets, it's crucial to remember that any active hard-coded secret represents a high-risk vulnerability until revoked. While this may seem obvious, it's important to prioritize this type of vulnerability when comparing it to other application risks, such as the well-known OWASP Top10 or the recently introduced OWASP Top 10 Risks for Open Source Software.

💡
Leaked secrets are a unique kind of vulnerability because the potential threat they pose has its own lifecycle, completely independent from the source hosting them.

What this means is that a leaked secret doesn't stop being a threat when:

  • the application where it is embedded stops running,
  • the code where it is hard-coded gets overwritten or made private,
  • the message where it was shared is deleted,
  • the disk where it is stored finally gets encrypted.
🚨
Again, the only way a leak stops being a high-risk vulnerability is when its permissions are revoked.

Why a high-risk vulnerability? Let's examine the industry standard for ranking the characteristics and severity of software vulnerabilities, the Common Vulnerability Scoring System (CVSS). The highest-ranking risk level is defined like that:

Critical (9.0 to 10.0): These vulnerabilities are usually easy for an attacker to exploit and may result in unauthorized access, data breaches, and system compromise or disruption. As with high-risk vulnerabilities, immediate attention and mitigation are advised. The major difference between critical and high-risk vulnerabilities is that exploitation of a critical-risk vulnerability usually results in root-level compromise of servers or infrastructure devices.

Leaked credentials that remain active fit this definition precisely and are critical vulnerabilities at best.

How to Stop Secrets from Becoming Zombies

Revoke them. It's easy to say, but not so easy to do. But it's got to be done. Here's a high-level overview of the process in three steps.

  1. Get your secrets management sorted. This could be as simple as using a .env, removing it from version control, or referring to an external secrets manager like CyberArk, AWS Secrets Manager, or HashiCorp Vault.
  2. Generate a new secret to replace the leaked ones. Test your refactored code thoroughly to make sure everything works properly with the new secrets and secrets management tools.
  3. Push the new code to production, taking the old secret(s) out of use.
  4. Revoke the leaky secret and remove all its associated permissions

For more detailed guidelines on the remediation procedure, take a look at this handy cheat sheet:

What to do if you expose a secret: How to stay calm and respond to an incident [cheat sheet included]
Learn how to respond to a secret leak incident effectively. Follow our step-by-step guide to understand the impact, rotate secrets safely, and prevent future leaks.
Zombie Leaks: Unrevoked Secrets Lurking on GitHubGitGuardian Blog – Code Security for the DevOps generationDwayne McDaniel
Zombie Leaks: Unrevoked Secrets Lurking on GitHub

If you need further information on remediating some of the most commonly leaked secrets, GitGuardian offers guides for a dozen of the most common secrets, ranging from AWS keys to SMTP credentials. And if there's one thing we cannot stress enough, it's to implement practices and procedures to discover and remediate leaked secrets before they become a lurking danger to your company and your customers.

Now that you know your zombies, better get hunting before they come back to haunt you!

*** This is a Security Bloggers Network syndicated blog from GitGuardian Blog - Code Security for the DevOps generation authored by Thomas Segura. Read the original post at: https://blog.gitguardian.com/zombie-leaks/

Thomas Segura
Avatar photo

Thomas Segura

What You Need to Scale AppSec Thomas Segura - Content Writer @ GitGuardian Author Bio Thomas has worked both as an analyst and as a software engineer consultant for various big French companies. His passion for tech and open source led him to join GitGuardian as technical content writer. He focuses now on clarifying the transformative changes that cybersecurity and software are going through. Website:https://www.gitguardian.com/ Twitter handle: https://twitter.com/GitGuardian Linkedin: https://www.linkedin.com/company/gitguardian Introduction Security is a dilemma for many leaders. On the one hand, it is largely recognized as an essential feature. On the other hand, it does not drive business. Of course, as we mature, security can become a business enabler. But the roadmap is unclear. With the rise of agile practices, DevOps and the cloud, development timeframes have been considerably compressed, but application security remains essentially the same. DevSecOps emerged as an answer to this dilemma. Its promise consists literally in inserting security principles, practices, and tools into the DevOps activity stream, reducing risk without compromising deliverability. Therefore there is a question that many are asking: why isn't DevSecOps already the norm? As we analyzed in our latest report DevSecOps: Protecting the Modern Software Factory, the answer can be summarized as follows: only by enabling new capacities across Dev, Sec and Ops teams can the culture be changed. This post will help provide a high-level overview of the prerequisite steps needed to scale up application security across departments and enable such capabilities. From requirements to expectations Scaling application security is a company-wide project that requires thorough thinking before an y decision is made. A first-hand requirement is to talk to product and engineering teams to understand the current global AppSec maturity. The objective at this point is to be sure to have a comprehensive understanding of how your products are made (the processes, tools, components, and stacks involved). Mapping development tools and practices will require time to have the best visibility possible. They should include product development practices and the perceived risk awareness/appetite from managers. One of your objectives would be to nudge them so they take into account security in every decision they make for their products, and maybe end up thinking like adversaries. You should be able to derive security requirements from the different perceptual risks you are going to encounter. Your job is to consolidate these into a common set for all applications, setting goals to align the different teams collaborating to build your product(s). Communicating transparently with all relevant stakeholders (CISO, technical security, product owner, and development leads) about goals and expectations is essential to create a common ground for improvement. It will be absolutely necessary to ensure alignment throughout the implementation too. Open and accessible guardrails Guardrails are the cornerstone of security requirements. Their nature and implementation are completely up to the needs of your organization and can be potentially very different from one company to the other (if starting from scratch, look no further than the OWASP Top10). What is most important, however, is that these guardrails are open to the ones that need them. A good example of this would be to centralize a common, security-approved library of open-source components that can be pulled from by any team. Keep users' accessibility and useability as a priority. Designing an AppSec program at scale requires asking “how can we build confidence and visibility with trusted tools in our ecosystem?”. For instance, control gates should never be implemented without considering a break-glass option (“what happens if the control is blocking in an emergency situation?”). State-of-the-art security is to have off-the-shelf secure solutions chosen by the developers, approved by security, and maintained by ops. This will be a big leap forward in preventing vulnerabilities from creeping into source code. It will bring security to the masses at a very low cost (low friction). But to truly scale application security, it would be silly not to use the software engineer's best ally: the continuous integration pipeline. Embed controls in the CI/CD AppSec testing across all development pipelines is the implementation step. If your organization has multiple development teams, it is very likely that different CI/CD pipelines configurations exist in parallel. They may use different tools, or simply define different steps in the build process. This is not a problem per se, but to scale application security, centralization and harmonization are needed. As illustrated in the following example CI/CD pipeline, you can have a lot of security control steps: secrets detection, SAST, artifact signing, access controls, but also container or Infrastructure as Code scanning (not shown in the example) (taken from the DevSecOps whitepaper) The idea is that you can progressively activate more and more control steps, fine-tune the existing ones and scale both horizontally and vertically your “AppSec infrastructure”, at one condition: you need to centralize metrics and controls in a stand-alone platform able to handle the load corresponding to your organization’s size. Security processes can only be automated when you have metrics and proper visibility across your development targets, otherwise, it is just more burden on the AppSec team's shoulders. In turn, metrics and visibility help drive change and provide the spark to ignite a cultural change within your organization. Security ownership shifts to every engineer involved in the delivery process, and each one is able to leverage its own deep (yet partial) knowledge of the system to support the effort. This unlocks a world of possibilities: most security flaws can be treated like regular tickets, rule sets can be optimized for each pipeline based on criticality, capabilities or regulatory compliance, and progress can be tracked (saved time, avoided vulnerabilities etc.). In simpler terms, security can finally move at the DevOps speed. Conclusion Security can’t scale if it’s siloed, and slowing down the development process is no longer an option in a world led by DevOps innovation. The design and implementation of security controls are bound to evolve. In this article, we’ve depicted a high-level overview of the steps to be considered to scale AppSec. This starts with establishing a set of security requirements that involve all the departments, in particular product-related ones. From there it becomes possible to design guardrails to make security truly accessible with a mix of hard and soft gates. By carefully selecting automated detection and remediation that provide visibility and control, you will be laying a solid foundation for a real model of shared responsibility for security. Finally, embedding checks in the CI/CD system can be rolled out in multiple phases to progressively scale your security operations. With automated feedback in place, you can start incrementally adjusting your policies. A centralized platform creates a common interface to facilitate the exchange between application security and developer teams while enforcing processes. It is a huge opportunity to automate and propagate best practices across teams. Developers are empowered to develop faster with more ownership. When security is rethought as a partnership between software-building stakeholders, a flywheel effect can take place: reduced friction leads to better communication and visibility, automating of more best practices, easing the work of each other while improving security with fewer defects. This is how application security will finally be able to scale through continuous improvement.

thomas-segura has 67 posts and counting.See all posts by thomas-segura