Security Bloggers Network 
SBN

How We Built a Supply Chain Security Watchtower: Meet SaaS-Sentinel

Avatar photo by Thomas Segura on April 27, 2023

How We Built a Supply Chain Security Watchtower: Meet SaaS-Sentinel

TL;DR: we have built SaaS-sentinel, the first (as far as we know!) SaaS watchtower. The platform allows you to be notified when your favorite tool may be under attack, helping you be on the lookout for ongoing supply chain attacks. Scroll down to the third part to understand the objective and how it works. Or read below to learn the full story!

The Story

In July 2022, GitGuardian announced ggcanary, an open-source project designed to help organizations detect compromised developer and DevOps environments. The goal was to democratize the fields of intrusion detection and deception techniques, making it easier for organizations to detect potential threats with high precision and low entrance cost.

The ggcanary project uses honeytokens, which are secrets left in the infrastructure to tempt attackers. When an attacker uses a honeytoken, it sends an alert, enabling organizations to detect potential breaches before they cause significant damage.

🍯
What are honeytokens? Honeytokens are secrets (like AWS API keys or other credentials) that are left in your infrastructure to tempt attackers to try and exploit them. Once an attacker uses a honeytoken, it sends an alert, and this lets you know you have an intruder in your systems. You can find a much more detailed description in this blog post.

The concept behind this mechanism is that when a system is breached, hackers typically search for easy targets to move laterally, escalate privileges, or steal data. In this context, AWS secrets are an ideal target for scanning as they have a recognizable pattern that starts with "AKIA…" or "ASIA…" and often contain useful information for the attacker. Therefore, they are a prime target for attackers to search for and exploit during a breach.

Here is a video tutorial if you want to get your feet wet with ggcanary:

GitGuardian has been working in the secrets detection space for years and has noticed two recurring facts about data breaches:

Therefore, the idea of decoy credentials looked very attractive as a way for defenders to turn the tables on attackers: by spreading out up to 5,000 AWS secrets in various locations, such as source code management, CI/CD pipelines, developer workstations, artifact registries, or other places, defenders can be alerted when a perimeter is breached in near real-time.

💡
We have now fully integrated the honeytoken technology into the GitGuardian platform! You can now benefit from zero-configuration, easy-to-roll-out, and scalable honeytokens to protect your assets, all from a centralized dashboard. Learn more about the new Honeytoken module here.

At this point, we didn't know that this idea would prove to be even more relevant to what was to come!

What We Didn’t See Coming: LastPass and CircleCI Fall for Supply Chain Attacks

Two incidents in early 2023 sparked the idea for SaaS-Sentinel.

First, we learned that LastPass had been targeted by a threat actor which exploited a third-party vulnerability to compromise a software engineer’s workstation and access a cloud environment where encryption keys, along with other data such as API keys and customer data, were stored.

💡
Did you know that, on average, it takes 327 days to identify a data breach? 

Then, we learned that CircleCI had also, unfortunately, fallen victim to a supply chain attack: in a similar scenario, an attacker leveraged malware deployed to a CircleCI engineer’s laptop to steal a valid, 2FA-backed SSO session and, from there, could escalate privileges to the company production infrastructure. It was later confirmed that the actor was able to exfiltrate customers’ environment variables, tokens, and keys.

In both cases, the customers of these two companies experienced a nightmare scenario: they had to work on rotating potentially sensitive data stored in the two SaaS services while investigating the damage and mitigating the consequences of the incident.

In the last story, something interesting happened. A security engineer, Daniel Hückmann, tweeted about investigating the unauthorized access to his company’s CircleCI account thanks to a canary token he had planted in the platform. The decoy AWS key had been triggered by the attacker, and Daniel had been notified before anyone else, including CircleCI itself.

The Birth of SaaS-Sentinel

SaaS-Sentinel was born from this idea: what if we planted honeytokens in the most used SaaS providers to alert people when a tripwire gets triggered, suggesting a possible breach?

In a world where third-party compromises and supply chain threats are becoming more and more common, leveraging a high-fidelity signal to win a few hours over threat actors sounded like a no-brainer.
With ggcanary, we already had the open-source technology, we just had to build a unique monitoring service!

How does it work?

SaaS Sentinel is based on the honeytoken technology: GitGuardian engineers have planted multiple honeytokens in a number of SaaS tools, such as source code management platforms (GitHub, GitLab, Bitbucket), continuous integration services (GitLab CI/CD, GitHub Actions, etc.), password managers (LastPass, Bitwarden), and of course, the GitGuardian platform itself!

💡
By the way, we are looking to extend our surveillance to many more SaaS tools. You can vote for the next service provider you would like to see monitored in the future and even suggest a service we haven’t thought of.
How We Built a Supply Chain Security Watchtower: Meet SaaS-Sentinel

When one of the honeytokens is triggered, subscribers to the (free) alerting service will receive a notification. However, to minimize the occurrence of false alerts and enhance the reliability of the signal, we have implemented a series of validation steps before sending out alerts to the public. The alerting flow can be better understood through the following diagram:

How We Built a Supply Chain Security Watchtower: Meet SaaS-Sentinel

There are two crucial points to note:

  1. If a honeytoken is triggered, the alert flow will first contact the affected service provider. We will then wait a minimum of 4 hours for a response before changing the service status indicator on the front page and sending an email notification to subscribers (manual process).
  2. Despite the fact that most service providers have implemented vulnerability disclosure programs, our flow intentionally circumvents this mechanism. This is because our primary objective in the event of a breach is to promptly notify downstream users and minimize the potential impact.
💡
We plan to provide information about the service providers’ vulnerability disclosure programs directly on the platform in a future release.

Summary

The idea for SaaS-Sentinel grew from GitGuardian's ggcanary project, which uses honeytokens to detect intrusions. The recent supply chain attacks on LastPass and CircleCI revealed the value of providing a monitoring platform to alert users as quickly as possible in the event of a compromise.

SaaS-Sentinel is a free monitoring platform that aims at helping organizations stay on top of their supply chain risks and reduce the Mean-Time-to-Detect as much as possible. It is GitGuardian’s contribution to democratizing the honeytoken technology as a way to turn the tables on attackers.

*** This is a Security Bloggers Network syndicated blog from GitGuardian Blog - Automated Secrets Detection authored by Thomas Segura. Read the original post at: https://blog.gitguardian.com/how-we-built-a-supply-chain-security-watchtower-meet-saas-sentinel/

Thomas Segura
Avatar photo

Thomas Segura

What You Need to Scale AppSec Thomas Segura - Content Writer @ GitGuardian Author Bio Thomas has worked both as an analyst and as a software engineer consultant for various big French companies. His passion for tech and open source led him to join GitGuardian as technical content writer. He focuses now on clarifying the transformative changes that cybersecurity and software are going through. Website:https://www.gitguardian.com/ Twitter handle: https://twitter.com/GitGuardian Linkedin: https://www.linkedin.com/company/gitguardian Introduction Security is a dilemma for many leaders. On the one hand, it is largely recognized as an essential feature. On the other hand, it does not drive business. Of course, as we mature, security can become a business enabler. But the roadmap is unclear. With the rise of agile practices, DevOps and the cloud, development timeframes have been considerably compressed, but application security remains essentially the same. DevSecOps emerged as an answer to this dilemma. Its promise consists literally in inserting security principles, practices, and tools into the DevOps activity stream, reducing risk without compromising deliverability. Therefore there is a question that many are asking: why isn't DevSecOps already the norm? As we analyzed in our latest report DevSecOps: Protecting the Modern Software Factory, the answer can be summarized as follows: only by enabling new capacities across Dev, Sec and Ops teams can the culture be changed. This post will help provide a high-level overview of the prerequisite steps needed to scale up application security across departments and enable such capabilities. From requirements to expectations Scaling application security is a company-wide project that requires thorough thinking before an y decision is made. A first-hand requirement is to talk to product and engineering teams to understand the current global AppSec maturity. The objective at this point is to be sure to have a comprehensive understanding of how your products are made (the processes, tools, components, and stacks involved). Mapping development tools and practices will require time to have the best visibility possible. They should include product development practices and the perceived risk awareness/appetite from managers. One of your objectives would be to nudge them so they take into account security in every decision they make for their products, and maybe end up thinking like adversaries. You should be able to derive security requirements from the different perceptual risks you are going to encounter. Your job is to consolidate these into a common set for all applications, setting goals to align the different teams collaborating to build your product(s). Communicating transparently with all relevant stakeholders (CISO, technical security, product owner, and development leads) about goals and expectations is essential to create a common ground for improvement. It will be absolutely necessary to ensure alignment throughout the implementation too. Open and accessible guardrails Guardrails are the cornerstone of security requirements. Their nature and implementation are completely up to the needs of your organization and can be potentially very different from one company to the other (if starting from scratch, look no further than the OWASP Top10). What is most important, however, is that these guardrails are open to the ones that need them. A good example of this would be to centralize a common, security-approved library of open-source components that can be pulled from by any team. Keep users' accessibility and useability as a priority. Designing an AppSec program at scale requires asking “how can we build confidence and visibility with trusted tools in our ecosystem?”. For instance, control gates should never be implemented without considering a break-glass option (“what happens if the control is blocking in an emergency situation?”). State-of-the-art security is to have off-the-shelf secure solutions chosen by the developers, approved by security, and maintained by ops. This will be a big leap forward in preventing vulnerabilities from creeping into source code. It will bring security to the masses at a very low cost (low friction). But to truly scale application security, it would be silly not to use the software engineer's best ally: the continuous integration pipeline. Embed controls in the CI/CD AppSec testing across all development pipelines is the implementation step. If your organization has multiple development teams, it is very likely that different CI/CD pipelines configurations exist in parallel. They may use different tools, or simply define different steps in the build process. This is not a problem per se, but to scale application security, centralization and harmonization are needed. As illustrated in the following example CI/CD pipeline, you can have a lot of security control steps: secrets detection, SAST, artifact signing, access controls, but also container or Infrastructure as Code scanning (not shown in the example) (taken from the DevSecOps whitepaper) The idea is that you can progressively activate more and more control steps, fine-tune the existing ones and scale both horizontally and vertically your “AppSec infrastructure”, at one condition: you need to centralize metrics and controls in a stand-alone platform able to handle the load corresponding to your organization’s size. Security processes can only be automated when you have metrics and proper visibility across your development targets, otherwise, it is just more burden on the AppSec team's shoulders. In turn, metrics and visibility help drive change and provide the spark to ignite a cultural change within your organization. Security ownership shifts to every engineer involved in the delivery process, and each one is able to leverage its own deep (yet partial) knowledge of the system to support the effort. This unlocks a world of possibilities: most security flaws can be treated like regular tickets, rule sets can be optimized for each pipeline based on criticality, capabilities or regulatory compliance, and progress can be tracked (saved time, avoided vulnerabilities etc.). In simpler terms, security can finally move at the DevOps speed. Conclusion Security can’t scale if it’s siloed, and slowing down the development process is no longer an option in a world led by DevOps innovation. The design and implementation of security controls are bound to evolve. In this article, we’ve depicted a high-level overview of the steps to be considered to scale AppSec. This starts with establishing a set of security requirements that involve all the departments, in particular product-related ones. From there it becomes possible to design guardrails to make security truly accessible with a mix of hard and soft gates. By carefully selecting automated detection and remediation that provide visibility and control, you will be laying a solid foundation for a real model of shared responsibility for security. Finally, embedding checks in the CI/CD system can be rolled out in multiple phases to progressively scale your security operations. With automated feedback in place, you can start incrementally adjusting your policies. A centralized platform creates a common interface to facilitate the exchange between application security and developer teams while enforcing processes. It is a huge opportunity to automate and propagate best practices across teams. Developers are empowered to develop faster with more ownership. When security is rethought as a partnership between software-building stakeholders, a flywheel effect can take place: reduced friction leads to better communication and visibility, automating of more best practices, easing the work of each other while improving security with fewer defects. This is how application security will finally be able to scale through continuous improvement.

thomas-segura has 62 posts and counting.See all posts by thomas-segura