Hot Topics
Security Bloggers Network 
SBN

Security Chats – Jon-Erik Schneiderhan, Senior SRE at a tech company

Avatar photo by Thomas Segura on May 25, 2022

Security Chats - Jon-Erik Schneiderhan, Senior SRE at a tech company

Jon-Erik has been using GitGuardian Internal Monitoring for 15 months with his team of three security engineers (and 80 developers). As Senior SRE, he was aware that many secrets could be hidden in the repositories he supervises, and new ones leaked every day.

He told Peerspot how and why he chose GitGuardian and the improvements his team received.

Rationale

Jon-Erik needed first and foremost a real-time secrets detection solution able to reduce the meantime to detection:

"Without GitGuardian, we wouldn't be doing real-time detection of secrets. It would be something that we did periodically. Maybe quarterly or semi-annually, we would review our code for secrets. This means that the meantime to detection would be much longer."

Before using GitGuardian, his team performed periodic code reviews than were not efficient:

“Previously, we only detected secrets when someone saw them, which was rare. Especially since a large portion of our secrets is in the Git history, not in the current state of the repository, we were only made aware of 10% of the secrets before.”

Return On Investment

Now, he estimates that the coverage has vastly improved thanks to the secrets detection capabilities of the solution:

“Now, we are probably in the 90 percentile.”

Not only that, but Jon-Erik also estimates that real-time monitoring saves his team an entire week per quarter in periodic code reviews:

"We don't have to do a periodic review to see if there are any secrets in our codebases. I would estimate, if we were to do that on a quarterly basis, we would be spending an entire week per quarter on it that we don't have to spend now. Therefore, it saves us a week every quarter in pure effort."

According to him, that represents a non-negligible amount of ROI:

“We have seen a return on investment. The amount of time that we would have spent manually doing this definitely outpaces the cost of GitGuardian. It is saving us about $35,000 a year, so I would say the ROI is about $20,000 a year.”

As he highlighted in his review, his initial objective of reducing MTTD and facilitating leaks remediation were achieved:

"GitGuardian reduces our mean time to detect substantially. In addition, we would be finding out about secrets much further away from the time that they were introduced into the codebase. We would be chasing people down to give us information about things that they did weeks or months ago. This would drastically reduce the effectiveness of us being able to triage and remediate the leaked secrets."

"The secrets detection and alerting is the most important feature. We get alerted almost immediately after someone commits a secret. It has been very accurate, allowing us to jump on it right away, then figure out if we have something substantial that has been leaked or whether it is something that we don't have to worry about. This general main feature of the app is great."

He summarizes the benefits :

"The solution efficiently supports our shift-left strategy."  

Features

Here are the core features that help Jon-Erik in his day-to-day security work:

Dismiss invalid secrets :

“Recently, they added a feature that checks the validity of leaked secrets. It will actually reach out and see if the secret that leaked was valid or not. I have found, over the past couple months, this to be a super useful feature. We can go through a lot of the secrets in our codebase, which have been detected, and dismiss them if we know that they are invalid secrets that can't be used anyway. This saves us a bunch of time, which is why this has been a really neat feature that has been useful.”

Don't waste time on false positives:

“I have found that I have been very satisfied with the breadth of the solution's detection capabilities. I don't think it has missed anything. The false positive rate has been very low. Every single time something is detected, it is something that we should look at. It does a very good job of detecting things that we should look at and make a decision on. We don't waste a lot of time chasing down false positives.“

Sort & prioritize incidents:

“The solution helps to quickly prioritize remediation because it allows us to tell which keys are valid versus which ones are invalid. We prioritize the valid ones first. It also lets us sort by detection type, e.g., what kind of secret is it detecting. There are ones that we would obviously prioritize over others, like SSH keys or AWS credentials, versus less sensitive credentials that aren't as concerning. I think it does a great job of helping us prioritize.”

Get feedback from the developers:

“GitGuardian provides a feedback form feature that we utilize heavily. When a secret is detected, our process is to generate a feedback form link in GitGuardian, then provide that to the developer. The developer will give us contextual information about the secret, then we can take action. They have also recently released a feature, which we haven't started using yet, called automated playbooks where you can set it up to automatically create that feedback form. Then, it will be emailed to the developer so they get automatically notified that they introduce a secret with a feedback form to fill out. I suspect this will improve our developer's ability to resolve the secrets faster.”

Jon-Erik also suggested a missing feature:

“They could give a developer access to a dashboard for their team's repositories that just shows their repository secrets. I think more could be exposed to developers.”

Good news: this much-awaited feature is coming very soon!  See our RBAC team management roadmap.

Lastly, we cannot say that we were not touched by the rating that Jon-Eric gave to our customer support ❤️

“I would rate them as nine out of 10. They respond to me almost immediately every time that I have a question, which has been great. I haven't experienced any delays or not had an issue solved.”

Summary: don't overlook code safety!

“We feel safe because we don't have valid credentials sitting in our code repositories. If any of our code was breached or any of our developer workstations were compromised or stolen, no one would be able to get valid API credentials out of the Git repositories on those workstations.”

"If you were to run a proof of concept with GitGuardian and see all of the things that it detects, then you would probably be very surprised. You can tell very quickly what the return on investment will be and how much risk a tool like this can mitigate."

"If you are not detecting secrets in code, then every developer's machine is a security breach waiting to happen."

Read the full review

*** This is a Security Bloggers Network syndicated blog from GitGuardian Blog - Automated Secrets Detection authored by Thomas Segura. Read the original post at: https://blog.gitguardian.com/customer-story-jon-erik/

Thomas Segura
Avatar photo

Thomas Segura

What You Need to Scale AppSec Thomas Segura - Content Writer @ GitGuardian Author Bio Thomas has worked both as an analyst and as a software engineer consultant for various big French companies. His passion for tech and open source led him to join GitGuardian as technical content writer. He focuses now on clarifying the transformative changes that cybersecurity and software are going through. Website:https://www.gitguardian.com/ Twitter handle: https://twitter.com/GitGuardian Linkedin: https://www.linkedin.com/company/gitguardian Introduction Security is a dilemma for many leaders. On the one hand, it is largely recognized as an essential feature. On the other hand, it does not drive business. Of course, as we mature, security can become a business enabler. But the roadmap is unclear. With the rise of agile practices, DevOps and the cloud, development timeframes have been considerably compressed, but application security remains essentially the same. DevSecOps emerged as an answer to this dilemma. Its promise consists literally in inserting security principles, practices, and tools into the DevOps activity stream, reducing risk without compromising deliverability. Therefore there is a question that many are asking: why isn't DevSecOps already the norm? As we analyzed in our latest report DevSecOps: Protecting the Modern Software Factory, the answer can be summarized as follows: only by enabling new capacities across Dev, Sec and Ops teams can the culture be changed. This post will help provide a high-level overview of the prerequisite steps needed to scale up application security across departments and enable such capabilities. From requirements to expectations Scaling application security is a company-wide project that requires thorough thinking before an y decision is made. A first-hand requirement is to talk to product and engineering teams to understand the current global AppSec maturity. The objective at this point is to be sure to have a comprehensive understanding of how your products are made (the processes, tools, components, and stacks involved). Mapping development tools and practices will require time to have the best visibility possible. They should include product development practices and the perceived risk awareness/appetite from managers. One of your objectives would be to nudge them so they take into account security in every decision they make for their products, and maybe end up thinking like adversaries. You should be able to derive security requirements from the different perceptual risks you are going to encounter. Your job is to consolidate these into a common set for all applications, setting goals to align the different teams collaborating to build your product(s). Communicating transparently with all relevant stakeholders (CISO, technical security, product owner, and development leads) about goals and expectations is essential to create a common ground for improvement. It will be absolutely necessary to ensure alignment throughout the implementation too. Open and accessible guardrails Guardrails are the cornerstone of security requirements. Their nature and implementation are completely up to the needs of your organization and can be potentially very different from one company to the other (if starting from scratch, look no further than the OWASP Top10). What is most important, however, is that these guardrails are open to the ones that need them. A good example of this would be to centralize a common, security-approved library of open-source components that can be pulled from by any team. Keep users' accessibility and useability as a priority. Designing an AppSec program at scale requires asking “how can we build confidence and visibility with trusted tools in our ecosystem?”. For instance, control gates should never be implemented without considering a break-glass option (“what happens if the control is blocking in an emergency situation?”). State-of-the-art security is to have off-the-shelf secure solutions chosen by the developers, approved by security, and maintained by ops. This will be a big leap forward in preventing vulnerabilities from creeping into source code. It will bring security to the masses at a very low cost (low friction). But to truly scale application security, it would be silly not to use the software engineer's best ally: the continuous integration pipeline. Embed controls in the CI/CD AppSec testing across all development pipelines is the implementation step. If your organization has multiple development teams, it is very likely that different CI/CD pipelines configurations exist in parallel. They may use different tools, or simply define different steps in the build process. This is not a problem per se, but to scale application security, centralization and harmonization are needed. As illustrated in the following example CI/CD pipeline, you can have a lot of security control steps: secrets detection, SAST, artifact signing, access controls, but also container or Infrastructure as Code scanning (not shown in the example) (taken from the DevSecOps whitepaper) The idea is that you can progressively activate more and more control steps, fine-tune the existing ones and scale both horizontally and vertically your “AppSec infrastructure”, at one condition: you need to centralize metrics and controls in a stand-alone platform able to handle the load corresponding to your organization’s size. Security processes can only be automated when you have metrics and proper visibility across your development targets, otherwise, it is just more burden on the AppSec team's shoulders. In turn, metrics and visibility help drive change and provide the spark to ignite a cultural change within your organization. Security ownership shifts to every engineer involved in the delivery process, and each one is able to leverage its own deep (yet partial) knowledge of the system to support the effort. This unlocks a world of possibilities: most security flaws can be treated like regular tickets, rule sets can be optimized for each pipeline based on criticality, capabilities or regulatory compliance, and progress can be tracked (saved time, avoided vulnerabilities etc.). In simpler terms, security can finally move at the DevOps speed. Conclusion Security can’t scale if it’s siloed, and slowing down the development process is no longer an option in a world led by DevOps innovation. The design and implementation of security controls are bound to evolve. In this article, we’ve depicted a high-level overview of the steps to be considered to scale AppSec. This starts with establishing a set of security requirements that involve all the departments, in particular product-related ones. From there it becomes possible to design guardrails to make security truly accessible with a mix of hard and soft gates. By carefully selecting automated detection and remediation that provide visibility and control, you will be laying a solid foundation for a real model of shared responsibility for security. Finally, embedding checks in the CI/CD system can be rolled out in multiple phases to progressively scale your security operations. With automated feedback in place, you can start incrementally adjusting your policies. A centralized platform creates a common interface to facilitate the exchange between application security and developer teams while enforcing processes. It is a huge opportunity to automate and propagate best practices across teams. Developers are empowered to develop faster with more ownership. When security is rethought as a partnership between software-building stakeholders, a flywheel effect can take place: reduced friction leads to better communication and visibility, automating of more best practices, easing the work of each other while improving security with fewer defects. This is how application security will finally be able to scale through continuous improvement.

thomas-segura has 48 posts and counting.See all posts by thomas-segura