Lessons in Securing Development Environments
The new world of software development is inherently collaborative — development teams are geographically dispersed and require easily accessible and automated tools to ship products and features quicker with confidence. The new generation of development and cloud-based tools now enables previously siloed teams to share easily and work together seamlessly, but it also brings with it new security risks. The pivot to CI/CD pipelines creates a new attack vector that can expose organizations’ networks, IT infrastructure and even source code to bad actors. An integrated and continuous security approach that scales with your new development effort is now more crucial than ever.
Securing CI/CD pipelines – and the software release process, in general – relies on three core components: people, process and technology. Only by combining the three elements together can you build a defense that ensures you stay protected. Here are three examples of recent software breaches and vulnerabilities that could have been prevented with better security in the development environment or along the CI/CD pipeline.
Lesson: People Should Collaborate on Security
Building, testing, deploying and securing your products is still very much a human-centered process. To fully secure your development environments, security awareness and training for the development teams is required. The security and DevOps teams must work together and establish collaborative practices.
Getting developers to take more responsibility for security and be part of the process of solving security issues is crucial for the security process and solutions to be effective.
The Impact of a Misconfiguration
In this example, a source code leak occurred due to a common misconfiguration where a default admin credential was left in use. This incident emphasizes the importance and impact of the developers on the security posture of the CI/CD pipeline. Nissan source code was leaked online after a Git repository misconfiguration. Swiss-based software engineer Tillie Kottmann said Nissan North America’s misconfiguration of a Bitbucket Git server led to the online leakage of the automaker’s source code for its mobile applications and internal tools. Nissan was allegedly running a Bitbucket Git server with the default credentials of admin/admin, which the developer should have modified as part of the system setup.
Security teams need to engage with the DevOps team and developers, build awareness around tools’ security risks and make them part of the security process. Such cooperation might take time to build, but overall, there’s been some initial success in this area.
Lesson: Focus on the Process
DevOps processes and CI/CD pipelines work at lightning speed and change continuously; therefore, security must be built in by design and move just as fast, if not faster. The security processes need to fit the test fast, fail fast mantra of the CI/CD process. Embedding security as part of the DevOps process at each step will maximize its effect and will create the right cooperation level with the development teams that is needed to make it successful.
If You Aren’t There, You Can’t Stop It
Embedding security controls as part of the DevOps process adds context and can identify unusual activities that are not in line with the usual pipeline process. In this example, that means the external pull requests, new pipeline script and the cryptomining activity on the GitHub servers.
Attackers mined cryptocurrency using GitHub servers, abusing the GitHub Actions automation workflow tool to facilitate the attack. This particular attack abused GitHub’s own infrastructure; it generated a pull request that executed the attacker’s code and instructed GitHub servers to retrieve and run a cryptominer on their servers.
Building security enforcement as part of the DevOps process is essential for the security to be effective and not delay the development process. Security needs to be part of the core CI/CD process and provide actionable information that is derived from understanding the process and its desired outcomes. That way, you are enabling development activities and increasing the adoption and participation of the development team.
Lesson: Technology Can Bolster Security
The tools and technologies used in the CI/CD pipeline are typically point solutions that come with limited security capabilities and do not speak to one another. This siloed approach does not provide a consolidated and unified way to view and analyze issues – which is required in complex environments such as CI/CD pipelines. This means that potential threats and breaches might be overlooked, as your current security only relates to one area of your environment (usually, this means scanning for images, vulnerabilities, secrets) and isn’t being viewed in the context of the entire pipeline process.
You Need to See and Understand it to Protect It
In a recent dependency confusion supply chain attack, a researcher managed to breach over 35 major companies’ internal systems, including Microsoft, Apple, PayPal, Shopify, Netflix, Yelp, Tesla and Uber. The attack uploaded malware to open source repositories including PyPI, npm and RubyGems, and the malware was then distributed downstream automatically into the company’s internal applications. The researcher discovered a design flaw that could be exploited in such a way that when a dependency package used by an application existed in both a public open source repository and a private build, the public package would get priority and be pulled instead — without any action from the developer.
To protect your CI/CD pipeline against such dependency confusion attacks, you need a security solution that is connected to all pipeline tools, understands the process and its dependencies and alerts on any change, such as an internal package being pulled from an outside repo, or on any new repository added to your pipeline.
Protecting a complex and ever-changing process such as the CI/CD pipeline requires a holistic security solution. Such a solution must take into consideration the pipeline process, tools and operation scripts. Such a solution must be embedded within the pipeline process to alert on vulnerabilities in real-time, prevent attacks and automatically remediate human mistakes and misconfigurations before real damage is done.
Only by combining strong security measures over the CI/CD pipeline with the right technology embedded into the DevOps process and involving the development teams in enforcing them can you create strong security posture for your development environments.
While it may seem hard to achieve, the recent security breaches used as examples are fueling enterprises’ awareness and understanding that holistic security over CI/CD pipelines is a must to improve overall organizational security posture.
