One of the main contributors to the weak security posture of development environments is the complexity and knowledge gap created by the number of tools and services involved in this process. With more than a hundred CI/CD tools to choose from and hundreds of plugins and services connected to those tools, no wonder security teams are having a hard time keeping tabs on the amount of information and staying on top of the security requirements of these environments.
It is not rare to see a CI/CD pipeline which includes 10 to 20 different tools and services; some of these cloud-based, others open source tools with a variety of plugins installed. It is impossible to manually keep track of this complexity, and it often results in an exposure of your environment, code, secrets and network through those tools and plugins’ vulnerabilities.
The DevOps tool sprawl continues as more and more companies expand their DevOps products and services. Development teams take advantage of these new CI/CD tools and services to build their pipelines and enhance the process, but by doing so they are also increasing their exposure to risks. Add to that the limited collaboration between development and security teams, and the lack of visibility and control over these services and it’s no surprise that CISOs and application security managers look puzzled when asked about the security of their CI/CD pipelines.
Hackers Continue to Damage Enterprises
The Japanese e-commerce giant Mercari and project management tool provider Monday.com are both victims of the Codecov attack which has already affected hundreds of major companies including HashiCorp, Confluent, Twilio and Rapid7, among others.
Mercari stated that the attack compromised records including financial and personal information from customers and partners, including:
- 17,085 records related to customer sales that occurred between August 5, 2014, and January 20, 2014, containing bank codes, branch codes, account numbers, account holder (kana) and transfer amount.
- 7,966 records of Mercari’s business partners, including names, birth dates, affiliation, e-mail address, etc.
- 2,615 employee records, including those working for a Mercari subsidiary. Names of some employees as of April 2021, company email address, employee ID, telephone number, date of birth, etc.
Just few days before, Monday.com had disclosed that it, too was impacted by the Codecov supply chain attack. After their investigation, they found unauthorized actors had gained access to a read-only copy of their source code.
Minimize Risk and Complexity to Avoid Supply Chain Attacks
The recent series of supply chain attacks affected tens of thousands of companies. Nowadays, CI/CD pipelines form the backbone of modern DevOps operations and, as we see this trend continue, we cannot ignore the urgent need to protect customers’ development environments from these pervasive attacks.
The complexity and collaborative nature of these environments provides an easy target for attackers, who can take advantage of vulnerabilities and misconfigurations within pipeline plugins and services. By gaining access to the CI/CD pipeline, attackers can hijack your updates, inject malicious code and gain a backdoor to your and your customers’ environments.
The latest Codecov attack showed us two alarming facts:
- Attackers can gain easy access to your most valuable data through your pipeline’s many services and plugins, which are usually not monitored.
- These attacks can go unnoticed for months, impacting thousands of companies and inflicting massive damage.
Organizations must proactively secure their software supply chain against such attacks to prevent access via these backdoors. This requires taking into account the complexity of the development environments, the various third-party plugins and services connected to it, and the sophisticated nature of today’s supply chain attacks.
Building a Strong CI/CD Pipeline Security Posture
Security and DevOps teams need to watch their pipeline dependencies closely to identify and respond to vulnerabilities and attacks against those add-ons, services and tools. Whenever a new service is connected to your pipeline, they need to be checked and monitored constantly for any vulnerability or suspicious activity. Any suspicion should automatically trigger an alert to the appropriate stakeholders to verify the integrity of the service and ensure there is no risk associated with it.