Home » Security Bloggers Network » Enterprise DevSecOps: New Series

Enterprise DevSecOps: New Series

by [email protected] (Securosis) on October 3, 2019

Posted under: Heavy Research

DevOps is an operational framework that promotes software consistency and standardization through automation. It helps address many of the nightmare development issues around integration, testing, patching and deployment by both breaking down the barriers between different development teams, but also by prioritizing things that make software development faster and easier.

DevSecOps is the integration of security teams and security tools directly into the software development lifecycle, leveraging the automation and efficiencies of DevOps to ensure application security testing occurs with every build cycle. This promotes security, consistency and ensure security is no less important that other quality metrics or feature. Automated security testing, just like automated application build and deployments, must be assembled with the rest of the infrastructure.

And therein lies the problem. Software developers have traditionally not embraced security. It’s not because they did not care about security, rather they were incentivized to to focus on delivery of new features and functions. DevOps is changing the priority on automating build processes to make them faster, easier and more consistent. But it does not mean they are going out of their way to include security or security tooling. That’s often because the security tools don’t easily integrate well with development tools and processes, and usually flood queues with unintelligible findings, and lack development centric filters to help prioritize work. Worse, the security platforms – and the security professionals who recommended them – were difficult to work with or even fail to offer API layer support to provide integration.

On the other side of equation are security teams, who are fearful of automated software processes, and commonly ask the question “How do we get control over development”. The very nature of this question misses both the spirit of DevSecOps, as well as the efforts of development organizations to get faster, more efficient and more consistency with each software release. The only way for security teams to cope with the changes occurring within software development, and to scale their relatively small organizations, is to become just as agile as Dev teams and embrace automation.

Why This Research Paper?

We typically discuss the motivation for our research papers to help readers understand our goals and what we wish to convey. This is doubly so when we update a research paper as it help us spotlight recent changes in the industry that have made older papers inaccurate or fall short in describing recent trends. As DevOps and DevOps option has matured considerably in four years, we have a lot to talk about.

This effort will be a major rewrite of our 2015 research work on Building Security into DevOps, with significant additions around common questions security teams ask about DevSecOps, and a thorough update to tooling and approaches to integration. Much of this research paper will reflect the 400+ conversations since 2017 across some 200+ security teams at Fortune 2000 firms. As such we will include considerably more discussion points derived from those conversations. As DevOps has been around for many years now, less discussion around what DevSecOps is or how it is beneficial, and a more pragmatic discussion on how to put together a DevSecOps program.

Now, let’s shake things up a bit.

Different Focus, Different Value

There are a plethora of new surveys and research papers available, and some of them a very good. And there are more conferences and on-line resources popping up than I can count. For example, Veracode recently released the latest iteration on their State of Software Security (SoSS) report, and it’s a monster, with loads of data and observations. The key takeaways are the agility and automation employed by DevSecOps teams provides demonstrable security benefits, including faster patching cycles, shorter duration of flaw persistence, faster reduction of technical debt, and ‘easier’ scanning meant faster problem identification. Sonatype recently released 2019 State of the Software Supply Chain report shows teams which shows teams that ‘Exemplary Project Teams’ who leverage DevOps principles drastically reduce code deployment failure rates, and remediate vulnerabilities in half the time as average groups. And we have events like All Day DevOps, where hundreds of DevOps practitioners share stories on cultural transformations, continuous integration / continuous deployment (CI:CD) techniques, site reliability engineering as well as DevSecOps. All of which is great, and offers a body of qualitative and quantitative data on why DevOps works and how practitioners are evolving their programs.

That’s not what this paper is about. Those resources are not addressing the questions I am being asked each and every week.

This paper is about putting together a comprehensive DevSecOps program. Overwhelmingly we are asked “How do I put a DevSecOps program together?” and “How does security fit into DevOps?”. They are not looking for justification, nor are they looking for point stories on nuances to address specific impediments. They want a security program that is in line with peer organizations and embraces ‘security best practices’. These audiences are overwhelmingly comprised of security and IT practitioners, largely left behind by development teams who have at the very least embraced Agile concepts if not DevOps outright. The challenge is to understand what development is trying to accomplish, integrate – in some fashion – with those teams, and figure out how to leverage automated security testing to be at least as agile as development teams are.

DevOps vs DevSecOps

Which leads us to another controversial topic and why this research is different: The name DevSecOps. It is our contention that calling out security – the ‘Sec’ in ‘DevSecOps’ – is needed given the lack of maturity and understanding on this topic.

Stated another way, practitioners of DevOps who have fully embraced the movement will tell you that there is no reason to add ‘Sec’ into DevOps, as security is just another ingredient. The DevOps ideal is to break down silos between individual teams (e.g.: architecture, development, IT, security and QA) to better promote teamwork and better incentivize each team member toward the same goals. If security is just another set of skills blended into the overall effort of building and delivering software, there is no reason to call it out any more than we call out quality assurance. Philosophically, these proponents are right. In practice, we are not there yet. Developers may embrace the idea, they generally suck at facilitating team integration. Sure, security is free to participate, but it’s up to them to learn where they can integrate, and typically asked to bring skills to the party they may not possess. It’s passive-aggressive team building!

Automated security testing, just like automated application build and deployments, takes time and skill to build out. In our typical engagements with clients the developers are absent from the call. A divide still exists, with little communication between security and what is usually dozens – to hundreds – of disperse development teams. When developers are present, they state that the security team can create scripts to integrate security testing into the build server; they can codify security policies; security can stitch together security analysis tools with trouble ticketing and metrics with a few lines of python code. After-all, many of the IT practitioners are learning to script for configuration management and build templates to define infrastructure deployments, so why not security? This completely misses the reality that few security practitioners are capable of coding at this level. Worse, most firms we speak with have a ratio of around 100 developers for every security practitioner, and there is simply no way to scale security resources across all development projects.

It does not help that many security professionals are early in their process of understanding DevOps and various methods developers have been adopting over the last decade to become more agile. Security is genuinely behind the curve, and it seems the bulk of the available research – mentioned above – is not aimed at tackling security’s introduction and integration.

Lastly, we have one more very important reason we choose to use the DevSecOps name: security efforts for code security are very different than the efforts to secure infrastructure and supporting components. The security checks to validate application code is secure (i.e.: DevSec), but are different than tooling and processes used verify the supporting infrastructure components (e.g.: network, data security, identity services, logging, code delivery mechanisms, operating systems) are secure. These are two different disciplines and should be discussed .

Common Questions

We went through all of our call notes from the last three years, and tallied the questions that we were asked. The following list is the most common questions, in the order of how often the question was asked.

We want to integrate security testing into the development pipeline, and are going to start with static analysis. How do we do this?
How do we build an application security strategy in light of automation, CI:CD and DevOps?
How to start building out an application strategy? What app security standards should I follow?
Development is releasing code to production every day. How do we get control over development? Can we realistically modify developers behavior?
What is the best way to introduce DevSecOps? Where do I start? What are the foundational pieces?
How do we get different units to adopt the same program (different teams all do things differently) when right now some are waterfall, some agile, some DevOps?
How should we (Security) work with Dev?
We understand shift left, but are the tools effective? What tools do you recommend to start with?

The questions all have some common threads; they all come from firms who have at least some DevOps teams already in place, that security has some understanding of the intent of DevOps, but they are all starting from scratch. Even teams that have security tests already built into the development pipeline are struggling with the value each tool provides, pushback from developers over flase positives, how to work with developers or how to scale across many development teams for consistency. And we find during calls and engagements that security is not quite in tune with why developers embrace DevOps, and miss the thrust of the effort, which is why they are commonly out of synch.

The following is a list of questions security we came up with that security teams should ask, but don’t.

How do we fit – culturally and operationally – into DevSecOps?
How do we get visibility into Development and Development practices?
How do we know changes are effective? What metrics should we collect and monitor?
How do we support Dev?
Do we need to know how to code?

Over the course of this series we will address both lists of questions. Next up, we will provide a brief introduction to DevSecOps principles and the role of security in DevOps.

– Adrian Lane
(0) Comments
Subscribe to our daily email digest