Gracefully Protecting Rapid Software Deployments

Gracefully Protecting Rapid Software Deployments — Part I


Software has changed. What used to be monolithic services on the backend are now the massive deployments of microservices that constantly are spawned up and torn down with shifting workload needs. They no longer run in controlled environments you provision, but in-turn run on VMs and containers on rent. Every sensitive information entered by a user on a web-form travels and transforms through layers of computing designed for ease of deployment and simplicity. But with all the elegance comes the burden of security. In this series of posts we will dive deeply and discover how modern software development and deployment pushes the security burden further, and the shortcomings of traditional security to cater to this model. We will break apart an application and see how it sees light the of day in the modern world, and the pain points everybody in the development and deployment chain feels.

The Software of Today

Of course, I can discuss the great virtues of modern software from straight of out of a textbook, someone’s presentation or research paper, but what fun is a blog if it doesn’t have a memorable story?

Just because I love pizzas, 🍕 We will create a PizzaCoin Bank in the modern software world. Everything is pizza here — their main back-end service is called Pizza Manager. People interact with the Pizza Manager through a web interface and deposit or withdraw their Pizza Coins. Now, this Pizza Manager as software has been there for a long time — since the dawn of Internet. We were told that in the old world, Pizza Manager was a single monolith that catered to all requests (~10 customers sending 2 request/day). Business was slow, but the engineers kept it running on their server racks in the basement. It was one single application interacting with one single database of hungry customers — keeping their ledger safe and updated. But the tides turned, Pizza Coins became a trend and business started booming. Soon they had 5,000 customers and the queries to the Pizza Manager were rapidly rising. More racks were mounted and soon everyone gave up and started renting hardware. Then the thing called cloud emerged and everyone started renting virtualized hardware. Another 5 years and a million requests/day later, the traditional monolith service model was breaking up now for Pizza Manager. The smart engineers decided its time to rewrite the back-end service as scalable microservices which would use the newly introduced containerization features as well.

The Glorious Pizza Manager Service — Then and Now. The databases can be distributed — one stores Customer Details and other stores the Pizza Coin Transactions and Customer Ledger.

Meanwhile, a thousand more engineers were hired. As the teams grew and ideas were shaped to features, new agile development models were incorporated. Fancy new features were being added everyday to the Pizza Manager service and they needed to be delivered as soon as they were developed to keep up the pace. Each pizza microservice now was developed with distributed teams and rapidly released to production. Or as the managers would say, the pizzas were being baked faster than usual.

‘Baking’ the Code

Lets take a closer look at this code development process. To keep up with the high churn-out rate, modern version control systems allow continuous integration (CI) tools such as Jenkins, Travis, Circle CI, etc. These tools automate per-commit builds and allow automated tests, tagging and releases. CI systems also make sure that the builds are reproducible and unique and the build environment is always same (can be a new container or a persistent VM with all build dependencies as required). CI tools also allow running automated tests and provide hooks to perform custom post-build actions. Once those are done and the version looks safe, the Pizza Service v1 is ready to be deployed automatically using Continuous Delivery (CD) tools such as Chef or Puppet or Ansible that ensure the latest versions are deployed and propagate to the core and the edge of the service mesh as required in the configuration. Usually, the journey of a Pizza Microservice through a CI/CD system would go like this:

This modern process is a cycle — with well-defined feedback at all stages. It also means that the code development and operations are converging and the high speed of delivery demands that code quality be maintained. To summarize, here are some characteristics of most modern software out there:

  • Developed in an agile fashion
  • Designed with scalability and portability in mind. It is not tied to the underlying infrastructure and is polyglot in nature of development
  • Deployed more often in production in an automated fashion
  • Designed to be resilient and re-entrant — fast bring-up and tear-down
  • Developed by distributed teams
  • Developed with dependencies on other components which the developers don’t necessarily control

Security Challenges in the DevOps World

While the idea of matching the dev speed with delivery is noble, modern security lacks any tools to adapt to this model, as they have traditionally been focused on production at deployment time. Moreover, modern software is now getting complex, and with multiple actors responsible for each section of the microservice code (or slice of the pizza), they introduce a lot of external dependencies — many of which may not be secure. With each version of the code being developed, known and unknown vulnerabilities may get introduced. In the past, they have been known to create havoc if left unnoticed. It is easy to have these vulnerabilities propagate throughout builds and deployments without a clear visibility of what is safe and what is unsafe to deploy.

Developers working in tandem on different components of Pizza Microservice inadvertently introducing a vulnerability through the usage of an affected (or infected?) dependency. Software versions are deployed in production.

Unknown Vulnerability: Having a vulnerability in your application or its external dependencies which has not been discovered or disclosed yet — but can be exploited.

It may be possible however that a vulnerable dependency may not actually be exploitable. As an example, Personally Identifiable Information (PII) such as street address input by a PizzaCoin Bank customer on the web application may not actually be flowing all the way through to the vulnerable section of the code. Maybe there is a sanitization function that removes such PII data. However, modern tools don’t support such advanced alerting mechanisms. The current software security model is build around reactive security while the modern world demands proactive security. On the spectrum of software development and release lifecycle, the responsibility now lies more so on the developer than it lies on the infrastructure. Security is now a bug in the system and it needs debugging. Debugging requires visibility and visibility is achieved through a thorough analysis and not duct-taping the data leaking pipe. In reality, absolute security is a myth but shifting the security left allows the attack surface to be greatly reduced. Rather than crying foul and bandaging a wound, it is wise to prevent an injury from happening — or at the very least know the risk of an injury beforehand.

In the next post, we will discuss in-depth how data flows through the code and how modern tools try to overcome the challenges of visibility. We will go through the process of bringing the advanced code analysis techniques to the modern world and see how it can be coupled with runtime analysis such that we protect applications and don’t lose the fast pace of development as well. Au revoir!

Gracefully Protecting Rapid Software Deployments was originally published in ShiftLeft Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

*** This is a Security Bloggers Network syndicated blog from ShiftLeft Blog - Medium authored by Suchakra Sharma. Read the original post at: