Home » Security Bloggers Network » GitOps Best Practices and the Kubernetes Guardrails You Need to Follow Them

GitOps Best Practices and the Kubernetes Guardrails You Need to Follow Them

by Andy Suderman on January 12, 2023

GitOps is a buzzword everyone is talking about, a term coined in 2017 by Weaveworks. It uses an operations by pull request approach to define and manage networking, infrastructure, application code, and the GitOps pipeline. GitOps is a way to manage cloud-native systems powered by Kubernetes.

What is GitOps

Weaveworks has a lot of cool open source tooling and has been in the Kubernetes space for a while. According to a story on the Weaveworks website, the team there invested a lot of energy into improving their Kubernetes operations by creating different staging areas, setting up multiple reliability zones, and creating built-in monitoring and security. At some point in 2016, an engineer informed the team that he was pushing out a change that might wipe everything out. It did. But because they had everything described in Git configuration files, they were able to restore their system in less than 45 minutes.

A lot of work went into building their system to make it possible to recover so quickly, which led them to identify the things that made that rapid recovery possible. That’s when they came up with the concept of GitOps. They made a list of principles to follow for operating their Kubernetes system — a system that made operations automatic for the entire system based on a model of that system that existed outside the system — in the open source distributed version control system Git.

GitOps saw industry-wide adoption in 2018, and in November 2020, Amazon, Codefresh, GitHub, Microsoft, and Weaveworks created the GitOps Working Group, which is now an open CNCF community project. This group’s goal is to define what GitOps is and make it vendor neutral.

The four main principles of GitOps

GitOps is declarative, which means that it focuses on facts and not instructions. Just like Kubernetes, you tell it what’s supposed to be, not how to make it that way.
GitOps uses Git as the source of truth and the Git source of truth is versioned and immutable. That’s one of the benefits of GitOps.
GitOps focuses on automatic pull mechanisms. In our standard flow, you do all your builds and tests in your CI pipeline; in there you have commands that push everything to your cluster. The GitOps philosophy is that your cluster should be pulling those changes itself, that you shouldn’t be pushing to it.
Your cluster is continuously reconciled. It takes advantage of control loops to keep an eye on how things are supposed to be and to keep your cluster in that state, much like the internal guts of Kubernetes itself, which is all about controllers that are watching the state of things versus the way that you want things to be.

At Fairwinds, we use GitOps internally for managing add-ons to our infrastructure, such as cert-manager, ingress-nginx, and external-dns. We use ArgoCD, which is one of a few different GitOps tools out there. You deploy ArgoCD in your cluster and configure it to watch a particular repo and then reconcile what’s in that repo with a particular cluster or clusters. It’s watching and then reconciling the cluster to maintain that state, which is in Git.

Sponsorships Available

Tooling to Support GitOps

Reckoner

One of the open source tools Fairwinds built and uses is Reckoner, a command line helper for Helm that creates a declarative syntax to manage multiple releases in a single place and allows the installation of charts from a git commit/branch/release.

Reckoner allows you to declare a whole bunch of Helm releases in a single YAML file. Each release is a Helm chart, along with its corresponding values. In the YAML file you can also tell Reckoner to generate an ArgoCD Application manifest. Most of the options available in ArgoCD Applications, such as autosync, are available in Reckoner as well.

Reckoner then creates a folder structure inside of the manifest directory with a folder for each app, as well as a single folder containing all of the ArgoCD applications. Then you can create an app of apps that points back to that folder and syncs the applications. So, the only thing you need to manage manually is the individual ArgoCD application manifest that defines your app of apps.

When you create a new application, it gets synced into ArgoCD and then it continues to sync that app going forward. The manifests that get templated into each folder are a declarative state of what should be applied to your cluster, so when you make changes to that, you can see the change in your Git pull/merge request. By templating out these manifests, you get a full diff of the exact changes that are going to happen before the changes get into your cluster. That increased transparency gives you all the information you need versus the more limited information that you would have to search for in your Helm charts or in an ArgoCD Helm Application.

The manifest also allows you to do static analysis and run CI/CD checks against the YAML in your CI/CD system. You should also create a check to validate that the Reckoner template was executed correctly. It verifies that there are no Git changes and that no YAML was modified manually. This supports the GitOps principle of making sure that there is a single source where there are no untracked changes.

Kubeconform

You also want to run a kubeconform check on your templated manifests, because while the Helm chart may have spit out some YAML, that YAML may not have been validated against the schema for Kubernetes. Helm doesn’t do that on a Helm template because it doesn’t have access to a cluster. Kubeconform is a Kubernetes manifest validator; it takes all the manifests and validates them against the schema from the Kubernetes repository. The other thing kubeconform can do is create a JUnit file, which you can send to your CI/CD system to have it show the results of each of your individual tests.

Polaris

When adopting GitOps, it is important to put the right guardrails in place and follow Kubernetes best practices to make sure that you are building in security and stability. Polaris helps you do that. It is another Fairwinds open source tool you can run in CI/CD. It checks for Kubernetes best practices, particularly related to security, efficiency, and reliability. Polaris includes over thirty built in configuration policies and will validate them and even remediate Kubernetes resources to ensure that configuration best practices are being followed. Polaris checks the whole repository and set of manifests for configuration issues, which helps you align to GitOps best practices.

Fairwinds Insights

Instead of writing individual checks for each tool, you can use Fairwinds Insights to get a full repository report, which will show you all the issues that were found in the manifest directory in one place. It includes:

A Trivy report that shows its findings: vulnerabilities, misconfigurations, secrets, software bill of materials in containers, Kubernetes, code repositories, clouds, and more.
A Polaris report that shows configuration issues.
A Pluto report of deprecated Kubernetes API versions in code repositories and Helm releases.
An Open Policy Agent (OPA) report so you can apply your own policies via Rego, apply them to your YAML, and add that into your checks.

You can turn on auto scan for Insights to automatically see those reports and create tickets based on the findings. You can also use a Jira integration to create tickets for findings and assign them to the team that needs to fix them. Fairwinds Insights has a free tier available for environments up to 20 nodes, two clusters, and one repo.

GitOps + Kubernetes Guardrails

By following these GitOps best practices and using tooling to align to Kubernetes best practices and implement guardrails, you can ensure that your GitOps processes are secure, reliable, and maintainable. This will help you keep your Kubernetes environment running as expected while making sure that any changes made are valid and won’t cause unexpected issues.

Watch the webinar to get a better understanding of how GitOps works and how these open source tools can help you implement Kubernetes guardrails that align with GitOps best practices.