×

A modern software development process consists of many steps built on each other. Two of the more significant steps in this process include integrating code from different developers into a single codebase and deploying that codebase to some kind of infrastructure.

With a CI/CD pipeline, we can automate most work related to integration and deployment, enabling high development velocity. These pipelines allow us to react to changes in the market quickly and ship new features without a manual approval process. They also help get bug and security fixes into production fast, so users aren’t annoyed or at risk of a breach in their private data.

But with all automation comes the requirement to see—from the outside—what’s happening. This is especially true if you deploy your applications to an external cloud provider’s infrastructure. That’s why it’s crucial to monitor your CI/CD pipeline, just like you would monitor your other software systems used in production.

The problem with monitoring is that it’s hard to know what to measure. You could gather many metrics for your pipeline and makeup infinitely more on your own. But nobody has time for trial and error here; that’s why we’re going to look at some of the most well-known metrics CI/CD pipelines in this article.

1. Uptime, Error Rate, and Infrastructure Costs

These are the basics. You should know how stable your system runs and how much you pay for it. Nobody wants to use a fragile system that is constantly offline or throws errors in masses.

Also, most solutions fail because they’re not financially viable. Suppose you pay too much for your pipeline—you should find out why. Do you need a different VM instance or to switch to another cloud provider? These are difficult questions to answer without hard numbers.

2. Cycle Time

The cycle time is the time your whole pipeline takes from start to functional deployment. Some companies take days to deploy a new version after all the coding is done, while others deploy hundreds of times a day.

The shorter your cycle time, the faster you can deliver. If you only need a few minutes for a deployment, you can fix a problem in production quicker, freeing you to try new things without the fear of losing hours of work. Taking advantage of these experiments, so often postponed out of fear of failure, are what will get you ahead of your competition.

3. Change Volume

How many changes are included in each cycle? Depending on your cycle time, you might need to optimize your change volume. The higher the overhead of each cycle, the more changes per cycle you’d want to make.

If the overhead is low, you might keep your change volume low because the changes not included in one run will be picked up more quickly later.

Bottlenecks can also form in your pipeline that wouldn’t develop if your change volume was lower. Sometimes, if you stay under a certain limit, a service you’re using might perform much better than when you go over it.

4. Deployment Frequency

How many deployments are done per period?

Again, some companies deploy hundreds of times per day; others only deploy once or twice a year. As your application accumulates features, it also gathers code. This code slows down your pipeline, making you less agile over time.

A slowdown in development velocity is expected, especially in the first year of a new application. But if you see this metric break down suddenly, there might be something seriously wrong in your codebase, and you should investigate further.

5. Rollback Speed

How quickly can you restore an older version in the event of a significant error? The faster you can do it, the less money you’ll lose when it happens. Even if every other metric is good, a slow rollback can still break a company. Things can and will go wrong, and if you take too long to recover, your customers might flee to a competitor.

At the same time, if your team members think a breaking error could cost them their job, they might pour more time into quality assurance than is needed, slowing down the whole development process. This can ultimately lead customers to choose a competitor anyway.

6. Team Retention Rate

What do the people who use the system think about it? Are they avoiding using it for their changes, preferring to “drop into VMs with SSH” instead? If your teams circumvent your pipeline, you should ask why. After all, it costs money to run a CI/CD pipeline, and it should save your team valuable time; if this isn’t the case, you don’t need the pipeline in the first place.

If the team retention rate goes down, ask your developers what’s wrong; sometimes, a few changes in the pipeline settings or a bigger VM can be enough to make people use it again.

7. Test Durations

Testing can quickly become the most significant step in your pipeline. More tests can lead to better quality, but they also slow your development velocity. It’s important to observe and optimize your tests to make sure they aren’t slowing down development and that they are providing you with the insights you need.

Monitoring systems like Foresight are tailored for automated testing and can help keep things in check. With Foresight, your tests are sorted by their execution time, so you know what’s going on at a glance. If you instrumented your production code with Thundra APM, you could even dissect slow tests to find out which code or service is causing the slowdown.

8. Test Failure Rate

The pipeline that runs our test is itself software that can have errors. The same goes for the software that is tested by your tests. You need to make sure that everything runs smoothly and find the causes of these errors quickly.

Again, Foresight has capabilities that can help mitigate the amount of work that usually goes into debugging tests. Foresight maintains a separate list for tests that errored, be it because the pipeline failed or the tested software was buggy.

9. Test Coverage

What code is covered by your tests? What isn’t?

You can’t be sure things are working if they aren’t tested.

While the relation of code quality and test coverage might be a matter of debate, there is one area where a near 100% coverage can help: test impact analysis.

If you know which code was touched by which test, you can filter out tests that don’t test changed code in each run. This way, you can increase your test suite to keep quality high without paying for it in pipeline slowdowns.

Conclusion

After automating all of your integration and delivery tasks, make sure they are—and keep—working correctly. This is just as true for your CI/CD pipeline as it is for your production software.

You can add countless metrics to your repertoire over time, but the ones we’ve covered here are a good place to start. With experience, you’ll gain an intuition for what’s missing and make more informed decisions about what metrics to remove or add.

If you’re concerned about your test suites, gain visibility in no time with Foresight.