How to Speed up Builds in Your CircleCI CI Environment
SUBSCRIBE TO OUR BLOG
Get our new blogs delivered straight to your inbox.
THANKS FOR SIGNING UP!
We’ll make sure to share the best materials crafted for you!
5 minutes read
How to Speed up Builds in Your CircleCI CI Environment
Written by Suleyman Barman
VP of Engineering at Thundra
In contemporary software creation, we want to work in an isolated environment. With so many dependencies needed to make a product work—many of them dependent on how other dependencies are configured—this makes total sense. An isolated environment can help protect the system at large while we solve these problems, hence the growing popularity of virtual environments and containers.
However, isolated environments require us to install or build a lot of dependencies each time we build and deploy our application, thus slowing down the CI/CD pipeline. And installing dependencies is just one reason CI/CD pipelines slow down. Other factors include too many tests, unutilized config management, tests running sequentially instead of parallel, and redundant dependencies.
If we don’t find ways to improve our CI/CD pipeline runtime, we can lose a lot of productivity and resources. This impacts the whole team, system administrators, and developers alike. Besides, watching running logs doesn’t help anyone’s mental health.
So how do we solve this problem? In this article, we’ll answer that. We’ll start by creating an example application and writing the tests for it. We’ll then define the CI/CD pipelines that build and deploy the application and run the integration tests afterward.
Example Project
For our example project, we’ve created a movie management service using Spring Boot. The service allows users to create, update, and retrieve information about movies, and delete the movie content. The service uses MySQL as the database, and for convenience, we’ll use the MySQL cloud provided by AWS RDS.
In real projects, it’s important to write unit tests for production code so that we can know whether any small changes we make have affected the existing functionality. Let’s take a look at an example unit test in a GitHub Gist. In the example, we’ll test the method used for retrieving movie content by its ID.
We can now trigger the service and interact with the new API. Just run the following command:
mvn spring-boot:runBelow is an example API call to create a new movie from the service.
Figure 1: Example API call to create new movie
Example Integration Test
The integration test for the movie management service will be implemented using Java with the help of REST-assured and jUnit, and we’ll record our example integration test on Gist. It will cover a case where we create a new movie and retrieve its content.
We’ll put the integration in the same repository with the application code, then push them to the GitHub repository.
Create an AWS EC2 Instance
The whole process can be a bit overwhelming, so we’ve outlined the steps required to prepare for deployment.
Step 1
First, set up a security group for the inbound and outbound rule (accept all for port 8081).
Figure 2: Inbound rule setup
Figure 3: Outbound rule setup
Step 2
Next, install Maven and Java.
Step 3
You’ll then set up a private key using the following command:
ssh-keygen -t rsa -b 4096 -C "[email protected]"Step 4
To copy your private key to CircleCI, go to project settings and choose “SSH Keys.”
Figure 4: SSH key is to be added to circle CI
Step 5
Next, you’ll add your public key to /authorized_keys in the EC2 instance with the following command:
cat <your_public_key_file> >> ~/.ssh/authorized_keys.Creating the .circleci/config.yml File with Jobs
Now, use the Thundra CircleCI orb to integrate Thundra with CircleCI. This will allow you to see integration test results in the Foresight dashboard.
Note that in order to use a third-party orb in CircleCI, we need to edit the security configuration. Here are the steps we’ll take:
Step 1
Get the project key ID and API key from Thundra.
Figure 5 : Retrieve project key ID and API key from Thundra
Step 2
To set the environment variable in CircleCI for Thundra credentials, go to project settings in CircleCI and select “Environment Variables” from the menu on the left.
Figure 6 : Set environment variables in CircleCI
Step 3
Define the CircleCI orb in the config.yml file as shown below.
orbs:
# Declare a dependency on the Thundra Orb
# Make sure to replace <VERSION> with the latest available
thundra: thundra-io/[email protected]
Step 4
Build the application using the following command:
build:
docker:
- image: cimg/openjdk:11.0
steps:
- checkout
- thundra/maven:
# Use environment variable name and set the secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
Step 5
Deploy the application to the AWS EC2 instance:
deploy:
machine:
enabled: true
steps:
- run:
name: Deploy Over SSH
command: ssh -o StrictHostKeyChecking=no [email protected] "kill $(lsof -t -i:8080) && cd ~/Projects/thundra-ci-speed && git pull origin main && cd movie-management-app && mvn clean install && nohup mvn spring-boot:run &"
Step 6
Run the integration test.
integration_test:
docker:
- image: cimg/openjdk:11.0
parallelism: 4
steps:
- checkout
- thundra/maven:
# Use environment variable name and set the secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
- run: cd integration-test && mvn clean test
Step 7
Finally, set up the workflow for the jobs to be run sequentially, one after the other.
workflows:
build_and_test:
jobs:
- build
- deploy:
requires:
- build
- integration_test:
requires:
- deploy
Troubleshooting a Slow CI Pipeline
From the results shown below, we found that the total time for the pipeline took roughly 3 minutes. That’s pretty high for building and deploying.
Figure 7: Overall result of the pipeline in CircleCI
We can also see that the time for the integration_test workflow took around 2 minutes. This is much too high. To help us troubleshoot, let’s look at Foresight to see how much time each test takes.
Figure 8: Monitor test execution in the Foresight dashboard
The tests for UserInfoTest and MovieTest classes are pretty fast, but the test for Authentication is slow.
Optimizing the Pipeline for Speed
So, how can we speed up our authentication test and gain valuable time? The first thing we can do is try to parallelize the tests as much as possible.
In order to do that, we’ll need to use Maven Surefire with options for running the tests in parallel.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
</configuration>
</plugin>
For this setup, we’ll tell Maven to test parallelization as much as possible for the maximum threads with 10. This should greatly reduce time spent in testing
Applying the Cache for Future Builds
The build time took longer than necessary because we had to download and install the library in the Maven project. If we save the cache and reuse it for the next build, we can save some time in the pipeline.
Below is how we update the pipeline’s config.yml file so that CircleCI can cache the previous build and restore it for the next one.
- restore_cache:
keys:
- v1-dependencies- # appends cache key with a hash of pom.xml file
- v1-dependencies- # fallback in case previous cache key is not found
- run: cd movie-management-app && mvn clean install
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-
Now, let’s commit this change and push it to GitHub to see if it worked.
Figure 7: CircleCI pipeline after apply parallel tests and caching build
After the pipeline is done, we can see that the execution time was reduced to just under 2 minutes, instead of the 3 minutes we had before optimizing. This is a great improvement.
Conclusion
Optimizing the build time of your CI/CD pipeline is important for moving fast and saving you the stress of watching the build. But without the proper tools to help us implement this optimization, we might fall into a pile of logs we can’t get out of.
By integrating CircleCI and Thundra Foresight, we can see the overall pipeline from CircleCI and the test executions in Foresight’s dashboard. With this level of observability, we can easily find which part of the build is taking too long and why. Instead of investigating and debugging applications, DevOps and developers can both spend their time building new features for their products instead.
Integrate Foresight into your CI/CD pipeline in a matter of seconds with this pre-defined Foresight orb for CircleCI configuration. Take your free Thundra account and start exploring yourself.
SHARE
×
SUBSCRIBE TO OUR BLOG
Get our new blogs delivered straight to your inbox.
THANKS FOR SIGNING UP!
We’ll make sure to share the best materials crafted for you!
5 minutes read
How to Speed up Builds in Your CircleCI CI Environment
Written by Suleyman Barman
VP of Engineering at Thundra
In contemporary software creation, we want to work in an isolated environment. With so many dependencies needed to make a product work—many of them dependent on how other dependencies are configured—this makes total sense. An isolated environment can help protect the system at large while we solve these problems, hence the growing popularity of virtual environments and containers.
However, isolated environments require us to install or build a lot of dependencies each time we build and deploy our application, thus slowing down the CI/CD pipeline. And installing dependencies is just one reason CI/CD pipelines slow down. Other factors include too many tests, unutilized config management, tests running sequentially instead of parallel, and redundant dependencies.
If we don’t find ways to improve our CI/CD pipeline runtime, we can lose a lot of productivity and resources. This impacts the whole team, system administrators, and developers alike. Besides, watching running logs doesn’t help anyone’s mental health.
So how do we solve this problem? In this article, we’ll answer that. We’ll start by creating an example application and writing the tests for it. We’ll then define the CI/CD pipelines that build and deploy the application and run the integration tests afterward.
Example Project
For our example project, we’ve created a movie management service using Spring Boot. The service allows users to create, update, and retrieve information about movies, and delete the movie content. The service uses MySQL as the database, and for convenience, we’ll use the MySQL cloud provided by AWS RDS.
In real projects, it’s important to write unit tests for production code so that we can know whether any small changes we make have affected the existing functionality. Let’s take a look at an example unit test in a GitHub Gist. In the example, we’ll test the method used for retrieving movie content by its ID.
We can now trigger the service and interact with the new API. Just run the following command:
mvn spring-boot:runBelow is an example API call to create a new movie from the service.
Figure 1: Example API call to create new movie
Example Integration Test
The integration test for the movie management service will be implemented using Java with the help of REST-assured and jUnit, and we’ll record our example integration test on Gist. It will cover a case where we create a new movie and retrieve its content.
We’ll put the integration in the same repository with the application code, then push them to the GitHub repository.
Create an AWS EC2 Instance
The whole process can be a bit overwhelming, so we’ve outlined the steps required to prepare for deployment.
Step 1
First, set up a security group for the inbound and outbound rule (accept all for port 8081).
Figure 2: Inbound rule setup
Figure 3: Outbound rule setup
Step 2
Next, install Maven and Java.
Step 3
You’ll then set up a private key using the following command:
ssh-keygen -t rsa -b 4096 -C "[email protected]"Step 4
To copy your private key to CircleCI, go to project settings and choose “SSH Keys.”
Figure 4: SSH key is to be added to circle CI
Step 5
Next, you’ll add your public key to /authorized_keys in the EC2 instance with the following command:
cat <your_public_key_file> >> ~/.ssh/authorized_keys.Creating the .circleci/config.yml File with Jobs
Now, use the Thundra CircleCI orb to integrate Thundra with CircleCI. This will allow you to see integration test results in the Foresight dashboard.
Note that in order to use a third-party orb in CircleCI, we need to edit the security configuration. Here are the steps we’ll take:
Step 1
Get the project key ID and API key from Thundra.
Figure 5 : Retrieve project key ID and API key from Thundra
Step 2
To set the environment variable in CircleCI for Thundra credentials, go to project settings in CircleCI and select “Environment Variables” from the menu on the left.
Figure 6 : Set environment variables in CircleCI
Step 3
Define the CircleCI orb in the config.yml file as shown below.
orbs:
# Declare a dependency on the Thundra Orb
# Make sure to replace <VERSION> with the latest available
thundra: thundra-io/[email protected]
Step 4
Build the application using the following command:
build:
docker:
- image: cimg/openjdk:11.0
steps:
- checkout
- thundra/maven:
# Use environment variable name and set the secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
Step 5
Deploy the application to the AWS EC2 instance:
deploy:
machine:
enabled: true
steps:
- run:
name: Deploy Over SSH
command: ssh -o StrictHostKeyChecking=no [email protected] "kill $(lsof -t -i:8080) && cd ~/Projects/thundra-ci-speed && git pull origin main && cd movie-management-app && mvn clean install && nohup mvn spring-boot:run &"
Step 6
Run the integration test.
integration_test:
docker:
- image: cimg/openjdk:11.0
parallelism: 4
steps:
- checkout
- thundra/maven:
# Use environment variable name and set the secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
- run: cd integration-test && mvn clean test
Step 7
Finally, set up the workflow for the jobs to be run sequentially, one after the other.
workflows:
build_and_test:
jobs:
- build
- deploy:
requires:
- build
- integration_test:
requires:
- deploy
Troubleshooting a Slow CI Pipeline
From the results shown below, we found that the total time for the pipeline took roughly 3 minutes. That’s pretty high for building and deploying.
Figure 7: Overall result of the pipeline in CircleCI
We can also see that the time for the integration_test workflow took around 2 minutes. This is much too high. To help us troubleshoot, let’s look at Foresight to see how much time each test takes.
Figure 8: Monitor test execution in the Foresight dashboard
The tests for UserInfoTest and MovieTest classes are pretty fast, but the test for Authentication is slow.
Optimizing the Pipeline for Speed
So, how can we speed up our authentication test and gain valuable time? The first thing we can do is try to parallelize the tests as much as possible.
In order to do that, we’ll need to use Maven Surefire with options for running the tests in parallel.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
</configuration>
</plugin>
For this setup, we’ll tell Maven to test parallelization as much as possible for the maximum threads with 10. This should greatly reduce time spent in testing
Applying the Cache for Future Builds
The build time took longer than necessary because we had to download and install the library in the Maven project. If we save the cache and reuse it for the next build, we can save some time in the pipeline.
Below is how we update the pipeline’s config.yml file so that CircleCI can cache the previous build and restore it for the next one.
- restore_cache:
keys:
- v1-dependencies- # appends cache key with a hash of pom.xml file
- v1-dependencies- # fallback in case previous cache key is not found
- run: cd movie-management-app && mvn clean install
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-
Now, let’s commit this change and push it to GitHub to see if it worked.
Figure 7: CircleCI pipeline after apply parallel tests and caching build
After the pipeline is done, we can see that the execution time was reduced to just under 2 minutes, instead of the 3 minutes we had before optimizing. This is a great improvement.
Conclusion
Optimizing the build time of your CI/CD pipeline is important for moving fast and saving you the stress of watching the build. But without the proper tools to help us implement this optimization, we might fall into a pile of logs we can’t get out of.
By integrating CircleCI and Thundra Foresight, we can see the overall pipeline from CircleCI and the test executions in Foresight’s dashboard. With this level of observability, we can easily find which part of the build is taking too long and why. Instead of investigating and debugging applications, DevOps and developers can both spend their time building new features for their products instead.
Integrate Foresight into your CI/CD pipeline in a matter of seconds with this pre-defined Foresight orb for CircleCI configuration. Take your free Thundra account and start exploring yourself.
SHARE
In contemporary software creation, we want to work in an isolated environment. With so many dependencies needed to make a product work—many of them dependent on how other dependencies are configured—this makes total sense. An isolated environment can help protect the system at large while we solve these problems, hence the growing popularity of virtual environments and containers.
However, isolated environments require us to install or build a lot of dependencies each time we build and deploy our application, thus slowing down the CI/CD pipeline. And installing dependencies is just one reason CI/CD pipelines slow down. Other factors include too many tests, unutilized config management, tests running sequentially instead of parallel, and redundant dependencies.
If we don’t find ways to improve our CI/CD pipeline runtime, we can lose a lot of productivity and resources. This impacts the whole team, system administrators, and developers alike. Besides, watching running logs doesn’t help anyone’s mental health.
So how do we solve this problem? In this article, we’ll answer that. We’ll start by creating an example application and writing the tests for it. We’ll then define the CI/CD pipelines that build and deploy the application and run the integration tests afterward.
Example Project
For our example project, we’ve created a movie management service using Spring Boot. The service allows users to create, update, and retrieve information about movies, and delete the movie content. The service uses MySQL as the database, and for convenience, we’ll use the MySQL cloud provided by AWS RDS.
In real projects, it’s important to write unit tests for production code so that we can know whether any small changes we make have affected the existing functionality. Let’s take a look at an example unit test in a GitHub Gist. In the example, we’ll test the method used for retrieving movie content by its ID.
We can now trigger the service and interact with the new API. Just run the following command:
mvn spring-boot:runBelow is an example API call to create a new movie from the service.
Figure 1: Example API call to create new movie
Example Integration Test
The integration test for the movie management service will be implemented using Java with the help of REST-assured and jUnit, and we’ll record our example integration test on Gist. It will cover a case where we create a new movie and retrieve its content.
We’ll put the integration in the same repository with the application code, then push them to the GitHub repository.
Create an AWS EC2 Instance
The whole process can be a bit overwhelming, so we’ve outlined the steps required to prepare for deployment.
Step 1
First, set up a security group for the inbound and outbound rule (accept all for port 8081).
Figure 2: Inbound rule setup
Figure 3: Outbound rule setup
Step 2
Next, install Maven and Java.
Step 3
You’ll then set up a private key using the following command:
ssh-keygen -t rsa -b 4096 -C "[email protected]"Step 4
To copy your private key to CircleCI, go to project settings and choose “SSH Keys.”
Figure 4: SSH key is to be added to circle CI
Step 5
Next, you’ll add your public key to /authorized_keys in the EC2 instance with the following command:
cat <your_public_key_file> >> ~/.ssh/authorized_keys.Creating the .circleci/config.yml File with Jobs
Now, use the Thundra CircleCI orb to integrate Thundra with CircleCI. This will allow you to see integration test results in the Foresight dashboard.
Note that in order to use a third-party orb in CircleCI, we need to edit the security configuration. Here are the steps we’ll take:
Step 1
Get the project key ID and API key from Thundra.
Figure 5 : Retrieve project key ID and API key from Thundra
Step 2
To set the environment variable in CircleCI for Thundra credentials, go to project settings in CircleCI and select “Environment Variables” from the menu on the left.
Figure 6 : Set environment variables in CircleCI
Step 3
Define the CircleCI orb in the config.yml file as shown below.
orbs:
# Declare a dependency on the Thundra Orb
# Make sure to replace <VERSION> with the latest available
thundra: thundra-io/[email protected]
Step 4
Build the application using the following command:
build:
docker:
- image: cimg/openjdk:11.0
steps:
- checkout
- thundra/maven:
# Use environment variable name and set the secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
Step 5
Deploy the application to the AWS EC2 instance:
deploy:
machine:
enabled: true
steps:
- run:
name: Deploy Over SSH
command: ssh -o StrictHostKeyChecking=no [email protected] "kill $(lsof -t -i:8080) && cd ~/Projects/thundra-ci-speed && git pull origin main && cd movie-management-app && mvn clean install && nohup mvn spring-boot:run &"
Step 6
Run the integration test.
integration_test:
docker:
- image: cimg/openjdk:11.0
parallelism: 4
steps:
- checkout
- thundra/maven:
# Use environment variable name and set the secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
- run: cd integration-test && mvn clean test
Step 7
Finally, set up the workflow for the jobs to be run sequentially, one after the other.
workflows:
build_and_test:
jobs:
- build
- deploy:
requires:
- build
- integration_test:
requires:
- deploy
Troubleshooting a Slow CI Pipeline
From the results shown below, we found that the total time for the pipeline took roughly 3 minutes. That’s pretty high for building and deploying.
Figure 7: Overall result of the pipeline in CircleCI
We can also see that the time for the integration_test workflow took around 2 minutes. This is much too high. To help us troubleshoot, let’s look at Foresight to see how much time each test takes.
Figure 8: Monitor test execution in the Foresight dashboard
The tests for UserInfoTest and MovieTest classes are pretty fast, but the test for Authentication is slow.
Optimizing the Pipeline for Speed
So, how can we speed up our authentication test and gain valuable time? The first thing we can do is try to parallelize the tests as much as possible.
In order to do that, we’ll need to use Maven Surefire with options for running the tests in parallel.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
</configuration>
</plugin>
For this setup, we’ll tell Maven to test parallelization as much as possible for the maximum threads with 10. This should greatly reduce time spent in testing
Applying the Cache for Future Builds
The build time took longer than necessary because we had to download and install the library in the Maven project. If we save the cache and reuse it for the next build, we can save some time in the pipeline.
Below is how we update the pipeline’s config.yml file so that CircleCI can cache the previous build and restore it for the next one.
- restore_cache:
keys:
- v1-dependencies- # appends cache key with a hash of pom.xml file
- v1-dependencies- # fallback in case previous cache key is not found
- run: cd movie-management-app && mvn clean install
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-
Now, let’s commit this change and push it to GitHub to see if it worked.
Figure 7: CircleCI pipeline after apply parallel tests and caching build
After the pipeline is done, we can see that the execution time was reduced to just under 2 minutes, instead of the 3 minutes we had before optimizing. This is a great improvement.
Conclusion
Optimizing the build time of your CI/CD pipeline is important for moving fast and saving you the stress of watching the build. But without the proper tools to help us implement this optimization, we might fall into a pile of logs we can’t get out of.
By integrating CircleCI and Thundra Foresight, we can see the overall pipeline from CircleCI and the test executions in Foresight’s dashboard. With this level of observability, we can easily find which part of the build is taking too long and why. Instead of investigating and debugging applications, DevOps and developers can both spend their time building new features for their products instead.
Integrate Foresight into your CI/CD pipeline in a matter of seconds with this pre-defined Foresight orb for CircleCI configuration. Take your free Thundra account and start exploring yourself.
In contemporary software creation, we want to work in an isolated environment. With so many dependencies needed to make a product work—many of them dependent on how other dependencies are configured—this makes total sense. An isolated environment can help protect the system at large while we solve these problems, hence the growing popularity of virtual environments and containers.
However, isolated environments require us to install or build a lot of dependencies each time we build and deploy our application, thus slowing down the CI/CD pipeline. And installing dependencies is just one reason CI/CD pipelines slow down. Other factors include too many tests, unutilized config management, tests running sequentially instead of parallel, and redundant dependencies.
If we don’t find ways to improve our CI/CD pipeline runtime, we can lose a lot of productivity and resources. This impacts the whole team, system administrators, and developers alike. Besides, watching running logs doesn’t help anyone’s mental health.
So how do we solve this problem? In this article, we’ll answer that. We’ll start by creating an example application and writing the tests for it. We’ll then define the CI/CD pipelines that build and deploy the application and run the integration tests afterward.
Example Project
For our example project, we’ve created a movie management service using Spring Boot. The service allows users to create, update, and retrieve information about movies, and delete the movie content. The service uses MySQL as the database, and for convenience, we’ll use the MySQL cloud provided by AWS RDS.
In real projects, it’s important to write unit tests for production code so that we can know whether any small changes we make have affected the existing functionality. Let’s take a look at an example unit test in a GitHub Gist. In the example, we’ll test the method used for retrieving movie content by its ID.
We can now trigger the service and interact with the new API. Just run the following command:
mvn spring-boot:runBelow is an example API call to create a new movie from the service.
Figure 1: Example API call to create new movie
Example Integration Test
The integration test for the movie management service will be implemented using Java with the help of REST-assured and jUnit, and we’ll record our example integration test on Gist. It will cover a case where we create a new movie and retrieve its content.
We’ll put the integration in the same repository with the application code, then push them to the GitHub repository.
Create an AWS EC2 Instance
The whole process can be a bit overwhelming, so we’ve outlined the steps required to prepare for deployment.
Step 1
First, set up a security group for the inbound and outbound rule (accept all for port 8081).
Figure 2: Inbound rule setup
Figure 3: Outbound rule setup
Step 2
Next, install Maven and Java.
Step 3
You’ll then set up a private key using the following command:
ssh-keygen -t rsa -b 4096 -C "[email protected]"Step 4
To copy your private key to CircleCI, go to project settings and choose “SSH Keys.”
Figure 4: SSH key is to be added to circle CI
Step 5
Next, you’ll add your public key to /authorized_keys in the EC2 instance with the following command:
cat <your_public_key_file> >> ~/.ssh/authorized_keys.Creating the .circleci/config.yml File with Jobs
Now, use the Thundra CircleCI orb to integrate Thundra with CircleCI. This will allow you to see integration test results in the Foresight dashboard.
Note that in order to use a third-party orb in CircleCI, we need to edit the security configuration. Here are the steps we’ll take:
Step 1
Get the project key ID and API key from Thundra.
Figure 5 : Retrieve project key ID and API key from Thundra
Step 2
To set the environment variable in CircleCI for Thundra credentials, go to project settings in CircleCI and select “Environment Variables” from the menu on the left.
Figure 6 : Set environment variables in CircleCI
Step 3
Define the CircleCI orb in the config.yml file as shown below.
orbs:
# Declare a dependency on the Thundra Orb
# Make sure to replace <VERSION> with the latest available
thundra: thundra-io/[email protected]
Step 4
Build the application using the following command:
build:
docker:
- image: cimg/openjdk:11.0
steps:
- checkout
- thundra/maven:
# Use environment variable name and set the secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
Step 5
Deploy the application to the AWS EC2 instance:
deploy:
machine:
enabled: true
steps:
- run:
name: Deploy Over SSH
command: ssh -o StrictHostKeyChecking=no [email protected] "kill $(lsof -t -i:8080) && cd ~/Projects/thundra-ci-speed && git pull origin main && cd movie-management-app && mvn clean install && nohup mvn spring-boot:run &"
Step 6
Run the integration test.
integration_test:
docker:
- image: cimg/openjdk:11.0
parallelism: 4
steps:
- checkout
- thundra/maven:
# Use environment variable name and set the secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
- run: cd integration-test && mvn clean test
Step 7
Finally, set up the workflow for the jobs to be run sequentially, one after the other.
workflows:
build_and_test:
jobs:
- build
- deploy:
requires:
- build
- integration_test:
requires:
- deploy
Troubleshooting a Slow CI Pipeline
From the results shown below, we found that the total time for the pipeline took roughly 3 minutes. That’s pretty high for building and deploying.
Figure 7: Overall result of the pipeline in CircleCI
We can also see that the time for the integration_test workflow took around 2 minutes. This is much too high. To help us troubleshoot, let’s look at Foresight to see how much time each test takes.
Figure 8: Monitor test execution in the Foresight dashboard
The tests for UserInfoTest and MovieTest classes are pretty fast, but the test for Authentication is slow.
Optimizing the Pipeline for Speed
So, how can we speed up our authentication test and gain valuable time? The first thing we can do is try to parallelize the tests as much as possible.
In order to do that, we’ll need to use Maven Surefire with options for running the tests in parallel.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
</configuration>
</plugin>
For this setup, we’ll tell Maven to test parallelization as much as possible for the maximum threads with 10. This should greatly reduce time spent in testing
Applying the Cache for Future Builds
The build time took longer than necessary because we had to download and install the library in the Maven project. If we save the cache and reuse it for the next build, we can save some time in the pipeline.
Below is how we update the pipeline’s config.yml file so that CircleCI can cache the previous build and restore it for the next one.
- restore_cache:
keys:
- v1-dependencies- # appends cache key with a hash of pom.xml file
- v1-dependencies- # fallback in case previous cache key is not found
- run: cd movie-management-app && mvn clean install
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-
Now, let’s commit this change and push it to GitHub to see if it worked.
Figure 7: CircleCI pipeline after apply parallel tests and caching build
After the pipeline is done, we can see that the execution time was reduced to just under 2 minutes, instead of the 3 minutes we had before optimizing. This is a great improvement.
Conclusion
Optimizing the build time of your CI/CD pipeline is important for moving fast and saving you the stress of watching the build. But without the proper tools to help us implement this optimization, we might fall into a pile of logs we can’t get out of.
By integrating CircleCI and Thundra Foresight, we can see the overall pipeline from CircleCI and the test executions in Foresight’s dashboard. With this level of observability, we can easily find which part of the build is taking too long and why. Instead of investigating and debugging applications, DevOps and developers can both spend their time building new features for their products instead.
Integrate Foresight into your CI/CD pipeline in a matter of seconds with this pre-defined Foresight orb for CircleCI configuration. Take your free Thundra account and start exploring yourself.
*** This is a Security Bloggers Network syndicated blog from Thundra blog authored by Suleyman Barman. Read the original post at: https://blog.thundra.io/how-to-speed-up-builds-in-your-circleci-ci-environment
