Guidelines for Optimizing CI-CD Pipelines

For a simple app, or a POC, you can straight-up publish and deploy without caching, without artifacts, without many build stages. But, as systems grow, the weaknesses in the systems’ CI-CD grow as well — and it’s time for optimizing your CI-CD pipelines.

When it’s time to optimize, you need to know what to optimize and how to track improvements. If you don’t track metrics, you’re shooting blindly in the dark.

What to Optimize?

So, what are the pain points in your CI-CD? In terms of optimizing CI-CD pipelines, do you want to…

Shorten the time the pipeline takes, meaning pipeline duration
Consume less storage when your ci-cd jobs run – storage consumption
Reduce memory and CPU consumption in your ci-cd runners – memory and CPU use
Decrease the volume of network traffic in your ci-cd runners, meaning network load
Reduce cognitive load – meaning, how hard it is for a new engineer to hold what’s going on in his or her head

The table below illustrates possible actions and what characteristics they will impact for the better.

Optimize/Reduce	Pipeline duration	Storage Consumption	Memory Use	CPU Use	Network Load	Cognitive Load
Cache external dependencies	Yes*	No	Yes	Yes	It Depends***	No
Use artifacts job to job	Yes*	No	Yes	Yes	It Depends***	No
Configure build size	Yes	Yes	Yes	Yes	Yes	No
Reduce dependencies between subsystems	Yes*	Yes**	Yes**	Yes**	Yes	Yes
Separate pipelines for subsystems	Yes*	Yes	Yes	Yes	Yes	Yes
Break apart large repos into smaller	Yes*	Yes	Yes	Yes	Yes	Yes

* Not useful for smaller projects

** The gains may not be noticeable here, depending on the sizes of the system and its subsystems and the dependencies

*** Dependent on the complexity of the ci pipelines, and the tech behind the pipelines. For instance, GitHub Actions will download the cache and upload/download artifacts per job, resulting in more network load, while other tech may hold cache or artifacts local to the pipeline runner.

How to Track?

Different methods are available to track metrics in your pipelines. Factors in choosing include your timeframe, cost, what tools are already at your disposal, and what you’re trying to track.

1. Quick and Home-Grown

You can add to your pipeline jobs certain statements to dump to the logging for the pipeline, and manually, visually compare the before and after. This serves to give an intuitive feel, but can be a nuisance because the previous pipeline’s metrics may not be available for long.

    steps:       
       
    - name: Checkout
      uses: actions/checkout@v3

    - name: Restore and build 
      run: |
        dotnet --version
        echo "Total solution storage consumption before job for ${PWD}:" 
        du -sh ./*
        dotnet build -c Release --no-incremental -o ./buildtest
        echo "Total solution storage consumption after job for ${PWD}:" 
        du -sh ./*

2. Home-Grown, Next Levels

You can take the logging from the previous approach, direct it to a file, and then keep the file as an artifact. You can even write a script to compare different logging artifacts and present the metrics to the console or more visually.

At a certain point, however, you will reach the point of diminishing returns. Meaning, you’re spending more time crafting the metrics collection versus the main work of your team.

Then, it may be time for a third-party metric collection.

3. Third-Party Tools

You might already have pipeline metric collection available in your tech stack, or perhaps one of these options is viable for your team and company.

Azure DevOps Pipeline Reports
Datadog CI Visibility for multiple CI providers
The GitHub action from Foresight
TeamCity from JetBrains

The above is a partial list. Check out your current tech stack and do a little Googling — there’s probably a pipeline metrics solution out there for your stack.