Portable Pipelines

CI shinobi masters

Nowadays we can find a lot of CI vendors. Let’s say that one day your team decided to switch from Jenkins to Gitlab CI or CircleCI, what would be the impact for this migration ? If your pipeline is very dependent on Jenkins plugins and every step is configured through the CI and not documented, this migration would be a sad story.

Then we come to test if our pipeline is portable and easy to migrate ? A portable pipeline for me would be the one which minimize the cost of migration from one vendor to another.

Following some best practices can help you a lot for this migration and achieve a portable pipeline.

Let’s take a look about this CI build step in Jenkins.

Migrating this would imply copying the same build step into the new vendor. A small refactor here can save you a lot of time.

Let’s break this into multiple components and combine them in the build directory that will look like this:

build
├── Dockerfile
├── build.sh
├── test.sh
└── docker-compose.yml

Where our bash scripts will be responsible to our CI steps:

# build/build.sh
docker-compose -f build/docker-compose.yml build
# build/test.sh
docker-compose -f build/docker-compose.yml sh -c bundle install \
  && bundle exec rake db:drop db:create db:migrate \
  && bundle exec rspec spec

Then we call these scripts from Jenkins:

Changing this will help you extract the build steps into the code repository. This is a quick improvement you can do; however, you have to configure this when you migrate to the new vendor.

To fix this we can abstract our steps into some configuration file which almost all the CI vendors can provide now. For Jenkins we can use Jenkins Pipeline. Here is an example:

pipeline {
  agent any
  stage('Build') {
    steps {
      script {
        sh "build/build.sh"
      }
    }
  }
  stage('Test') {
    steps {
      script {
        sh "build/test.sh"
      }
    }
  }
}

Moving to another platform other than Jenkins means we have to change this configuration file which now exists in the repository itself.

Pushing every CI configuration into the repo would make the pipeline visible to the whole team and also a good documentation tool.

So to ease CI migration from vendor to another, make sure to:

• Push every pipeline step into the repository as scripts. Using bash is always a good option but going to ruby or python is also a great candidate when things get more complex.
• Use the vendor configuration file as much as possible and minimize the usage of manual configuration from the UI.