How to setup CodeClimate and CircleCI 2.0 parallel builds to run RSpec with SimpleCov and JUnit formatter

Published in

ITNEXT

6 min readJun 3, 2019

How to merge CodeClimate reports for your RSpec test suite executed with parallel builds on CircleCI 2.0? You will learn how to run RSpec parallel tests for your for Ruby on Rails project using CircleCI and how to send test coverage merged from parallel jobs into CodeClimate. We will cover config examples for:

How to use SimpleCov needed by CodeClimate Test Reporter to prepare RSpec test coverage summary on each parallel job and then how to merge it so you will be able to send it to CodeClimate dashboard.
How to use JUnit formatter for RSpec running on parallel jobs. Thanks to JUnit formatter you can see nice tests results in CircleCI UI view. For instance, when your tests fail then CircleCI will show you failing tests at the top of your CI build steps.
How to split your RSpec tests across parallel jobs using Knapsack Pro Queue Mode. At the end of the article, you will see a video explaining how Queue Mode in knapsack_pro ruby gem dynamically distributes specs across parallel jobs to ensure each job takes a similar amount of time to ensure CI build is as fast as possible (to get you optimal CI build time).

CodeClimate and parallel builds on CircleCI 2.0

Let’s start with the full YAML config for CircleCI 2.0. You will find comment for each important step and what it does. You may have already configured some of the stuff in your project so looking at a full example below might be more familiar to you. If you are just adding CodeClimate to your project for the first time then except below config you will need to setup simplecov gem and it’s covered in next section.

Here is the full CircleCI 2.0 example for parallel builds using RSpec and CodeClimate.

# .circleci/config.yml
version: 2
jobs:
  build:
    # set here how many parallel jobs you want to run.
    # more parallel jobs the faster is your CI build
    parallelism: 2
    docker:
      # specify the version you desire here
      - image: circleci/ruby:2.6.3-node-browsers
        environment:
          PGHOST: 127.0.0.1
          PGUSER: rails-app-with-knapsack_pro
          RAILS_ENV: test
          RACK_ENV: test

          # API token should be set in CircleCI environment variables settings instead of here
          # KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: rspec-token

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      - image: circleci/postgres:9.4.12-alpine
        environment:
          POSTGRES_DB: rails-app-with-knapsack_pro_test
          POSTGRES_PASSWORD: ""
          POSTGRES_USER: rails-app-with-knapsack_pro

    working_directory: ~/repo

    steps:
      - checkout

      # create directory for xml reports created by junit formatter
      - run: mkdir -p tmp/test-reports/rspec/queue_mode/

      - run:
          name: Install Code Climate Test Reporter
          command: |
            curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
            chmod +x ./cc-test-reporter
      - run: ./cc-test-reporter before-build

      # Download and cache dependencies
      - restore_cache:
          keys:
          - v2-dependencies-bundler-{{ checksum "Gemfile.lock" }}
          # fallback to using the latest cache if no exact match is found
          - v2-dependencies-bundler-

      - run:
          name: install ruby dependencies
          command: |
            bundle install --jobs=4 --retry=3 --path vendor/bundle

      - save_cache:
          paths:
            - ./vendor/bundle
          key: v2-dependencies-bundler-{{ checksum "Gemfile.lock" }}

      # wait for postgres to be available
      - run: dockerize -wait tcp://localhost:5432 -timeout 1m
      # Database setup
      - run: bin/rails db:setup

      # Run RSpec tests with knapsack_pro Queue Mode and use junit formatter
      # junit formatter must be configured as described in FAQ for knapsack_pro Queue Mode
      # this is also described in this article later
      # https://github.com/KnapsackPro/knapsack_pro-ruby#how-to-use-junit-formatter-with-knapsack_pro-queue-mode
      - run: bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/test-reports/rspec/queue_mode/rspec.xml]"

      - run:
          name: Code Climate Test Coverage
          command: |
            ./cc-test-reporter format-coverage -t simplecov -o "coverage/codeclimate.$CIRCLE_NODE_INDEX.json"

      # store coverage directory with CodeClimate reports prepared based on simplecov reports
      # it's special step used to persist a temporary file to be used by another job in the workflow
      - persist_to_workspace:
          root: coverage
          paths:
            - codeclimate.*.json

      # store test reports created with junit formatter in order to allow CircleCI 
      # show info about executed tests in UI on top of CI build steps
      - store_test_results:
          path: tmp/test-reports

      # store test reports created with junit formatter in order to allow CircleCI 
      # let you browse recorded xml files in Artifacts tab
      - store_artifacts:
          path: tmp/test-reports

  upload-coverage:
    docker:
      - image: circleci/ruby:2.6.3-node
    environment:
      # you can add your CodeClimate test report ID here or in CircleCI
      # settings for environment variables
      CC_TEST_REPORTER_ID: use-here-your-codeclimate-test-report-id
    working_directory: ~/repo

    steps:
      # This will restore files from persist_to_workspace step
      # Thanks to it we will have access to CodeClimate test coverage files from
      # each parallel job. We need them in order to merge it into one file in next step.
      - attach_workspace:
          at: ~/repo
      - run:
          name: Install Code Climate Test Reporter
          command: |
            curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
            chmod +x ./cc-test-reporter
      - run:
          # merge CodeClimate files from each parallel job into sum coverage
          # and then upload it to CodeClimate dashboard
          command: |
            ./cc-test-reporter sum-coverage --output - codeclimate.*.json | ./cc-test-reporter upload-coverage --debug --input -

workflows:
  version: 2

  commit:
    jobs:
      # run our CI build with tests
      - build
      # once CI build is completed then we merge CodeClimate reports
      # from each parallel job and upload summary coverage to CodeClimate
      - upload-coverage:
          requires:
             - build

SimpleCov configuration for RSpec

When you use simplecov gem in order to create test coverage for RSpec then you need to remember about one additional thing when you want to run tests in parallel on many CircleCI jobs. You set a unique name for the simplecov report with SimpleCov.command_name.

# spec/rails_helper.rb or spec/spec_helper.rb
require 'simplecov'
SimpleCov.start

# this is needed when you use knapsack_pro Queue Mode
KnapsackPro::Hooks::Queue.before_queue do
  SimpleCov.command_name("rspec_ci_node_#{KnapsackPro::Config::Env.ci_node_index}")
end

JUnit formatter for RSpec

In order to show in CircleCI UI info about your test suite like failed tests, you need to generate xml report for your RSpec test suite using JUnit formatter.

You can use junit formatter for RSpec thanks to gem rspec_junit_formatter.

# knapsack_pro Queue Mode
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/test-reports/rspec/queue_mode/rspec.xml]"

The xml report will contain all tests executed across intermediate test subset runs based on work queue. You need to add after subset queue hook to rename rspec.xml to rspec_final_results.xml thanks to that the final results file will contain only single xml tag with all tests executed on the CI node. This is related to the way how Queue Mode works. Detailed explanation is in the issue.

# spec_helper.rb or rails_helper.rb

# TODO This must be the same path as value for rspec --out argument
# Note the path should not contain sign ~, for instance path ~/project/tmp/rspec.xml may not work. Please use full path instead.
TMP_RSPEC_XML_REPORT = 'tmp/test-reports/rspec/queue_mode/rspec.xml'
# move results to FINAL_RSPEC_XML_REPORT so the results won't accumulate with duplicated xml tags in TMP_RSPEC_XML_REPORT
FINAL_RSPEC_XML_REPORT = 'tmp/test-reports/rspec/queue_mode/rspec_final_results.xml'

KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
  if File.exist?(TMP_RSPEC_XML_REPORT)
    FileUtils.mv(TMP_RSPEC_XML_REPORT, FINAL_RSPEC_XML_REPORT)
  end
end

This example is based on FAQ.

Summary and Queue Mode to do dynamic test suite split

CI builds can be much faster thanks to leveraging parallel jobs on Circle CI 2.0 and CI parallelisation on any CI provider (see more parallelisation examples for your CI providers). You can check Knapsack Pro tool for CI parallelisation and learn more about Queue Mode and what problems it solves in below video.

Installation guide for knapsack_pro gem can be found here in order to setup your RSpec test suite.

— — —

Originally published at: https://docs.knapsackpro.com/2019/codeclimate-and-circleci-2-0-parallel-builds-config-for-rspec-with-simplecov-and-junit-formatter

How to setup CodeClimate and CircleCI 2.0 parallel builds to run RSpec with SimpleCov and JUnit formatter

CodeClimate and parallel builds on CircleCI 2.0

SimpleCov configuration for RSpec

JUnit formatter for RSpec

Summary and Queue Mode to do dynamic test suite split

Written by Artur Trzop