Running Rails 5 System Tests on Travis CI with Chromedriver

Rails 5.1 introduced the concept of system tests using Capybara and Selenium. System tests takes the practice of "feature" tests that many Rails apps were already doing and brings it into an official part of the Rails testing strategy. Since system tests utilize a browser, they can execute Javascript, something that Rails integration tests could not do.

System tests work great out of the box on local machines, but getting them set up to run on a remote server like Travis CI is a little harder. There's not a lot of good documentation out there for what needs to be done, and a lot of solutions have to be dug up through Stack Overflow and GitHub comment threads.

In this post, I'll show how I set up system tests to run on Travis CI using headless Chrome and Chromedriver.

Rails 5.1.6

Although system tests were announced with 5.1, you really should make sure you are using Rails 5.1.6 to get the best experience. This is because after the initial release there were a few other small quality of life improvements made to the framework that are great to have.

Among these are a change to ActiveRecord that allows the Capybara thread and the test server thread to use the same database connection (see rails/rails#28083). This is important when using transactions because otherwise the two threads would not be able to see DB changes made by the other thread.

In the past, you would have had to work around this by turning off transactions and adding Database Cleaner or some similar gem to clean up data created during test runs. But this would have meant that either Rails would need to pull Database Cleaner in as a dependency, or anyone wanting to use system tests would need to add another gem to their Gemfile manually. Neither of those were good for the future adoption of system tests, so instead the Rails team updated ActiveRecord to allow the connection to be shared.

RSpec 3.7

I prefer using RSpec as my test framework. RSpec 3.7 added support for system tests into the framework, replacing the previous feature specs.

If you are using RSpec, you will need to make sure you are using version 3.7 in order to use system tests.

Installing Chromedriver

Chromedriver allows Chrome to be controlled by Selenium and is necessary in order for us to use Chrome as the browser in system tests.

You can download Chromedriver from the project's website, or it can be installed with Homebrew Cask:

$ brew cask install chromedriver

Removing chromedriver-helper

If you have worked with Chromedriver in the past, you may have installed the Ruby gem chromedriver-helper. If you want to use Chromedriver directly, you will need to remove chromedriver-helper and it's executables from your system.

If you're using rbenv, you can use the commands I used:

$ gem uninstall chromedriver-helper
$ ls ~/.rbenv/versions/*/bin/* | grep chromedriver | xargs rm
$ rbenv rehash

Note: Removing chromedriver-helper may or may not be necessary for you, depending on your setup. I found it was causing problems for me, so I decided to remove it.

Selecting a driver

System tests abstract Capybara's drivers using the driven_by method. With regular system tests, you would include driven_by in your ApplicationSystemTestCase. RSpec, however, does not use ApplicationSystemTestCase, so you need to configure the driver in a before block.

The easiest way to do this is by adding a global before(:each) to rails_helper.rb:

RSpec.configure do |config|
  config.before(:each, type: :system) do
    driven_by :selenium, using: :chrome
  end
end

This will set all system tests to be driven by Selenium using Chrome.

type: :system indicates that this before block will only apply to tests marked as system tests. RSpec will infer that a test is a system test if it is located in spec/system, or you can tag specs manually.

Now running bundle exec rspec should open a Chrome window and run your system specs. Cool!

Going headless

Opening a browser window works fine on your local machine, but it won't work on a remote server because there's no monitor to render the window. Instead, we have to use Chrome in headless mode.

To use headless Chrome, we can pass some extra arguments to driven_by:

 RSpec.configure do |config|
   config.before(:each, type: :system) do
-    driven_by :selenium, using: :chrome
+    driven_by :selenium, using: :chrome, options: { args: ["headless"] }
   end
 end

There are other ways to use headless mode, and the downside of this method is that it causes Selenium to throw a deprecation warning. For reasons we'll see later, we have to use this method of passing arguments.

Now running bundle exec rspec should not open a browser window. Neat!

Running on Travis

Getting the tests to run on Travis is the trickiest part. For my build, I wanted to use Travis's Docker infrastructure with sudo: false in my .travis.yml file. This meant that I had to work around several limitations.

Installing Chrome and Chromedriver

To install packages on Travis, you need to use the addons section in your config file. Although Travis has an addon specifically for Chrome, I ended up having some problems using this. Instead, I used the apt addon to install both Chrome and Chromedriver:

addons:
  apt:
    - google-chrome-stable
    - chromium-chromedriver

Selenium expects to find Chromedriver on the PATH, but the default installation on Travis does not put the binary on the PATH. To fix this, I created a symlink in the before_script step:

before_script:
  - ln -s /usr/lib/chromium-browser/chromedriver ~/bin/chromedriver

Now the Travis environment is set up correctly.

Chrome crashes on launch

Even with the environment set up correctly, my tests were still failing because Chrome would immediately crash on launch. After a lot of digging, I found this troubleshooting page.

There are two takeaways:

1. We need to run Chrome with --no-sandbox in order for it to work properly on Travis.
2. We need to disable usage of /dev/shm because Docker provisions too little space in that device and Chrome crashes if there's not enough space.

In order to do this, we need to be able to pass custom arguments to Selenium, which is why we are using the driven_by options that we have.

 RSpec.configure do |config|
   config.before(:each, type: :system) do
-    driven_by :selenium, using: :chrome
+    driven_by :selenium, using: :chrome, options: { args: ["headless", "disable-gpu", "no-sandbox", "disable-dev-shm-usage"] }
   end
 end

The disable-gpu argument is there on recommendation from this page.

Now your system tests should successfully run on Travis! Awesome!

Extra credit: Filter system tests with metadata

As we have it set up now, RSpec will run all system tests whenever we run the full test suite. But system tests are slow and we probably don't want to run them all the time. Luckily RSpec has a way to filter out certain specs from the run based on the metadata on the spec.

Let's tag all specs in spec/system with the browser tag. To do this, we can use define_derived_metadata:

RSpec.configure do |config|
  config.define_derived_metadata(file_path: /spec\/system/) do |metadata|
    metadata[:browser] = true
  end
end

Then we can use filter_run_excluding to filter browser specs by default:

RSpec.configure do |config|
  config.filter_run_excluding browser: true
end

Now when running bundle exec rspec, system tests will be filtered out. To run the system tests, we can use the -t option to include browser specs.

$ bundle exec rspec -t browser

Remember to change your .travis.yml to use -t to run the system tests. Having system tests split off from the rest of the test suite allowed me to parallelize running the slow system tests with running the rest of the suite.

Conclusion

System tests are a really easy way to get started with feature testing. The default configuration shipped with Rails should work for most people, and extending it to use a different browser or pass different options is straightforward. What was harder was trying to figure out how to use system tests on Travis, with not a lot of documentation or examples to go off of.

With this post, I have gathered together everything I learned in getting system tests running on Travis CI. I hope it's useful!