Fast(er) tests with Stunt Doubles and VCR Recordings.

This blog originally appeared here.

Making Tests Specific

One benefit of writing specific tests is that it requires your code to be well organised. For example, earlier on in the application, the code which talks talk to the Shopify APIs were scattered all over the application. But, because I wanted each of my tests to be specific and do only one thing, I had to move this part of the code which is used a lot in communicating with the APIs into its own section - a ShopifyWrapper module - whose only purpose is to interface with Shopify. By clearly having these application boundaries in my code, it helped with writing better tests. And in my application, I only need to interact with the Shopify wrapper - and the wrapper takes care of communicating with the Shopify APIs.

For example, in the background job below, I only have to call ShopifyWrapper::Download.all_products and then deal with the response from it. The wrapper talks to the Shopify APIs.

class ShopifyApi::DownloadAllProductsJob < ActiveJob::Base
  queue_as :default

  around_perform do |job, block|
    shop =  Shop.find_by id: job.arguments.first
    shop.with_lock do
      block.call
    end
  end

  def perform(shop_id, job_created_time)
    shop =  Shop.find_by id: shop_id
    if shop.started_downloading_products == false 
      shop.update(started_downloading_products: true)
      download = ShopifyWrapper::Download.all_products(:shop => shop)
      if download.successful?
        shop.update(sync_time: DateTime.now, downloaded_success: true)
      else
        shop.update(sync_time: DateTime.now, downloaded_success: false)
      end
    end
  end
end

This made testing easier - as I could write Shopify API tests in one place, and test it thoroughly. While writing these ShopifyWrapper tests, there were two different approaches I could take.

1. I could write tests that cover everything upto hitting the Shopify APIs, and then fake the response coming back from the Shopify APIs.
2. Or I could fully integrate with the Shopify APIs - and go all the way, and hit their servers in real time and get the real data coming back from the requests. Choosing this option means the wrapper spec will have to handle and interpret the data coming back from Shopify in real time.

For testing the Shopify Wrapper, I chose the second option - i.e to truly integrate with the Shopify APIs and go all the way. For example, the below code shows the RSpec tests for testing the different download methods of the Shopify Wrapper.

require 'rails_helper'

describe ShopifyWrapper do 
  describe ShopifyWrapper::Download do 
    describe ".all_products" do 
      context "with a valid shop" do
        let(:shop) { Fabricate(:shop) } 

        it "returns the products successfully" do 
          response = ShopifyWrapper::Download.all_products(:shop => shop)

          expect(response).to be_successful 
        end

        it "saves the products to the shops database" do 
          shop.products.destroy_all

          response = ShopifyWrapper::Download.all_products(:shop => shop)
          shop.reload

          expect(shop.products.size).to eq(response.response.size)
        end

        it "saves the product images to the shops database" do 
          shop.products.destroy_all

          response = ShopifyWrapper::Download.all_products(:shop => shop)
          shop.reload

          expect(shop.products.first.images.size).to eq(response.response.first.images.size)
        end
      end

      context "with an invalid shop" do
        let(:shop) { Fabricate(:shop, shopify_domain: 'invalidDomain.com', shopify_token: 'invalidToken') } 
        let(:response) { ShopifyWrapper::Download.all_products(:shop => shop) } 

        it "does not return the products" do 
          expect(response).to_not be_successful 
        end

        it "contains an error message" do 
          expect(response.error_message).to eq("There was an error while downloading the products from Shopify.")
        end
      end
    end

    describe ".single_product" do 
      context "with a valid product" do
        let(:shop) { Fabricate(:shop) } 
        let(:product) { shop.products.create(title: "Shirt", shopify_product_id: "4568022220932") } 

        it "it downloads the products successfully" do 
          response = ShopifyWrapper::Download.single_product(:shop => shop, :product => product)

          expect(response).to be_successful 
        end

        it "saves the products images to the database" do 
          product.images.destroy_all

          response = ShopifyWrapper::Download.single_product(:shop => shop, :product => product)
          product.reload
          expect(product.images.first.src).to eq(response.response.images.src)
        end
      end
    end
  end
end  

In all other areas of the code which deals with Shopify APIs, I just had to test until my application boundary, and then fake the response from SHopify. I could do this because the ShopifyWrapper spec tests the Shopify APIs thoroughly.

Making Tests Fast with VCRs and cassettes

While running the ShopifyWrapper spec, it takes 9 seconds for actually running the spec.

The reason it is taking so long is that it hits the live Shopify APIs 12 times in this spec. Six times to create a new Shopify session, and six times to download the products from the Shopify store. And as my application grows, I am going to add more and more tests in this spec, and it will take longer to execute.

The only way I could solve this problem and make it faster was to not hit the Shopify API every time - and instead to have a dummy response. I could do this in two ways.

1. Webmocks - This is a library that does exactly this. It stubs the HTTP requests at the http client library level and you can tell exactly what the response is.

For e.g. ruby stub_request(:post, "www.example.com") with(body: "abc", headers: { 'Content-Length' => 3 })


But the problem with this is that I have to change the code in the tests manually. And stubbing the response this way is not ideal - I do want to hit the Shopify API and get back the response from them. What if Shopify changes the way they send data back via their APIs? I wouldn't be able to catch these changes with Webmocks, as Webmocks does this on a low-level. Ideally, I need something at a higher level, and which also integrates with RSpec nicely - so that I don't have to manually change the test code.

2. VCR - This is where VCR is really useful. VCR is a library that records your test's HTTP interactions and replays them during future test runs for faster tests. This is perfect - as it hits the Shopify APIs and gets the response back in real time the first time the test runs. It then records all this information and saves them into a cassette! So, the next time the test runs again, it will just play back the recording - and it will not hit the Shopify endpoints for any subsequent runs.

It is also highly customisable. I can change it to hit the end points all the time if I want to make sure that the Shopify APIs have not changed the way they send data back via their endpoints. Having this option is good because I can switch between these modes as and when required. For the majority of the time, I will be using the VCR recordings - and every once in a while I can hit the end points in real time. This way my tests are going to be secure and run faster.

I added VCR, and you can see that when I run the ShopifyWrapper spec again it takes only 1.06 seconds to run because it did not go to the actual APIs, it just used the local recordings.

I saved 8 seconds from one spec alone by implementing VCR

You can see that VCR automatically created a ShopifyWrapper_Download directory under `spec/cassettes/ShopifyWrapper' and for each context it creates a data file that is uses for the tests. This is super helpful - and it integrated really well with RSpec.

Making Tests Faster with Test Doubles and Method Stubs

In the Test Desiderata videos, Kelly Sutton talks about why speed is important in testing - and also one of the reason they want faster tests is so that they can remain in flow (and switch off the monkey-brain). Paul Graham talks about the makers schedule vs the managers schedule - and how it is important for makers to have dedicated blocks of time just for making.

"For someone on the maker's schedule, having a meeting is like throwing an exception. It doesn't merely cause you to switch from one task to another; it changes the mode in which you work." - Paul Graham

This maybe a bit exaggerated - but I feel slow tests and meetings do exactly the same thing. While I am waiting for the tests to complete, my mind wanders - thinks about lunch, what to buy, plans for the evening, the game last evening - and many other things before I have to re-focus and get it back to programming mode. So, even if I have a dedicated 4-5 hour maker schedule, and no one else is disturbing me - slow tests can interrupt the flow with context switching.

In my application, I talk with the Shopify APIs a lot. And in many different places. I am calling the ShopifyWrapper::Download class itself 14 times in my application. Apart from that, there are other classes in the ShopifyWrapper which are called repeatedly.

Using VCRs does make testing faster. But I can make it even faster by using Test Doubles and method stubs.

This is how the ShopifyWrapper::Download class code looks like.

As I already tested the Shopify Wrapper thoroughly, I don't really need to call the `ShopifyWrapper::Download' class again in other places (14 times!) in my tests. Calling it each time will in-turn call the Shopify APIs (or use VCR recordings). Instead I could stub this method to return something specific - which in this case is going to be a successful download.

So, I started by created a test double object called 'download' --- This is basically a fake object which stands for the actual return value from ShopifyWrapper. I then stubbed the successful method of this class and made it return the 'download' test double we created earlier.

download = double('download')
download.stub(:successful?).and_return(true)

ShopifyWrapper::Download.stub(:all_products).and_return(download)

In another context, when I want to test for an un-successful download, I can again stub this method and make it return false instead. I can also stub the test double with the error message.

download.stub(:error_message).and_return("There was an error connecting to Shopify.")

The reason I am doing this is because I am calling the ShopifyWrapper multiple times in my code. For example, when a new user signs up, and after I create a Shop, the after_commit method calls a job, which in-turn calls the ShopifyWrapper.

class Shop < ActiveRecord::Base
  ...

  after_commit :set_up_shop_first_time

  def set_up_shop_first_time
    ShopifyApi::DownloadAllProductsJob.perform_later(self.id, DateTime.now)
  end

  def api_version
    ShopifyApp.configuration.api_version
  end
end


class ShopifyApi::DownloadAllProductsJob < ActiveJob::Base
  ...

  def perform(shop_id, job_created_time)
    ...
    download = ShopifyWrapper::Download.all_products(:shop => shop)
    if download.successful?
      shop.update(sync_time: DateTime.now, downloaded_success: true)
    else
      shop.update(sync_time: DateTime.now, downloaded_success: false)
    end
  end
end

Now, if I didn't stub the wrapper - it will actually execute the code and hit Shopify apis (or VCR recordings) again. But, because I have stubbed it, it will always return the test double that we created.

This makes my tests faster... I am mainly using Test doubles and stub method for two reasons:

1. Hitting the Shopify APIs every time is expensive (takes a long time) - so I can replace it with something that is very cheap (Test doubles and stub methods). I first created a test double, and then stubbed the test double itself to make it in a state that I want.

2. The other reason I am using Test doubles is because the Shopify Wrappers have been thoroughly tested before. And by not repeating the testing in the current spec, it keeps my tests focussed and noise-free.

Speed limits

Even though I wanted my tests to run really fast, I had to slow it down to include feature tests. Feature tests simulate real user experiences - adding new images, re-ordering positions, bulk editing images, and going all the way to the Shopify APIs to update a product with the right image, and then coming back to show the success or the error messages of the response.

For example, here is one feature test - that tests for dragging images between different products, and re-arranging it's position.

In each of my feature tests, I only tested for the happy path. I didn't test for errors or intentionally break things in feature tests. These were already covered in the other tests, which runs much faster than feature tests.

This was a happy compromise I made while writing tests. Even though it made my tests slower, it gave me more confidence in my code.

Do check out the videos from Kent Beck and Kelly Sutton to learn more about the best practices while testing. I like the fun way they break down important concepts into smaller videos. Also, reading the docs from Rspec helped a lot.

Thanks for reading.

Arjun