Introduction

Designing a good test suite is critical to the development of a reliable application, but it involves constant decisions about how closely to replicate your end product. For example, it might seem like the best test suite is one that only tests the application by simulating user actions through the front end, however, test suites like this are one of the most common types of technical debt. They are often very brittle and take orders of magnitude longer to run than test suites mostly composed of unit tests and integration tests.

Another place where this decision is challenging is when your app integrates with outside APIs. In some cases, this is completely fine because connecting to the outside API in your test suite gives the most up to date and realistic information, however, there are times where this is a bad idea. Some APIs have high costs per API call, others have strict rate limiting, and other times pulling live information can force you to create tests that only pass intermittently.

This was the situation we found ourselves in when designing a Rails app that integrated with the Shopify REST API. The Shopify API is powerful and easy to use with great documentation. We love using it to build e-commerce stores for our clients and create apps for the app store. However, integrating with the test suite directly in our app caused a number of problems

The Problems With Directly Referencing the Shopify API in Rspec

First, the Shopify REST API has a fairly strict rate limit for each store (2 calls per second) that we was causing intermittent errors and forcing tests that should have been testing "happy paths" to instead go down error recovery paths.

A second problem with connecting directly to the Shopify API was performance. The Shopify API is not slow by itself, but when a large percentage of our tests were making HTTP calls to an outside service, our relatively small test suite was taking upwards of 20 minutes to finish, which was slowing down our deploy process and our development significantly.

Related to performance, the test suite requiring an outside service meant developers needed internet access to run the test suite. Obviously, this was not our biggest issue our team is always online, but it is sometimes convenient to be able to write and test code offline or when internet is spotty, so it was a small pain point.

Finally, with the Shopify API calls directly in our test suite, we had to create development stores with API keys for our developers to test on. Managing that information quickly became a hassle and we began to research the best way to mock out the Shopify API in the majority of our tests.

Our Solution Using a Custom Sinatra App

The problems listed above have a number of different solutions. Some common solutions are using middleware such as Rack::Test or using a gem like VCR to record and playback your app's HTTP calls or Thoughtbot's gem called capybara_discoball. However, while researching these options, we became inspired by a separate gem from Thoughtbot called fake_stripe.

The fake_stripe gem uses a small Sinatra app and the Webmock gem to simulate connecting to Stripe without ever actually hitting their servers.

Building off of this idea, we created our own fake_shopify gem which uses a similar architecture to turn off HTTP requests in our test suite and instead return specific JSON data when a call is made to the Shopify API URL.

We left a small number of "sanity" tests that do connect to Shopify's actual API to ensure we have a few failing tests in the case that something breaks in that part of our codebase.

The Sinatra App Architecture

The Sinatra app is very basic and only has 3 real pieces: the route files, the JSON fixture files, and the code connecting the fixtures to the stubbed HTTP request. All of this code is reviewable on Github, so we'll describe it quickly below.

The first piece you'll need is to route your HTTP traffic into your Sinatra app. In our gem, we create a FakeShopify module with a single stub_shopify method to do this. The /domain.myshopify.com/ is the hardcoded URL that we'll eventually direct all traffic to when we want a stubbed response.

module FakeShopify
  extend Configuration

  def self.stub_shopify
    stub_request(:any, /domain.myshopify.com/).to_rack(FakeShopify::StubApp)
  end
end

Next, we add the routes that the Shopify API uses. A good chunk of this work was done manually taking information from the API Shopify reference into our routes files.

For the fake_shopify gem, there are dozens of these routes, but they all follow the same pattern as this example:


module FakeShopify
  class StubApp < Sinatra::Base
    # RecurringApplicationCharge API Endpoints
    get "/admin/api/:api_version/recurring_application_charges.json" do
      json_response 200, fixture(params[:api_version], "billing/recurring_application_charges/index")
    end
  end
end

These routes contain two helper methods fixture and json_response for building the fixture and formatting the json_response. The Shopify API releases a new version roughly every 3 months, and the version needs to be added to the API URL, so we pass that value in to our fixture method.

    def fixture(api_version, file_name)
      file_path = File.join(FakeShopify.fixture_path, api_version.to_s, "#{file_name}.json")
      File.open(file_path, "rb").read
    end

    def json_response(response_code, response_body)
      content_type :json
      status response_code
      response_body
    end

The final piece is adding the fixtures themselves. The Shopify API has excellent example JSON responses for all of their routes, but you can also generate these fixtures by copying over any JSON response from an API call.

Conclusion

We split the fixture files by api version and then use the naming conventions provided by Shopify themselves, but these fixtures can be stored in any local file as long as your routes are providing the correct file path.

We've found a large improvement after switching our test suite from connecting directly with the Shopify API to a stubbed out Sinatra API that gives us more control and better performance.

We highly recommend that other applications that depend closely on 3rd party APIs try out a similar approach of mocking out that API in their test suite.

If you have a Shopify App on the App Store or just a private app you use for running custom tasks, give our fake_shopify gem a try, and let us know how it works for you.

If you'd like more advice on what improvements you could make to your test suite, please reach out to us and we'd be happy to help you. Feel free to also check out our offerings focused on the health of your company's CI/CD pipelines or custom application development as a whole.