Xcode 10 Random And Parallel Tests

Xcode 10 introduces three new options for running your tests. These allow you to randomize the execution order, speed up the tests by running them in parallel on multiple simulators and finally to control if Xcode automatically adds new tests to a scheme.

You’ll find all three new test options in the scheme editor. Look in the test phase of the scheme and select the Options button for a test target:

The defaults match Xcode 9 behavior. Let’s look at each option:

Randomize Execution Order

By default, Xcode runs tests based on the alphabetic order of the test method names. So testA() will always run before testB(). This can hide subtle dependencies on state between your tests. To help detect these dependencies Xcode 10 can randomize the execution order if you select the option in the scheme:

Parallel Test Execution (iOS Simulator)

Xcode 9 allowed you to run tests on multiple destinations at the same time from the command-line:

$ xcodebuild test -scheme MyApp \
  -destination "plaform=iOS,name=iPhone 8" \
  -destination "platform=iOS,name=iPhone 8 Plus"

This ran all of the tests in the scheme on all destinations at the same time. This speeds up running tests on multiple destinations, but the tests are still executed in (alphabetical) order, one at a time, on each destination.

Xcode 10 introduces parallel distributed testing to run tests in parallel for the same destination. Xcode launches multiple test runners based on the available cores. Each test runner creates a clone of the original simulator for the destination and receives a test class to execute (so if you keep all your unit tests in a single test class you may want to do some refactoring).

As with the execution order, edit the scheme to enable parallel running using the options for the test target:

You can run iOS unit or UI tests in parallel on the simulator (Xcode can’t clone your physical devices). When you run parallel tests you’ll see Xcode launch clones of the simulators:

Separate Bundle For Non-Parallel Tests

You set the options to parallelize and randomize tests for a target in the scheme. You cannot override the settings for individual tests. If you have tests that you don’t want to run in parallel put them in a separate bundle with parallel running disabled (the default for a new scheme):

Automatically Add New Tests

In Xcode 9 any new tests you added to the test target were automatically included when you ran the tests unless you manually deselected the tests in the scheme:

This is a pain if you create a new scheme to run a subset of the available tests. You are forced to manually deselect any new tests you add after creating the scheme.

In Xcode 10 you can edit the scheme to prevent it automatically adding new tests to the scheme. For example, suppose I wanted a scheme that ran a subset of unit and UI tests:

Create the new scheme and add the unit and UI test targets.
Select the tests you want to run from each target.
Disable the option to include new tests (for both targets) automatically:

Now if you add an extra unit or UI test, Xcode will not automatically include it when you run this scheme.