Test isolation with XCTestExpectation

Asynchronous tests are hard and recently I found a new rough edge when using XCTestExpectation. Take a look at these examples and try to guess the outcome from the following options:

• No tests fail
• testA fails
• testB fails
• Both testA and testB fail

class AsyncTestsTests: XCTestCase {
    func testA() throws {
        let expectation = self.expectation(description: "async work completed")
    
        asyncFunction { result in
            XCTAssertEqual(0, result)
            expectation.fulfill()
        }
        
        waitForExpectations(timeout: 1)
    }
    
    func testB() throws {
        let expectation = self.expectation(description: "async work completed")
        
        asyncFunction { _ in
            expectation.fulfill()
        }
        
        waitForExpectations(timeout: 3)
    }
}

func asyncFunction(completion: @escaping (Int) -> Void) {
    DispatchQueue.main.asyncAfter(deadline: .now() + 2, execute: { completion(42) })
}

On Xcode 12 both of these tests are flagged as failing, which was very suprising at first. The code above follows the same flow that the Apple example code demonstrates here, where there are assertions performed within the async callback. Anecdotally when checking for blog posts on how to use XCTestExpectation they seemed to mostly follow the pattern in the Apple docs with a few exceptions that used the proposed solution below.

What’s happening?

If the tests are run without the random order/concurrent options then testA will be executed before testB.

testA fails because the expectation times out so you get a failure with Asynchronous wait failed: Exceeded timeout of 1 seconds, with unfulfilled expectations: "async work completed"..

The code in testB shouldn’t actually fail but because of the way the assertion in testA is performed in the async callback the timing means that the assertion fails whilst testB is in progress. This results in Xcode associating the failed assertion with testB.

The following diagram shows a timeline of the above tests and how the assertion from testA bleeds into the execution of testB.

How do I avoid this?

You can avoid this by not performing assertions in the async callback but instead capturing any results and performing the assertions after the wait.

func testA() throws {
    let expectation = self.expectation(description: "async work completed")
    
    var capturedResult: Int? = nil

    asyncFunction { result in
        capturedResult = result
        expectation.fulfill()
    }
    
    waitForExpectations(timeout: 1)
    
    XCTAssertEqual(0, capturedResult)
}

After making this change the tests now behave as I would expect with testA failing and testB passing.