XCTest passes when it should fail using expectations

Question

I am testing a method that runs in background and executes a code block when it finishes. I am using expectations to handle the asynchronous execution of the tests. I wrote simple a test that shows the behaviour:

- (void) backgroundMethodWithCallback: (void(^)(void)) callback {
    dispatch_queue_t backgroundQueue;
    backgroundQueue = dispatch_queue_create("background.queue", NULL);
    dispatch_async(backgroundQueue, ^(void) {
        callback();
    });
}

- (void) testMethodWithCallback {
    XCTestExpectation *expectation = [self expectationWithDescription:@"Add collection bundle"];
    [self backgroundMethodWithCallback:^{
        [expectation fulfill];

        usleep(50);
        XCTFail(@"fail test");
    }];
    [self waitForExpectationsWithTimeout: 2 handler:^(NSError *error) {
        if (error != nil) {
            XCTFail(@"timeout");
        }
    }];
}

The XCTFail(@"fail test"); line should fail for this test but the test is passing.

I also noticed that this only happens when the code ran on the callback takes an amount of time (in my case, I was checking some files on the file system). This is why the usleep(50); line is necessary to reproduce the case.

score 6 · Accepted Answer · answered Mar 31 '15 at 10:21

The expectation must be fulfilled after all the test checks. Moving the line to the end of the callback block is enough to make the test fail:

- (void) testMethodWithCallback {
    XCTestExpectation *expectation = [self expectationWithDescription:@"Add collection bundle"];
    [self backgroundMethodWithCallback:^{

        usleep(50);
        XCTFail(@"fail test");
        [expectation fulfill];
    }];
    [self waitForExpectationsWithTimeout: 2 handler:^(NSError *error) {
        if (error != nil) {
            XCTFail(@"timeout");
        }
    }];
}

I did not find explicit documentation about this but in the apple developer guides, the fulfill message is sent at the end of the block and it makes a lot of sense.

Note: I first found an example in swift where the fulfill method is called at the start of the callback. What I don't know is if the example is not correct or there is a difference with Objective-C.

Rob · Answer 2 · 2015-03-31T11:32:35.093

The block called by backgroundMethodWithCallback is immediately fulfilling the expectation, thereby letting the test finish before XCTFail is called. If the block fulfills the expectation before it finishes performing other actions, you end up with race condition, in which the behavior of the test is conditional upon the speed with with the rest of the block is performed. But one shouldn't reasonably expect XCTFail to be captured if the test, itself, has already finished.

Bottom line, if you move the [expectation fulfill] to the end of the block, this race condition is eliminated.

XCTest passes when it should fail using expectations

2 Answers2