Month: October 2017

Did you know that there’s a regression in Xcode 9’s support for automated tests involving the Keychain?

To show you, I’ve updated the Secrets test application to have two unit test targets, “Secrets Hosted Tests” and “Secrets Unhosted Tests”. They are both unit test targets and, as advertised, the first runs against the Secrets application, and the second does not, relying on Apple’s built-in mechanism to run unit tests. This means the second target needs to include the necessary SAMKeychain files instead of relying on the app to provide them.

Both targets execute only one test, the exact same one: after checking and trying to delete any previous entry, it tries to save a string to the Keychain:

func testSave() {
    do {
        let _ = try SAMKeychain.password(forService: "MyService", account: "MyTestAccount")

        try SAMKeychain.deletePassword(forService: "MyService", account: "MyTestAccount")
    } catch let error as NSError {
        if (error.code != Int(errSecItemNotFound)) {
            print("\(error)")
        }
    }

    XCTAssertNoThrow(try SAMKeychain.setPassword("Foo", forService: "MyService", account: "MyTestAccount"),
                     "Throws!")
}

If you run these two tests in Xcode 8 in the Simulator against iOS 9 and against iOS 10, they succeed under iOS 9 and they succeed under iOS 10. Good so far.

But if you run them in Xcode 9, with the exact same Simulators, only the hosted tests succeed under all versions of the OS: iOS 9, iOS 10, and iOS 11. The unhosted tests only succeed under iOS 9. They fail under iOS 10 with the error -34018 (“A required entitlement isn’t present”), and they fail under iOS 11 with the error -50 (“One or more parameters passed to a function were not valid”).

Puzzling, isn’t it? It can’t be an entitlements issue or a parameter issue, because the exact same thing works when tested with a host application, and works under an earlier Xcode.

I have some anecdotal evidence that Xcode 8 in its early releases had similar issues with Keychain testing in the Simulator, fixed in later versions, so here’s hoping Apple can once again fix these issues in later versions of Xcode 9. I’ve filed a Radar about it.