Putting It to the Test

I’ve been thinking about automated tests recently, and I’ve come up with a number of observations based on my experience.

I’ve rarely if ever seen unit tests break.
In theory, the tests exist so that when you add functionality to a class or a group of classes in a way that changes the code paths of existing functionality, you’re sure the existing functionality still works. In practice, I’ve seen that either those classes are never touched again, or they’re only added to, not modified, or the modifications are simple enough that they just don’t lead to breakage.

Rewrites don’t bring the tests along.
In my experience, when you’re doing a rewrite of your GUI application code, for, let’s say, a new major release, you’re generally changing the logic in the new code enough that the automated tests don’t come along. So any exhaustive, pre-existing tests you have need to be thrown out. For example, if you’re moving your model to Core Data for version 2 of your app, much of your existing model classes tests are now outdated.

I’ve only had one case where a rewrite kept the same logic, and that was for a recent side project that was more of an example than anything else. And in that case, I wrote the exhaustive tests right before doing the rewrite. That worked really nicely, but, if anything, it’s an argument for putting off writing such exhaustive tests until you know you need them for something specific.

Regression tests tend to run into sorting issues.
I’ve written extensive regression tests for some projects, and I’ve tended to run into problems with sorting. NSArray’s sorting APIs are unstable. So I’ve often had to write extra logic into the production code to provide reproducibility that it doesn’t actually need, for the sake of testing.

Whenever I talk to any colleagues about automated testing, there’s always a wistful quality to it; yes, we don’t have automated tests or we only have a few, but we should be testing everything. We’ll get to it when we have time. You’re not a real developer unless you’ve done it or at least feel inadequate about it.

But in my, possibly limited experience (I did, after all, work on the same app for almost 6 years), I just don’t see them as the panacea that others do.