Tag: Test-Driven Development

As I’ve said before, for my small home project, where the most interesting work lies at the intersection of AppKit and a cross-platform C++ engine, I can’t write code the “unit test” way – writing the tests first – because those intersections are precisely what can’t be isolated from the entire application.

But I have changed some habits.

Now, I move the little snippets of logic I’ve written from their organically-grown locations to separate stand-alone functions.

These functions are grouped by purpose in their own files, such as UTF8 Functions.c or Geometry Translation Functions.m.

Because they aren’t classes, they need all their context passed in as parameters. This can be a pain in the ass, but it also prevents me from grouping too much functionality together; it would be too tedious to add all the necessary parameters.

It also means that I can access the logic directly from unit tests. I can try out in isolation all the little variations I need to really ensure the code is sound.

This works well for a small, home project. Splitting functionality out into lots of additional files may work less well in large-scale endeavors, where there are already hundreds of files in a project.

Then again, in such huge projects, the data model may be elaborate enough that there are more classes that, without modification, can be tested in isolation from AppKit (or each other).

I was getting a strange smearing effect when I tried to scroll in the main view of my application’s window.

I knew my content-drawing routines were being invoked correctly. I knew that the effect seemed to be happening somewhere in the depths of Cocoa, not in my code. But that’s all I knew.

Turns out, it’s because I was setting my view frames to fractional values (they’re floats, after all), but my content engine was still drawing things per integer coordinates.

This is the kind of thing that unit testing won’t help.

My goal was “make resizing the window and then scrolling through the main view work.” Can’t write a test for that because “work” means “look right,” and you can’t test for that when you don’t know how to make it look right yet.

It’s one part experimenting and one part perseverance and one part lightbulb going off after things go wrong.

And I find myself working through these kinds of issues a lot.

Anyone who says “Always write your tests first” can stick that in their pipe and smoke it.

Doctor: So don’t laugh.

In addition to being a joke, the above is also a succinct summary of Michael Feathers’ book Working Effectively with Legacy Code, recommended to me by Chris Hanson.

“Legacy” in this case means “anything without unit tests,” which means just about all code out there. Including, I’m ashamed to admit, most of the codebase of the Neutrino project I’m currently working on.

Why, after all that effort I put into adding unit test capabilities to my project, have I mostly failed to follow through? Feathers’ book describes it well. Currently, neither programming languages nor external libraries and frameworks make it easy to separate the unique logic of your code from all the dependencies that it requires to actually get its work done.

My codebase, for example, does a lot of custom drawing to a view, and does that drawing in response to calls from the cross-platform engine that actually drives the process. So I’ve got two huge dependencies: the engine, and the Cocoa framework. How do you get around that?

But it’s not just Neutrino. I prefer to work on code that’s in the middle of complicated systems, doing interesting and complicated things.

The classes in Kent Beck’s Test Driven Development: by Example were small data model classes that dealt only with each other. Well, sure, that would be easy! Real world code is rarely so straightforward. Beck describes a world so far removed from mine it might as well be in another galaxy.

Feathers, on the other hand, gets dirty in the trenches. But he doesn’t reveal any magic formulas. Instead, over and over again, he says: stop laughing. Change the code.

Does the class you want to test rely on a well-protected singleton? Change the singleton so it isn’t, even though that allows others to use it incorrectly.

Does the class you want to test have some private methods? Make them public.

Does the class you want to test rely on other classes from a library or framework? Change your code not to refer to such classes directly, even at the cost of added complexity and overhead, so you can break the dependency and use dummy objects instead for your tests.

All these are ugly solutions, design-wise. And Feathers knows it; he makes multiple apologies for his techniques. But he stresses that they work.

Well-designed code, code that only allows itself to be used in the way it needs to be for production scenarios, turns out to be anathema to unit testing.

When I programmed primarily in C++, I looked askance at the typeless nature of Objective-C. How will the compiler check that you’re doing the right thing? What I’ve found since is that very few errors result from this extra freedom from type that you have in Objective-C. Maybe all those safeguards and extra complexity in C++ weren’t necessary after all?

Similarly, I wonder if the next major programming language might be optimized for unit tests. It will have even fewer compile-time restraints, even fewer security restraints, but many mechanisms to make it easier to test what you’ve just written – maybe even require such tests, or auto-generate such tests.

Until then, I’m going to see what the results are when I make my Neutrino code a lot less jolly.

1. Open Xcode 2.1

2. Make a Cocoa application project. I actually have the OCUnit stuff installed from a previous Xcode version, so my list of new projects included “Cocoa Application + Test”, which I chose (but which you shouldn’t, see below). I called my project “Testy”.

3. Add a C++ class with a member function that’s defined in the .cpp file. I called my class “Testy” as well, and the member function “test()”.

4. In the build settings for the application target, turn off ZeroLink for all configurations: choose “All Configurations” as your configuration in the target’s Info window, select the ZeroLink setting, and hit the delete key. This should turn it from bold to non-bold, and the checkbox to unchecked.

5. Build and run the application, be sure it runs OK, then quit it.

6. Add a Unit Test Bundle target. Add your app target as a direct dependency. Set the target’s Bundle Loader and Test Host settings as specified in Chris Hanson’s wonderful Unit Testing Cocoa Applications post.

7. Add a new Objective-C test case class file. Be sure it has an .mm suffix. In that file, instantiate your C++ class, and call the member function from that instance.

8. Build the Unit Test Bundle target.

It should work, right? You’re using a default project, default targets, following the instructions carefully. But it doesn’t work. The unit test bundle can’t find the C++ symbols from the application.

And the reason why is an additional little wrinkle I discovered today.

9. In your application target’s Info window build settings pane, search on “symbols hidden”. Turn it off. Rebuild your application and your unit test bundle.

Now it works.

Step 9 isn’t needed for applications with Objective-C classes and Objective-C unit test bundles. But it is needed for applications with C++ classes and Objective-C unit test bundles.

So if you have that combination, as I do, don’t forget step 9!

P.S. Second lesson: if you’ve got the old OCUnit stuff on your system, get rid of it before using Xcode 2.1-style unit tests. They don’t go well together. Because I didn’t, I had a lot of “scratch head, fiddle with it some more” sub-steps in the above that you should be spared.

I’d like to explain the "complicated and fragile" comment I made in my last post.

What was complicated and fragile about it? Let me count the ways:

-SenTest All-the-Time
In order to make the tests take place when the application is launched, I had to put the -SenTest All argument on the project’s MyDocApp executable. I didn’t find a way to conditionalize that to a target: When I instead added $(MY_BUILD_SETTING) as the argument, and then set MY_BUILD_SETTING to -SenTest All for my Tests aggregrate target, no argument was passed in to the application.

This constraint means that I can’t, for instance, put any tests in MainMenu.nib, because if I did they would always be invoked, even if my Tests framework was not present. Not so cool.

Anybody know a way conditionalize such executable arguments?

Why not just add a second custom executable to the project? It could have a different set of arguments. Use one executable for Tests, and one for No-Tests.

Unfortunately, such a custom executable reference in the target has to be specified by full path, which is a deal-breaker. When I tried to set the path for such a custom executable relative to the shared build directory, I got circular dependency errors, probably because the executable in question was already a product of one of the project’s targets.

Subscript
Both the copying of the Tests framework to the application and deleting them from the application only work if you specify the application name in build phase locations directly by string. This means duplication, and breakage if you only change the name in one place and not the others.

The to location is the Subpath section of the Copy Files build phase of the Tests target. I can’t use $(PRODUCT_NAME).$(WRAPPER_EXTENSION) there because those build setting values aren’t set for the Tests target, where the copying has to take place. They’re set for the MyDocApp target, which can be used by either Tests or No-Tests.

The from location is in the script text in the Shell Script Files build phase of the No-Tests target. If you look at that text, you’ll see that I actually delete two things: the framework in $(TARGET_BUILD_DIRECTORY), which is where it was originally built, and the framework inside the MyDocApp application bundle inside $(TARGET_BUILD_DIRECTORY), which is where the framework was copied to.

The Simple Life
The first two points describe the fragility of the setup: the changes I can make are more constricted or error-prone than I would like.

But the general complexity of the setup also needs to be noted. My setup has two entirely new embedded frameworks that the OCUnit template does not have. It has five targets instead of two. There are 22 steps in the README to change a generic Cocoa project into this.

Is it worth it? I’ll let you know.

You may have received the impression from my last post, in my ongoing saga to get a OCUnit testing setup I’m happy with, that I had everything built and working correctly.

You would be wrong.

What I had working correctly was the modification of my – rather complex – project file to use two embedded frameworks for, respectively, most of the app’s source code and its tests.

But my testing framework was still strongly linked to the application.

What I’ve done since then is gotten my project to work with a weakly-linked testing framework, and made aggregate targets that can, without any recompilation, either run the application with the tests or run the application without the tests, merely by copying in or removing the testing framework.

It’s still far more complicated and fragile than I would like, but it works! And in addition, in order to placate my demanding readers, I have made a far simpler sample Cocoa document-based application, called MyDocApp, which has all the same changes I made to my complicated project, and a README file which explains what those changes were. It’s a long list!

MyDocApp is available at http://umbar.com/macdev/MyDocApp.zip.

Enjoy!

Wolf (a.k.a. “Jon”) pointed out to me that I don’t want to compile my tests into my application all the time, which my last post assumed I could do. Why? Because with the amount of tests you’re supposed to write, this could easily double the size of your application. But I don’t want target A that builds my code, and target B that builds my code+tests. I only want to build the code once!

The solution is to put as many of the tests as you can in a separate embedded framework. When you run or start a debugging session with the app, the framework is loaded at startup. OCUnit automatically runs the tests. Adding the tests is a copy phase, not a rebuild. Stripping the tests is as simple as not copying the framework into the application bundle. (This only works if you weak-link the framework; if you link the test framework regularly, then you’ll get linker errors when you run the app without it.)

I tried this with my much-abused TestExtras example from OCUnit, and immediately ran into a problem. The test framework needs to refer to the objects it’s testing, but you can’t link a framework against an application. (At least, not that I know of.) To get around this, my TestExtras application has two frameworks. One for the classes used by the application. Another for the tests on those classes.

I added the Person class from the first example in the OCUnit tutorial to my app framework, called Person.framework. I then added Person’s test class, PersonTest, to my Test.framework.

I linked against SentTestingKit.framework both in my Test.framework and in TestExtras.app itself, so that I can run both TestPerson and the tests TestExtras was originally created for, a CompletionTest instance instantiated in MainMenu.nib, at the same time. It worked! –Although the failure of the CompletionTest tests meant that the application quit immediately.

I’m pretty happy with this, so it’s time to try it out for real.

The reason I spent so much effort, chronicled in my previous OCUnit post, to get the OCUnit tutorial working was because of a claim it made near the end: “It is true that many things [about an application’s user interface] are difficult or impossible to test automatically, but some things can still be tested.”

OCUnit is fairly easy to use with frameworks; you build the tests into the framework, and pass the whole framework as a command-line argument to the otest utility, which automatically finds and runs the tests.

But it seems to me that it’s a much more difficult proposition to test applications cleanly and quickly, especially applications and situations with involved preparatory steps.

The third OCUnit example is an application, so how does it do it? The tests are embedded in the application as a top-level object in MainMenu.nib. Top-level nib objects are instantiated when the nib is loaded, and MainMenu.nib is loaded at application startup. Perfect!

The top-level test object is an instance of SenInterfaceTestCase, which runs its tests when its awakeFromNib method is invoked – not all the time (that would be bad), but only if the proper user default has been set.

In the tutorial’s third example, this user default is set by using a third build style, after development and deployment. This is the deal-breaker, for me. I don’t want to have to maintain a different build style. I don’t want to have to, say, keep the deployment and the test builds styles manually in sync. Removing human error from the tests was the whole point of this exercise!

I tried to get around this by making a new custom executable, which I called Run Tests Executable but which was really just the same TestExtras build product from the development and deployment styles. Then I set the arguments for that custom executable so that it would run its tests. Now testing could be done with either deployment or debug styles.

Only one problem: you need a full path to the custom executable. D’oh! I don’t want a full path. Full paths can’t be checked in and used by other people on your team.

So close! But you know the old saying. Close only counts in….

Is there any way around this? Stay tuned.

I’ve had more trouble getting up to speed with OCUnit and its tutorial than I’d like.

My initial response, presented below, is to document the steps I took to compensate for the problems I found.

My subsequent response will probably be to send updated project, code, and tutorial files to Sen:te, which should make the rest of this post irrelevant. Until then, I present to you, with apologies to O’Reilly, OCUnit: The Missing Manual.

(Speaking of O’Reilly, their MacDevCenter has a decent simple tutorial Unit Testing with OCUnit, but it doesn’t address any of the problems discussed here.)

This is going to be a bit disjointed, because it is not meant to stand alone. Rather, it augments OCUnit’s documentation and tutorial.

Before you get started, be sure you have the latest version of OS X 10.3 (as of this writing, OS X 10.3.8) and the latest version of Xcode for 10.3, which is Xcode 1.5.

For an initial introduction to OCUnit, and to download the latest version, currently 39, go to http://www.sente.ch/software/ocunit/.

There are two OCUnit downloads, “OCUnitHome” and “OCUnitRoot”. I will only be discussing “OCUnitHome” right now.

If you follow the instructions in its top-level ReadMe file, the “OCUnitHome” disk image (currently named “OCUnitHome-v39.dmg”) installs its contents in various locations inside your home directory, ak.a. ~/.

On the plus side, this makes it easier to manually find and delete these files later, since these locations, such as ~/Developer/, are not generally used and so will only contain OCUnit files. (And OCUnit comes with no automated uninstaller, so easy manual deletion of its files is not to be sneezed at.)

On the minus side, this installs the SenTestingKit.framework, which you will use for all your OCUnit tests, in a somewhat non-standard location: ~/Library/Frameworks/. According to Apple’s thorough and well-written Installing Your Frameworks guide, the standard location is /Library/Frameworks/. Frameworks in ~/Library/Frameworks/ will be found at runtime; the problem is at compile/link time. In your Xcode projects, you have to specify an absolute path to the frameworks you link to. If you specify a non-standard location for a framework, such as all the “OCUnitHome” example projects do, others will have to fiddle with your project, or move their frameworks on their end, to make it compile successfully. So with that warning, we will forge ahead.

The examples’ tutorial will be at <where-you-dragged-the-OCUnit-folder-to>/OCUnit/Documentation/Tutorials/OCUnitWWDC2002.rtfd.

The examples themselves are at ~/Developer/Examples/OCUnit/ (except for those that aren’t, see below). I would recommend deleting the projects with the .pbproj suffix if there is a corresponding project in the same location with a .xcode suffix. Note a .pbproj suffix doesn’t necessarily mean such a project is out of date, since Xcode recognizes this suffix and can update such files to be as current as any .xcode project. But in this case, the .pbproj projects have non-native targets, outdated paths to the OCUnit utilities, references to the obsolete SenFoundation.framework, and other problems that you would have to correct. Instead, just use the corresponding .xcode projects, which need no special steps. If you open the .xcode projects after opening the .pbproj projects of the same name, be sure to do a clean build to clear out any outdated indexing information.

Now, when you see a name such as OCUnitWWDC2002.rtfd that includes “2002,” you might think that such a document would be a little outdated, and you’d be right. For example, it mentions SenFoundation.framework, which (as mentioned above) is obsolete. And there are a couple more gotchas I describe below.

But before we get to the individual examples, a rather long-winded build window tip. The OCUnit unit tests are run as an additional shell script build phase in the main target. Errors are integrated by default into Xcode’s results; you get a red entry in the upper pane of your build window if a test fails. (Pay no attention to the part about how to achieve this integration manually for Project Builder, and keep in mind that Xcode’s build pane looks a little different than the Project Builder screenshot.) But to see the full output of the unit test script, you need to open the build log pane of the build window. Do this by clicking the “Show build log” icon in the build window’s middle bar. The icon looks like a tiny note and is the third icon from the left. Then, you will see the list of test suites that were run.

First Example: Person
The first example is for the Person project. Note that the proposed solutions for the fullName tests are not the only possible solutions. For example, they change the results of the firstName method. If you already had tests for firstName that assumed certain results, changing firstName for the sake of fullName might cause some firstName tests to break.

Also, their solution involves the NSString API stringByTrimmingSpace, which no longer exists, if it ever did. Instead, you need to use the far more verbose NSString API stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceCharacterSet]. In the PersonComplete project that fully implements the Person tutorial tests, they got around this API problem by implementing stringByTrimmingSpace in the Person class, which also changes how the method needs to be invoked. Either follow the PersonComplete example, or use the API mentioned above, to complete the Person example without errors.

Second Example: SenRandomTest
The test method the tutorial introduces, testSenRandomUnsignedLong, is actually already present, though slightly better implemented, in the provided file SenRandomConformanceTest.m. You don’t have to type anything in yourself.

Third Example: TestExtras
This example requires TextExtras, available at http://www.lorax.com/FreeStuff/TextExtras.html (The tutorial’s URL is slightly different, but you can still navigate to TextExtras from it.)

For me, the OCUnit TestExtras project folder was not installed correctly sometimes when I ran the installer. It is in the OCUnit/Examples folder of the “OCUnitHome” disk image, and can be moved to ~/Developer/Examples/OCUnit/ manually.

The TestExtras project is not available in .xcode form, which means all the improvements we avoided making for the other projects, now we have to do to make it compile at all.

Step 1: Choose the menu item Project→Upgrade All Projects in Target to Native.

Step 2: Go to the shell script build phase Info window (open the Targets→TestExtras (Upgraded) node under Groups & Files, select the last entry under it, Shell Script Files, and hit Cmd-I) and change the script path there, which starts with $SYSTEM_DEVELOPER_DIR/, to the path: ~/Developer/Tools/RunTargetUnitTests (replace the whole path).

Step 3: Delete the reference to the SenFoundation.framework under TestExtras→Frameworks→Linked Frameworks.

Step 4: Delete the line #import <SenFoundation/SenFoundation.h> from the file CompletionTest.m.

Step 5: Enable Objective-C exceptions, by selecting the Targets→TestExtras (Upgraded) node, hitting Cmd-I to go to its Info window, going to the Build tab, finding the checkbox for “Enable Objective-C Exceptions”, and checking it.

Now that the project compiles, you need to clean up its code warnings. In CompletionTest.m, NSString is assumed to have a method called isEmpty. You have to replace that with isLength] == 0.

The test method the tutorial mentions, testForward, is, as with the second example, already in the provided source code file.

The test method that actually fails is testCycle. The tutorial goes on to explain how to debug that failure: instead of running the tests at compile time via a shell script build phase, you run them in a debugging session by specially triggering the tests during a normal application run.

They mention a target “Executables” tab: in Xcode, this is the top-level Executables node under Groups & Files. You need to open the Info window for the TestExtras (Upgraded) leaf under the Executables node and look at the “Arguments to be passed on launch” table at the top.

The tutorial then says, “Choose to use the ‘SenTest Self’ argument,” but what that really means with Xcode is to add a new row to the “Arguments” table via the “+” button, and then set the row’s string to -SenTest Self. (Important: note the initial dash.)

Now, you will be able to follow the rest of the tutorial and stop at a breakpoint to debug what’s wrong with the test.

That’s it! At least for the OCUnit tutorial. I have more to say about OCUnit, though, so stay tuned.

Update 3/15: Updated descriptions of second and third examples to be more accurate. Also, I’ve sent my modified files to Sen:te. We’ll see if their spam filter nabs my email, which is (a) unsolicited and (b) has both a URL and an attachment in it.