Month: March 2005

I’d like to explain the "complicated and fragile" comment I made in my last post.

What was complicated and fragile about it? Let me count the ways:

-SenTest All-the-Time
In order to make the tests take place when the application is launched, I had to put the -SenTest All argument on the project’s MyDocApp executable. I didn’t find a way to conditionalize that to a target: When I instead added $(MY_BUILD_SETTING) as the argument, and then set MY_BUILD_SETTING to -SenTest All for my Tests aggregrate target, no argument was passed in to the application.

This constraint means that I can’t, for instance, put any tests in MainMenu.nib, because if I did they would always be invoked, even if my Tests framework was not present. Not so cool.

Anybody know a way conditionalize such executable arguments?

Why not just add a second custom executable to the project? It could have a different set of arguments. Use one executable for Tests, and one for No-Tests.

Unfortunately, such a custom executable reference in the target has to be specified by full path, which is a deal-breaker. When I tried to set the path for such a custom executable relative to the shared build directory, I got circular dependency errors, probably because the executable in question was already a product of one of the project’s targets.

Subscript
Both the copying of the Tests framework to the application and deleting them from the application only work if you specify the application name in build phase locations directly by string. This means duplication, and breakage if you only change the name in one place and not the others.

The to location is the Subpath section of the Copy Files build phase of the Tests target. I can’t use $(PRODUCT_NAME).$(WRAPPER_EXTENSION) there because those build setting values aren’t set for the Tests target, where the copying has to take place. They’re set for the MyDocApp target, which can be used by either Tests or No-Tests.

The from location is in the script text in the Shell Script Files build phase of the No-Tests target. If you look at that text, you’ll see that I actually delete two things: the framework in $(TARGET_BUILD_DIRECTORY), which is where it was originally built, and the framework inside the MyDocApp application bundle inside $(TARGET_BUILD_DIRECTORY), which is where the framework was copied to.

The Simple Life
The first two points describe the fragility of the setup: the changes I can make are more constricted or error-prone than I would like.

But the general complexity of the setup also needs to be noted. My setup has two entirely new embedded frameworks that the OCUnit template does not have. It has five targets instead of two. There are 22 steps in the README to change a generic Cocoa project into this.

Is it worth it? I’ll let you know.

You may have received the impression from my last post, in my ongoing saga to get a OCUnit testing setup I’m happy with, that I had everything built and working correctly.

You would be wrong.

What I had working correctly was the modification of my – rather complex – project file to use two embedded frameworks for, respectively, most of the app’s source code and its tests.

But my testing framework was still strongly linked to the application.

What I’ve done since then is gotten my project to work with a weakly-linked testing framework, and made aggregate targets that can, without any recompilation, either run the application with the tests or run the application without the tests, merely by copying in or removing the testing framework.

It’s still far more complicated and fragile than I would like, but it works! And in addition, in order to placate my demanding readers, I have made a far simpler sample Cocoa document-based application, called MyDocApp, which has all the same changes I made to my complicated project, and a README file which explains what those changes were. It’s a long list!

MyDocApp is available at http://umbar.com/macdev/MyDocApp.zip.

Enjoy!

Wolf (a.k.a. “Jon”) pointed out to me that I don’t want to compile my tests into my application all the time, which my last post assumed I could do. Why? Because with the amount of tests you’re supposed to write, this could easily double the size of your application. But I don’t want target A that builds my code, and target B that builds my code+tests. I only want to build the code once!

The solution is to put as many of the tests as you can in a separate embedded framework. When you run or start a debugging session with the app, the framework is loaded at startup. OCUnit automatically runs the tests. Adding the tests is a copy phase, not a rebuild. Stripping the tests is as simple as not copying the framework into the application bundle. (This only works if you weak-link the framework; if you link the test framework regularly, then you’ll get linker errors when you run the app without it.)

I tried this with my much-abused TestExtras example from OCUnit, and immediately ran into a problem. The test framework needs to refer to the objects it’s testing, but you can’t link a framework against an application. (At least, not that I know of.) To get around this, my TestExtras application has two frameworks. One for the classes used by the application. Another for the tests on those classes.

I added the Person class from the first example in the OCUnit tutorial to my app framework, called Person.framework. I then added Person’s test class, PersonTest, to my Test.framework.

I linked against SentTestingKit.framework both in my Test.framework and in TestExtras.app itself, so that I can run both TestPerson and the tests TestExtras was originally created for, a CompletionTest instance instantiated in MainMenu.nib, at the same time. It worked! –Although the failure of the CompletionTest tests meant that the application quit immediately.

I’m pretty happy with this, so it’s time to try it out for real.

The reason I spent so much effort, chronicled in my previous OCUnit post, to get the OCUnit tutorial working was because of a claim it made near the end: “It is true that many things [about an application’s user interface] are difficult or impossible to test automatically, but some things can still be tested.”

OCUnit is fairly easy to use with frameworks; you build the tests into the framework, and pass the whole framework as a command-line argument to the otest utility, which automatically finds and runs the tests.

But it seems to me that it’s a much more difficult proposition to test applications cleanly and quickly, especially applications and situations with involved preparatory steps.

The third OCUnit example is an application, so how does it do it? The tests are embedded in the application as a top-level object in MainMenu.nib. Top-level nib objects are instantiated when the nib is loaded, and MainMenu.nib is loaded at application startup. Perfect!

The top-level test object is an instance of SenInterfaceTestCase, which runs its tests when its awakeFromNib method is invoked – not all the time (that would be bad), but only if the proper user default has been set.

In the tutorial’s third example, this user default is set by using a third build style, after development and deployment. This is the deal-breaker, for me. I don’t want to have to maintain a different build style. I don’t want to have to, say, keep the deployment and the test builds styles manually in sync. Removing human error from the tests was the whole point of this exercise!

I tried to get around this by making a new custom executable, which I called Run Tests Executable but which was really just the same TestExtras build product from the development and deployment styles. Then I set the arguments for that custom executable so that it would run its tests. Now testing could be done with either deployment or debug styles.

Only one problem: you need a full path to the custom executable. D’oh! I don’t want a full path. Full paths can’t be checked in and used by other people on your team.

So close! But you know the old saying. Close only counts in….

Is there any way around this? Stay tuned.

I’ve had more trouble getting up to speed with OCUnit and its tutorial than I’d like.

My initial response, presented below, is to document the steps I took to compensate for the problems I found.

My subsequent response will probably be to send updated project, code, and tutorial files to Sen:te, which should make the rest of this post irrelevant. Until then, I present to you, with apologies to O’Reilly, OCUnit: The Missing Manual.

(Speaking of O’Reilly, their MacDevCenter has a decent simple tutorial Unit Testing with OCUnit, but it doesn’t address any of the problems discussed here.)

This is going to be a bit disjointed, because it is not meant to stand alone. Rather, it augments OCUnit’s documentation and tutorial.

Before you get started, be sure you have the latest version of OS X 10.3 (as of this writing, OS X 10.3.8) and the latest version of Xcode for 10.3, which is Xcode 1.5.

For an initial introduction to OCUnit, and to download the latest version, currently 39, go to http://www.sente.ch/software/ocunit/.

There are two OCUnit downloads, “OCUnitHome” and “OCUnitRoot”. I will only be discussing “OCUnitHome” right now.

If you follow the instructions in its top-level ReadMe file, the “OCUnitHome” disk image (currently named “OCUnitHome-v39.dmg”) installs its contents in various locations inside your home directory, ak.a. ~/.

On the plus side, this makes it easier to manually find and delete these files later, since these locations, such as ~/Developer/, are not generally used and so will only contain OCUnit files. (And OCUnit comes with no automated uninstaller, so easy manual deletion of its files is not to be sneezed at.)

On the minus side, this installs the SenTestingKit.framework, which you will use for all your OCUnit tests, in a somewhat non-standard location: ~/Library/Frameworks/. According to Apple’s thorough and well-written Installing Your Frameworks guide, the standard location is /Library/Frameworks/. Frameworks in ~/Library/Frameworks/ will be found at runtime; the problem is at compile/link time. In your Xcode projects, you have to specify an absolute path to the frameworks you link to. If you specify a non-standard location for a framework, such as all the “OCUnitHome” example projects do, others will have to fiddle with your project, or move their frameworks on their end, to make it compile successfully. So with that warning, we will forge ahead.

The examples’ tutorial will be at <where-you-dragged-the-OCUnit-folder-to>/OCUnit/Documentation/Tutorials/OCUnitWWDC2002.rtfd.

The examples themselves are at ~/Developer/Examples/OCUnit/ (except for those that aren’t, see below). I would recommend deleting the projects with the .pbproj suffix if there is a corresponding project in the same location with a .xcode suffix. Note a .pbproj suffix doesn’t necessarily mean such a project is out of date, since Xcode recognizes this suffix and can update such files to be as current as any .xcode project. But in this case, the .pbproj projects have non-native targets, outdated paths to the OCUnit utilities, references to the obsolete SenFoundation.framework, and other problems that you would have to correct. Instead, just use the corresponding .xcode projects, which need no special steps. If you open the .xcode projects after opening the .pbproj projects of the same name, be sure to do a clean build to clear out any outdated indexing information.

Now, when you see a name such as OCUnitWWDC2002.rtfd that includes “2002,” you might think that such a document would be a little outdated, and you’d be right. For example, it mentions SenFoundation.framework, which (as mentioned above) is obsolete. And there are a couple more gotchas I describe below.

But before we get to the individual examples, a rather long-winded build window tip. The OCUnit unit tests are run as an additional shell script build phase in the main target. Errors are integrated by default into Xcode’s results; you get a red entry in the upper pane of your build window if a test fails. (Pay no attention to the part about how to achieve this integration manually for Project Builder, and keep in mind that Xcode’s build pane looks a little different than the Project Builder screenshot.) But to see the full output of the unit test script, you need to open the build log pane of the build window. Do this by clicking the “Show build log” icon in the build window’s middle bar. The icon looks like a tiny note and is the third icon from the left. Then, you will see the list of test suites that were run.

First Example: Person
The first example is for the Person project. Note that the proposed solutions for the fullName tests are not the only possible solutions. For example, they change the results of the firstName method. If you already had tests for firstName that assumed certain results, changing firstName for the sake of fullName might cause some firstName tests to break.

Also, their solution involves the NSString API stringByTrimmingSpace, which no longer exists, if it ever did. Instead, you need to use the far more verbose NSString API stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceCharacterSet]. In the PersonComplete project that fully implements the Person tutorial tests, they got around this API problem by implementing stringByTrimmingSpace in the Person class, which also changes how the method needs to be invoked. Either follow the PersonComplete example, or use the API mentioned above, to complete the Person example without errors.

Second Example: SenRandomTest
The test method the tutorial introduces, testSenRandomUnsignedLong, is actually already present, though slightly better implemented, in the provided file SenRandomConformanceTest.m. You don’t have to type anything in yourself.

Third Example: TestExtras
This example requires TextExtras, available at http://www.lorax.com/FreeStuff/TextExtras.html (The tutorial’s URL is slightly different, but you can still navigate to TextExtras from it.)

For me, the OCUnit TestExtras project folder was not installed correctly sometimes when I ran the installer. It is in the OCUnit/Examples folder of the “OCUnitHome” disk image, and can be moved to ~/Developer/Examples/OCUnit/ manually.

The TestExtras project is not available in .xcode form, which means all the improvements we avoided making for the other projects, now we have to do to make it compile at all.

Step 1: Choose the menu item Project→Upgrade All Projects in Target to Native.

Step 2: Go to the shell script build phase Info window (open the Targets→TestExtras (Upgraded) node under Groups & Files, select the last entry under it, Shell Script Files, and hit Cmd-I) and change the script path there, which starts with $SYSTEM_DEVELOPER_DIR/, to the path: ~/Developer/Tools/RunTargetUnitTests (replace the whole path).

Step 3: Delete the reference to the SenFoundation.framework under TestExtras→Frameworks→Linked Frameworks.

Step 4: Delete the line #import <SenFoundation/SenFoundation.h> from the file CompletionTest.m.

Step 5: Enable Objective-C exceptions, by selecting the Targets→TestExtras (Upgraded) node, hitting Cmd-I to go to its Info window, going to the Build tab, finding the checkbox for “Enable Objective-C Exceptions”, and checking it.

Now that the project compiles, you need to clean up its code warnings. In CompletionTest.m, NSString is assumed to have a method called isEmpty. You have to replace that with isLength] == 0.

The test method the tutorial mentions, testForward, is, as with the second example, already in the provided source code file.

The test method that actually fails is testCycle. The tutorial goes on to explain how to debug that failure: instead of running the tests at compile time via a shell script build phase, you run them in a debugging session by specially triggering the tests during a normal application run.

They mention a target “Executables” tab: in Xcode, this is the top-level Executables node under Groups & Files. You need to open the Info window for the TestExtras (Upgraded) leaf under the Executables node and look at the “Arguments to be passed on launch” table at the top.

The tutorial then says, “Choose to use the ‘SenTest Self’ argument,” but what that really means with Xcode is to add a new row to the “Arguments” table via the “+” button, and then set the row’s string to -SenTest Self. (Important: note the initial dash.)

Now, you will be able to follow the rest of the tutorial and stop at a breakpoint to debug what’s wrong with the test.

That’s it! At least for the OCUnit tutorial. I have more to say about OCUnit, though, so stay tuned.

Update 3/15: Updated descriptions of second and third examples to be more accurate. Also, I’ve sent my modified files to Sen:te. We’ll see if their spam filter nabs my email, which is (a) unsolicited and (b) has both a URL and an attachment in it.

Two things:

(1) I remember reading somewhere, probably a weblog, about how test-driven development was acknowledged to be a Bad Idea for UI and security. I think it was security. I know UI was one of them. Anybody know where this was originally written? I’ve already garnered some link karma, so help me out, LazyWeb!

(2) I have been reading about test-driven development for the sake of a specific project. A project, it has occurred to me, which involves both Objective-C and C++.

For Objective-C, there are already at least two unit-testing frameworks:

• sen:te’s OCUnit
• James Duncan Davidson’s UnitKit

Both have Xcode integration.

C++ seems less clear-cut. In fact, a Google search on “test-driven development C++” turns up this link: “Much harder than it should be”. Still, the article mentions two frameworks:

• CppUnit, preferred but still merely “about it”
• TUT Framework, which requires “strongly template-compliant compiler” – a good sign

Since my project uses Xcode, and OCUnit et al already integrates with Xcode, of course I want a C++ framework that also integrates with Xcode.

Anybody doing this already?

Update: It occurs to me that I can use OCUnit or UnitKit to test my C++ code – even if all the code involved is in C++, the test method can still be Objective-C. That’s probably the route I’ll go, just to save time.