Tagged: Test-Driven Development

I’ve Been Framed!: Solving the OCUnit App Test Problem with Frameworks

Wolf (a.k.a. “Jon”) pointed out to me that I don’t want to compile my tests into my application all the time, which my last post assumed I could do. Why? Because with the amount of tests you’re supposed to write, this could easily double the size of your application. But I don’t want target A that builds my code, and target B that builds my code+tests. I only want to build the code once!

The solution is to put as many of the tests as you can in a separate embedded framework. When you run or start a debugging session with the app, the framework is loaded at startup. OCUnit automatically runs the tests. Adding the tests is a copy phase, not a rebuild. Stripping the tests is as simple as not copying the framework into the application bundle. (This only works if you weak-link the framework; if you link the test framework regularly, then you’ll get linker errors when you run the app without it.)

I tried this with my much-abused TestExtras example from OCUnit, and immediately ran into a problem. The test framework needs to refer to the objects it’s testing, but you can’t link a framework against an application. (At least, not that I know of.) To get around this, my TestExtras application has two frameworks. One for the classes used by the application. Another for the tests on those classes.

I added the Person class from the first example in the OCUnit tutorial to my app framework, called Person.framework. I then added Person’s test class, PersonTest, to my Test.framework.

I linked against SentTestingKit.framework both in my Test.framework and in TestExtras.app itself, so that I can run both TestPerson and the tests TestExtras was originally created for, a CompletionTest instance instantiated in MainMenu.nib, at the same time. It worked! –Although the failure of the CompletionTest tests meant that the application quit immediately.

I’m pretty happy with this, so it’s time to try it out for real.

Horseshoes and Hand Grenades: OCUnit App Testing, Continued

The reason I spent so much effort, chronicled in my previous OCUnit post, to get the OCUnit tutorial working was because of a claim it made near the end: “It is true that many things [about an application’s user interface] are difficult or impossible to test automatically, but some things can still be tested.”

OCUnit is fairly easy to use with frameworks; you build the tests into the framework, and pass the whole framework as a command-line argument to the otest utility, which automatically finds and runs the tests.

But it seems to me that it’s a much more difficult proposition to test applications cleanly and quickly, especially applications and situations with involved preparatory steps.

The third OCUnit example is an application, so how does it do it? The tests are embedded in the application as a top-level object in MainMenu.nib. Top-level nib objects are instantiated when the nib is loaded, and MainMenu.nib is loaded at application startup. Perfect!

The top-level test object is an instance of SenInterfaceTestCase, which runs its tests when its awakeFromNib method is invoked – not all the time (that would be bad), but only if the proper user default has been set.

In the tutorial’s third example, this user default is set by using a third build style, after development and deployment. This is the deal-breaker, for me. I don’t want to have to maintain a different build style. I don’t want to have to, say, keep the deployment and the test builds styles manually in sync. Removing human error from the tests was the whole point of this exercise!

I tried to get around this by making a new custom executable, which I called Run Tests Executable but which was really just the same TestExtras build product from the development and deployment styles. Then I set the arguments for that custom executable so that it would run its tests. Now testing could be done with either deployment or debug styles.

Only one problem: you need a full path to the custom executable. D’oh! I don’t want a full path. Full paths can’t be checked in and used by other people on your team.

So close! But you know the old saying. Close only counts in….

Is there any way around this? Stay tuned.

OCUnit: The Missing Manual

I’ve had more trouble getting up to speed with OCUnit and its tutorial than I’d like.

My initial response, presented below, is to document the steps I took to compensate for the problems I found.

My subsequent response will probably be to send updated project, code, and tutorial files to Sen:te, which should make the rest of this post irrelevant. Until then, I present to you, with apologies to O’Reilly, OCUnit: The Missing Manual.

(Speaking of O’Reilly, their MacDevCenter has a decent simple tutorial Unit Testing with OCUnit, but it doesn’t address any of the problems discussed here.)

This is going to be a bit disjointed, because it is not meant to stand alone. Rather, it augments OCUnit’s documentation and tutorial.

Before you get started, be sure you have the latest version of OS X 10.3 (as of this writing, OS X 10.3.8) and the latest version of Xcode for 10.3, which is Xcode 1.5.

For an initial introduction to OCUnit, and to download the latest version, currently 39, go to http://www.sente.ch/software/ocunit/.

There are two OCUnit downloads, “OCUnitHome” and “OCUnitRoot”. I will only be discussing “OCUnitHome” right now.

If you follow the instructions in its top-level ReadMe file, the “OCUnitHome” disk image (currently named “OCUnitHome-v39.dmg”) installs its contents in various locations inside your home directory, ak.a. ~/.

On the plus side, this makes it easier to manually find and delete these files later, since these locations, such as ~/Developer/, are not generally used and so will only contain OCUnit files. (And OCUnit comes with no automated uninstaller, so easy manual deletion of its files is not to be sneezed at.)

On the minus side, this installs the SenTestingKit.framework, which you will use for all your OCUnit tests, in a somewhat non-standard location: ~/Library/Frameworks/. According to Apple’s thorough and well-written Installing Your Frameworks guide, the standard location is /Library/Frameworks/. Frameworks in ~/Library/Frameworks/ will be found at runtime; the problem is at compile/link time. In your Xcode projects, you have to specify an absolute path to the frameworks you link to. If you specify a non-standard location for a framework, such as all the “OCUnitHome” example projects do, others will have to fiddle with your project, or move their frameworks on their end, to make it compile successfully. So with that warning, we will forge ahead.

The examples’ tutorial will be at <where-you-dragged-the-OCUnit-folder-to>/OCUnit/Documentation/Tutorials/OCUnitWWDC2002.rtfd.

The examples themselves are at ~/Developer/Examples/OCUnit/ (except for those that aren’t, see below). I would recommend deleting the projects with the .pbproj suffix if there is a corresponding project in the same location with a .xcode suffix. Note a .pbproj suffix doesn’t necessarily mean such a project is out of date, since Xcode recognizes this suffix and can update such files to be as current as any .xcode project. But in this case, the .pbproj projects have non-native targets, outdated paths to the OCUnit utilities, references to the obsolete SenFoundation.framework, and other problems that you would have to correct. Instead, just use the corresponding .xcode projects, which need no special steps. If you open the .xcode projects after opening the .pbproj projects of the same name, be sure to do a clean build to clear out any outdated indexing information.

Now, when you see a name such as OCUnitWWDC2002.rtfd that includes “2002,” you might think that such a document would be a little outdated, and you’d be right. For example, it mentions SenFoundation.framework, which (as mentioned above) is obsolete. And there are a couple more gotchas I describe below.

But before we get to the individual examples, a rather long-winded build window tip. The OCUnit unit tests are run as an additional shell script build phase in the main target. Errors are integrated by default into Xcode’s results; you get a red entry in the upper pane of your build window if a test fails. (Pay no attention to the part about how to achieve this integration manually for Project Builder, and keep in mind that Xcode’s build pane looks a little different than the Project Builder screenshot.) But to see the full output of the unit test script, you need to open the build log pane of the build window. Do this by clicking the “Show build log” icon in the build window’s middle bar. The icon looks like a tiny note and is the third icon from the left. Then, you will see the list of test suites that were run.

First Example: Person
The first example is for the Person project. Note that the proposed solutions for the fullName tests are not the only possible solutions. For example, they change the results of the firstName method. If you already had tests for firstName that assumed certain results, changing firstName for the sake of fullName might cause some firstName tests to break.

Also, their solution involves the NSString API stringByTrimmingSpace, which no longer exists, if it ever did. Instead, you need to use the far more verbose NSString API stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceCharacterSet]. In the PersonComplete project that fully implements the Person tutorial tests, they got around this API problem by implementing stringByTrimmingSpace in the Person class, which also changes how the method needs to be invoked. Either follow the PersonComplete example, or use the API mentioned above, to complete the Person example without errors.

Second Example: SenRandomTest
The test method the tutorial introduces, testSenRandomUnsignedLong, is actually already present, though slightly better implemented, in the provided file SenRandomConformanceTest.m. You don’t have to type anything in yourself.

Third Example: TestExtras
This example requires TextExtras, available at http://www.lorax.com/FreeStuff/TextExtras.html (The tutorial’s URL is slightly different, but you can still navigate to TextExtras from it.)

For me, the OCUnit TestExtras project folder was not installed correctly sometimes when I ran the installer. It is in the OCUnit/Examples folder of the “OCUnitHome” disk image, and can be moved to ~/Developer/Examples/OCUnit/ manually.

The TestExtras project is not available in .xcode form, which means all the improvements we avoided making for the other projects, now we have to do to make it compile at all.

Step 1: Choose the menu item Project→Upgrade All Projects in Target to Native.

Step 2: Go to the shell script build phase Info window (open the Targets→TestExtras (Upgraded) node under Groups & Files, select the last entry under it, Shell Script Files, and hit Cmd-I) and change the script path there, which starts with $SYSTEM_DEVELOPER_DIR/, to the path: ~/Developer/Tools/RunTargetUnitTests (replace the whole path).

Step 3: Delete the reference to the SenFoundation.framework under TestExtras→Frameworks→Linked Frameworks.

Step 4: Delete the line #import <SenFoundation/SenFoundation.h> from the file CompletionTest.m.

Step 5: Enable Objective-C exceptions, by selecting the Targets→TestExtras (Upgraded) node, hitting Cmd-I to go to its Info window, going to the Build tab, finding the checkbox for “Enable Objective-C Exceptions”, and checking it.

Now that the project compiles, you need to clean up its code warnings. In CompletionTest.m, NSString is assumed to have a method called isEmpty. You have to replace that with isLength] == 0.

The test method the tutorial mentions, testForward, is, as with the second example, already in the provided source code file.

The test method that actually fails is testCycle. The tutorial goes on to explain how to debug that failure: instead of running the tests at compile time via a shell script build phase, you run them in a debugging session by specially triggering the tests during a normal application run.

They mention a target “Executables” tab: in Xcode, this is the top-level Executables node under Groups & Files. You need to open the Info window for the TestExtras (Upgraded) leaf under the Executables node and look at the “Arguments to be passed on launch” table at the top.

The tutorial then says, “Choose to use the ‘SenTest Self’ argument,” but what that really means with Xcode is to add a new row to the “Arguments” table via the “+” button, and then set the row’s string to -SenTest Self. (Important: note the initial dash.)

Now, you will be able to follow the rest of the tutorial and stop at a breakpoint to debug what’s wrong with the test.

That’s it! At least for the OCUnit tutorial. I have more to say about OCUnit, though, so stay tuned.

Update 3/15: Updated descriptions of second and third examples to be more accurate. Also, I’ve sent my modified files to Sen:te. We’ll see if their spam filter nabs my email, which is (a) unsolicited and (b) has both a URL and an attachment in it.

Testing++

Two things:

(1) I remember reading somewhere, probably a weblog, about how test-driven development was acknowledged to be a Bad Idea for UI and security. I think it was security. I know UI was one of them. Anybody know where this was originally written? I’ve already garnered some link karma, so help me out, LazyWeb!

(2) I have been reading about test-driven development for the sake of a specific project. A project, it has occurred to me, which involves both Objective-C and C++.

For Objective-C, there are already at least two unit-testing frameworks:

Both have Xcode integration.

C++ seems less clear-cut. In fact, a Google search on “test-driven development C++” turns up this link: “Much harder than it should be”. Still, the article mentions two frameworks:

  • CppUnit, preferred but still merely “about it”
  • TUT Framework, which requires “strongly template-compliant compiler” – a good sign

Since my project uses Xcode, and OCUnit et al already integrates with Xcode, of course I want a C++ framework that also integrates with Xcode.

Anybody doing this already?

Update: It occurs to me that I can use OCUnit or UnitKit to test my C++ code – even if all the code involved is in C++, the test method can still be Objective-C. That’s probably the route I’ll go, just to save time.

An Exercise for the Reader

A while back, I bought Test Driven Development: by Example by by noted banjoist Kent Beck.

On p. 5, after presenting me with the first code of the book, he writes, “I’ll explain where and how we type it in later, when we talk more about the testing framework, JUnit.” He then proceeds to go on for pages, describing more and more code changes and additions, without ever coming back to JUnit.

Odd.

Still, I decided to keep going. I spent some time getting JUnit set up on my own – Java CLASSPATHs are evil evil evil – and got back to the book a couple of days ago.

It didn’t take me long to realize that even with JUnit in front of me, I wouldn’t be able to follow along with the book. It’s just not written that way. There isn’t anywhere near enough detail provided: what his classes inherit from, how they should be arranged for the sake of later steps, etc.

Now, one of the benefits of TDD as he describes it is its immediacy – you can get to work right away knowing that any mistakes you make will be corrected as you go along. But his book doesn’t allow you to experience this benefit. It just lets you peek in through the window as he experiences it.

Now, JUnit appears to include the example he describes, but it’s the finished example. You’d have to reverse engineer it to get it to the point where he starts his work on it. Basically, you’d have to learn it on your own before you learn it from him. So what do I need the book for?

I’ve heard very good things about TDD from various people, so I will still be giving it a try, but this was a frustrating first experience from a recommended resource.