“Complicated and Fragile”

I’d like to explain the "complicated and fragile" comment I made in my last post.

What was complicated and fragile about it? Let me count the ways:

-SenTest All-the-Time
In order to make the tests take place when the application is launched, I had to put the -SenTest All argument on the project’s MyDocApp executable. I didn’t find a way to conditionalize that to a target: When I instead added $(MY_BUILD_SETTING) as the argument, and then set MY_BUILD_SETTING to -SenTest All for my Tests aggregrate target, no argument was passed in to the application.

This constraint means that I can’t, for instance, put any tests in MainMenu.nib, because if I did they would always be invoked, even if my Tests framework was not present. Not so cool.

Anybody know a way conditionalize such executable arguments?

Why not just add a second custom executable to the project? It could have a different set of arguments. Use one executable for Tests, and one for No-Tests.

Unfortunately, such a custom executable reference in the target has to be specified by full path, which is a deal-breaker. When I tried to set the path for such a custom executable relative to the shared build directory, I got circular dependency errors, probably because the executable in question was already a product of one of the project’s targets.

Both the copying of the Tests framework to the application and deleting them from the application only work if you specify the application name in build phase locations directly by string. This means duplication, and breakage if you only change the name in one place and not the others.

The to location is the Subpath section of the Copy Files build phase of the Tests target. I can’t use $(PRODUCT_NAME).$(WRAPPER_EXTENSION) there because those build setting values aren’t set for the Tests target, where the copying has to take place. They’re set for the MyDocApp target, which can be used by either Tests or No-Tests.

The from location is in the script text in the Shell Script Files build phase of the No-Tests target. If you look at that text, you’ll see that I actually delete two things: the framework in $(TARGET_BUILD_DIRECTORY), which is where it was originally built, and the framework inside the MyDocApp application bundle inside $(TARGET_BUILD_DIRECTORY), which is where the framework was copied to.

The Simple Life
The first two points describe the fragility of the setup: the changes I can make are more constricted or error-prone than I would like.

But the general complexity of the setup also needs to be noted. My setup has two entirely new embedded frameworks that the OCUnit template does not have. It has five targets instead of two. There are 22 steps in the README to change a generic Cocoa project into this.

Is it worth it? I’ll let you know.


You may have received the impression from my last post, in my ongoing saga to get a OCUnit testing setup I’m happy with, that I had everything built and working correctly.

You would be wrong.

What I had working correctly was the modification of my – rather complex – project file to use two embedded frameworks for, respectively, most of the app’s source code and its tests.

But my testing framework was still strongly linked to the application.

What I’ve done since then is gotten my project to work with a weakly-linked testing framework, and made aggregate targets that can, without any recompilation, either run the application with the tests or run the application without the tests, merely by copying in or removing the testing framework.

It’s still far more complicated and fragile than I would like, but it works! And in addition, in order to placate my demanding readers, I have made a far simpler sample Cocoa document-based application, called MyDocApp, which has all the same changes I made to my complicated project, and a README file which explains what those changes were. It’s a long list!

MyDocApp is available at http://umbar.com/macdev/MyDocApp.zip.


When We Last Saw Our Hero…

…he was riding into the sunset with the cry, “It’s time to try it out for real!”


The good news: after spending lots of time converting my application to the two-framework setup I talked about in my I’ve Been Framed! post, I have it working again.

The bad news: I continued having problems even after I solved the ones I talked about in my Looks Can Be Deceiving post.

Now, a lot of this is because I decided, as long as I was embedding my own frameworks, in addition I would try to embed the Omni frameworks I was also using. Not only embed those frameworks, but also put dependencies on those framework projects in my own project, so everything would always get built correctly.

Using that setup, I kept getting strange compilation errors that would go away or change after each clean rebuild. Undefined symbol errors, code logic compilation errors that only showed up after the third rebuild, inability to find the embedded frameworks. You name it.

In general, my advice is: keep fiddling with it, you’ll get it to work eventually. But don’t expect a cakewalk.

Specifically, about the app framework setup:

  1. main.c*/m* must be in the app itself. All other code can be in the app framework.
  2. MainMenu.nib must be in the app itself. All other nibs can be in the app framework.
  3. The Finder icons for the application and any of its documents must be in the app itself.

2. means that you may need to put a few tests in the app itself, not just in the tests framework, if you want to test something in MainMenu.nib.

And specifically, about the Omni frameworks:

  1. Add the Omni projects to your project and use the framework references within those projects – don’t also add the frameworks to your project under the Frameworks group.
  2. I found no way around modifying the Omni projects themselves to have an INSTALL_PATH of “@executable_path/../Frameworks”. It would be nice if you didn’t have to touch those projects at all for this purpose, but it looks like you do.
  3. You must link the Omni frameworks to both your app framework and the app itself, or you may get linker errors.
  4. As I warn in my Don’t Use the Brown Project File post, at this time, you really really should use the *.pbproj versions of the Omni projects, not the *.xcode versions.

Looks Can Be Deceiving: Xcode and Those Trees Again

One problem I had tonight trying to set up an Xcode project for my previous post was in trying to set up the embedded framework.

The instructions on Apple’s Web site, Creating a Framework: Creating a Private Embedded Framework, are good as far as they go.

They say to add your embedded framework to two build phases in your application’s target. First, the Frameworks & Libraries phase, which makes sure the framework gets linked properly. And second, the Copy Files phase that copies the framework into the application bundle.

Now, the instructions emphasize that, for step one, you have to drag from the Products node, not the Target node. But they don’t emphasize the same thing for step two. I tried to accomplish step two by dragging the framework icon that was now in the Frameworks & Libraries phase into the Copy Files phase.

What I was expecting this to do was put a second representation of the framework in Copy Files. What it actually did was move the framework representation from Frameworks & Libraries to Copy Files. This gave me the impression you could only have one framework representation in your build phases – maybe putting it in the Copy Files phase was enough, it would automatically link it? (Nope.)

It turns out that if I’d dragged the framework from the Products node again, I would’ve gotten the framework representation duplicated in both phases. But what I also found, from playing around with it, is that starting the drag from Frameworks & Libraries, then holding down the Option key, turned the drag into a copy drag (cursor turns into a green plus sign), which has the same effect.

A copy-drag?? I complained here about using a single tree to represent a series of unlike objects. But this is even worse; unlike objects that look exactly the same. The same framework icon appears under the project sub-tree and the Targets sub-tree, but it represents two fundamentally different things – an actual file versus a build phase value. Which means different drag behavior. Even though they look exactly the same.

I’ve Been Framed!: Solving the OCUnit App Test Problem with Frameworks

Wolf (a.k.a. “Jon”) pointed out to me that I don’t want to compile my tests into my application all the time, which my last post assumed I could do. Why? Because with the amount of tests you’re supposed to write, this could easily double the size of your application. But I don’t want target A that builds my code, and target B that builds my code+tests. I only want to build the code once!

The solution is to put as many of the tests as you can in a separate embedded framework. When you run or start a debugging session with the app, the framework is loaded at startup. OCUnit automatically runs the tests. Adding the tests is a copy phase, not a rebuild. Stripping the tests is as simple as not copying the framework into the application bundle. (This only works if you weak-link the framework; if you link the test framework regularly, then you’ll get linker errors when you run the app without it.)

I tried this with my much-abused TestExtras example from OCUnit, and immediately ran into a problem. The test framework needs to refer to the objects it’s testing, but you can’t link a framework against an application. (At least, not that I know of.) To get around this, my TestExtras application has two frameworks. One for the classes used by the application. Another for the tests on those classes.

I added the Person class from the first example in the OCUnit tutorial to my app framework, called Person.framework. I then added Person’s test class, PersonTest, to my Test.framework.

I linked against SentTestingKit.framework both in my Test.framework and in TestExtras.app itself, so that I can run both TestPerson and the tests TestExtras was originally created for, a CompletionTest instance instantiated in MainMenu.nib, at the same time. It worked! –Although the failure of the CompletionTest tests meant that the application quit immediately.

I’m pretty happy with this, so it’s time to try it out for real.

Horseshoes and Hand Grenades: OCUnit App Testing, Continued

The reason I spent so much effort, chronicled in my previous OCUnit post, to get the OCUnit tutorial working was because of a claim it made near the end: “It is true that many things [about an application’s user interface] are difficult or impossible to test automatically, but some things can still be tested.”

OCUnit is fairly easy to use with frameworks; you build the tests into the framework, and pass the whole framework as a command-line argument to the otest utility, which automatically finds and runs the tests.

But it seems to me that it’s a much more difficult proposition to test applications cleanly and quickly, especially applications and situations with involved preparatory steps.

The third OCUnit example is an application, so how does it do it? The tests are embedded in the application as a top-level object in MainMenu.nib. Top-level nib objects are instantiated when the nib is loaded, and MainMenu.nib is loaded at application startup. Perfect!

The top-level test object is an instance of SenInterfaceTestCase, which runs its tests when its awakeFromNib method is invoked – not all the time (that would be bad), but only if the proper user default has been set.

In the tutorial’s third example, this user default is set by using a third build style, after development and deployment. This is the deal-breaker, for me. I don’t want to have to maintain a different build style. I don’t want to have to, say, keep the deployment and the test builds styles manually in sync. Removing human error from the tests was the whole point of this exercise!

I tried to get around this by making a new custom executable, which I called Run Tests Executable but which was really just the same TestExtras build product from the development and deployment styles. Then I set the arguments for that custom executable so that it would run its tests. Now testing could be done with either deployment or debug styles.

Only one problem: you need a full path to the custom executable. D’oh! I don’t want a full path. Full paths can’t be checked in and used by other people on your team.

So close! But you know the old saying. Close only counts in….

Is there any way around this? Stay tuned.

OCUnit: The Missing Manual

I’ve had more trouble getting up to speed with OCUnit and its tutorial than I’d like.

My initial response, presented below, is to document the steps I took to compensate for the problems I found.

My subsequent response will probably be to send updated project, code, and tutorial files to Sen:te, which should make the rest of this post irrelevant. Until then, I present to you, with apologies to O’Reilly, OCUnit: The Missing Manual.

(Speaking of O’Reilly, their MacDevCenter has a decent simple tutorial Unit Testing with OCUnit, but it doesn’t address any of the problems discussed here.)

This is going to be a bit disjointed, because it is not meant to stand alone. Rather, it augments OCUnit’s documentation and tutorial.

Before you get started, be sure you have the latest version of OS X 10.3 (as of this writing, OS X 10.3.8) and the latest version of Xcode for 10.3, which is Xcode 1.5.

For an initial introduction to OCUnit, and to download the latest version, currently 39, go to http://www.sente.ch/software/ocunit/.

There are two OCUnit downloads, “OCUnitHome” and “OCUnitRoot”. I will only be discussing “OCUnitHome” right now.

If you follow the instructions in its top-level ReadMe file, the “OCUnitHome” disk image (currently named “OCUnitHome-v39.dmg”) installs its contents in various locations inside your home directory, ak.a. ~/.

On the plus side, this makes it easier to manually find and delete these files later, since these locations, such as ~/Developer/, are not generally used and so will only contain OCUnit files. (And OCUnit comes with no automated uninstaller, so easy manual deletion of its files is not to be sneezed at.)

On the minus side, this installs the SenTestingKit.framework, which you will use for all your OCUnit tests, in a somewhat non-standard location: ~/Library/Frameworks/. According to Apple’s thorough and well-written Installing Your Frameworks guide, the standard location is /Library/Frameworks/. Frameworks in ~/Library/Frameworks/ will be found at runtime; the problem is at compile/link time. In your Xcode projects, you have to specify an absolute path to the frameworks you link to. If you specify a non-standard location for a framework, such as all the “OCUnitHome” example projects do, others will have to fiddle with your project, or move their frameworks on their end, to make it compile successfully. So with that warning, we will forge ahead.

The examples’ tutorial will be at <where-you-dragged-the-OCUnit-folder-to>/OCUnit/Documentation/Tutorials/OCUnitWWDC2002.rtfd.

The examples themselves are at ~/Developer/Examples/OCUnit/ (except for those that aren’t, see below). I would recommend deleting the projects with the .pbproj suffix if there is a corresponding project in the same location with a .xcode suffix. Note a .pbproj suffix doesn’t necessarily mean such a project is out of date, since Xcode recognizes this suffix and can update such files to be as current as any .xcode project. But in this case, the .pbproj projects have non-native targets, outdated paths to the OCUnit utilities, references to the obsolete SenFoundation.framework, and other problems that you would have to correct. Instead, just use the corresponding .xcode projects, which need no special steps. If you open the .xcode projects after opening the .pbproj projects of the same name, be sure to do a clean build to clear out any outdated indexing information.

Now, when you see a name such as OCUnitWWDC2002.rtfd that includes “2002,” you might think that such a document would be a little outdated, and you’d be right. For example, it mentions SenFoundation.framework, which (as mentioned above) is obsolete. And there are a couple more gotchas I describe below.

But before we get to the individual examples, a rather long-winded build window tip. The OCUnit unit tests are run as an additional shell script build phase in the main target. Errors are integrated by default into Xcode’s results; you get a red entry in the upper pane of your build window if a test fails. (Pay no attention to the part about how to achieve this integration manually for Project Builder, and keep in mind that Xcode’s build pane looks a little different than the Project Builder screenshot.) But to see the full output of the unit test script, you need to open the build log pane of the build window. Do this by clicking the “Show build log” icon in the build window’s middle bar. The icon looks like a tiny note and is the third icon from the left. Then, you will see the list of test suites that were run.

First Example: Person
The first example is for the Person project. Note that the proposed solutions for the fullName tests are not the only possible solutions. For example, they change the results of the firstName method. If you already had tests for firstName that assumed certain results, changing firstName for the sake of fullName might cause some firstName tests to break.

Also, their solution involves the NSString API stringByTrimmingSpace, which no longer exists, if it ever did. Instead, you need to use the far more verbose NSString API stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceCharacterSet]. In the PersonComplete project that fully implements the Person tutorial tests, they got around this API problem by implementing stringByTrimmingSpace in the Person class, which also changes how the method needs to be invoked. Either follow the PersonComplete example, or use the API mentioned above, to complete the Person example without errors.

Second Example: SenRandomTest
The test method the tutorial introduces, testSenRandomUnsignedLong, is actually already present, though slightly better implemented, in the provided file SenRandomConformanceTest.m. You don’t have to type anything in yourself.

Third Example: TestExtras
This example requires TextExtras, available at http://www.lorax.com/FreeStuff/TextExtras.html (The tutorial’s URL is slightly different, but you can still navigate to TextExtras from it.)

For me, the OCUnit TestExtras project folder was not installed correctly sometimes when I ran the installer. It is in the OCUnit/Examples folder of the “OCUnitHome” disk image, and can be moved to ~/Developer/Examples/OCUnit/ manually.

The TestExtras project is not available in .xcode form, which means all the improvements we avoided making for the other projects, now we have to do to make it compile at all.

Step 1: Choose the menu item Project→Upgrade All Projects in Target to Native.

Step 2: Go to the shell script build phase Info window (open the Targets→TestExtras (Upgraded) node under Groups & Files, select the last entry under it, Shell Script Files, and hit Cmd-I) and change the script path there, which starts with $SYSTEM_DEVELOPER_DIR/, to the path: ~/Developer/Tools/RunTargetUnitTests (replace the whole path).

Step 3: Delete the reference to the SenFoundation.framework under TestExtras→Frameworks→Linked Frameworks.

Step 4: Delete the line #import <SenFoundation/SenFoundation.h> from the file CompletionTest.m.

Step 5: Enable Objective-C exceptions, by selecting the Targets→TestExtras (Upgraded) node, hitting Cmd-I to go to its Info window, going to the Build tab, finding the checkbox for “Enable Objective-C Exceptions”, and checking it.

Now that the project compiles, you need to clean up its code warnings. In CompletionTest.m, NSString is assumed to have a method called isEmpty. You have to replace that with isLength] == 0.

The test method the tutorial mentions, testForward, is, as with the second example, already in the provided source code file.

The test method that actually fails is testCycle. The tutorial goes on to explain how to debug that failure: instead of running the tests at compile time via a shell script build phase, you run them in a debugging session by specially triggering the tests during a normal application run.

They mention a target “Executables” tab: in Xcode, this is the top-level Executables node under Groups & Files. You need to open the Info window for the TestExtras (Upgraded) leaf under the Executables node and look at the “Arguments to be passed on launch” table at the top.

The tutorial then says, “Choose to use the ‘SenTest Self’ argument,” but what that really means with Xcode is to add a new row to the “Arguments” table via the “+” button, and then set the row’s string to -SenTest Self. (Important: note the initial dash.)

Now, you will be able to follow the rest of the tutorial and stop at a breakpoint to debug what’s wrong with the test.

That’s it! At least for the OCUnit tutorial. I have more to say about OCUnit, though, so stay tuned.

Update 3/15: Updated descriptions of second and third examples to be more accurate. Also, I’ve sent my modified files to Sen:te. We’ll see if their spam filter nabs my email, which is (a) unsolicited and (b) has both a URL and an attachment in it.


Two things:

(1) I remember reading somewhere, probably a weblog, about how test-driven development was acknowledged to be a Bad Idea for UI and security. I think it was security. I know UI was one of them. Anybody know where this was originally written? I’ve already garnered some link karma, so help me out, LazyWeb!

(2) I have been reading about test-driven development for the sake of a specific project. A project, it has occurred to me, which involves both Objective-C and C++.

For Objective-C, there are already at least two unit-testing frameworks:

Both have Xcode integration.

C++ seems less clear-cut. In fact, a Google search on “test-driven development C++” turns up this link: “Much harder than it should be”. Still, the article mentions two frameworks:

  • CppUnit, preferred but still merely “about it”
  • TUT Framework, which requires “strongly template-compliant compiler” – a good sign

Since my project uses Xcode, and OCUnit et al already integrates with Xcode, of course I want a C++ framework that also integrates with Xcode.

Anybody doing this already?

Update: It occurs to me that I can use OCUnit or UnitKit to test my C++ code – even if all the code involved is in C++, the test method can still be Objective-C. That’s probably the route I’ll go, just to save time.

My Audigone: NetBuild

Updated: see bottom.


It’s taken me longer than I would like, but I have finally gotten around to releasing my Audigone as Open Source.

My foray into Audigone territory was not a solo one, nor was it even my idea.

In early 2003, my then-coworker Mac Murrett saw that Xcode had a problem. It took forever to compile large-scale C++ projects, even on the fastest Macintosh hardware.

Why not, he thought, replace the cc executable used by Xcode with our own pass-through utility, which would distribute the individual file to a network of machines for compilation?

Now, this is not a new idea. Windows has several commercial distributed-compilation products, and there is an open-source utility, distcc, as well. But our version would use Rendezvous to build and maintain its list of helper machines dynamically and would work seamlessly with Project Builder.

Oh, and we were going to sell it for $450 a seat.

Mac did the heavy lifting for our cc replacement, called nbcc, and the Rendezvous mechanism. He deserves the credit for getting the system up and running.

I did – well, I did other stuff. A PackageMaker-based installer with a lot of bash shell script custom logic (PackageMaker really sucks, by the way). A Preferences plug-in to allow users to turn compiling on and off. A good bit of the writing for the Web site, which my designer friend Mark Abrams half-jokingly said looked like it had been put together by a college student.

iTunes left panel, including Library, playlist, shared music tree, and more

But finally we were ready to go, under the company name “Distributed Experts”. We planned to hand out buttons made with the graphic you see above at MacHack 2003, and get our product listed in ADC News.

It was when we tried to arrange the latter that we realized something was up. At first, our ADC request seemed to be going fine. Then: nothing. We didn’t show up in the ADC newsletter, we didn’t hear from anyone. MacHack came and went with no news. Then came WWDC….

At WWDC, Apple introduce Xcode 1.0, and one of its featured was a distributed build system. It was based on distcc, as it turned out, and anecdotal evidence suggested that our system was more efficient, but we could see the writing on the wall.

We closed down the Web site without having made a single sale.

Now, as stated above, I’m sending NetBuild where it seems many commercial projects go to die, the heaven of Open Source. It is available at my site, Umbar.com.

Addendum: Yes, it occurred to me when I started writing about Audigones that I had invented a category especially for my own experience. Certainly I haven’t received an avalanche of Audigone stories, obscure or otherwise. A false generalization? Has Apple really not slain dozens and dozens of companies? Or have I just not reached the right people to tell the tales?

And hey: anybody want a button?