-no-cpp-precomp: the compiler flag that time forgot

I’m often surprised how long software can keep trying to use compatibility features ages after their best-by date.

Now that GCC 4.8 builds on Tiger1, I’ve been testing as much software with it as I can. When building ncurses using GCC 4.8.1, though, I came across a strange error message:

gcc-4.8: error: unrecognized command line option ‘-no-cpp-precomp’

It built fine with the gcc-4.0 that came with the OS. GCC rarely removes support for flags like this, so I assumed it must be an Apple-only flag. Unfortunately, it wasn’t listed in the manpage at all, and the interenet was no help either – the search results were full of users confused about the same build failures, or trying to figure out what it does. All I could find was confirmation that it’s an obsolete Apple-only flag.

Not finding anything, I decided to find out straight from the horse’s mouth and try source-diving. Luckily Apple publishes the source code for all obsolete versions of their tools at their Apple Open Source site.

Recent versions of Apple GCC don’t include the flag anywhere in their source. The only place it’s still referenced is in a few configure scripts and changelogs, such as these:

In several releases prior to that, for instance gcc-5493, the flag is explicitly mentioned as being retained for compatibility and is a no-op:

The last time it was actually documented was in gcc-1765‘s install.texi, shipped as a part of the WWDC 2004 Developer Preview of Xcode, which also provides a hint as to what the flag actually did:

It’s a good idea to use the GNU preprocessor instead of Apple’s @file{cpp-precomp} during the first stage of bootstrapping; this is automatic when doing @samp{make bootstrap}, but to do it from the toplevel objdir you will need to say @samp{make CC=’cc -no-cpp-precomp’ bootstrap}.

So this partially answers our question: Apple shipped an alternate preprocessor, and -no-cpp-precomp triggers the use of the GCC cpp instead. I can only assume this was a leftover that had yet to be excised, because the flag itself was still a no-op at that time. To actually find a version where the flag does something, we have to go all the way back to the December 2002 developer tools, whose gcc-937.2 actually has code that uses the flag. This particular build of GCC is Apple’s version of gcc-2.95, and it appears to be the very last where it had any effect. Interestingly, the #ifdef that guards this particular block of code is “#ifdef NEXT_CPP_PRECOMP” – suggesting that this dates back to NeXT, rather than Apple.

To actually find out what this means, O’Reilly’s Mac OS X for Unix Geeks, from September 2002, has a nice explanation in chapter 5:

Precompiled header files are binary files that have been generated from ordinary C header files and that have been preprocessed and parsed using cpp-precomp. When such a precompiled header is created, both macros and declarations present in the corresponding ordinary header file are sorted, resulting in a faster compile time, a reduced symbol table size, and consequently, faster lookup. Precompiled header files are given a .p extension and are produced from ordinary header files that end with a .h extension.

Chapter 4 also provides a nice explanation of why -no-cpp-precomp was desirable:

cpp-precomp is faster than cpp. However, some code may not compile with cpp-precomp. In that case, you should invoke cpp by instructing cc not to use cpp-precomp.

So there we have it – -no-cpp-precomp became somewhat widely used in Unix software as a compatibility measure to prevent Apple’s cpp-precomp feature from breaking their headers, and has stuck around more than a decade since the last time it’s actually done anything.

  1. More on that in a future blog post. []

Software archaeology: Apple’s cctools

One of the things I’ve been working on in Tigerbrew is backporting modern Apple build tools. The latest official versions, bundled with Xcode 2.5, are simply too old to be able to build some software. (For example, the latest GCC version available is 4.0.)

In the process, I’ve found some pretty fascinating bits of history littered through the code and makefiles for Apple’s build tools. Here are some findings from Apple’s cctools1 package:

This comment from near the top of cctools’s Makefile lists some of the valid build targets, which includes:

  • Kodiak, which was the Mac OS X public beta from September, 2000
  • Gonzo (Developer Preview 4), Bunsen (Developer Preview 3), and Beaker (PR2)
  • Rhapsody (internal name for the OS X project as a whole), Hera (Mac OS X Server 1.0, released 1999), and teflon (unknown to me)
  • OPENSTEP, NeXT’s implementation of their own OpenStep API

From further down in the same Makefile:

A lot of familiar cats here, along with a couple of early iOS versions (SugarBowl, BigBear) and a lot of names I’m not familiar with. (Please leave a comment if you have any insight!) As far as I know “Vail” was the Mac LC III from 1993 with no NeXT connection, but I’m sure it must be referring to something else.

From elsewhere in the tree, there’s code to support various CPU architectures. Aside from the usual suspects (PPC, i386), there are some other interesting finds:

  • HP/PA, aka PA-RISC, a CPU family from HP; some versions of NeXTSTEP were shipped for this
  • i860, an Intel CPU used in the NeXTdimension graphics board for NeXT’s computers
  • M680000, the classic Motorola CPU family, used in the original NeXT computers
  • M880000, a Motorola CPU family; NeXT considered using this in their original hardware but never shipped a product using it
  • SPARC, a CPU family from Sun; some versions of NeXTSTEP were shipped for this

I find it fascinating that, even now, cctools still carries the (presumably unmaintained) code for all of these architectures Apple no longer uses.

  1. Apple’s equivalent of binutils. []

Tiger’s `which` is terrible; or, Necessity is the mother of invention

One of the most useful things about running software in unusual configurations is that sometimes it exposes unexpected flaws you never knew you had.

The which utility is one of those commandline niceties you never really think about until it’s not there anymore. While sometimes implemented as a shell builtin1, it’s also frequently shipped as a standalone utility. Apple’s modern version, which is part of the shell_utils package and crystallized around Snow Leopard, works like this:

  • If the specified tool is found on the path, prints the path to the first version found (e.g., the one the shell would execute), and exits 0.
  • If the specified tool isn’t found, prints a newline and exits 1.

This version of the tool is really useful in shell scripts to determine a) if a program is present, and b) where it’s located, and until fairly recently Homebrew used it extensively. Unfortunately, early on in my work on Tigerbrew, I discovered that Tiger’s version was… deficient. It works like this:

  • If the specified tool is found on the path, prints the path to the first version found, and exits 0.
  • If the specified tool isn’t found, prints a verbose message to stdout, and exits 0.

The lack of a meaningful exit status and the error message on stdout are both pretty poor behaviour for a CLI app, and broke Homebrew’s assumptions about how it should work.

To work around this, I replaced Homebrew’s wrapper function with a two-line native Ruby method for Tigerbrew, like so:

As it turns out, not only does it work better on Tiger, but this method is actually faster2 than shelling out like Homebrew did; process spawning is relatively expensive. As a result, I ended up using the new helper in Homebrew even though it wasn’t strictly necessary.

(And as for the commandline utility, Tigerbrew has a formula for the shell_cmds collection of utilities.)

  1. zsh does; bash doesn’t. []
  2. On the millisecond scale, at least. []

Adventures with Ruby 1.8.2

Homebrew has always used the version of Ruby which comes with OS X,1 a design decision I decided to keep with Tigerbrew. Tiger comes with Ruby 1.8.2, built on Christmas Day, 2004, and with a version of Ruby that old I went in steeling myself for the inevitable ton of compatibility issues.

On the whole I was pleasantly surprised. Most of what Homebrew uses is provided in exactly the same form, and while there are differences that range from puzzling2 to major3, pretty much everything Just Works.

Except, at first, for Pathname. Ruby’s Pathname class, which is an object-oriented wrapper around the File and Dir classes, is at the heart of Homebrew’s file management. The first time I tried to install something with the newborn Tigerbrew, I was quickly treated to a strange exception with an equally mysterious backtrace: Errno::ENOTDIR: Not a directory.

Curious, I dug in. I soon discovered that the bug occurred while Homebrew was unlinking an existing version of a package before beginning to install an upgrade. (For those not in the know, Homebrew installs software into isolated versioned prefixes. The active version of a given package is symlinked into the standard /usr/local locations.) Most of the files were linked and unlinked just fine, but a few files caused the method Pathname#unlink to throw an exception every time. Eventually I noticed a pattern — every symlink that Pathname choked on represented a directory. Once I noticed that, it clicked.

For those who don’t know, symlinks are actually treated on the filesystem level as special files containing their target as text. For most operations, symlinks transparently act as their targets. However, applications which hit the filesystem directly will see them as files — even when they point to directories. Since Pathname handles files and directories differently, handing its instance methods off to File or Dir as appropriate, the bug happened something like this:

  1. The #unlink method is called on a Pathname object representing a symlink to a directory.
  2. Pathname examines the object to see if it represents a file or directory, in order to determine whether to call File.unlink or Dir.unlink.
  3. In doing so, Pathname follows the symlink to its target and examines the properties of the target.
  4. Seeing that the target is a directory, Pathname calls Dir.unlink on the original symlink.
  5. Dir.unlink raises Errno::ENOTDIR because, of course, the symlink isn’t a directory.

Ruby 1.8.2 is of course quite old at this point, but it’s pretty mindboggling to see a bug this major in a programming language’s standard library. Thankfully Ruby supports monkeypatching, so I was able to override the method with a version which acts sanely.

The overridden version of the method can be found here. The rest of Tigerbrew’s current backports are in Tigerbrew’s file extend/tiger.rb, for the curious.

  1. For predictability, and so the user doesn’t have to install Ruby before installing Homebrew. []
  2. String’s [] operator always returns the sliced character’s ASCII ordinal, not a string. []
  3. File#flock doesn’t exist in any form. []

Introducing Tigerbrew

Some of you may know that my other gig is Homebrew, the package manager for Mac OS X. Over the last few months, I’ve been spending some time on a fork of Homebrew that’s starting to become usable enough that I think it’s ready to be announced.

When I was attending the AMIA1 conference in December, my partner and I were travelling together; while I was at the conference during the day, she worked from various places in Seattle on her laptop. Since it’s practically impossible to attend a modern conference without a laptop, and she uses a desktop at home, I dug out my 2005-era PowerBook G4 to take notes. It may be eight years old, but as soon as I opened it up I remembered why I loved that laptop so much. It’s still in great shape, and it feels like a crime to leave it sitting unused so much of the time.

It’s slow by modern standards, of course, but the thing really keeping it from being usable all the time is software. Apple’s left PowerPC behind as of Mac OS X Leopard2, and so have nearly all developers at this point. There are still a few developers carrying the torch (shoutouts to TenFourFox), but as a commandline junkie what I really need is an up-to-date shell3 and CLI software. And as big Homebrew fan, as well as a developer, MacPorts just wasn’t going to cut it. Tigerbrew was born.

The first version of Tigerbrew was pulled together over an evening at the hotel after the first day of the conference, and I’ve been plugging away at it regularly since. At this point I’m proud to say that a significant number of packages build flawlessly,4 and thanks to some backports from newer versions of OS X5 Tigerbrew can supply a much more modern set of essential development tools than Apple provides.

Tigerbrew’s still very much an alpha, and there’s some more work needed until it’s mature, but at this point I consider it ready enough to announce to the world.6 If you have a PowerPC Mac yearning to be used again, why not give it a go?

  1. Association of Moving Image Archivists []
  2. And many hardcore PowerPC users stick with their old Macs for OS 9 compatibility, which was last supported in Tiger. []
  3. bash 2.5 doesn’t cut it. []
  4. Even complex software with a lot of moving parts, like FFmpeg. []
  5. I’m very indebted to the MacPorts developers, whose portfiles served as a reference for the buildsystems for several of these. []
  6. Development’s been happening in the public for months, of course, and there are already a few other users out there. []

PSA: homebrew-digipres repository now available!

Outside of archivy, I’m also a collaborator on Homebrew, the awesome, lightweight package manager for OS X. I’ve been building a private repository of niche packages which aren’t available in the core repository for some reason or another, and ended up collecting enough digital preservation tools to create a new digital preservation-focused repository. You can find the new homebrew-digipres here: https://github.com/mistydemeo/homebrew-digipres I’d welcome any contributions if you want to improve an existing formula, submit updates, or add a new package! Fork away.

Sigma SD1 – update

Heartbreak :’(



Apologies for the long hiatus. I didn’t announce it on the blog, but I began a new job in January and moved to another province to take it.

I plan to get back to posting in the next bit. It will be awhile until things get set up, but I promise I’ll have cool stuff to share – and I’ll get back to posting digitization tutorials in the near future as well.