DIY versus rapid development

Not so long ago, Slashdot had a link to something by the CEO of Wind River: five things you can do to avoid becoming roadkill. The first of his five things to do (I didn't see any particular reason to assume that he considered it the most important) was to "get over Do It Yourself".

He claimed that using your own code instead of off-the-shelf code is too risky and too time consuming.

Like all generalizations (including this one), this is wrong.

One of the Big Things They Don't Teach In School is that everything in life is a trade-off. Ignoring the wider applicability of this, just look at a few of the computer science examples: using more space to save time, spending more time now to save time later, implementing a more complex algorithm to save time or space, implementing wanted new functionality at the cost of stability (hopefully only in the short term), implementing new functionality at the risk of unwanted interactions with existing functionality, adding ugly hacks for compatibility with something widespread but broken... We all do these things, pretty much all the time.

And writing your own code versus using someone else's is no different. I don't tend to think very concretely, so as practice I'll force myself here to explain what I mean with an example. Since 1997, I've used an editor, Edit that I wrote with friends of mine. Initially, it was a Java implementation of wily, itself a Free Unix/X11 implementation of Pike's acme. Since then it's diverged quite a bit.

The first version used a home-grown text component. Swing wasn't out yet, and the off-the-shelf solution of the time, AWT's TextArea didn't offer a suitable interface (both from a UI perspective and from an implementation perspective) for a lot of the stuff we wanted to do. So at that point, there was no choice.

After a few years, we decided that Swing's JTextArea was probably good enough, and that rather than fix a few bugs that had been annoying us, and rather than do anything about our unusual acme-like scroll bars, we'd just switch to JTextArea. Which we did. And that was okay, except now we didn't have our bugs; we had Sun's bugs. And sometimes there were work-arounds, and sometimes there weren't. When there were, they cost us both in the time and experimentation to find them, and in the time and resulting complexity of using the work-arounds. As I'm sure you know, work-arounds aren't generally the nicest chunks of code in any project.

A good example of a hard problem is that the up-arrow on the top line of a JTextArea doesn't work. Any other text component interprets this as moving the insertion point to the beginning of the document. JTextArea just does nothing. I think it did work in 1.3, but it's been broken ever since, with a bug on the bug parade that I can't be bothered to find right now, and it's still broken in 1.5. And if you look at trying to work around it, you'll see that the particular method that needs fixing could hardly be in a more awkward place. I think it's a static method in a package-private class. Working around that kind of problem generally involves a lot of copy and paste, which introduces its own problems. Especially if other stuff gets fixed in the meantime, or you want to work on multiple Java versions.

If you think that's easy, I'd like to see a work-around that always (rather than mostly) works for the problem of the selection highlight not extending to the right-hand margin when the selection includes a the whole line.

After a while, we decided that we'd like to experiment with colored text. As you may know, JTextArea doesn't support that, but JTextPane does. Only there's a huge leap there from a relatively-lightweight plain text component to a big, slow, buggy text component intended to be able to implement word processors and HTML viewers. But we tried it anyway and vastly reduced the maximum size of file we could open, and vastly increased the time various operations took. The best part, though, was that we couldn't really implement the coloring we wanted, because we couldn't manage to safely batch style mutations to avoid causing a lot of redrawing. I know there's a toy example in the SwingSet demo, but it's a joke. "Please wait" while it applies the coloring? And editing disabled? They may as well have a notice there saying "we know this is too slow to be useful, but we don't intend fixing it, so get lost".

If you can't re-color the whole of a 22,000 line C++ source file from Mozilla because someone's opened a comment near the start so quickly that it doesn't feel any different from typing the first character in an empty file, you're not useful. Because the world is full of poor bastards who have to edit those kinds of files. (Much of Edit's focus is on making it easier to get around large files and large codebases.)

So we went back to JTextArea. And now we're working on a text component of our own. A less buggy, slightly better-featured equivalent of JTextArea. We considered using whatever JEdit uses, but when we looked, it didn't support wrapping (that seems to be another preference that distinguishes Unix and MS Windows users), and it was really tied into that particular editor. We want something we can use in all our other projects, and that others can use in their projects too, without too much hassle.

I think, if we were starting Edit today, we'd start with JTextArea. There's no question that it's good enough for an initial implementation, and the advantage of being able to spend your time working on the editor's features rather than on a text component is a huge one. And don't get me wrong: I learned from the Swing text components that treating the selection as a special case of a more general notion of highlighters is very powerful, and I wouldn't want to use a text component that didn't support highlighters.

But sooner or later, we'd end up writing our own. There would be enough pressure for coloring, say, or we'd get sufficiently sick of all those little bugs that don't seem so important when the text component isn't the main focus of your application, but when you're using it all day every day can really get to you. (The JTextArea bug where sometimes a selected line will only render the selection highlight, and not the text, is the scariest.)

And yes, I do see the humor in using Yet Another Text Editor as an example in an explanation of why there will always be room for DIY. But James Gosling wrote (since he started blogging, most of his papers have been unavailable, so I'll have to paraphrase what I remember rather than quote and link) that it's naive to expect to not have to implement your application's fundamental class yourself. This may be because you're doing something sufficiently new that by definition you can't use someone else's code, or it may be because you need millions of something [I think Java's lack of user-defined value types was his specific context]; the traditional example of not building a spreadsheet from your system's grid container of your system's text field, because neither was likely to have been designed for use on the scale you're thinking of.

So, after explaining myself through an example, I'm left wondering if what the guy really meant was "you probably want to use an off-the-shelf OS", but didn't want to say that as an OS vendor, and abstracted his message to a point where it made no sense. That interpretation would certainly explain "when you contribute to differentiated value, you're also differentiating yourself. You become visible and invaluable instead of out of date, cranky, quaint." He's recommending knowledge of something like RTLinux instead of your own task scheduler, rather than suggesting that you won't learn anything from having written an editor or a compiler, or even your own task scheduler. (And pointing out that although the implementor of the home-grown scheduler learns useful lessons, those who have to use it just learn arcana that no-one cares about and that won't be much use anywhere else.)

It may have taken me several weeks and over 1,000 words, but I think I get what he was trying to say now. If only he'd had the confidence to say what he meant. I'm sure no-one would have held it against him if he'd mentioned (say) RTLinux every time he'd mentioned his own product.