elliotth's blog

Beware convenience methods

2012-01-01T23:09:00.000Z

The Android documentation links from various methods to Beware the default locale. It's great advice, but most people don't know they need it.

I've never been a great fan of convenience methods (or default parameters), and methods that supply a default locale are a prime example of when not to offer a [supposed] convenience method.

The problem here is that most developers don't even understand the choice that's being made for them. Developers tend to live in a US-ASCII world and think that, even if there are special cases for other languages, they don't affect the ASCII subset of Unicode. This is not true. Turkish distinguishes between dotted and dotless I. This means they have two capital 'I's (one with and one without a dot) and two lowercase 'i's (one with and one without a dot). Most importantly, it means that "i".toUpperCase() does not return "I" in a Turkish locale.

The funny thing is, toLowerCase and toUpperCase just aren't that useful in a localized context. How often do you want to perform these operations other than for reflective/code generation purposes? (If you answered "case-insensitive comparison", go to the back of the class; you really need to use String.equalsIgnoreCase or String.CASE_INSENSITIVE_ORDER to do this correctly.)

So given that you're doing something reflective like translating a string from XML/JSON to an Enum value, toUpperCase will give you the wrong result if your user's device is in a Turkish locale and your string contains an 'i'. You need toUpperCase(Locale.ROOT) (or toUpperCase(Locale.US) if Locale.ROOT isn't available). Why doesn't Enum.valueOf just do the right thing? Because enum values are only uppercase by convention, sadly.

(The exception you'll see thrown includes the bad string you passed in, which usually contains a dotted capital I, but you'd be surprised how many people are completely blind to that. Perhaps because monitor resolutions are so high that it looks like little more than a dirt speck: I versus İ.)

In an ideal world, the convenience methods would have been deprecated and removed long ago, but Sun had an inordinate fondness for leaving broken glass lying in the grass.

The rule of thumb I like, that would have prevented this silliness in the first place, is similar to Josh Bloch's rule of thumb for using overloading. In this case, it's only acceptable to offer a convenience method/default parameter when there's only one possible default. So the common C++ idiom of f(Type* optional_thing = NULL) is usually reasonable, but there are two obvious default locales: the user's locale (which Java confusingly calls the "default" locale) and the root locale (that is, the locale that gives locale-independent behavior).

If you think you're safe from these misbegotten methods because you'd never be silly enough to use them, you still need to watch out for anything that takes a printf(3)-style format string. Sun made the mistake of almost always offering those methods in pairs, one of which uses the user's locale. Which is fine for formatting text for human consumption, but not suitable for formatting text for computer consumption. Computers aren't tolerant of local conventions regarding the use of ',' as the decimal separator (in Germany, for example, "1,234.5" would be "1.234,5"), and it's surprisingly easy to write out a file yourself and then be unable to read it back in! (There are a lot more of these locales, though, so in my experience these bugs get found sooner. The Enum.valueOf bug pattern in particular regularly makes it into shipping code.)

Where locales are concerned, there's really no room for convenience methods. Sadly, there's lots of this broken API around, so you should be aware of it. Especially if you're developing for Android where it's highly likely that you actually have many users in non-en_US locales (unlike traditional Java which ran on your server where you controlled the locale anyway).

ThinkPad "plugged in, not charging"

2011-12-23T18:30:00.000Z

The web is full of all kinds of advice if you find yourself with a ThinkPad running Windows 7 that refuses to charge. The message "plugged in, not charging" being what you see in the tool tip if you hover over the battery icon in the system tray. (The Windows battery icon, that is, not the Lenovo one next door. I love the way Lenovo ship their own battery and wifi widgets, so you get two of each, and I really love the way that -- unlike the Microsoft ones -- the Lenovo ones get stretched and blurry at screen resolutions that Lenovo ship. Nice touch. With attention to detail like that from their competitors, it's no wonder Apple has such a hard time in the laptop market.)

Anyway, none of the advice I saw on the web helped (though all the BIOS updating and other fun was hopefully not completely wasted). The trick turned out to be that there's a little catch on the underside of the laptop, next to the battery. When you push the battery into place, that catch goes from "unlocked" to somewhere in the middle, just by the mechanical force of battery insertion. But you actually have to physically move the catch all the way over to the "locked" side before you can charge. There's nothing in the UI to indicate this, not even if you run the system tests.

Like I said: with attention to detail like that from their competitors, it's no wonder Apple has such a hard time in the laptop market.

How can I get a thread's stack bounds?

2011-09-22T05:18:00.003Z

I've had to work out how to get the current thread's stack bounds a couple of times lately, so it's time I wrote it down.

If you ask the internets, the usual answer you'll get back is to look at the pthread_attr_t you supplied to pthread_create(3). One problem with this is that the values you get will be the values you supplied, not the potentially rounded and/or aligned values that were actually used. More importantly, it doesn't work at all for threads you didn't create yourself; this can be a problem if you're a library, for example, called by arbitrary code on an arbitrary thread.

The real answer is pthread_getattr_np(3). Here's the synopsis:

#include <pthread.h>

int pthread_getattr_np(pthread_t thread, pthread_attr_t* attr);

You call pthread_getattr_np(3) with an uninitialized pthread_attr_t, make your queries, and destroy the pthread_attr_t as normal.

pthread_attr_t attributes;
errno = pthread_getattr_np(thread, &attributes);
if (errno != 0) {
  PLOG(FATAL) << "pthread_getattr_np failed";
}

void* stack_address;
size_t stack_size;
errno = pthread_attr_getstack(&attributes, &stack_address, &stack_size);
if (errno != 0) {
  PLOG(FATAL) << "pthread_attr_getstack failed";
}

errno = pthread_attr_destroy(&attributes);
if (errno != 0) {
  PLOG(FATAL) << "pthread_attr_destroy failed";
}

Note that you don't want the obsolete pthread_attr_getstackaddr(3) and pthread_attr_getstacksize(3) functions; they were sufficiently poorly defined that POSIX replaced them with pthread_attr_getstack(3).

Note also that the returned address is the lowest valid address, so your current stack pointer is (hopefully) a lot closer to stack_address + stack_size than stack_address. (I actually check that my current sp is in bounds as a sanity check that I got the right values. Better to find out sooner than later.)

Why should you call pthread_getattr_np(3) even if you created the thread yourself? Because what you ask for and what you get aren't necessarily exactly the same, and because you might want to know what you got without wanting to specify anything other than the defaults.

Oh, yeah... The "_np" suffix means "non-portable". The pthread_getattr_np(3) function is available on glibc and bionic, but not (afaik) on Mac OS. But let's face it; any app that genuinely needs to query stack addresses and sizes is probably going to contain an #ifdef or two anyway...

Language-induced brain damage is better than the alternative

2011-05-29T00:03:00.000Z

Whenever old uncle Edsger had had a skinful, he'd rant about the harmful effects of bad languages on programmers. As a kid, I was never particularly convinced. He was too old to be all that concerned about C, and Ada hadn't even been invented yet, let alone Java, so his rants about Fortran, PL/I, Cobol and APL all seemed comically anachronistic. The languages he'd talk about were all pretty moribund anyway, at least for the purposes of a software engineer (as opposed to, say, a physicist).

BASIC, his other bête noire, never really seemed that bad. I grew up with a pretty good BASIC, the main deficiency of which seemed to be the lack of a garbage collected heap (except for strings, which were special). Even as a kid, it was pretty clear that fixed-size arrays were evil, so it was distressing that one's language forced one into the habit. But to me, that disproved Ed's claim: I was clearly a child of the BASIC era, and yet wasn't I sophisticated enough to recognize the problems? Wasn't this ability to recognize the flaws of one's language almost a test of one's latent aptitude, and thus useful in distinguishing those with real potential from those without?

In the years since, I've used a lot of languages. It's hard to imagine a well-rounded programmer who hasn't been exposed to an assembly language, a Lisp, Haskell or an ML, SQL, C, C++, and a Java-like managed language. And it's probably a shame too few encounter a Smalltalk. Even if you don't like these languages, and even if you wouldn't use them commercially, I think they each influence the way we think about computation and programming. And I think that that makes us better at doing our jobs, regardless of which language we're actually using.

(I deliberately omitted logic programming languages -- both deductive and the even less common inductive -- because if they did have an effect on me or my thinking, I've absolutely no idea what it was, and if they didn't I've absolutely no idea what I've missed.)

So it seems to me like there's a trade-off. Yes, learning a new class of language will change the way you think, but it will be both for better and worse. I don't think you can avoid this, and I think that deliberately remaining ignorant is worse than just accepting the mental scarring as a fact of life. Hell, I even think that learning absolutely appalling languages like Ada, S, and Javascript is an important learning experience. Those who cannot remember the past are condemned to repeat it.

But what I think is really interesting, and another reason it was hard to believe Ed's claim, is that pretty much by definition you can't see the damage a language does to you as clearly as you can see the good. You're likely to remember that language X taught you Y, but you don't even know that it failed to expose you to Z. So back in my BASIC days, I never bemoaned the lack of a sequence type or a map type. I almost missed the former, but would have been over-specific in my demands: I wanted to dynamically size arrays. What I thought I wanted was something like C's realloc(3), not C++'s std::vector. It wasn't until I was a C programmer and had realloc(3) that I realized how small an advance that is, and it wasn't until I was a C++ programmer that I realized that, really, I wanted a managed heap. (Not always, of course, because someone has to implement the managed language's runtime, but another thing that learning plenty of languages teaches you is the importance of always using the highest-level one you can afford for any given task.)

I was reminded of this topic recently when someone sent round a link to a Javascript x86 emulator. The interesting part to me was Javascript Typed Arrays. Javascript is very definitely in the class of languages that I'd never voluntarily use, but that doesn't mean I'm not interested to see what they're up to. And, as maintainer of an implementation of java.nio buffers, I was interested to see the equivalent functionality that Javascript users are getting.

If you don't know java.nio buffers, they're one of Java's many ill-conceived APIs. I say this as a fan of managed languages in general, and a long-time Java user, but having both used and implemented java.nio buffers, there's very little love lost between me and them. They're almost exactly not what I would have done. Surprisingly to me, given my admitted dislike of Javascript, Javascript's typed arrays are pretty much exactly what I would have done.

If I were asked to point to the most damaging design error in java.nio buffers, it would be one that I think was a side-effect of the kind of brain damage that C causes. Specifically, I contend that C programmers don't usually have a clear mental distinction between containers and iterators. I think that was one of the things that C++'s STL really taught us: that containers and iterators (and algorithms) are distinct, and that it's important to maintain these distinctions to get a high-quality library. The design of ICU4C suffers greatly from an ignorance of this idea (ICU4C is the C/C++ equivalent of the heinous java.text classes and such all-time API war crimes as java.util.Calendar, brought to you by the same people).

Java programmers ought not to be ignorant of this important lesson, but it took two attempts to get half-decent collections in the library (three if you count the late addition of generics), and iteration has been such a dog's breakfast in Java that I don't think the lesson to students of Java is nearly as clear as it is to students of C++.

(Dog's breakfast? Enumeration versus Iterator versus int indexes; raw arrays versus collections; the awful and verbose Iterator interface; and most of all the modern Iterable turd which makes the "enhanced" for loop less generally useful than it should have been and encourages the confusion between collections and iterators because the modern style involves an anonymous and invisible iterator. From arguments I've had with them, I think those responsible were hampered by the brain damage inflicted C and their ignorance of C++, an ignorance of which they're bizarrely boastful.)

But java.nio buffers are far far worse. There, rather than offering any kind of iterators, the collections themselves (that is, the buffers) have an implicit position. (Buffers have other state that really belongs in an iterator, and that is inconsistently inherited by related buffers, but that's beyond the scope of this discussion.) You can simulate iterators by making new buffers (with Buffer.duplicate, say) but it's awkward and ass-backward, leading to ugly and intention-obscuring calling code, and leading to another generation of programmers with this particular kind of brain damage.

(At this point you might argue that the ideal model is one of collections and ranges rather than iterators, since C++ iterators do tend to come in pairs, and from there you might argue that a range is really just another way of expressing a view, and from there that a view is best expressed as a collection, and from there that the containers-are-iterators model I'm complaining actually makes sense. It's one of those "how did we get into Vietnam"-style arguments, where any individual step isn't entirely unreasonable in itself, but where the final result is utterly fucked. The problem here being not so much a land war in Asia but having all collections have an implicit position to support iterator-style usage. Which in practice means that you've got a new orthogonal axis of "constness" to worry about, and that it's a lot harder to share containers. It's actively working against what I think most people consider to be one of the big lessons Java taught us: design for immutability. In a functional language, always working with views and relying on referential transparency might be fine, but Java is not that language, and many of the mistakes in the collections API are, I think, down to trying to pretend that it is. Which I hope makes it clear that I'm not blaming C any more than I'm blaming Haskell: I'm just giving examples of mistakes caused by transferring concepts into situations where they no longer make sense.)

The Javascript DataView class is sorely missing from java.nio's mess too. It's a really common use case that's very poorly served by java.nio. My java.nio implementation has something similar internally, but it's really nice to see the Javascript guys exposing what appears to be a lean, sane, task-focused API.

I do think there's a really nice comparative programming languages book to be written, and I think one of the most interesting chapters would be the one about iteration styles. I don't know whether it's surprising that something so fundamental should differ so wildly between languages (I didn't even touch on the Smalltalk style, which is something completely different again from any of the styles I did touch on), or whether it's exactly in the fundamentals that you'd expect to find the greatest differences.

If this is the brain-damage uncle Ed was so keen to warn me about, all I can say is "bring it on!". As far as I can tell, more different kinds of brain damage seems to lead to better outcomes than staying safe at home with the first thing that ever hit you over the head.

Fuck Sony

2011-05-19T04:55:00.000Z

So the only reason I use my PS3 any more is for Netflix. It sucks hard when it comes to anything else. It's a crap DVD player compared to the 360, cross-platform games are usually better on the 360, the platform exclusives are usually more interesting on the 360, and the PS3 doesn't even work as reliably with my Harmony remote as the 360 does.

And that's ignoring the PS3's UI shitfest.

There's nothing wrong with the 360's Netflix player (though it does tend to lag behind the PS3's because it's beholden to Microsoft's slow OS update schedule rather than downloading each time you start it). I use the PS3 for Netflix because I resent paying Microsoft $60/year protection money for Netflix on the Xbox 360.

The flaw in my logic is that paying Microsoft's protection money only upsets me one day each year. Using the PS3 upsets me every single night.

If I'm not being forced to install some stupid update that benefits Sony but not me, I'm being forced to fail to log in twice to their broken online gaming network so I can use Netflix, the only thing on the PS3 that doesn't suck.

Tonight, I'm told my password is no longer valid. (A not unreasonable thing to declare when your broken online gaming network has just lost millions of your users' passwords and credit card numbers.) I'm told that a link has been mailed to my email address. I wait for it to arrive, click it, and get:

PLAYSTATION

Site Maintenance Notice

The server is currently down for maintenance.

We apologize for the inconvenience. Please try again later.

メンテナンスのお知らせ

現在、サーバーのメンテナンス中です。

大変申し訳ございませんが、しばらくしてから再度接続してください。

So now my PS3 is completely useless. I've no idea when they'll fix this, but if it's anything like as fast as they fixed their last network problem, I've got about 30 days to wait.

Typical fucking Sony bullshit. And I'm finally sick of it. The PS3 is dead to me. I don't know whether I'm actually going to support Microsoft's protection racket, but I'd rather watch Netflix on my Nexus S than put up with any more of the PS3's passive-aggressive nonsense.

Ubuntu 11.04

2011-05-15T20:39:00.000Z

If Mac OS is the continuing evolution of Steve Jobs' vision of how we should use our computers, it's becoming increasingly clear that Ubuntu is Mark Shuttleworth's indirect request that we all just fuck off and get ourselves an OS from someone who actually gives a shit.

Rise
I was a big fan of Ubuntu in the beginning. I liked Debian in principle, but hated having to choose between the "stable" and "testing" branches, the former of which was literally years out of date, while the latter was too unstable for my taste (leading me to dub the choice "stale" or "broken"). Ubuntu at the time seemed to strike a happy medium: a reasonably well-tested 6-month snapshot of Debian "testing". As far as I recall, my only real complaint in the early days was that its color scheme had been decided upon by someone we can only assume was legally blind. Turd brown with safety orange highlights: no sighted person's first choice.

It also seemed, in those early days, as if Canonical was adding some value. They were acting as editors, shielding us from the petty internecine freetard religious wars. So, for example, those of us who just wanted to be able to play our mp3s didn't have to have an opinion on exactly which of the 100 not-quite-shippable music apps to choose, nor did we have to trawl through them all trying to find one that we'd consider acceptable: someone at Canonical had made a good-enough choice for us.

Decline...
Then things turned bad. Each release was less stable than the last. Only the LTS ("Long Term Support") releases were even half-way reasonable, and then they started fucking them up too, changing major components shortly before release, swapping in things that couldn't be considered stable. (And, of course, the user who restricts themselves to LTS releases gets to relive the old Debian "stable" days. Given that Debian is no longer as pathologically bad at shipping as it once was, such a user would have to ask themselves "why not Debian?".)

The usual volunteer disease afflicted Ubuntu too: people would only work on stuff that interested them. Which basically means that the same three components (window manager, desktop, music system) get rewritten over and over and over, each one being replaced before it can actually mature to the state where an impartial observer might call it good.

...and Fall
And now we have Ubuntu 11.04. The worst release yet. A release so bad even noted free software apologist Ryan Paul hates it.

I've no idea what the underlying stuff is like, because the surface layer of crap is so bad that it's taken away all my will to use it, and I'm spending my time surfing the web trying to decide which other distro to jump ship for. (Presumably Debian, but if I'm going to go to all the trouble of reinstalling, I may as well do the legwork.)

Misguided netbook focus
What sucks? There's yet another implementation of a dock from someone who appears to know nothing of the competition that can't be gleaned from screenshots. The old task-bar-thing has moved to the top of the screen (and apparently can't be moved back). The old menus are gone, and so are the buttons representing windows (the latter of which never worked very well anyway, compared to Mac OS or Windows). My system monitor and weather thingies disappeared (and if they can be added back, it's not in any way I can find), the rather nice world map used for the world clock is gone, my launcher for Chrome was replaced by Firefox and random crap I've never used like OpenOffice (and if I can add my own launchers, I couldn't work out how). The replacement for the apps menu appears to be an enormous search box that -- despite using almost a quarter of the area of my 30" display -- somehow only manages to show four apps at a time.

(Despite all this upheaval, there's no attempt to introduce users to the new system.)

The reason for moving the task-bar-thing to the top of the screen is because they've tried to switch to a Mac-style screen menu bar (rather than a Windows-style per-window menu bar). Unfortunately, this doesn't work with any of the apps I actually use. The only thing I've found that it did work with was the on-line help which I tried to use, but that inexplicably starts in some kind of full-screen mode, making it really frustrating to try to actually follow the instructions in the help.

I'm sure some of this must be semi-reasonable on a 10" netbook screen, but can only assume that none of the freetards responsible was able to get their mothers to buy them a 30" display. For example, even on Mac OS, the per-screen menu doesn't work very well on a 30" display. The screen's just too damned big.

ChromeOS: the netbook done right
But why would I be running Ubuntu on a netbook? Why wouldn't I be running ChromeOS? The only reason I can think of is if the netbook was my only computer. But that would be pretty stupid for the kind of person who even considers Linux. Sure, I have the Linux kernel on my Android phone, my Android tablet, my ChromeOS netbook (sorry, "Chromebook"), and my big-ass make -j16 desktop. But there's only one of those devices I'd consider using a Linux distro or desktop on, and honestly that's only for lack of an alternative.

I was hugely skeptical of ChromeOS until I acquired a Cr-48 and started using it. It's replaced my MacBook Pro at work. It hasn't replaced any of my Android devices, nor my work or home desktops, but that's fine and hardly unexpected. An Android-based netbook might be an interesting idea, but it would represent a different trade-off. For example, ChromeOS' multi-account model is its multi-user model. Pro: you can safely let friends or strangers log in to your Chromebook. Con: if you personally have multiple accounts (one for work, one for talking to the wife, and one for talking to the mistress, say), it's awkward to switch between them because you have to actively log back in. Android doesn't have a multi-user model, but supports multiple accounts being logged in simultaneously. Pro: you don't have to log in and out. Con: you can't log in and out, so an Android device is something you no more want to hand out than you would your wallet.

This whole Ubuntu netbook mania just seems like a way to screw your real users with no realistic hope of gaining new users. Not happy ones, anyway. Sadly, it looks like we're going to have this stuff forced down our throats whether we like it or not; GNOME Shell looks to be pretty much the same.

A work-around
As a work-around until you install something less lossy, here's how to go back to the pre-11.04 desktop. Click the "power off" button to get to "System Settings". Why wasn't I able to find that myself? I must be stupid, not trying the "power off" button!

signal(2) versus sigaction(2)

2011-05-04T22:00:00.002Z

Normally, I'm pretty gung-ho about abandoning old API. I don't have head space for every crappy API I've ever come across, so any time there's a chance to clear out useless old junk, I'll take it.

signal(2) and sigaction(2) have been an interesting exception. I've been using the former since the 1980s, and I've been hearing that it's not portable and that I should be using the latter since the 1990s, but it was just the other day, in 2010, that I first had an actual problem. (I also knew that sigaction(2) was more powerful than signal(2), but had never needed the extra power before.) If you've also been in the "there's nothing wrong with signal(2)" camp, here's my story...

The problem

I have a bunch of pthreads, some of which are blocked on network I/O. I want to wake those threads forcibly so I can give them something else to do. I want to do this by signalling them. Their system calls will fail with EINTR, my threads will notice this, check whether this was from "natural causes" or because I'm trying to wake them, and do the right thing. So that the signal I send doesn't kill them, I call signal(2) to set a dummy signal handler. (This is distinct from SIG_IGN: I want my userspace code to ignore the signal, not for the kernel to never send it. I might not have any work to do in the signal handler, but I do want the side-effect of being signalled.)

So imagine my surprise when I don't see EINTR. I check, and the signals are definitely getting sent, but my system calls aren't getting interrupted. I read the Linux signal(2) man page and notice the harsh but vague:

The only portable use of signal() is to set a signal's disposition to
SIG_DFL or SIG_IGN. The semantics when using signal() to establish a
signal handler vary across systems (and POSIX.1 explicitly permits this
variation); do not use it for this purpose.

POSIX.1 solved the portability mess by specifying sigaction(2), which
provides explicit control of the semantics when a signal handler is
invoked; use that interface instead of signal().

It turns out that, on my system, using signal(2) to set a signal handler is equivalent to using the SA_RESTART with sigaction(2). The Open Group documentation for sigaction(2) actually gives an example that's basically the code you'd need to implement signal(2) in terms of sigaction(2).) The SA_RESTART flag basically means you won't see EINTR "unless otherwise specified". (For a probably untrue and outdated list of exceptions on Linux, see "man 7 signal". The rule of thumb would appear to be "anything with a timeout fails with EINTR regardless of SA_RESTART", presumably because any moral equivalent of TEMP_FAILURE_RETRY is likely to lead to trouble in conjunction with any syscall that has a timeout parameter.)

Anyway, switching to sigaction(2) and not using the SA_RESTART flag fixed my problem, and I'll endeavor to use sigaction(2) in future.

Assuming I can't stay the hell away from signals, that is.

Alternative solutions

At this point, you might be thinking I'm some kind of pervert, throwing signals about like that. But here's what's nice about my solution: I use a doubly-linked list of pthreads blocked on network I/O, and the linked list goes through stack-allocated objects, so I've got no allocation/deallocation, and O(1) insert and remove overhead on each blocking I/O call. A close is O(n) in the number of threads currently blocked, but in my system n is currently very small anyway. Often zero. (There's also a global lock to be acquired and released for each of these three operations, of course.) So apart from future scalability worries, that's not a bad solution.

One alternative would be to dump Linux for that alt chick BSD. The internets tell me that at least some BSDs bend over backwards to do the right thing: close a socket in one thread and blocked I/O on that socket fails, courtesy of a helpful kernel. (Allegedly. I haven't seen BSD since I got a job.) Given Linux's passive-aggressive attitude to userspace, it shouldn't come as a surprise that Linux doesn't consider this to be its problem, but changing kernel is probably not an option for most people.

Another alternative would be to use shutdown(2) before close(2), but that has slightly different semantics regarding SO_LINGER, and can be difficult to distinguish from a remote close.

Another alternative would be to use select(2) to avoid actually blocking. You may, like me, have been laboring under the misapprehension that the third fd set, the one for "exceptional conditions", is for reporting exactly this kind of thing. It isn't. (It's actually for OOB data or reporting the failure of a non-blocking connect.) So you either need to use a timeout so that you're actually polling, checking whether you should give up between each select(2), or you need to have another fd to select on, which your close operation can write to. This costs you up to an fd per thread (I'm assuming you try to reuse them, rather than opening and closing them for each operation), plus at least all the bookkeeping from the signal-based solution, plus it doubles the number of system calls you make (not including the pipe management, or writing to the pipe/pipes when closing an fd). I've seen others go this route, but I'd try incest and morris dancing first.

I actually wrote the first draft of this post last August, and the SA_RESTART solution's been shipping for a while. Still, if you have a better solution, I'd love to hear it.

Java benchmarks

2010-09-14T05:14:00.000Z

Performance is half my job. The other half is correctness. Both are important, but it's performance that tends to grab the headlines.

The main tool I use for both is Vogar. Think of it as the "run stuff" side of an IDE, but for the command line: you can point it at a Java source file with a main method, at a particular class in a JAR file, at a source file containing JUnit tests, at a source file containing Caliper benchmarks, or a bunch of other stuff... Vogar will just do the right thing. By default, since it was written by Android developers for our own use, code will run on your attached Android device, but you can also use --mode jvm to run on a desktop JVM, or --mode host to run on a desktop dalvikvm (assuming you've built one). (We might want to make that default sensitive to whether you have the Android SDK on your path; the current default makes Vogar seem more Android-specific than it actually is.)

At its simplest, then, it's a replacement for javac test.java && java test that also works on Android (doing all the dexing and pushing and adb shelling behind the scenes). But it goes way beyond that, and all without requiring you to think. You just give it the path to the source, and it works out what to do.

Caliper is the best way to write Java benchmarks. There are examples in the Caliper project itself, but what I want to mention here is the collection of benchmarks in the code.google.com Dalvik project. That's kind of a misleading name for the project; the Dalvik VM doesn't live there. (If that's what you're looking for, the Dalvik VM lives here instead.) The code.google.com project is a dumping ground for other source from the Dalvik and core libraries team. In particular, that's where our benchmarks live.

As you may have noticed, we rewrote the Designing for Performance documentation for Froyo. Previously it was a bunch of stuff that may have been true at some point, but had long ceased to bear any relationship to reality. In Froyo, every single claim in the document is backed by a benchmark to prove (or, in future, disprove) it. You can peruse the "Designing For Performance" benchmarks in your browser.

When we're not writing documentation, we also do a lot of performance work. To guide our efforts, and to be able to recognize whether we're making forward progress, we write Caliper benchmarks. You can see the currently checked-in regression benchmarks.

These aren't always completely up-to-date. I've spent the last couple of weeks rewriting our nio implementation, for example, and my benchmark is huge. But until Sunday it wasn't checked in. That said, because the code.google.com dalvik project is on no particular schedule, you'll likely have access to the any given benchmark before you have access to the improvements it inspired/helped us develop.

I think this collection of benchmarks is interesting for anyone who's interested in Caliper (and anyone writing Java benchmarks – for any platform, not just Android – should be using Caliper), and interesting for Android developers to see what benchmarks we're using to improve the core libraries.

If you want to file a particularly convincing performance bug against the Android core libraries, you can't do much better than providing us with a Caliper benchmark we can point Vogar at.

I admit: not all comments are crap

2010-07-03T18:42:00.003Z

This very week in meatspace, I claimed that adding a commenting facility to a website is like handing out marker pens to everyone who enters a public toilet. The results are obvious, and with very rare exceptions, don't really improve anything for anyone.

Here's a rare exception. Ross Anderson (who I've mentioned before, and whose book should be required reading) used comments on his group's blog to liveblog a security conference. There's more quality in these eight comments than all the other comments I've read this year: Ross Anderson's Security and Human Behavior 2010 liveblog.

Of course, since it took me days to find the time to read these, I question the value of a liveblog. And using the comments rather than updating the post means that anyone using Google Reader (or similar tools) to read the blog won't necessarily have understood that the the trivial snippet they saw was merely the gravestone for the body that lay buried beneath. So although I still don't think that comments are the right way to do things, but if this is how he chooses to present the information and my choice is take it or leave it, I look forward to 2011's comments.

(Although I value this kind of work -- Cliff Click provides a similar service for VM-related conferences -- I'd kill to have links to PDFs of the corresponding papers, where they exist.)

Keep Enjoying Great Gold Benefits

2010-04-18T01:56:00.002Z

Judging by the way my Xbox keeps hinting I might want to renew my Xbox Live Gold subscription (USD50/year), I'm guessing my Xbox Live Gold subscription is soon to run out. To help persuade me, there's a sort of "click here for an ad" thing in the Xbox's UI, and when I click (or, in this case, hit OK on the remote), I get, under the title "Keep Enjoying Great Gold Benefits" this list:

Online multiplayer
facebook
twitter
NETFLIX
last.fm

You'll notice something about those benefits. It's related to the fact that most of them are in all-lowercase, a branding decision doubtless chosen based on the fact that they are, in fact, web sites. Even NETFLIX is a web site. Here's the simplified version:

Online multiplayer
We won't break the internet if you pay us

Except even that's not quite true, because even if you pay them, they'll still only let you use less than a handful of sites; the ones they deem appropriate. Four, to be precise.

If the question were only "is the Xbox UI sufficiently better than the alternatives available to you that you're prepared to pay anew each year for it?", these "benefits" might still be convincing. But the voice in the back of my head keeps telling me the question is really "do you want to support these business practices, this walled-off worse-than-Apple internet?". And the answer to that one is very different.

(Though I despise the PS3's UI almost as much as it clearly despises me, the equivalent services on the PS3 are all free. In both cases, you need to provide your own internet connection, of course.)

A system software update is required

2010-03-20T04:26:00.003Z

I turned the PS3 on this evening, for the first time in months. I wanted to try the Heavy Rain demo. My PS3 didn't want to play with me:


A system software update is required.

Go to [Settings] > [System Update] and perform the update.

                         o Back

Every other system I use tries to sugar-coat the bitter pill when it has an update for me. Whenever they have to come to me with news of an update, they directly offer to perform the update for me. The PS3 once again shows the care and attention that's gone into polishing its interface to a perfect shine of "fuck you".

If the PS3's ridiculously over-endowed multi-dimensional menu weren't bad enough, the bastard won't even traverse it for me, though it trivially could. (Ignoring the relative cost of translator time versus software engineer time, how much longer do you think it took to write and localize that insulting "dialog" than it would have taken to just do the right bloody thing?)

I won't tell you how many clicks the actual update took, nor the number of lines in the EULA I was made to scroll through yet again, nor that the default option was "shut down system after update"; you wouldn't believe me. Unless you own a PS3, in which case you already know of its curious lurking hostility.

And it's still not setting the fucking time off the fucking internet, so it's a fucking hour off now. How are these clowns still in business?

The only enjoyable part of the whole interaction was the almost koan-like nature of its initial pronouncement, before it even knew it was going to get the chance to mess with me: "An Ethernet cable is not connected".

A video game console is not played.

A 360 is returned to.

Review: Uncharted (PS3)

2009-12-17T05:02:00.002Z

I didn't get "Uncharted" from playing just the first half hour, it took me a while longer. In fact, I didn't really get it until I had to shoot a barrel on the back of truck, causing both the barrel and the truck to explode, at which point the truck somersaulted toward me, falling neatly into a gap between two stone pillars and allowing me to safely pass from one pillar to another.

In retrospect I should have noticed earlier, but it wasn't until that point that it finally clicked: "I'm playing an action movie!".

With that vital clue, I was finally able to enjoy the game. No longer was I playing some bizarre mish-mash of lame shooter with lame platformer with lame puzzle game with lame driving game... I was playing something that made sense, in it's own nonsensical way. If Halo famously gives you the same 30 seconds of fun, over and over, Uncharted attempts to keep your attention by constantly switching from one thing to another.

And it works equally well. Uncharted is the best action movie ever. It's every bit as stupid as other members of the genre, but it's a lot more fun because you're taking part yourself.

Whatever kind of game you do or don't like, there's some of it in Uncharted. But it chops and changes so fast that nothing outstays its welcome. Take survival horror, for example. I don't like survival horror. Either I don't care, in which case what's the point, or I do care, and this thing I'm doing to relax is actually going out of its way to cause me stress, in which case... well, that's weird too. Halo (which isn't just the same 30s over and over) has the Flood to provide its survival horror interludes, but for my liking those sections were too long. If I had a 14 year old neighbor, I'd definitely have paid them to finish the sphincter level in Halo 3 for me. Not my cup of tea. But the equivalent in Uncharted? Not a problem, because it's over so quickly and you're on to something else before you know.

Speedboats or jetskis or something.

The end was appropriately ridiculous too. In the cinema, it would have made me reflect that I'd just wasted two hours and ten dollars on mindless pap. In my living room, after probably more like eight hours and $25, it just made me laugh, because it was hard not to see it as some sort of comment on the action movie genre.

Can you imagine coming out of an eight-hour action movie thinking "yeah, I'll definitely give the sequel a go"? I can, because I did.

It's not worth buying a PS3 for, but if you have a PS3, it's worth buying.

Review: PS3

2009-11-02T02:41:00.003Z

The day Netflix and Sony announced Netflix streaming videos were coming to the PS3, my 360 died. Not metaphorically; it hung while playing a DVD, and power-cycling got me the red ring of death I thought I'd escaped.

So, after requesting my free repair, in a small act of consumer retaliation, I bought a PS3. I also bought a couple of games: "Prince of Persia", which I'd been considering getting for the 360, and "Uncharted", one of the few PS3 exclusives I could muster any enthusiasm for. (The 360's exclusives are much more my taste; it's no coincidence I bought a 360 shortly after "Halo 3" came out.)

Unboxing
The PS3 slim is much nicer-looking than its "fat" predecessors. It beats the 360 on looks too, mainly because it's just a small black box that you don't really notice. The 360 is a large white box that's pretty hard to ignore. It's slot-loading rather than tray-loading, which I don't really like, but having suffered a flaky tray loader on my original Xbox (Microsoft's hardware screw-up for that generation), maybe it's a blessing. The power and eject buttons are fine; I mention this because the power button on the 360 is mushy and unconvincing, and coupled with the console's slow reaction to a press, happens to be one of the 360's minor annoyances.

The box contains a composite video cable, which is tres fucking lame in 2009. The back of the console does have completely standard HDMI video and TOSLINK audio ports, though, which is a step up from the 360's proprietary crap. There's an ethernet port too; no cable in the box, but the system (unlike the 360) has built in WiFi, so it's perhaps not unreasonable to assume that the kind of old-timers like me who're going to connect a cable already have a house full of the stuff.

The controller connects via USB, but it's wireless, so that's mainly useful for charging. Personally, I'd rather have the 360's non-rechargeable controllers and pay $5/year for a bulk pack of batteries.

There's no power brick. Unlike the 360 and original PS3, both of which really guzzled power, the PS3 slim stays under 100W.

Installation
PS3 installation is the worst consumer electronics installation experience I've had in years. Sony's on-screen keyboard is bad enough (much more fiddly than Microsoft's), but having to use it the first time I turn the device on is worse. (I gave in and connected one of the USB keyboards I had lying around, but the fact I can do that in no way makes up for the fact I felt compelled to.)

If you think the days of being asked to set the date and time were long gone, think again. The PS3 can set the time from the internet, but it won't by default, and even when it does it won't get DST right ("do it manually").

If you think seeing your ethernet interface's MTU is an integral part of setting up any game console, you'll absolutely love the PS3. If you're a general consumer, you'll probably just keep hitting "OK" until it stops pestering you with trivia it should just sort out for itself. If you're in-between, the installer is like something from a UI hall of shame. (Including Sony's own cover of the ever-popular "this setting will take effect after the next reboot".)

Reboot
The console reboots a lot, actually. Or seems to. If you play a DVD, there's a visible "reboot" you don't get with a 360. Luckily, the PS3 does whatever it's doing quicker than the 360 reboots. The orchestra-tuning-up sound that accompanies the "full" reboots is very classy, much nicer than the 360's beeps. (I'd recognize the 360 start-up sound if you played it, but I couldn't sing it. It's totally non-memorable.)

UI
Anyway, when you're finally through the installation torture, you're looking at an Ubuntu-like turd-brown screen. I shit you not. Sure, there are some bits of ash or something drifting across the screen and there's some kind of horizontal brown flow that gives the overall air of a shampoo commercial's "pseudo-science bit". The UI is very much like an uglier version of the PSP interface (which is purple). Where that interface works well on a small screen, I'm unconvinced it scales. Just like the Mac's screen menu bar being genius on small laptop screens but awkwardly distant on a dual-30" setup, Sony's UI works well on a small display where you can take everything in at a glance, but is much less convincing on a huge modern TV, where you feel like you're constantly having to move your eyes.

Poor "dialog" design exacerbates this. Coming from the original Xbox to the 360, I thought this was something that had been solved by the current generation of consoles, but it seems not. The 360 associates actions with buttons, and since each relevant button has a distinct color, the mapping from action to button is easily and clearly signaled by use of a little colored blob representing the button. (That's what it looks like on an SD screen. On an HD screen, you can clearly see the letter, but that's superfluous for anyone with normal color vision.)

Sony's buttons each have a different color, but they seem to be paired: the two horizontal buttons (light pink and dark pink) and the two vertical buttons (pastel blue and pastel green) are quite similar to one another, and it's only the outline of the symbol (circle, square, triangle, and cross) that's colored, rather than the entire button. They're not even filled symbols.

The most distracting part for an outsider, though, is that each dialog's explanation of the buttons is shown in the bottom corners of the (huge!) screen, miles away from where you want to focus your attention. Presumably the habitual PS3 user's response is to tune out this noise, but as a beginner it means I'm slowed down on every decision to confirm that, yes, cross means "OK" and circle means "Cancel". (Though that "OK" might actually mean "Cancel" if you've selected a negative option first, for example in a "quit?" dialog with "yes" and "no" selected by position and "OK" and "Cancel" selected by choice of button.)

The Sony store is a lot less polished than Microsoft's. The front page of PlayStation Home has the overall aesthetic of a C64 bulletin board: a cluttered mess in 8-bit color (the 8-bit blues and yellows really do it).

Why do I mention the store? Readers with long memories will remember that one thing I don't like about the 360 is that it acts like a dipshit compared to the original Xbox. Put in a DVD and the Xbox would play it. Put in a game and the Xbox would play it. The 360 takes you to a screen where you have to press "green" ("A", for black and white viewers, "OK" for users with remotes) to actually play the DVD or game.

By default, the PS3 assumes you want PlayStation Home. That is, you want to see scrolling ads for games you don't have. Great.

Pop in a DVD, and the PS3 will start playing it. Just what I wanted, right? Well, no. Not what I've (for my sins) just popped in a "Star Trek: DS9" or "Battlestar Galactica" DVD, with multiple episodes. If you watch one episode per night, it's annoying having the console start playing the first episode on the disc every night. It wouldn't kill it to remember where I left off.

I have the 360 remote, so it's not fair to compare that aspect of using the PS3 as a DVD player, but I read that the PS3 remote doesn't have backlit buttons, and eats batteries, which isn't encouraging. (I haven't had to change the batteries in the 360 remote yet, and it does have backlit buttons, which is useful when you're watching DVDs in the dark.)

The stupid behavior with multi-episode discs is probably going to keep the PS3 from becoming my default DVD player.

Actually, there's one positive thing I can say about the PS3 as a DVD player. It's well known that the 360 uses a lot of power, and power means heat, and heat means fan noise. I'd stopped noticing the noise the 360 makes, without even realizing. It took me several minutes to work out what I found unnerving about watching a DVD on the PS3: it's too quiet. I can't hear the console at all. As a long-term 360 user, my subconscious conclusion was "something must be wrong".

Games
I forgot to mention how things play out immediately after the device installation. You've been through all this unnecessary bullshit, and you pop your game disc in and... you're told you need to download an update. Okay, that's not too different from the 360, except the updates seem much bigger. So you have the "downloading" progress bar and then the "installing" progress bar, and then the game starts... and you get another progress bar. Another long one. My assumption, having read reviews back when the PS3 first came out, is that this is something to do with caching stuff onto hard disk. But there's nothing to tell you what's going on, not even "loading...", and it's slow. When it finishes, comically enough, then you get "loading..." and another progress bar.

I'd be curious to know why they didn't just implement a regular cache, or even background prefetching during play, instead of this weird pre-play stop-the-world prefetching. (Start Halo after spending different amounts of time at the loading screen after boot to see masters at work.)

Games look about the same as 360 games. "Prince of Persia" looks exactly like a 360 game, and "Uncharted" looks kind of crappy by 2009's standards, but okay for a 2007 game, which is fair enough, because that's exactly what it is.

Games also have a bizarre habit of starting by telling you that a particular icon means "saving progress" and that you shouldn't turn your console off. The two I have actually wait for you to acknowledge this "helpful" tip, and they're not doing anything useful while they wait. The "loading..." progress bar comes after you acknowledge. (The 360 solves this by using text instead of an icon, and saying something like "Saving progress. Do not turn off console." Sometimes a few words are worth a lot more than an icon no-one recognizes.)

Controller
The controller is too small. It's like a miniature controller for tiny little girl hands. "Uncharted" uses what 360 users would call the bumpers as triggers, perhaps because the PS3 controller doesn't really have triggers. It has buttons where the triggers should be, but they're not long enough, and they curve the wrong way: they're convex rather than concave! That neither of the two games in my extensive survey really use the triggers for anything suggests that everyone knows the controller's triggers are fucked beyond repair. (Sony even calls them L2 and R2, the bumper buttons being L1 and R1.)

I find the analog sticks harder to use too, but I'm not sure if that's lack of practice, or that one of them is in the wrong place (the left analog stick is swapped with the d-pad), or that the controller as a whole is too small, or that the analog sticks aren't as good.

Microsoft (or Sony) could sell me a 360-like controller for the PS3 in a flash. Japanese miniaturization is all well and good, but I need a full-sized controller. With proper bloody triggers.

How can they sell a game like "Rainbow Six" on this platform without including a custom controller? Or is there some secret about plugging in a keyboard and mouse and pretending you're a PC gamer no-one's told me about?

Summary
The PS3 is more invisible than the 360 (both in appearance and noise), games seem about the same (though the 360 has better exclusives), the 360 is a better DVD player and has a better controller.

I was hoping to be more won over by the PS3. I'd have liked to relegate the 360 to "second console", brought out only for future 360 exclusives. (Assuming there are any more interesting ones before it's time for the next generation.) As it is, I don't think the 360 will find it hard to retake its place, though it's weak that my main problems could be fixed in software (UI fiddliness, DVD retardation) and by selling optional better peripherals (controller and remote).

As for the games, "Prince of Persia" is great fun if you liked collecting orbs in "Crackdown", the middle-east theme of "Assassin's Creed", and the improbable momentum-defying acrobatics of both. Don't tell Sgt. Johnson, but I'm actually enjoying spending my time with a frigging fairy princess. "Uncharted", well, it's okay, but there's nothing in the first hour to make me want to play any more until I've finished "Prince of Persia".

Giving up on .dmg

2009-10-04T18:47:00.002Z

Alexander Limi recently wrote Improving the Mac installer for Firefox (and also a part 2), where he basically explains all the ways in which .dmg is an awful choice.

John Gruber followed up with How Should Mac Apps Be Distributed?, in which he praised "the resurgent trend of delivering Mac apps as simple .zip archives".

It's not like .dmg hasn't always been awful; in 2006, Cédric Luthi 's zip vs dmg complained about how slow .dmg is compared to .zip, and though he could have better emphasized that he meant the awkwardness of the entire process rather than just CPU time, it's clear he was really talking about the same problem.

The jessies.org applications have been shipped as .dmg files for years now, and since most of us build from svn and run straight from the file system, we don't often feel the pain of installation. Watching others struggle with .dmg, though, is a pretty convincing argument that this madness be stopped, Apple's recommendations be damned.

It's worth noting that Apple doesn't have to suffer its own recommendation, because Apple's stuff either uses .pkg or is installed with the OS [using .pkg behind the scenes] and updated by Software Update.

We've also had trouble since 10.5 with occasional bogus "the disk image you are opening may be damaged" messages. These are repeatable on any given machine, but the MD5 of the downloaded bits is exactly the same as on another machine where everything works fine. And later builds seem to work on machines that have had problems previously, and new machines that used to be fine start having trouble. This hasn't been fixed by 10.6. It doesn't seem to matter whether the .dmg is generated on 10.4, 10.5, or 10.6. (As far as I know, we've never seen the warning on a 10.4 machine, so whatever the bug introduced in 10.5 is, it wasn't included in 10.4 security updates.)

Anyway, so many people finally standing up and agreeing that the emperor has no clothes has given me the courage to revisit what I said in 2007's Choosing the best disk image format on Mac OS, where .zip was the clear winner even if it wasn't taken seriously as a contestant, and switch us over to .zip.

Review: "JavaScript: The Good Parts"

2009-09-21T22:33:00.000Z

JavaScript has always been a bit of a mystery to me. Every time I've had any contact with it, I've been left wondering why anyone in their right mind would have anything to do with it. Let alone what kind of criminal mind would have originally conceived of such an abortion. I've never found (or heard tell of) anything good about it, other than its ubiquity. The fact that JavaScript is built in to every browser, but no alternative is. As far as I can tell, it's this ubiquity that helps JavaScript cling to life like a daglock, despite being little more than hair matted with dried shite.

I've always been curious, though, if only thanks to a weak belief in the "no-one's useless; they can always serve as a bad example" principle. So when I stumbled across a book with the title "JavaScript: The Good Parts", I thought I'd take a look.

It is, as you'd imagine, a very slim volume. It's about 140 pages, but even that exaggerates the amount of content. There's plenty of whitespace in the book (a good thing) and many things are taken too slowly (a bad thing, because many tricky things are taken too fast).

It starts off quite amusingly, but as early as page 3 I was starting to worry about the author. The verse numbering here is my own. In the original, this is one paragraph with no breaks:

1. The fashion in most programming languages today demands strong typing.

2. The theory is that strong typing allows a compiler to detect a large class of errors at compile time. The sooner we can detect and repair errors, the less they cost us.

3. JavaScript is a loosely typed language, so JavaScript compilers are unable to detect type errors. This can be alarming to people who are coming to JavaScript from strongly typed languages.

4. But it turns out that strong typing does not eliminate the need for careful testing.

5. And I have found in my work that the sorts of errors that strong type checking finds are not the errors I worry about.

6. On the other hand, I find loose type checking to be liberating. I don't need to form complex class hierarchies.

7. And I never have to cast or wrestle with the type system to get the behavior that I want.

You'll notice he starts off in verse 1 with a unsubstantiated pejorative that "fashion demands" strong typing. Not that programmers find it useful. No, "fashion" "demands" it.

He mentions the "theory" of "strong typing" like a creationist might mention the "theory" of "evolution", without bothering to point out which of the logical steps in his presentation of the "theory" he considers to be incorrect.

I assume it's the devil who puts errors in our code.

Actually, while we're going with this analogy, let's throw in the old "missing link" fallacy: verse 4 helpfully points out that stronger typing can't catch all errors. As if anyone ever claimed it would.

We're given no details of what proportion of errors are type errors, or whether it's more efficient (by any metric) to catch those errors via type signatures/annotations than via manually writing unit tests (which, of course, also contain errors).

In verse 5, we have a nice example of "argument by anecdote" made especially worthless by the rather vague term "errors I worry about". Note that he doesn't say "errors I make" or "errors commonly made by a cross-section of programmers". He says "errors I worry about". Does this mean he's happy to ignore type errors? Or that JavaScript gives him so many opportunities for new classes of error (like scope errors and binding errors and whitespace errors and RTTI errors and floating point arithmetic errors and equality errors and so on) that he just doesn't have time to worry about old-fashioned stuff like type errors that people writing code in sane languages have the luxury of concerning themselves with? Who knows?

Verse 6, without examples, would appear to be an admission that the author isn't any good at class-based OO programming. I'm not sure what the relevance is, other than a warning to the reader.

I realize this dissection of a single paragraph is of only tangential relevance to the book review this is supposed to be. But I'm sick of hearing this same old crap time and time again. You want to fight? Okay, let's fight. But let's fight with science, like grown ups. Fighting with religion like mystics is demeaning to us both, and fundamentally pointless. Show me the data, or shut the fuck up.

I left verse 7 in to show that I'd like to be above this kind of name-calling. He has a point, that most type systems aren't as expressive as we'd like at times. But there are notable exceptions, and whenever I see a "loose typist" say things like this, I think "there goes someone who had the misfortune never to have used Haskell or SML or whatever". Sure, all popular languages' type systems have their weaknesses, but I don't understand why anyone would assume that the solution is to throw the whole thing out. It's not like we have proof that the problem can't be satisfactorily solved. Or that, returning to the horse you thought I'd already flogged quite to death, that half an eye isn't better than no eye. Quite the opposite.

And it's not like anyone bothers to back up their anti-type system prejudice with any research. Do we stop testing because testing can't catch all errors, or do we try harder, and use other things to help us catch errors too?

The onus is on the "loose typists" to demonstrate that they're not crazy, which is exactly why they always resort to this kind of sophistry. It's so much easier.

(As a final aside before I return to the book, many of the best programmers I know understand programs via the types. This too is often overlooked by "loose typists", and sometimes leads me to wonder if there's a deeper difference in the way the two groups see, understand, and reason about the world. In which case demonstrating, say, a software-quality superiority of one style over the other would solve nothing.)

Anyway, ignoring the nonsense – bad sign though it is, when setting out one's stall – and turning the page, that's chapter 1 ("Good Parts") done with. On to "Grammar", a chapter which takes 15 pages, mostly consumed by "railroad diagrams" of the kind used in Pascal textbooks from the 1980s, bizarrely enough. Sadly, these are even harder to read than the traditional kind because there are no arrowheads on the arcs. You're faced with directed graphs where you have to work out the directionality yourself.

How very fitting.

I don't understand why some variant of BNF wasn't used. This book is aimed at people who can already program, so it wouldn't seem unreasonable to assume they can already read a variant of BNF, or learn one in a couple of paragraphs. (If you make it to the end, pages 125-135 repeat the diagrams from this chapter, to no particular advantage.) But don't skip this chapter: in between the diagrams there are many important gotchas revealed. If you noticed me talk about "whitespace errors" earlier, for example, read the section on "return" and weep. (Though note that the text here is misleading. There's a clearer and more accurate explanation later in the book. So on second thoughts, maybe you should skip this chapter after all.)

Skipping ahead to page 36, we learn that the one thing JavaScript appears superficially to have got right, C-like syntax, is also fundamentally broken. Those curly-brace delimited blocks don't introduce a new scope. Strangely, the chapter spends more time on currying and memoization than it does on this, which is one of the things so broken about JavaScript that I'd heard of it even before reading the book. Currying and memoization are presented seemingly for their coolness rather than their usefulness to JavaScript programmers. If you want to be a functional programmer, I can't think of a worse language than JavaScript. And if you must run in a browser, real functional programmers are smart enough to write SML-to-JavaScript compilers in Haskell anyway. (Given a decade and a handful of PhD students.)

Chapter 5 talks about how JavaScript's prototype-based system of inheritance is "more expressive" than the usual class-based systems. Without, of course, dwelling on why class-based systems are more common. Or why previous prototype-based systems (NewtonScript, say, or Self) never went anywhere. No. What we have instead is a section that explains how to imitate class-based inheritance (but pointing out that it doesn't really work out thanks to some bad decisions in JavaScript's design), a so-called "functional" scheme whose purpose is seemingly to paper over the lack of privacy in JavaScript, and a presentation of mixins by a JavaScript-specific name ("parts").

The "Arrays" chapter seems to claim that arrays are really just maps, and hence slower than you'd imagine an array to be, but surely that's a quality of implementation detail? Surely an implementation is free to special-case the (presumably common) case where the keys are consecutive (or at least dense) integers? The "delete" operator is weird (I don't like Python's either), while the "splice" method just seems badly-named. The "splice" method also seems like a good example of how loose typing seduces people into making one method do too much. For an array "for in" is even more screwed up than we've already been shown it is for objects. Strangely, JavaScript seems to have no range type or literals, so you're reduced to:


  var i;
  var v;
  for (i = 0; i < array.length; ++i) {
    v = array[i];
    // ...
  }

Going back to the lack of block scope, you could move the "var v" inside the "for" loop, but it would mean the same as the code shown here, and thus be more confusing.

(You'll note I'm left speculating about a lot of things. The author doesn't really go into implementation issues, and when he does, he doesn't actually come across as someone intimately familiar with the implementations anyway.)

Page 61 contains the kind of code that makes you not want to work with its author. It shows you how to write a home-made heuristic test for whether a given object is really an array. Why, you might wonder, would someone who doesn't go in for type systems care? So they can "write functions that do one thing when passed a single value and lots of things when passed an array of values". Except these functions won't be 100% sure they've really got an array. Cunning plan, Baldrick.

There follows more wishful thinking along functional programming lines, leaving me honestly curious to know how much this crops up in the stuff JavaScript's actually used for. Or maybe JavaScript programmers spend most of their time fantasizing that they're actually programming in some other language. A language that doesn't suck.

Odd though it may seem to have a single 12-page chapter on regular expressions that doesn't really deal with language-specific quirks, there's one in this book. I doubt that anyone who didn't understand regular expressions going in could gain much from the presentation, and given there was nothing of value for people who already know regular expressions (beyond the fact of the Ruby-like slash-delimited syntax and the exact details of the three possible trailing flags), I'm not sure what purpose this chapter serves.

Chapter 8 is a nice brief summary of the built-in types' methods, though much space is wasted on implementing functions in terms of each other. Here's most of the author's explanation of push, for example:


  Array.method('push', function () {
    this.splice.apply(
        this,
        [this.length, 0].
            concat(Array.prototype.slice.apply(arguments)));
    return this.length;
  });

I'm sure that's really helpful.

Some of the text in this chapter (which is otherwise pretty clear and to the point) is annoyingly vague, too. We're told, for example, that "shift is usually much slower than pop", but what does that mean? That some implementations back arrays with vectors but other ones use dequeues? That bad implementations use arrays and good ones dequeues? That standards-conforming ones use arrays and non-conforming ones use dequeues? Something else entirely? (You'll recall that we were told in the beginning of chapter 6 that JavaScript arrays are more like hashes, though there too we were left on our own as to what that really means.)

I mentioned earlier that JavaScript doesn't have a range type, which means that counted loops end up written out in longhand. In this book, they're even longer than usual, looking like this:


  for (i = 0; i < 4; i += 1) {
    ...
  }

Not because JavaScript doesn't have operator++. It does. But Crockford doesn't like operator++: they "have been known to contribute to bad code by encouraging excessive trickiness" and "are second only to faulty architecture in enabling viruses and other security menaces" (page 122). Is this true of JavaScript? We're given no reason to believe it is. There's a longer version of this same madness on page 112:

The increment and decrement operators make it possible to write in an extremely terse style. In languages such as C, they made it possible to write one-liners that could do string copies:
  for (p = src, q = dst; !*p; p++, q++) *q = *p;
They also encourage a programming style that, as it turns out, is reckless. Most of the buffer overrun bugs that created terrible security vulnerabilities were due to code like this.

In my own practice, I observed that when I used ++ and --, my code tended to be too tight, too tricky, too cryptic. So, as a matter of discipline, I don't use them any more. I think that as a result, my coding style has become cleaner.

Allow that to sink in for a moment.

Nice way to destroy your own credibility, dude. I mean, there may be some terrible problem with JavaScript's ++ and --, but if there is, he's completely failed to tell us what it is. He's asserted that it's a bad thing in C, but this isn't a book about C. This is a book about JavaScript.

And, since I'm already shouting and red in the face, I'll add that what he says isn't true of C, either. It's not the increment of an index or pointer that's the problem: it's the lack of a bounds check before using the index or pointer. If you're not sure who to believe, Mr Published Expert or me, Random Internet Guy, ask yourself this simple question: would an incorrect C program with a buffer overrun be fixed by switching from "++" to "+= 1"? Would the buffer overrun have been less likely in any way, shape, or form if the programmer had used "+= 1" instead of "++"? Of course not.

By this kind of bogus "logic", I shouldn't use the word "gift" in English, because "Gift" means "poison" in German, and, as you know, most of the poisonings that cause terrible illnesses in people are due to poison.

I haven't been subjected to such utter gobshite in a computer book since I gave up on Herbert Schildt.

If you've got a stomach for tripe, and you still think you might like this book, I'd recommend a glance at appendix E, which contains an implementation, presumably Crockford's, of a JSON parser. There's a "break" indented 9 levels and inside a loop made more readable by saying "i += 1" instead of "++i", of course; along with an assignment and a "delete" statement in another loop, both indented another 9 levels. I'll grant you it's a crude measure, but as far as style goes, deepest level of indentation is a pretty reliable indicator of a programmer's taste, and this man has absolutely none.

In fact, if you're seriously considering wasting money on this book, I strongly recommend you look at the last page of content (page 145), and the function on it. If you think the opinions of the man who wrote that code in a book primarily about style are worth your time, you need more help than any book can offer. There may be parts of the function that are, thanks to JavaScript's awfulness, unavoidably bad. But there are parts that could be trivially improved. (See the big comment for one hint; never say in comments what you can say in code!) And there are other seemingly obvious improvements that if they're not valid, it would really be worth explaining why not. For example: the author repeatedly tells us that JavaScript's for-each is broken and you need to make a hasOwnProperty test too, and the author has a huge hard-on for functional programming in all kinds of places where it doesn't help, but he won't write a higher-order forEach function?

Bah.

I still think the idea behind "JavaScript: The Good Parts" was a good one, but the implementation was flawed, and the author was perhaps not a man suited to the task. The language itself may or may not actually have enough good parts to leave a working language behind if you really did avoid the bad parts, but Douglas Crockford is not a man of sufficient perspicacity for us to find out.

If this book had been a blog post somewhere, I could have happily passed over it with little more than a "meh". But the thought that those bastards over at O'Reilly killed trees for this, and unknowing punters like you or me give them money for it... Unrelated to this, someone at work the other day claimed that O'Reilly hadn't printed a decent book in the last ten years, and there was general agreement. I asked that "Java Generics and Collections" be considered the sole exception, but it's bumf like this that's ruined/ruining O'Reilly's reputation.

So what did I learn about JavaScript? That I'm never going any closer to it than the sensible end of a AnythingButJavaScript-to-JavaScript compiler. And I shall do my best to avoid even that.

Farewell to Java 5?

2009-09-06T21:52:00.000Z

You've probably read that Mac OS 10.6 has finally made Java 6 the default, and you may also have read that there are both 32-bit and 64-bit JVMs. (That latter part will only seem surprising to you if you'd been using Java 6 on 10.5.)

What I didn't know until I read Things Removed in Snow Leopard was that Java 1.4 and Java 5 have actually been removed from 10.5. (Cocoa-Java has been removed too, but we were told to stop using that years ago, and we know that what Apple deprecates in one version, Apple tends to remove in the next.)

If you've been paying attention to Ubuntu 9.10 – which I hadn't, but someone else pointed this out – a quick sun-java5-jre package search shows that while Ubuntu 9.04 ("jaunty") offered the sun-java5-jre package, 9.10 ("karmic") doesn't. Both have Java 6 though: compare the sun-java6-jre package search.

In the Ubuntu case, you'll still be able to install a Java 5 JDK/JRE downloaded direct from Sun, but in terms of ease (and likely default), it looks like we're finally entering a Java 6 world.

The people least likely to be happy about this are those running Java applications on a Unix box, displaying on a Cygwin X11 server. Sun's Java 6 and Cygwin's X11 server really don't get on well.

Mac OS 10.4 users might not be too happy as apps start requiring Java 6 since for them that means an OS upgrade: there are no practical third-party JVMs they can use. And for some, an OS upgrade will entail a hardware upgrade because 10.6 is the first version that doesn't support PowerPC. But now these users are two major OS releases behind, Apple's "deprecate one, remove one" tempo means lots of stuff's going to start breaking for them anyway.

I'll be curious to see what happens to the Omni Software Update Statistics over the next couple of months. For 10.5 to have only overtaken 10.4 in 2009-02 is quite shocking for a group so traditionally upgrade-rabid as Mac users. (Though these numbers aren't necessarily representative of all Mac users.)

Java on a thousand cores

2009-08-21T00:05:00.004Z

Cliff Click wrote a recent blog post that started "warning: no technical content", which wasn't true. Buried 3/4 of the way through was a link to the PDF slides for the "Java on 1000 cores" talk he's been giving recently. I was in the audience of the specific instance of the talk he mentioned, but if you weren't, you can now watch the video on YouTube.

I tend to avoid talks because so many of them are a complete waste of time. This one, though, was so good it had me thinking about why I've mostly given up on talks.

For one thing, most speakers speak far too slowly. I know one of the rules everyone's taught about public speaking is to speak more slowly. That because you're nervous you'll speak faster than you realize, and that sounds like you're rushing, and that's bad. Which might have made sense in the 1700s when public speaking likely meant boring your congregation to sleep every Sunday morning, where the point was less the content than the submission to authority. Going slow might make sense if you're trying to educate beginners (though there I think repetition and usage is what matters, not slow presentation). I can even admit that going slow is downright necessary if you're walking someone through a physical procedure, but it's an awful way to present summaries or reviews (in the sense of "review paper"). And most talks are, one way or another, summaries or reviews.

As if looking for a less literal way in which they can speak too slowly, few speakers have a good feel for how many words any particular point deserves. You know these people. You've been in their talks. These people have their four obvious bullet points on their slide, four bullet points that you grasp in the second or two after the slide is up, and somehow they manage to spend five minutes laboriously wading their way through these points while adding nothing to your understanding. And all the time, you stare at the screen willing the next slide to appear. Wishing you'd brought your Apple remote. (They're not paired by default, you know.)

Don't be afraid to have a lot of stuff on your slides, either. The people who tell you not to have more than a couple of points per slide have nothing to say. At best, they're salesmen. (See Steve Jobs' slides and presentations. Things of beauty both, but designed for selling, not transfer of technical information and ideas.) The naysayers also assume you're making all the usual mistakes too, when they should instead tell you to speak fast and only speak useful words, in which case it won't matter how much stuff is on each slide, because no-one will be bored enough that they're sat there just reading the slides. If they're not listening, taking away the slides is not a fix for your problem.

Another bad habit is not asking anything of your audience. I don't mean questions. I mean prerequisites. Seriously, those five people at your talk about building STL-like containers who don't know what the STL is? Tell them to fuck off and stop wasting everybody's time. Or, if you think that's a bit much, just start with "this talk assumes a basic familiarity with X, Y, and Z; if you don't have this, I'm afraid you probably won't get much out of this talk". You're not helping anyone by spending ten minutes rambling on about stuff that anyone who's going to get anything out of your talk already knows anyway. Unless you're someone like Steve Jones, you probably don't need to explain science to the layman. It's much more likely you're talking to an audience of your peers, and you should respect their time as you'd expect them to respect yours.

Also, please don't waste my time with talks where the only thing I learn is how much you like the sound of your own voice, or talks that only exist because you get some kind of merit badge for having given a talk. In the former case, get a blog. Then people can decide for themselves whether they like the sound of your voice, and subscribe if they do and ignore you if they don't. In the merit badge case, corporate life is full of bogus achievements; I'm sure you can find one that minimizes waste of other people's time. Hell, if it helps, I'll make you a little "I didn't waste anyone's time giving a gobshite talk" certificate.

Et cetera.

Anyway, getting back to Cliff Click's talk... his was a great example of what talks should be like. His content was interesting and concentrated. He assumed his audience knew the basics of the areas he was talking about. He spoke fast. Fast enough that I was often still digesting his previous point while he was on to the next. (And for all you MBAs: this is a good thing. Better too much content than too little. I can always re-read the slides/re-watch the video afterwards. And sometimes thinking about the last thing I was interested in helps me coast through a bit I'm less interested in.)

One thing I particularly like about Cliff Click's stuff in general is his practitioner's point of view. He asks the important practical questions: "Have you successfully built one?", "Did it do what you expected?", "What is it good for?", "What isn't it good for?", "Is it worth it?".

As for this particular talk, there were several new ideas to me (I hadn't heard of the optimistic escape detection [as opposed to escape analysis] he mentioned in passing, for example), several things I found surprising (that write buffers turned out to be unimportant with Azul's CLZ [unrelated to ARM's CLZ], for example), and a lot of familiar stories about mixed hardware-software designs, interesting mainly because it hadn't occurred to me that they might be common to all hardware-software companies.

Plus it was a model of how to give a good talk.

Removing the title bar from your Android Activity

2009-07-27T05:42:00.000Z

Sometimes you have a good use for the title bar – the gray strip beneath the system status bar – in your Android app. The Gmail app, for example, uses it to tell you which folder you're looking at and how many threads with unread mails are in that folder, for instance: "Inbox (52)".

Other times, you've got nothing worth saying. If you're actually using it as a title bar rather than as a status bar, you're probably wasting precious screen space that could be used for more context or bigger (easier to hit) buttons or something.

If you ask the internets, the usual advice is to call requestWindowFeature in onCreate:

  @Override public void onCreate(Bundle state) {
    ...
    requestWindowFeature(Window.FEATURE_NO_TITLE);

This works, but it's really meant for the exceedingly rare case when you don't know ahead of time whether or not you want a title bar. If you do know, you're better off saying so in your AndroidManifest.xml:

...
  <activity android:name=".MyMainClass"
            android:label="@string/app_name"
            android:theme="@android:style/Theme.NoTitleBar">
...

(You can specify the theme on a per-activity or per-application basis.)

To see the advantage of this, compare what happens when you first start an app that uses each method. In the requestWindowFeature case, the system provides a title bar announcing your app name until the code to turn it off is actually run a few hundred milliseconds later. If you then go back to the home screen and re-enter the app, the title bar is already gone. So not only does this scheme look rather sloppy, it's not even consistent.

In the AndroidManifest.xml case, though, you never get a title bar. Your app looks the same whether it's starting afresh or simply un-pausing, never looks like it's been caught with its pants down, and psychologically (to me at least) feels like it's starting faster. Or at least as if the pause is the system's fault, rather than your app's fault.

(It seems likely to me that part of the reason the worse solution is so popular is that it's mentioned in the developer.android.com FAQ. I've sent a patch.)

Why find(1) doesn't like big directories

2009-07-16T05:14:00.000Z

A friend had trouble recently trying to use find(1) on a directory containing millions of entries. He expected it to start pumping out results straight away, but found instead that it went away for a long time. A quick strace(1) showed it steadily reading directory entries.

We could speculate about what's going on, or we could look at the source. Let's start by finding find(1):

$ which find
/usr/bin/find

Then we ask the Debian packaging system which package installed that file:

$ dpkg -S /usr/bin/find
findutils: /usr/bin/find

And then we ask for the source code for that package:

$ cd /tmp
$ apt-get source findutils

A quick search for "opendir", "readdir", and "closedir" – the three POSIX functions responsible for directory iteration – suggests lib/savedirinfo.c as a likely candidate, and if you look at xsavedir in that file, you'll see that it does indeed iterate over the entire directory and returns an array of all the entries to its caller.

Easily fixed, right? If we're not sorting, there's no reason to return a collection rather than an iterator, and any caller who wants to sort can still implement that on top of the iterator. Conversely, you can't implement a low-memory or low-latency iterator on top of a collection (because you've already paid for all the memory and all the latency while building the collection), nor can you implement a cheap "isEmpty" that reads at most three directory entries (three because you may get "." and "..", neither of which count).

In the single-directory case, an iterator probably is the way to go. But things aren't so simple in general, which is why most modern APIs, though built on the POSIX iterator primitives, offer a collection.

Suppose you're implementing find(1) using iterators. And suppose, for simplicity's sake, you're doing a depth-first traversal. What do you do when you come to a child directory? Well, the important part is that you create a new iterator for that child's entries. And if it has child directories, you'll do the same for them. So what?

Memory isn't the scarce resource here. File descriptors are. Specifically, the one opendir(3) needs for each directory between the directory you started in and the deepest directory. Your new pathological case then is a deep directory tree. How deep? The default per-process limit on my Ubuntu 9.04 box appears to be 1024 file descriptors. And remember that your program is probably already using a bunch of file descriptors, and that most code doesn't cope well with running out of them.

You can increase this limit (the separate per-system limit is sufficiently high that you probably don't need to worry about it), but then all programs using the iterator-based code to walk a file tree needs to worry about setting this.

You can try the trick of keeping an explicit stack of iterators and closing old ones if you feel you've used too many, re-creating them when you recurse back up. See telldir(3) and seekdir(3) for how you might implement this or nftw(3) for an API that actually works like this. Note that the OpenBSD nftw(3) just ignores the file descriptor limit (i.e. it uses the naive approach) while glibc 2.9 uses a hybrid approach of collecting all remaining entries for each iterator (that is, file descriptor) it has to close.

So, yeah, it's a bit more awkward than it looks.

Still, there's little excuse for non-recursive non-sorting ls(1) to have the same behavior, but it does (and if you look at the source for that, you'll see it's got pretty much the same code as find(1) to collect all the entries before touching any of them). Is not the main reason for "ls -U" the fact that sometimes you have really big directories and just want the names as quick as possible?

jvisualvm

2009-06-11T05:58:00.000Z

For some reason, I was under the impression that jvisualvm(1) was just a rebranding of jconsole(1). It turns out that it isn't, and that – unlike jconsole(1) – it's not a waste of space.

It's not very clever about dividing up space between its various graphs, but other than that it looks pretty nice. I thought I wanted a quick heap profile to see why one of my applications ate so much memory that my machine always felt swappy afterwards. When jvisualvm(1) showed that there wasn't much heap retention going on, I was almost ready to dismiss it as broken, but thought I'd play about and see what else it could do. Turning to the "threads" tab, I saw my actual problem: I had several hundred idle threads.

It turned out I'd once again forgotten that "an unused ExecutorService should be shut down to allow reclamation of its resources".

I had code like this (only without the call to shutdown):

  ExecutorService service = Executors.newFixedThreadPool(threadCount);
  for (Input input : inputs) {
    service.execute(new Job(input));
  }
  service.shutdown();

Every time this ran, which was pretty frequently, I'd leak another 8 or so threads, that I'd never use again.

I should write myself some kind of "do this batch of jobs on n threads and then kill the threads" wrapper.

Anyway, although I was disappointed not to find any heap retention in any of my applications, I was quite pleased with jvisualvm(1) for finding a performance problem, even though it would have been as easy to spot with control-\ — I think that having all the tools together in one place makes me a bit more likely to bother to poke about now and again.

If you have trouble installing plugins like VisualGC, try quitting and restarting. I found that – contrary to the error message – they had actually installed okay, and appeared in the UI when I restarted. They even seem to work.

xargs -P

2009-06-10T07:07:00.000Z

Another entry in my occasional series of options Unix never had when I were a lad (last time: tail -F)...

I needed to use xargs(1) the other day to run lots of really cheap jobs, each of which had a high latency.

This was taking a long time until a friendly youngster suggested I use GNU xargs' -P option. Starting again with -P 32, my job finished in less time than I'd already wasted.

Mac OS 10.6

2009-06-08T22:17:00.003Z

I'm pleased to see Mac OS 10.6 is coming in September 2009, and I'm pleased it's going to be $29 instead of $129, because most Mac users I know stuck with 10.4 which was "good enough" and which they'd already paid for.

I'm slightly surprised by the "Mac computer with an Intel processor" requirement. At least I won't have to feel quite as bad about dropping PowerPC support myself.

I'm disappointed at the lack of detail about Java. "Complete Java JDK, including javac, javadoc, ANT, and Maven tools" doesn't tell me whether Java 6 will finally be the default, which is what I'm hoping for. PowerPC support doesn't cost me much (except the ability to build on 10.4 or 10.5 with the same make rules, which isn't a big problem as long as all the Mac OS-using developers are still on 10.4).

But being tied to Java 5 is increasingly annoying (and limiting). Reflection and other hacks only get you so far, and -source 1.5 doesn't protect you against accidentally using API that wasn't in Java 5 — String.isEmpty having caught me out just yesterday.

Netbeans + OpenJDK = frustration

2009-05-25T06:23:00.000Z

Although I sometimes claim to try out the various IDEs every few years, what I really mean is that I try out Eclipse every few years. I've only tried Intellij once, recently, and to be honest I think I killed it after a couple of minutes waiting for it to start. I'm pretty sure I never got round to actually doing anything with it.

As for Netbeans, I'm not sure I've even seen it since 1999. Not on my desktop or anyone else's. When I complain of bloated, slow, unresponsive IDEs, I always think of Netbeans. But when I've complained of crashy, bloated, slow, and unresponsive IDEs, well, I'm more likely to have been thinking of Eclipse.

Today I thought I'd fix a few of Sun's simpler Swing bugs in JColorChooser. The kind that have been "fix understood" since 2003 but which shouldn't take more than a few minutes to fix. The aggravating "have the bastards ever tried to actually use this shit?" stuff.

Going to the OpenJDK website, I saw they were advertising Netbeans (currently version 6.5.1) as an easy way to get into OpenJDK development. Fair enough, I thought, I'll give that a go.

The instructions should have warned me off. Don't expect any special OpenJDK-building integration. Netbeans doesn't let you deal with Mercurial forests without manually installing the forest extension; the instructions go straight to downloading a .zip source bundle. Okay. They then go on to explain where to find Netbeans project files to open, but they seem to have moved since the instructions were written. find(1) found them, and I opened the small "swing" project...

My first problem was where Netbeans hung my X11 server. I was selecting text at the time, in the first file I double-clicked on, just to see what the editor was like. Thankfully I was using Linux, so I could switch to a different virtual terminal, log in and kill Netbeans, and then switch back to the virtual terminal with X11.

After restarting Netbeans, I tried to build the "swing" project, but it won't build out of the box (this was OpenJDK 1.7.0-b59). Something to do with Nimbus. (Of course!) So naively I thought I'd open the "world" project and build that. Only Netbeans couldn't really open that project on my puny machine. 4 cores and 4 GiB of RAM just isn't enough, seemingly. I sat waiting for it to finish (because the UI was really unresponsive during its "scanning" process) but it took long enough (using only one core) that the screensaver came on. And, presumably for reasons related to my first problem, I couldn't unlock the screensaver until I switched virtual terminal to kill Netbeans.

Don't get me wrong, I saw hints of interesting features in Netbeans. Its behavior when you return to a file you've edited externally but not in Netbeans, for example, feels better than Evergreen's behavior; Evergreen alerts you to the modification by drawing a watermark behind the content, and requires you to manually "Revert to Saved", showing you a diff when you do so. This is sometimes useful — if you've got a generated file open, for example, it's really nice to be able to keep one golden version while you experiment with the code that generates the file, seeing a diff after each run from the convenience of your editor. But automatically and silently reverting to saved seems a lot more convenient for the more common case where you've simply pull new changes from version control, or you used Vim over SSH on your bus ride home or whatever.

Netbeans was also pretty good-looking for a Swing application, although somewhat uneven; the directory chooser looked like some SwingSet Metal horror that probably hadn't changed since 1999.

And Netbeans' uninstall.sh script (in the directory you installed Netbeans in) seems to work perfectly, too, which was a relief. But overall this was a bad enough experience that I fear it will be 2019 before I give Netbeans another try.

The silly part is that building OpenJDK is pretty quick and easy. (At least compared to fighting Netbeans.) Download and install a source .zip, download and install the remaining binary plug (some SNMP crap), install the things you don't have that the README-builds.html lists, and "make". I also had to install libxtst-dev, which I didn't see mentioned, and I found that the Makefiles don't appear to be "make -j"-safe.

So there's not much difficult about building OpenJDK any more. How about slow?

Even on my too-puny-for-Netbeans 2007-vintage Core 2 Quad with 4 GiB of RAM, a clean build only took about 45 minutes. (With the build seemingly never using more than two cores.)

Despite it being something any monkey could do, if they were of a mind to, it's strangely gratifying to run your own JVM at the end:

$ ./build/linux-amd64/bin/java -version
openjdk version "1.7.0-internal"
OpenJDK Runtime Environment (build 1.7.0-internal-elliotth_2009_05_24_21_45-b00)
OpenJDK 64-Bit Server VM (build 16.0-b03, mixed mode)

If you're wondering why I used a source .zip rather than using Mercurial to check out the forests, it's not that I haven't tried Mercurial a few times. It's just "too hard".

I'll try Mercurial again when I'm running an OS that has a sufficiently up-to-date package. Hopefully one day Sun's choice of Mercurial will look like a wise long-term decision, but it's a bummer that it makes life hard for casuals like me right now. Hopefully, just as building OpenJDK was once "too hard", but is now – once you have the source – perfectly reasonable, we'll be able to say the same about Mercurial.

Confessions of a Windows hater

2009-05-13T04:08:00.000Z

I've slightly changed my working environment recently, for the first time in years. Other than adding the System Monitor applet to the panel at the top of my GNOME desktop, I've basically been using the default setup since I switched to GNOME a few years ago.

The default setup has a panel at the top of the screen and another at the bottom. The top panel has a few menus, an icon that gets you a new web browser, a few other icons, a clock, and a weird log out/shut down/reboot button duplicating part of one of the menus. The bottom panel has a button to hide all windows, a big space for buttons representing windows, a little representation of your virtual desktops, and a trash can.

Type "ubuntu" into your favorite image search.

Of these things, the only things I used frequently were the icon that gets me a new web browser and the clock. I used the buttons representing windows too, to some extent, but I didn't really like them and often felt like they were costing me more time than they were saving me. Certainly the experience was nothing like as smooth as Mac OS' dock.

What I didn't know was that the top panel and the bottom panel aren't fundamentally different. They're both the same kind of container, different only in the stuff they happen to have in them. I learned this when I saw someone else's desktop and noticed that they'd moved the buttons representing windows into their top panel, and removed their bottom panel entirely. Even on a 30" display, pixels are valuable, and vertical pixels especially so.

I immediately switched to just one panel. Being a Unix/Mac OS kind of person, I kept the top panel. The bottom panel had always bothered me a bit, possibly because it's so Windows-like, but having tried my single panel in both places, I have to say I much prefer it at the bottom of the screen. I can actually feel myself having to look up when it's at the top. You could argue that's good, because it's not particularly useful, and you may as well have it out of the way, but being a miserable sod, I find that just makes it even more annoying when i do have to use it. I also find it easier to hit the web browser icon when it's at the bottom rather than the top; possibly just because it's easier for my eyes to keep up with my mousing.

What about the sides? Good question. On Mac OS, I've only ever used the dock at the side. Left or right. I don't much care. I love this, because it takes the least space, and that space is horizontal space, which I have more of anyway. It works well on Mac OS because everything in the dock is basically square. Any text only shows when you point, and it shows over the desktop, where there's plenty of space anyway. The other reason it works is that the Mac's dock only has one entry per application, rather than one per window.

If you find the exact right pixel to click (right-click the little grip to the left of the buttons; the nice big space on the right gets you a completely different menu) you can set the "Always group windows" property to get a lame-ass imitation of this. It's okay in terms of saving space, but the behavior shows the usual lack of respect for the user and what they might be trying to do. In particular, you can't easily bring all an application's windows to the front. That's not a big problem for something like a web browser or terminal emulator where an application's various windows aren't closely related other than by parentage, but it's a pain for applications that have a small number of genuinely related windows: you can't conveniently bring "the application" to the front. It's even quite awkward to hide (minimize) all of an application's windows. Certainly nothing approaching the "click to show, command-H to hide" I so miss from Mac OS. What else is wrong? The visual indication of minimized windows could be more distinct (as it is, the title is simply surrounded by square brackets). The title shown in the panel is the application title if you have more than one window, but the window title if you only have one window. I'm sure this was well-intentioned, and it does sound reasonable until you use it, but it just doesn't work if you keep opening and closing windows. The text you're looking for keeps changing, and its position changes too, though I'll admit that I honestly can't work out what the sorting criteria are. Random shuffle? Finally, most difficult to fix, you only get an entry per window. So if you use tabs, you'll only see the title of the currently-visible tab in each window. This really sucks, but even the non-free OSes don't get this right yet.

Even with those reservations, I'm still happier like this. I have enough windows open at any given time that the other style just isn't useful. Fifty little buttons all with nothing more than an icon and "..." may as well not be there at all for all the good it does. (Which strikes me as odd; surely users who don't open many windows don't really need the buttons anyway?)

Just as I thought I was done messing with configuration, I accidentally removed my "Applications", "Places", and "System" menus. Which was 99% fine by me, but I do actually need them once a week or so. So I choose "Add to Panel..." and add "Main Menu" back. Only it turns out that "Main Menu" is just an Ubuntu logo. Clicking on it gets you a single menu that's basically the "Applications" menu with the semi-useful bits of the other menus tacked on the bottom. So there's a few hundred more pixels you can save: remove the menus and add "Main Menu" back for no loss of functionality but plenty more space. (The new menu is nice and easy to hit, too, being right in the corner.)

The one change I made after this was to swap the relative positions of the system monitor and my web browser and terminal emulator icons. I originally had the icons right after the wordy menus, and the system monitor in the wasteland to the right. Even when I switched to one panel at the bottom, I saw no reason to change this. But making the menus really small moved those icons further from the center of the screen where, on average, my mouse pointer is. Swapping the two moves the buttons closer to hand. (Or, if you prefer, brings them within range of a single mouse sweep rather than requiring two, which is far more aggravating than you'd credit.)

Anyway, here are my suggestions again:

1. One panel is enough for the stuff that actually does anything.

2. The bottom of the screen seems to be a better place for this panel than the top.

3. "Always group windows" isn't as stupid as the default behavior.

4. Removing the menus and adding "Main Menu" back looks and feels better.

The only other things I have in my panel are the "lock screen" icon and the date/time/weather thingy. I used to have the "lock screen" icon with the other two, but I found myself sometimes confusing the terminal emulator icon and the lock screen icon, or at least pausing to make sure which one I wanted to press (they're both basically small dark squares representing monitors), which is stupid when one of them I want a hundred times a day and the other maybe twice. (No, I don't lock my machine when I go for a piss. I piss in the corner of the office.) Moving the lock screen icon out next to the clock thingy cured that.

I will admit that, all in all, I've made my desktop look a lot more like Windows than it did. But I'm man enough to admit it's better for it. And I'm grumpy enough to wish I didn't have to waste my declining years pissing about with stuff that shouldn't be getting in my way in the first place.

Maybe next year I'll learn how to use gconftool(1) to stop the window manager from opening every window in the top left corner right on top of the last one, as soon as the screen is too "full" to place the window in completely empty space. (You know, ten seconds after you log in.) Presumably GNOME developers have 15" displays and only open one maximized window at a time, like the Windows users of Windows-haters' myth?

Playing mp3s from Java

2009-05-07T03:33:00.000Z

JNI

It's pretty obvious that mp3d needs to be able to actually play mp3s. This was another reason why I initially chose C++, my assumption being that there would be plenty of C++ mp3-playing libraries to choose from.

In a way, there were. But, perhaps because audio code isn't very portable and perhaps because Linux's audio API is abandoned for something new every six months or so, the libraries I came across were mainly focused on decoding mp3 data in some raw form. Presumably if I'd looked hard enough I'd have found the other pieces of the puzzle and been able to put them together, but after a brief search I gave up and started looking at command-line tools instead.

Spawning subprocesses

This is how the first version worked. If all you want to do is play an mp3 from start to finish and have no real control over – or feedback about – what's going on, this is a really easy option. But I was surprised by the strange programmatic interfaces offered. mplayer(1) seemed the best overall, and does indeed have a mode where it expects to communicate via a pipe, but the command set available seemed to assume a greater autonomy for mplayer than I was looking for. I wanted to keep the "playlist" in my server, for example, and basically use mplayer as my low-level API. It wasn't clear that I could really do that, and the various bits of calling code I could find on the web didn't suggest otherwise.

When I used mplayer from Java, I actually used Unix signals to control it, but I soon decided I wanted more control, and didn't particularly want to be tied to running on Linux (even though personally I'm unlikely to ever run mp3d anywhere else).

javax.sound

You might think that Java would have built-in support for the most popular audio format, but it doesn't. Presumably licensing problems. I hope this will be fixed by OpenJDK, at least for Linux, where it seems possible that whatever Free library they use for audio will have out of the box support for mp3 (and ogg and all kinds of other thus-far neglected formats).

Sun's bug database considers JMF (remember that?) to be the answer.

JavaZOOM JLayer

Despite the awful name (though I of all people should be careful about throwing stones where naming is concerned) JavaZOOM's JLayer is "the" Java library for decoding/playing mp3s. There may be others, but this is the one that everyone seems to use, so it was the one I went for too.

The API is a bit odd. Some of the names suggest no native speakers of English were involved. You stop a Player by calling "close", for example.

The API is also rather limited. If seems like more of a demo for the underlying mp3 decoder than an real API anyone's actually expected to use. If all you want to do is play an mp3 from start to finish, without pause or fast-forward (though you can stop prematurely), you're fine. Anything else and you'll be using the low-level API to shuffle frames from the decoder to the audio device.

Even the trivial case isn't well documented, so here's the minimal sample:

  String filename = "muzak.mp3";
  AudioDevice audioDevice = FactoryRegistry.systemRegistry().createAudioDevice();
  Player player = new Player(new FileInputStream(filename), audioDevice);
  player.play();

The call to "play" will block until the mp3's finished playing, but you can call "close" in another thread if you want.

(I'd have sent in patches, but the project doesn't have any visible source repository and appears pretty dormant.)

This is actually pretty much the code I'm using in mp3 at the moment. I keep expecting to want more, but I'm not yet convinced. I will admit to having used SIGSTOP and SIGCONT from a shell on a couple of occasions, but "pause" doesn't make as much sense as you might think on the web. Maybe if I bound my "media keys" on all my machines to POST appropriate form data I'd be more convinced. That would actually be pretty cool.

Without moving to something distinctly more AJAXy, though, I'm not sure "pause" fits the UI. And you probably would have to present it for the benefit of other users.

Using com.sun.net.httpserver

2009-03-31T05:23:00.000Z

One of the things I needed for mp3d was an HTTP server. Sun added one to JDK6 – com.sun.net.httpserver – but the documentation's a bit sparse, there's no example code (and the snippet in the package description doesn't compile), and judging by the results of my web searches, it's not widely used.

Despite these obstacles, it was pretty easy to do everything I needed. Here are a few notes from my experiences.

Getting query parameters

If you're using HTML forms with the GET method, you'll need access to your query parameters. The obvious way to get them would be to call HttpExchange.getRequestURI and then call URI.getQuery on the result. Unfortunately, that's broken: URI's built-in decoder only copes with %xx escapes, and doesn't know that '+' needs to be translated to ' '. Bizarre.

What you need to do is use URLDecoder on the result of URI.getRawQuery instead.

(Either way you need to break the string into key-value pairs yourself.)

Getting form data

If, on the other hand, your HTML forms use the POST method, you'll need access to your form data. Here the obvious scheme works fine: just read the input stream. Again, I'm surprised there's no convenience method to help with this common task.

Setting headers in the HTTP response

If you want to set an HTTP header in your response, use the HttpExchange.getResponseHeaders method. That gives you a Headers object which is just a map. You can use Headers.put as if it were a plain old HashMap or whatever, but I preferred to be explicit and use Headers.add or Headers.set to make my exact intention clear.

(Getting the response headers object and calling a setter on that is pretty obvious when you know, but it took me longer to find than it should have done, in part because I wasn't sure the functionality even existed, but mainly because I was looking to start my journey with a setter, not a getter.)

Exception handling

Exceptions not caught by your HttpHandler implementations are reported, but in very spartan fashion, without stack traces. There's also no way to set an uncaught exception handler (as there is with AWT, say). You'll probably want to write your own code to deal with this. An obvious possibility is to write your own abstract class implementing HttpHandler with a "handle" method that catches and reports anything thrown by your class' new abstract method that does the actual handling of the request.

HTTP response codes

If you want constants for HTTP response codes so your code's not full of magic numbers such as 404 and 307, look at HttpURLConnection (from the standard java.net package). Then you can use, for example, the more obvious HttpURLConnection.HTTP_NOT_FOUND instead of 404.

GZIP content encoding

There's no built-in support for the GZIP content encoding, but it's pretty easy to do it yourself. Given an HttpExchange 't':

t.getResponseHeaders().set("Content-Encoding", "gzip");
t.sendResponseHeaders(HttpURLConnection.HTTP_OK, 0);
final GZIPOutputStream os = new GZIPOutputStream(t.getResponseBody());
os.write(...);
os.finish();
t.close();

You might want to check the request headers to ensure that the client can cope with this encoding, but I couldn't be bothered. As the Zen master said, a bug's not a bug until it inconveniences somebody.

How does HttpServer choose the appropriate HttpContext?

When you set up your HttpServer, you add "contexts", which are basically associations between a path and the HttpHandler that corresponds to that path. You can just have one that parses the path itself, if you like. The current implementation is an unsorted linked list, but that's fine for the kinds of things you'd sensibly use this code for.

Is it kosher?

You'll have noticed that the HTTP server packages are com.sun.* packages. Although they're mentioned in the Networking Features part of the JDK6 release notes, they're not indexed in the Java Platform API Specification.

The rules for com.sun.* packages aren't mentioned in theJDK6 documentation's Note About sun.* Packages.

As I understand it, com.sun.* differsfrom sun.* in that Sun supports the former but not the latter. In neither case are you guaranteed that it'll be available in other implementations. (For the record, Apple's Mac OS 10.5 JDK6 does include the com.sun.net.httpserver packages.)

Past experience suggests that a com.sun.* API in one JDK release won't necessarily be in the next, so you might not want to base your business on this API. But for a weekend hack that would be equally fine with an 80-line HTTP server using little more than a ServerSocket, it's a handy jump start.

Sample code?

Even I have better things to do than write you a nice little tutorial, but the mp3d source currently contains working examples of all the above.

I'm pretty sure I spent longer writing this post than I did working any of this stuff out or writing the code.

Why do I keep writing C++?

2009-03-27T07:08:00.000Z

When you think of Unix daemons, you probably think of C or C++. When you think of cobbling together a program from a handful of libraries on Linux, you probably think of all the C and C++ libraries just a dependency away.

This, at least, was how I ended up writing the very first version of mp3d in C++. And it was a huge mistake. Throwing away two days of work and starting again in Java was the best move I made.

Aren't all daemons written in C/C++?

Tradition is a state, not a reason. Just because the daemons we grew up with (or wrote) years ago were in C doesn't imply that C was particularly suited to writing daemons, or that we should use it to write any more.

I think part of the problem here was that I could pretty much see the boilerplate, ready to flow out of my fingers. And that boilerplate was in C++. The opposite would have been the case if I'd been thinking of a GUI application. It wouldn't have even crossed my mind to consider C++ for one minute. The relevant boilerplate would have been ready, and this time it would have been in Java.

In the server case, such "thinking" is short-sighted. If the boilerplate is the hard part of what you're about to do, what you're about to do probably isn't worth doing at all.

But all those great libraries...

I think what really convinced me, though, was the idea that I'd be able to grab one of any number of HTTP libraries and id3v2 tag libraries if I chose C++.

Never mistake quantity for quality.

The HTTP library wasn't much of a problem, though the selection was much more limited than I'd imagined. The least unpromising choices were very much C libraries, with all the ugliness that that entails. Manual memory management everywhere, and no notion of stream or string. Three fairly fundamental things that even C++ programmers take for granted. I won't even waste breath on my chosen library's lack of const-correctness.

The id3v2 tag library situation was far worse. The situation there appears to be that there are several alternatives, but none of them is actively maintained. As far as I can tell those applications that manage to work with real-world id3v2 tags without crashing are using their own forked/patched versions of these libraries. And even though you can get a C++ (rather than C) library for handling id3v2 tags, it's quite the runniest kind of shite. It's the kind of C++ you see written by people who don't show any sign of understanding why C++ is better than C. ("Is it the // comments, perhaps?")

Don't expect to see any use of the standard library, not even std::string.

And even if they had used C++ to its fullest, you soon run up against the edges of that "fullness".

You want to cope half-way sensibly with non-ASCII text? You'd better learn and use ICU4C then. (Sure, C++ gets along okay-ish on its own in an environment where its input is UTF-8 and it doesn't actually have to process text so much as just pass it around. But as soon as you need to either deal with encodings other than UTF-8 or need to actually process the text, you need ICU4C.)

Or maybe you'd like threads? Or maybe you need concurrent collections? Perl regular expressions?

I can't remember the last program I wrote that didn't need all of these things.

I'm not saying that all Free Java libraries are great. Of course they're not. 90% of everything is crud. But you don't need nearly as many third-party libraries because what you get "built in" is mostly pretty good, and certainly covers a lot of ground. And it's much easier to write a decent Java library than a decent C/C++ library. I've yet to see a Java library invent its own string class, for example. And there's much less effort expended on questions of ownership (though there are certainly traps for the unwary, should they hand out mutable objects they intend to continue using).

Why do you keep fooling yourself?

I fear C and C++ fall into the same class as awkward editors. Why do people use emacs or vi rather than a proper editor? Why do people make scale models of intricate buildings using only matchsticks? Sometimes the very fact that a thing is hard is what makes it fun. (Actress, bishop.)

I evidently don't know how to stop myself falling into this trap, but maybe it will help I come out in public more often and confess. Along the lines of "hi, I'm Elliott, and I wasted a couple of days trying to start my project in C++, fighting an awkward language and a stunted standard library and several really badly-built libraries, and in the end I realized I had nothing to show for these two days, came to my senses, started again in Java, and after just a couple of hours I had something I could actually use".

It's not like using sensible tools takes all your fun away; it's just a way of trading a puerile unproductive kind of fun for the fun of doing a good job of something useful, in a shorter time than the job would otherwise have taken. Trading short-term gratification for long-term satisfaction.

I guess I'll be fighting that part of human nature for a long time.

mp3d

2009-03-26T02:59:00.000Z

The few lines of code in last post, Registering a service with DNS-SD from Java were lifted directly from one of the things I'm working on at the moment. As you could tell from the service name, this program is called mp3d. If you know what an mp3 is, and you know that Unix tends to name daemons something ending in 'd', you've probably got the picture already.

All I wanted was a little web server that would let me play my music from any computer, with the sound coming from the one computer that has decent speakers.

At this point, you may well be muttering "reinventing the wheel again, dude?". And you'd be forgiven for thinking so. I certainly assumed that the internet must be positively dripping with examples of the program I was looking for. Indeed, I spent as long searching the web for such a thing as it actually took me to cobble together the first version when I finally gave up looking.

(If you ignore my unfortunate detour into the world of C++ that I may talk about some other time. For now though, let's pretend I was a sensible boy and chose Java from the outset.)

Distraction: RhythmWeb

The first candidate I found on the web was a plugin for RhythmBox, which sounded ideal. I was at that time using RhythmBox as my mp3 player, and although RhythmBox isn't great, I could have lived with a web interface to it.

Unfortunately, RhythmBox and RhythmWeb didn't get on very well with each other. I'd hoped for some kind of unified state, but they seemed to be fighting amongst themselves and keeping track of their combined state (and thus what my actions would do) was pretty confusing.

I could have stopped using RhythmBox locally and used the web browser even on the box with the speakers, but there was a worse problem: RhythmWeb was slow. So slow it was almost impressive. I don't have a large mp3 collection – less than 5000 tracks – but RhythmWeb couldn't cope with that collection. Not in any reasonable time frame, anyway. Not even when only asked to display a few matches. Python: all the elegance of a silly walk, and all the speed of a dead parrot.

RhythmWeb did look quite nice, though, in a minimalist kind of way. I've tried to keep something of that style in mp3d.

Distraction: mpd

The only other candidate I came across that seemed close to what I was looking for was mpd. But that turned out to not be quite what I expected. My idea was basically "a web server that lets me play my music". What I found in mpd was more "a server that would play my music if I ran an appropriate client to make the necessary RPC calls".

There followed an interlude where I looked at a bunch of clients before eventually giving up and deciding to write my server.

Frankenstein's Mp3 Player: mp3d

I started off with directory-indexing code copied & pasted from one of my other projects. I started with the id3v2 tag parser recommended by Jesse Wilson in his roundup (though I eventually wrote my own, which I may talk about another time). I used Sun's com.sun.net.httpserver.HttpServer which was a blessed relief after suffering libmicrohttpd in the C++ version. I remembered just about enough HTML to knock together a TABLE. I copied RhythmWeb's CSS. I spawned mplayer(1) to actually play the mp3s. That was the first version, done in a couple of hours.

The code's all in code.google.com/p/enh/, home to projects that are still in early states (and dumping ground for dead projects). mp3d's been through a few changes since the first version, and it's got a way to go still, but it's already the only mp3 player used on any of the computers at home. You're welcome to try it if you're interested, but I'd no intention of packaging it in the near future, so you'll have to build from source.

I bring it up mainly as an introduction to the next few posts, which will cover various things that cropped up while working on mp3d.

Registering a service with DNS-SD from Java

2009-03-24T06:29:00.000Z

DNS-based service discovery (DNS-SD, but better known as Rendezvous, Bonjour, or ZeroConf) is a great way for services to announce themselves, and for potential clients to see what services are available. Mac OS has mDNSResponder, Linux has Avahi, and Windows – as far as I know – has its own competing alternatives.

I've played with this stuff a few times in the past, but never really needed it. Recently though, I wrote a small web service that will mainly be used from Mac web browsers on the same network. What's the easiest way of finding the server? Probably the fact that registered _http._tcp services automatically appear in both Camino and Safari's bookmarks menus. Unfortunately, Firefox does not support anything like this, even on Linux. Nor, as far as I'm aware, does Chrome. (Epiphany does, though, if you're looking for a Linux browser with DNS-SD support.)

So. What's the easiest way to get a Java HTTP service registered?

avahi-publish(1)

Since my server is running on Linux, and I have the avahi-utils package installed, avahi-publish(1) is a convenient choice. Given my ProcessUtilities class, it's just one extra line to spawn a child that publishes news of my service. And, conveniently, that child dies when I die, so I don't need to do anything special about deregistering. Normal process semantics take care of things well enough.

Here's the single line it took to get my service (called "mp3d") registered:

ProcessUtilities.spawn(null,
    "/usr/bin/avahi-publish", "-f",
        "-s", "mp3d", "_http._tcp", Integer.toString(portNumber));

Pros: trivial to use, from any language.

Cons: Linux-only.

Avahi bindings

I did look at Avahi's bindings, but they don't include Java. And since that wouldn't actually solve the "con" of avahi-publish(1), it's not really obvious that I'd gain anything by going down such a route even if it were open to me.

Pros: none.

Cons: neither useful nor written yet.

mDNSResponder

Apple published mDNSResponder as open source under the APL 2. It supports Mac OS (even the prehistoric pre-Unix variant that it's believed the cavemen used), Windows, and "POSIX platforms". This does come with Java bindings, but they're just that: bindings to the native code using JNI.

This would provide portability, but by this time I'd realized that ideally, I wanted a pure Java solution.

Pros: somewhat portable.

Cons: awkward JNI dependency, recompilation requirements.

waiter

The waiter project is very proud about being written using Test Driven Development and what kind of mocks it uses, but not terribly interested in telling you how to use it. Or what state of development it's at. The code looked a bit odd, too. More like a Java-hater's parody of Java than anything.

Pros: none.

Cons: no documentation or sample code, weird attitude, not obviously alive.

mahalo-mdns

The mahalo-mnds mahalo-mdns project was a fork of JmDNS.A fork on which apparently nothing of substance happened.

Pros: none.

Cons: dead.

JiveDNS

The JiveDNS project was a fork of JmDNS.A fork that apparently didn't get as far as actually committing any code.

Pros: none.

Cons: dead.

JmDNS

So what about the project everyone else seems to fork but then not do anything with? This appears to be the best of the bunch, though it's not without problems.

I tried version 2.1, and the API looked reasonable. More Java-like than the others, certainly. The documentation seems to think the API uses constructors, but the code itself has changed. Luckily, the example programs it ships with have been kept up to date.

I noticed that the project's site has bug reports about deadlock and dodgy use of JVM shutdown hooks going back years. Given that the second of their example programs did indeed deadlock in a shutdown hook, leaving me having to kill -9 the JVM after suspending it, I'm ready to believe those reports.

Anyway, here's the single line I needed after dropping the jmDNS JAR file into my project's lib/jars/ directory:

JmDNS.create().registerService(ServiceInfo.create("_http._tcp.local.", "mp3d",
                                                  portNumber, ""));

And sure enough, if you add something like that to your program, you'll find your program too will deadlock if interrupted. Suboptimal.

Under other circumstances, I'd suggest jmDNS was ripe for a forking, but that doesn't seem to have worked in the past!

Pros: portable.

Cons: deadlock-prone, under-maintained.

Conclusion

So what am I going to use? avahi-publish(1). I was only expecting to run on Linux anyway, so I'll worry about portability when I come to need it.

But... it turns out that I can't use DNS-SD for what I originally wanted it for. My plan was to switch from a hard-coded port number for my HTTP server (8888, just like we used to use in the good old days) to just accepting whatever ephemeral port I got. I thought that if I announce my service using DNS-SD it wouldn't matter what port I was on.

To a certain extent that's true. The missus doesn't need to know the port number if she uses the Bonjour bookmark to get there. But if she wants to get there via the history and the server's restarted, she's screwed. Likewise if the server restarts while she has a browser window open, when she comes back to that window, it's "broken". So it turns out I can't really get away with accepting an ephemeral port after all. This seems like a real pity. Maybe one day we'll be able to use DNS-SD service names as virtual port names, as in http://lithium.local.:mp3d./ say. But not today.

So it's back to the four fat ladies of my hard-coded port number, but I'll leave the DNS-SD registration in anyway. I'm used to it now, and it'll be a comfort in my old age, when my memory's completely gone.

How to fix dnsmasq after an Ubuntu upgrade

2009-03-17T02:24:00.001Z

I mentioned before in Desktop Linux suckage: DNS caching that I use dnsmasq as a local DNS cache to work around lame ISP DNS, because Linux doesn't tolerate lame DNS as well as Mac OS will. (I never even noticed I had a problem until I switched from Mac OS to Linux.)

Using dnsmasq as a local DNS cache worked well, and I had no trouble until I upgraded to Ubuntu 8.10. Here, for my own reference next time I upgrade, is what I needed to do to fix things. (I'm assuming that this breakage will happen on every upgrade, because it would be rather sad if literally the only difference I noticed between 8.04 and 8.10 is that my local DNS cache stopped working.)

The symptom was that "nameserver 127.0.0.1" was missing from /etc/resolv.conf. Hasty optimistic fool that I was, I manually reinserted the missing line, and all was well... for several hours, when my DHCP lease was renewed. At which point it appears dhcp3-client rewrote /etc/resolv.conf for me.

(I really wish the perpetrator had had the decency to write a comment in /etc/resolv.conf saying what was going on. It might also have been nice to make the file read-only, as an added hint that humans shouldn't waste their time editing the file.)

So the real fix, it seems, is to edit /etc/dhcp3/dhclient.conf and uncomment this line:

prepend domain-name-servers 127.0.0.1;

I thought the daemon (dhclient3) might watch its configuration file for changes, but it didn't appear to. The daemon also appears to die if you send it SIGHUP, a signal those daemons too stupid to notice changes to their configuration files used to take as an indication that they should re-read their configuration.

Anyway, this – though perhaps more forceful than necessary – did the trick and saved me from wasting any more time on this nonsense:

sudo /etc/init.d/networking restart

Yeah, I know: "here's a nickel, kid".

Apple Keyboard first impressions

2009-03-07T04:28:00.001Z

This time, it's war

The box containing my "old" Apple Keyboard with Numeric Keypad had a sticker on the side that said something to the effect of "this keyboard requires Mac OS 10.4.7 or later". Apart from the fact I happened to be running a buggy Linux kernel at the time (thanks, Ubuntu), it actually worked fine on my Linux boxes. It worked fine on my Macs, too, though they were running suitable versions of Mac OS.

Anyway, the new Apple Keyboard (MB869LL/A), the one without the numeric keypad, its box says something like "blah blah Mac OS 10.5.6 or later". (I'd quote exactly, but the poltergeist that throws my stuff out while leaving its own stuff lying around appears to have paid another visit while I was at work yesterday.)

I scoffed when I saw this, but it turns out it's actually true.

Mac OS 10.4 compatibility

On Mac OS 10.4.11, the function keys' functions don't match their glyphs. If you have fond memories of the good old days when it was F12 to activate Dashboard, you'll love this keyboard. Despite the Dashboard glyph being on F4 and F12 having the increase volume glyph, F12 activates Dashboard.

Also, despite the fact that this keyboard looks exactly like a MacBook keyboard, unlike the MacBook, you can't use fn and the arrow keys to get home/end and page up/down. Presumably you can if you're running 10.5, but those of us planning on going straight to 10.6 are out of luck.

Linux compatibility

It should go without saying that if Mac OS 10.4 users are out of luck, Linux users are screwed, but none of Ubuntu 6.06, 8.04, or 8.10 have any clue what to do with this keyboard. I've mailed USB id updates to the guy who maintains the list (so at least lsusb(1) will show what you've got connected), but I had the strangest sense of deja vu as I was doing so, and a strong feeling that my previous submissions never made it into the list.

Windows compatibility

I don't have a Windows machine, but I see no reason to assume things are any different there. The box certainly didn't say anything about Windows compatibility. This probably presents quite a problem for Windows users. I can suffer without home/end and page up/down, as I used to on my sort-of-Happy Hacking keyboard, but as I understand it, Windows users can't even log in without a delete key. And though Mac keyboards say delete, they mean backspace, and if you really want delete, you need a working fn key.

So if you're a Windows user who likes Apple's keyboards, I'd be really cautious about this particular one, though the Apple Keyboard with Numeric Keypad had two delete keys, one of which actually was delete, so you'd probably be fine there. I certainly used to use it on Linux, judging by the number of times I've hit the desk where that key ought to be over the past couple of days.

First impressions

Having the mouse closer to hand is great. That feels good. The downside is that I'm actually having to use the mouse more because I'm without the editing keys.

I'm coping with the scrunched-up arrow keys, though I still have to look for them, and sometimes hit the wrong one, even with my rather slender fingers.

I'm not missing home and end much, though I'd rather I still had them. I'm missing page up/down more, though I can cope. The surprise winner of most-missed-key is actually delete (aka forward delete, as opposed to backspace). I'd really like a working fn key, and can't really recommend this keyboard to anyone not running Mac OS 10.5 right now, because that seems to be the only OS with the right magic. I don't know enough about USB keyboards to understand the problem here, but it seems quite an unfortunate situation. Surely pseudo-modifiers like fn should be handled in the keyboard itself, and some kind of uniform key code (plus uniform set of modifiers) sent to the computer? Surely requiring each OS to understand each keyboard's layout (and its international variants) is something from the dark ages?

I wasn't expecting the fn key to be where it is. Despite the fact that it's in exactly the same place on my MacBook Pro. I guess I just don't really use my MacBook Pro enough to have noticed or cared; it's a glorified web tablet, really. A heavy battery-hungry one at that.

Anyway, the trouble with the fn key's position isn't where it is so much as that it moves the control key across one place. Which moves the alt key across one place. So if you've ever experienced modifier-key confusion, expect to experience it even worse now. On the other hand (the other metaphorical hand, that is), the control key's new position actually makes it possible for me to type control-shift combinations without ridiculous discomfort. Previously I'd have to bend my left thumb so far back to reach the right edge of the control key that it literally hurt. The combination of a more accessible control key and an out-of-place alt key might actually get me to switch to using control-shift instead of alt in Linux terminal emulators.

Assuming I stick with the new keyboards. I'm left more than ever wishing for the "Apple Keyboard without Numeric Keypad but with all those Useful Keys that come between Keyboard and Numeric Keypad". If I had a working fn key, that would be one thing. But genuinely losing my home/end and page up/down keys is harsh.

And I don't even know whether I need a new kernel, X server, xkb configuration, or some GNOME thing to fix my problem. Too many layers of bureaucracy fighting for control of a triviality.

Apple Keyboard unboxing

2009-03-06T07:42:00.001Z

So my 3 Apple Keyboards arrived today. These are small keyboards. Put one next to a laptop, and it looks small. Put it next to a full-size keyboard, and it looks tiny.

Each keyboard comes in a box 8.5cm by 32cm by 13.5cm. I ordered three of these, and their three boxes came inside another box. This other box was 31cm by 32cm by 56cm. So the keyboards took up 7% of the space inside this enormous box.

Or, alternatively, I could have ordered 30 keyboards and have them arrive in the same box.

How did they not rattle about? The rest of the box was filled with a paper snake, like the ones you see Chinese people dancing about in during Jackie Chan movies. To be precise, one 1080cm by 76cm sheet of brown paper, scrunched into a tube, and then bent back on itself many times to fit inside the box. I pulled out so much paper that I feared the box wouldn't actually contain my keyboards.

To get an idea of the size of this paper snake, imagine you pull your arm out of your sleeve and let the sleeve hang like Nelson (the Vice-Admiral, not the Simpsons character). Then stuff your sleeve with paper, so it's as full as if it still contained a rather chubby arm. Then extend your sleeve and arm until they're eleven meters long. That's the length of six men standing on each other's heads. That's long enough to reach to the bottom of an olympic diving pool and back to the surface again. That's about the length of a whale penis.

By anyone's standards, that's pretty big.

Apple Keyboard

2009-03-04T06:17:00.001Z

I've complained about the numeric keypad for well over a decade now, and it's been a particularly frustrating decade. When I have a software problem, I can either decide I don't really care all that much, or get pissed off enough to actually fix it myself. With hardware problems, I just have to take it.

Most recently, just last year, I complained that Apple's current metal keyboards only offered two choices: wireless without numeric keypad, or wired with. Hater of batteries and numeric keypads that I am, I wanted wired without.

Today, Apple quietly added that missing keyboard to their lineup.

Even more interestingly, they changed the name of the old wired keyboard to "Apple Keyboard with Numeric Keypad". The old wireless keyboard with no numeric keypad remains the "Apple Wireless Keyboard". The new wired keyboard with no numeric keypad is the "Apple Keyboard".

This is it, friends. This is the beginning of the end of the accountants' reign of numeric keypad tyranny!

Actually, the beginning was probably the generation for whom computers were mostly laptops, and desktops were either their parents' computers, or the boring computers work makes them use.

But what's important is that "with numeric keypad" is now an exception worth pointing out in your product description. At least for Apple. But even Dell eventually gives up on parallel ports and floppy drives and VGA and whatever. (I mention Dell only because the last computer I saw with a floppy drive happened to be a Dell. Really I mean "all the computer companies that just drift on the tide". At least Dell never made a transparent computer; I'll give them credit for ignoring that bizarre fashion.)

Some of you will remember that what I actually wanted was the full keyboard without the numeric keypad. That is, I wanted to keep the "editing pad" (or whatever you call the bit between the main part of the keyboard and the vestigial numeric keypad). Maybe I'll learn to live with cramped-up arrow keys and no separate page or home/end keys. I lived similarly before, when I had a Happy Hacking keyboard that didn't have arrow keys at all, so the question is really whether I'll learn to be happy living like this.

Whatever, my mouse will relish the extra desk space, and I'll not have to bother with batteries or Bluetooth.

I'll take three, please, Steve.

Swing 2: Pissing in the Wind

2009-02-25T03:42:00.001Z

As nerds, it's all too easy for us to focus on technical merits to the exclusion of all else. Sadly, here in the real world, technical merits aren't the be-all and end-all. Repugnant though it may sound to some ears, most things come down to economics.

Take Swing 2, for example. The "vision" is the somewhat ungrammatical "Leverage current Swing know-how to create modern version Swing". But that's not what they mean. What they mean is "use Java 5 language features to perturb the Swing API". Specifically, the current focus is on cleaning up Swing's admittedly old-fashioned API: using enumerations instead of integer constants, using generics, and adding "varargs support". (I'm not sure quite what's meant by the last one, but it doesn't matter for the purposes of this post.)

Sun – sensibly enough – is not interested in making these changes, so the "Swing 2" guys plan to fork.

Why wouldn't Sun be interested? Surely it's embarrassing that JList has a constructor that takes a Vector<?>, for example? Surely there's no doubt that an enum is a better way to specify an enumeration than a bunch of ints? Surely we should splash some varargs around, now we have them?

Danny Coward responded but appears to have been too polite to get the point across.

That's where I come in.

If you read Danny Coward's response, check out the bits in bold. That's what a sane man sounds like. He talks about making it "easier to develop with Swing" and making the "runtime deploy and perform better". As long as "perform" covers look and feel as well as space/time performance, I don't think anyone could disagree that those are the most important possible goals for Swing.

The other important goal would probably be "enable developers to build stuff they can't currently build", but that seems to be the one Sun always ignores. They're too busy catching up with functionality that was old hat even in the days of Windows 95. (An about box, say, or a font dialog, or an embeddable browser component, or any number of other fairly basic things we still don't have in Java in 2009.)

I've written and worked on numerous Swing applications now, and can honestly say that none of the ideas mentioned so far for the third-party "Swing 2" would have helped in any way...

Enums What would making BorderLayout.EAST and friends enum constants buy me? I'll grant you that the Font constructor can be troublesome (which is it; size then flags, or flags then size?) but Font.deriveFont is a perfectly adequate work-around for that. Plus Font doesn't count because it's part of AWT, not Swing.

Generics Translating to or from a legacy collection is the work of a single line. More importantly, using a collection directly as your model is hardly defensible as good practice, so this just isn't an issue that comes up. (If you really want to improve the Swing model situation, you probably want to be thinking more along the lines of something like Glazed Lists.)

Varargs I'm baffled there. Varargs methods are occasionally useful, but Arrays.asList and/or Collection.toArray ensure that their absence is never a major problem. And see Bloch's Effective Java 2e (item 42) for warnings about over-using varargs.

Nothing in the current "Swing 2" plan has any real value to the developer of Swing applications, or to the developer's users.

It doesn't make existing stuff any easier to write.

It doesn't make existing stuff work any better.

It doesn't make it possible to write stuff you couldn't write before.

And what does "Swing 2" cost me, a Swing developer who's thinking of using "Swing 2" despite the fact that it doesn't do anything useful? Well, it costs me an extra dependency. A dependency on a fork of something that's under continued development. A dependency that won't auto-update itself with future Java releases; you'll have to re-release your application too. Oh, and I have to rewrite my source to be incompatible with what everyone else is using.

And that's as complex as your economic insight needs to be in this case. You look at what you're getting, and you look at what it's costing you, and you decide whether it's worth the bother.

The Nimbus LAF is an interesting contrast. It's equally worthless (if I wanted a LAF that looks like ass and shouts "crappy Java app", I'd use Metal), but Nimbus doesn't cost me anything except the engineering time that was wasted on it, and I had no influence over that anyway. So who cares?

You could accuse me of Python 3K blindness here. When that came out, existing Python users complained that it changed stuff unnecessarily, and even broke the "hello world" that's in all the tutorials. Am I not making the same complaint about "Swing 2"? No. What the Python old-timers were missing, I think (as someone who doesn't like Python, but thinks Python 3K is at least a little less bad than Python 2), is that while in the short term all the tutorials are broken, they'll soon be fixed, and everyone will be better off with a slightly less grody language. Python 3K aims to make things better for the future – where it'll be called just "Python" – at some cost to the past.

"Swing 2" doesn't improve the future, though, because it's a fork that will wither and die for lack of purpose. No tutorials will be rewritten, because they'll always have to refer to the One True Swing anyway. Anyone foolhardy enough to change their code to rely on "Swing 2" will end up changing their code back again, but don't worry: there will be precious few such people, and they won't be people writing actual applications anyway — those people will be worrying about stuff that actually makes a difference rather than unfortunate trivia.

If you want to do something that will make a difference, there are tons of things that are a real pain in Swing at the moment. JFileChooser, JOptionPane, JTable, and SwingWorker are all obvious examples of classes it's pretty hard to write a serious Swing application without, and yet all of which need to be taken outside and shot. And don't forget the ugly stepsisters JTextArea and JTextPane. You could write a decent replacement for any or all of those. Or you could look at some of the deeper stuff like the roles of ActionMap and InputMap and whether menus should really deal in Actions, and how the "Edit" menu works in Cocoa compared to how it never really works right in Swing. Or you could work on giving us abilities we don't currently have; an embeddable web browser with a sensible interface, that's available on all platforms, and that doesn't crash, say. Or you could try to address some of the difficulties of writing cross-platform Swing applications.

If you honestly think that using enums, generics, and varargs in the existing code are anything more than pissing in the wind, you need to read and write more Swing applications.

Or maybe try living without your little toolkit you've been using as a crutch that's prevented you from seeing the real problems. I personally never fight JOptionPane any more, for example, because I've written my own wrapper that beats some sense into it. But that doesn't mean JOptionPane is anything above awful. (This is the Python 3K blindness: it's harder for old-timers to appreciate how broken things are because they've got a good feel for where the dangerous swamps are, and ready-made duckboards for the areas they can't avoid. And aren't you supposed to get a bit wet anyway?)

The only advantage I can see of attacking the non-problems of enums, generics, and varargs is that they're easy problems. Imagine, by comparison, the amount of work involved in writing and maintaining your own JTextArea replacement (like the jessies.org one). And that's a tiny project compared to any reasonable "Swing 2". If you don't have the stomach for one of these wafer-thin mints, you can forget about anything as grandiose as "Swing 2".

Don't get me wrong: you can "fix" these trivial API warts, but you need to provide something worthwhile to go with those changes. If people were already using "Swing 2" because, say, it had a JTable replacement based on Glazed Lists, they might then swallow the trivia. But try to do things the other way round, and your project is stillborn.

Although, given that the currently perceived "enemy" of Swing is JavaFX, you might even want to revisit the very premise of Swing and ask whether we shouldn't really be talking about "AWT 2" instead of "Swing 2". In a sense, JavaFX is Sun making the Swing mistake all over again. (The main difference is probably that last time they were too early to the party; this time they're too late. And at least this time I haven't heard them say it'll be okay for the desktop too.)

Mistakes MS makes when asking for my money

2009-02-07T05:12:00.001Z

Ars Technica recently ran an article "Four things Sony can do to turn the tide for the PS3 in 2009". I don't have a PS3, but I do have some ideas for Microsoft about how they could get more of my money without making me feel like I'm constantly fending off their incompetent attempts to mug me.

No special cables
Feel free to sell special cables, because there are idiots buying $200 HDMI cables right now from the tech equivalent of homeopaths, and they may as well throw their money at Microsoft. But don't screw the rest of us by designing the back of your console so we can't get optical audio and HDMI out without hacking our component cable to bits.

No special hard disks
Feel free to sell special hard disks, but let me attach my own, too. $140 for a 120GiB drive? I can get a 1TiB drive for less than that. I'd have to make an effort to find a drive as small as 120GiB, but it looks like the going rate's about $40. Charging twice that would be cheeky, but three times?

Fix the music UI
Of course, I'm happy with the 20GiB drive drive that came with my 360, because the UI for navigating music is so awful I'd have no intention of putting music on the 360 even if I didn't have a house full of machines with more than enough storage.

Stop charging for Netflix
I'm a Netflix subscriber. I pay Netflix money each month, and they send me DVDs. They also let me watch stuff on my Mac. I can plug my Mac into the TV, and I can play the DVDs on my 360 (or the Mac). So it's not like Microsoft is adding any value there, from my perspective. So why do I have to pay Microsoft an additional annual $50 fee before I can watch Netflix movies on my 360?

I realize it's not a large sum, but you've not done anything to deserve it, so asking for it makes you look bad.

Stop making me wait for demos
When I first got a 360, all you had to do to download demos was sign up. And I thought that sucked. Why should I have to tell you all those details about myself? Great way to put off a casual user.

But it got worse. At some point, Microsoft decided that only paying users should have the demos straight away. Free users should be punished by being made to wait a week.

The advertising doesn't present it like that, of course. Just today, I was told I could get "early access" to some demo. But I know I only have to wait a week, so why would I pay anything for this? Again, there's a distinct lack of class evident here.

Mandate demos
So here's why making me wait for demos is stupid...

When a game first comes out, it's $60. If you're over the age of 12, you'll have noticed that sooner or later, that $60 game will be $50. Then $40. Then $30. And then it's a toss-up between going down to $20 or just disappearing. And these are the new prices I'm talking about. You can also buy used, or rent, or borrow (look for these options to disappear when games leave physical distribution behind).

So let's assume you want to sell at $60. How do you do that? You get people excited. How do you do that? You give them a little taste. Make them want more.

In more reputable lines of business, this is known as "the first one is always free". In video games, it's called "playable demo".

And here's the funny thing: Microsoft has this big list of things you must and must not do to be allowed to sell a game for their console. So you must support "achievements", say, and you must not show erect penises. Stuff like that. And one of the things Microsoft don't see fit to insist upon is "you must offer a playable demo, at no cost, via Xbox live".

Maybe this is to protect games that stink, but I doubt it. And who's to say which games stink anyway? I like FPSes, you like flight simulators, she likes RPGs, he likes driving games. I liked "Fracture" more than "Star Wars: Force Unleashed". I liked "Crackdown" more than either, but – because it was labeled as a driving game (which it really isn't; you can finish the game without driving at all, though you can't get all the achievements without getting in a vehicle or two) – I would never have bought "Crackdown" if I hadn't been able to play a free demo via Xbox live.

There's a related situation playing out right now. Reviews say "Left 4 Dead" is great, despite the illiterate boy-band title. I'm mildly interested. The similar game "Resident Evil 5", which hasn't been reviewed yet because it isn't out yet, has a free demo available via Xbox live. I have played this game, and I am excited about it already, and I'm going to buy it. If the missus plays the demo and doesn't hate the controls too much, I'll buy it even sooner.

"Left 4 Dead"? Left for dead. No demo, you see.

The really funny part is that I almost forgot to play the "Resident Evil 5" demo. I mean, it's labeled "survival horror", and I hate survival horror, but I make an effort to play all co-op demos. How did I almost forget? Because MS made me wait a week. A week's a long time when you're a grown-up with a job.

Speaking of survival horror, I love sci-fi, but the survival horror side of "Dead Space" meant I knew ten seconds in to the demo that this was not for me. And that's a good thing for EA, because I won't buy that game and think "EA sucks". I'll buy "Mirror's Edge" instead, and think "EA's okay".

Don't get me wrong: I know you can lie in a demo. But the essence always seems to show through. Some of the shine was taken off "Mirror's Edge" for me when the guns came out and people started shooting. I just wanted to run about. And the demo for "Star Wars: Force Unleashed" showed me both what I'd like and what I'd hate about the game. "Stranglehold" was probably the most misled I've felt. If the game had been like the demo only longer, I'd have been happy. But the game got harder and more arbitrary until it was just a frustrating chore. The demo was too short to show that.

But demos are a game's best chance of getting me to buy them before they hit the bargain bin.

Look to the future
The DVD is the last read-only physical medium. BluRay may have won the battle against HD-DVD, but it lost the war against irrelevance. Large amounts of local storage are cheap. Bandwidth is plentiful and becoming more so. Start selling me a subscription service now. Games, like movies (and unlike music), are perfect for subscription. Only a hardcore few want to play the same game over and over, year after year. Most people will play a game, "finish" it, and cast it aside. Right now, many will do that in a way – selling it used – that lets other people buy and play the game without anyone involved in making the game seeing any of that money.

So let me subscribe, and let my subscription allow me to play whatever games I want, whenever I want. This is the only reason I'd ever pay for Xbox live. I don't want to play online with juveniles who'll call me a Mexican Jew lizard. I'm insulted by the suggestion I should pay money to be allowed access to demos, the sole purpose of which is to get me excited about paying money to buy the real thing.

But a "Netflix instant" for games? That would put your attach rate through the roof. It would get me hooked on more of your platform exclusives, just as the promise of "Halo 3" and "Fable II" made the choice of which console to replace the original Xbox with pretty darn easy.

(The funny part here is that the missus is living in this world already, happily playing free Flash games in the browser, via our "subscription" to the internet. And that's definitely eating into her 360 time, and the less time she spends on the 360, the less progress she makes in her 360 games, and the less progress she makes, the longer until we buy her next 360 game. I don't know if Microsoft understand yet that it's not just the PS3 and Wii they're competing with; it's MySpace and YouTube and a bazillion websites with crappy Flash games. Hell, you can't even browse the web on the 360, for no obvious technical reason. An internet-connected device in 2009 that can't browse the web?)

Let me skip tracks
Finally, for now at least, one last request. Let me skip the boring bits. As the Ars Technica article said (meaning something different entirely) "remember that games are for fun". Don't force me to keep replaying the same tedious boss battle, let alone the five minutes of tedium between the last checkpoint and the boss battle itself, or the unskippable two-minute cut scene some genius sandwiched between the two, or make me suffer any of the "shitty bits" I keep finding in my games. Let me say "skip this", rather than forcing me to suffer, or, in some cases, forcing me to just give up on the game entirely. Boss whose arbitrary rule you can't work out in a game no-one else liked well enough to write a walkthrough for? Sorry. Game with automatic checkpointing saved your game when you had no health/ammo or were lacking the necessary weapon/item for the next section? Sorry.

Looking back, all of these strike me as reflecting a basic lack of respect for the customer. We're just a bunch of nameless, faceless chumps to be milked (or mugged) for everything they can get. It may be the traditional way of business, but I'm unconvinced it's a healthy way, and I very much doubt it's even the most efficient way of transferring our money into their slush funds.

I suppose I should be grateful the current generation of consoles aren't coin-operated.

Swing Tip: binding an action to a JList

2009-01-24T05:56:00.002Z

I don't use JList much. It rarely makes it past the experimental version of a UI. A single-column JTable (which is one way of looking at JList) just don't crop up much. JList is easier to use, though, and doesn't look too bad out of the box — in stark contrast to JTable.

(JTable has been enough of a pain in my ass that sometimes I'll even go to the trouble of writing a fancy custom ListCellRenderer to avoid having to use a JTable. But those are stories for other days.)

But when I do use JList, I all too often make the mistake of thinking of it as a component to be double-clicked on, not used from the keyboard. You can see from the lack of any addActionListener method that its designers weren't too sure how to interact with it.

Handling double-click is easy, and it's documented in the class comment for JList. There's basically only one way to do it anyway, so I've never seen anyone get this wrong.

But what about handling enter? That's often neglected, and if you search the web you'll find all kinds of bad advice, from addKeyListener to subclassing JList!

What you really want to do is use the JList's ActionMap and InputMap. If you don't have an Action, just an ActionListener, you'll need to fabricate one, but AbstractAction makes that easy. You can just replace this:


  new ActionListener() {
    public void actionPerformed(ActionEvent e) {
      // whatever...
    }
  };

With this instead:


  new AbstractAction() {
    public void actionPerformed(ActionEvent e) {
      // whatever...
    }
  };

You'll need a name, too, because that's how you link the entry in the InputMap with the entry in the ActionMap. Something like:


  final String name = "user-hit-enter";
  list.getInputMap().put(KeyStroke.getKeyStroke(KeyEvent.VK_ENTER, 0), name);
  list.getActionMap().put(name, new AbstractAction() {
    public void actionPerformed(ActionEvent e) {
      // whatever...
    }
  });

Of course, not being a filthy copy-and-paste merchant, you'll wrap that up in one of your utility classes somewhere. And personally, I'd rather deal with ActionListener in this instance, since the actionPerformed method is all I want from the Action, and an Action is-a ActionListener anyway, so the user can still supply a full-on Action if they happen to have one.

The "clever" part of my tip, though, is to basically have your own JList.addActionListener, and have it take an ActionListener but bind it both to double-click (via addMouseListener) and enter (via getInputMap/getActionMap). That way, you get into the habit of providing both by default.

As usual, there's code for this in my handy library of Java stuff linked to from the side of the page.

2009-01-21T05:27:00.004Z

Item 55 of Scott Meyers' (excellent) "Effective C++" is "Familiarize yourself with Boost". Good advice. The trouble is, those crafty Boost buggers keep adding new stuff.

In 2004, for example, they added <boost/algorithm/string.hpp> to Boost version 1.32. I didn't find out until this past weekend when I needed several of the usual string utilities, and didn't have any of many implementations I've written before (or were written for me by other C++ programmers at the various companies I've worked for).

If you've used C++ and Java, you'll know that C++'s std::basic_string is full of orthogonal but mostly useless methods (find_last_not_of, say) and missing most of the really useful stuff you actually want from your strings (no starts_with, say). The sort of string an academic might design.

Java's java.lang.String, on the other hand, is big bag of random really useful stuff (startsWith, say) whose non-orthogonality can drive you mad (no startsWithIgnoreCase, say). The sort of string people who actually write programs would arrive at.

Boost being what it is, you can now have a big bag of orthogonal useful stuff.

So there's starts_with and ends_with, just like Java, but there's istarts_with and iends_with too.

And you don't just get contains, you get icontains too — something I wanted from Java just yesterday.

There are split functions too, including a regular expression variant. There's even a join in case you change your mind and want to put humpty back together again.

There's a full family of nicely orthogonal trim functions. Not that I personally have ever needed anything more than Java's plain old trim, but it brings a gleam to my shiny icicle of a heart to see all the possible variants laid out before me with sane names, neatly arranged like well-designed German surgical instruments designed as a coherent set by someone with a plan. This in stark contrast to the usual slowly-accreted collection of stains in some god-forsaken header file in a directory called something like "common" or "utils", squirted there by a thousand unthinking deadbeat parents who barely remember their progeny themselves.

Boost's case conversion stuff looks a bit dodgy, so you might end up with upset Turks and Azerbaijanis on your hands if you use it (or, by implication, any of the case-insensitive functions) without checking that out first, but otherwise what I used looked pretty sound. (And if you care about languages other than Seventh Edition New Jersey ASCII, you're probably writing something that should be in Java rather than C++.)

Being Boost, these are all templates too, so they'll work just fine with your home-grown basic_string-that-doesn't-own-the-memory-it-points-to, whatever you call it. (I still haven't found that class in Boost anywhere, which is surprising, because I keep tripping over them everywhere else. Google's variant is called StringPiece, which will have British readers rolling in the aisles. The wide-character variant is sadly not called STGaryGlitter.)

Anyway, I'll endeavor not to write my own C++ starts_with ever again, especially on Linux, where it's trivial to add a build dependency on Boost. And if you read this thinking "I knew that already", why didn't you tell me?

Accessing GET arguments with libmicrohttpd

2009-01-19T21:03:00.002Z

The tutorial for libmicrohttpd talks about handling POST arguments, but what if your FORM uses GET? The solution's easy, but I had to work to find it, so this might save someone some trouble in future...

The tutorial contains this example to dump the HTTP headers:

  MHD_get_connection_values(connection, MHD_HEADER_KIND, print_out_key, NULL);

What it doesn't mention is the alternatives to MHD_HEADER_KIND. You don't need to parse the "Referer:" header yourself, as I was about to fall back on; you just need to look at the MHD_GET_ARGUMENT_KIND values. (Or, if it's cookies you're wanting, MHD_COOKIE_KIND.)

Even better, given that you probably know the name of the cookie, GET argument, or header you're looking for in any real code, there's MHD_lookup_connection_value. Here's how you'd get the value of a GET argument called "q", for example:

  const char* q = MHD_lookup_connection_value(connection, MHD_GET_ARGUMENT_KIND, "q");

Don't go putting that straight into a std::string, because libmicrohttpd returns NULL for missing values.

Why am I using libmicrohttpd? Well, that's a story for another day. I'm writing this post because I don't start dealing with the little jobs as soon as they come to me, I'm going to suffer stack overflow.

A History of US Communications Security

2008-12-31T23:00:00.000Z

I just finished reading A History of US Communications Security [pdf], the NSA's "David G Boak lectures" from 1973. (According to the introduction, the lectures were actually given in 1966. I'm also not sure whether David G Boak should be considered the lecturer, or that this is some NSA equivalent of the BBC's Reith Lectures.)

These lectures may have been given a long time ago, but this really spoke to me across the decades:

But do not lose sight of the real world where your ultimate product must be used, and beware of security features so intricate, elaborate, complex, difficult, and expensive that our customers throw up their hands and keep on communicating in the clear...

As did this:

In retrospect, the demise of the obsolescent, inefficient, and insecure systems seems natural, easy, inevitable, and relatively painless. But the fact of the matter is that it is usually quite difficult to get the users to relinquish any equipment once it is solidly entrenched in their inventories — especially if it works well [...]; but even if it doesn't [...] The reluctance to junk old systems stems from a number of causes, I think. First of all, they represent a large investment; secondly, the users have developed a supporting logistic base for the systems, have trained personnel to operate and maintain it — they've used it. Finally, the introduction of a new system is a slow and difficult business requiring new budgetary and procurement action, new training, the establishment of a new logistics base, and – increasingly these days – a costly installation job [...].

And this remains excellent advice in every field:

I suggest that newcomers to the business not jump on board whichever side of [a] controversy your viscera may first direct. Rather, take the other side – whichever it is – and go through the exercise of building its defense. You are likely to be surprised at how elaborate and involuted the arguments become either way and might lead you to my personal conclusion that the best way to achieve a net gain in our [endeavor] is through compromise.

That said, I've often found it interesting to play at being a zealot to see where it leads, as long as you're careful to eventually move on; keeping what was useful, dropping what wasn't, and having a better understanding of the limitations and trade-offs than you would if you'd never pushed the limits to find out where the limits are.

In fact, I'd be rather perturbed to find someone in their 20s who wasn't over-zealous on at least some issues relating to their profession (my assumption otherwise being that they aren't really interested in what they do). And I'm always deeply suspicious of someone older who doesn't, as a matter of course, talk in terms of trade-offs (my assumption otherwise being that they haven't really learned anything but dogma).

The description in the lectures of the US military's repeated re-discoveries of the problem of compromising emissions is interesting in contrast with the global view in Ross Anderson's book "Security Engineering", the first edition of which said:

The first appearance of compromising emanations in warfare seems to date to 1914. Field telephone wires were laid to connect the troops with their headquarters, and these often ran for miles, parallel to enemy trenches that were only a few hundred yards away. These wires used a single-core insulated cable and earth return in order to halve the weight and bulk of the cable. It was soon discovered that earth leakage caused a lot of crosstalk, including messages from the enemy side. Listening posts were quickly established and protective measures were introduced, including the use of twisted-pair cable. By 1915, valve amplifiers had extended the earth-leakage listening range to 100 yards for telephony and 300 yards for Morse code. It was found that the tangle of abandoned telegraph wire in no-man’s land provided such a good communications channel, and leaked so much traffic to the Germans, that clearing it away become a task for which lives were spent. By 1916, earth-return circuits had been abolished within 3,000 yards of the front. When the United States joined the war, the techniques were passed on.

Passed on, maybe, but seemingly not really taken on board until rediscovered by Bell Labs during WWII in relation to the specific technology of their time, and then forgotten again until the cold war, and then... you get the idea.

There's a strikingly pragmatic tone to the lectures, and a clear understanding of the effects of economics and politics on their work, in addition to engineering and scientific limitations. Refreshing to see an understanding that, at any given time, there will be some problems you cannot solve, for one reason or another, and that you shouldn't then waste undue effort in those directions because there will be other problems that you can solve. A kind of alcoholics anonymous creed for security theater addicts.

I guess they don't give these lectures any more.

Speaking of Ross Anderson, if you didn't already know, it's worth mentioning that the second edition of Security Engineering came out mid-2008. I haven't read the second edition yet, but the first edition (which you can read free online by following that last link) was a must-read. "Security Engineering" is the best book on security ever, by far. This isn't just a CS classic, it's a non-fiction classic that everyone should read. The subject matter is sufficiently important, and the coverage sufficiently fundamental and wide-ranging it should be required reading in schools. If you're too tight/broke to buy the second edition, at least promise me you'll read the first edition?

Given how sparing I usually am with praise, you should have a fair idea of just how good Ross Anderson's "Security Engineering" is.

Anyway, if you like Anderson, you'll like "A History of US Communications Security" because it gives some interesting 1970s US military specifics that go well with Anderson's much broader picture and perspective. It would probably be easier to read Anderson's "Security Engineering" first, and reading that book might awaken an interest in security engineering you didn't even know you had. It certainly did for me.

Will stdio outlive us?

2008-12-20T04:19:00.003Z

I'm reading "Advanced Programming in the Unix Environment" at the moment. I'll post a review at some point, when I've finished. Right now, I've just reached chapter 5, "Standard I/O Library", and it's reminding me how much I hate stdio.

I realize it's somewhat unfair to criticize an API that's as old as I am. I think much of Unix has the advantage that it was only ever meant to provide solid primitives, and for the most part that stuff's stood the test of time well. Higher-level stuff (by the standards of the time) like C and stdio have fared significantly less well because our expectations of "high level" have changed so much. It's the continued ubiquity of this nasty turdlet that frustrates me; the fact that not only are people still using it, but that I'm still seeing people misuse it in the same ways they were misusing it 20 years ago, when I first encountered it.

getchar
This was actually where my hatred of stdio began, despite the fact that it's not stdio's fault that so much buggy getchar(3)-calling code got written. The interface is pretty sound, in a Unixy way. The mistake people make with getchar(3) is more a reflection of the environment that surrounds stdio: C.

By various accidents, I've spent most of my life programming on platforms whose char is unsigned. If you've led a more normal life, not involving ARM or PowerPC, you may not even have been aware that C's "char" type can be signed or unsigned, at the implementation's whim.

I had to learn about signed char versus unsigned char twenty years ago, learning C on an ARM-based home computer. The trouble was that the world (especially Herb "gobshite" Schildt's world) was full of code like this:


 char ch;
 while ((ch = getchar()) != EOF) { // BROKEN!
  // ...
 }

This works fine on signed-char platforms, but not on unsigned char platforms because 255 != -1. The solution is to use int instead of char, at least until you've determined that you're not dealing with EOF. (There were other hacks, but they were bad ideas, and they're the reason why, to this day, I know what character is represented by 0xff in ISO-8895-1, even though I've never seen anyone or anything deliberately use that character.)

Could stdio have fixed this? Not convincingly; taking an int* or returning a struct are the only ideas that come to mind. C might usefully have renamed "char" to "byte" and only offered an unsigned variant. For its part, stdio could usefully have deemphasized character I/O. (More on that later.)

getchar versus getc versus fgetc
A trivial thing, but even as a kid the redundancy of getchar(3)/getc(3)/fgetc(3) and their putting counterparts bothered me. I was born a minimalist. The maybe-macro versus definitely-not-macro distinction seemed particularly unconvincing and ugly. Maybe it had made more sense to offer a maybe-macro in the 1970s, I remember thinking.

But the implicit-stdin/stdout variants bothered me too. One of the things I really liked about Unix was the way it treated the console like just another kind of file (and the convention of using the filename "-" to mean the console; I remember being surprised that fopen(3) didn't return the appropriate FILE* when given that special filename, and disappointed that all programs that wanted to support the convention had to do so manually).

Encouraging people to write code that only works for stdin/stdout seemed like a mistake, even in the years before I met more sophisticated stream abstractions.

getchar versus EOF versus error
Before we leave getchar(3) and friends, it's worth remembering that it returns "EOF" both when you're at the natural end of the file and when an error occurs. This isn't necessarily a problem if you're careful with your idiom, looping until you get EOF and then check ferror(3), but people forget that second part all the time. Or they make a mess out of trying to write something fancier.

I've come to wonder if the seductive simplicity of offering a per-character interface isn't also a mistake. There are relatively few programs whose natural unit of input is the character (rather than the line or block) and in retrospect it's starting to look as if we might be better off if the most notable of those, lexical analyzers for programming languages, weren't character-based. (Treating whitespace as insignificant doesn't seem to suit humans particularly well, and definitely leads to whitespace wars.)

And what is a character, anyway? Sadly, there's precious little support for character encodings in stdio. (Though that's not a criticism of the decisions made in the 1970s so much as a criticism of the continued use of stdio.)

fread and fwrite
Like its character-based friends, fread(3) also fails to distinguish between natural ends of files and errors, with the same consequences for correctness. Relatedly, now we're dealing with more than a single byte at a time, have you ever seen a caller of fread(3)/fwrite(3) check ferror(3) and then check errno for EINTR, assuming that having read/written zero "objects" means they've read/written zero bytes? Is it possible to retry an fwrite(3) at the application level? No. At least, not if you're writing objects larger than bytes. (So if you really must use this awful API, be sure to write n "objects" each 1 byte long, rather than 1 object n bytes long.)

ungetc
It was a few years before I was sophisticated enough to be concerned that buffering wasn't a separate concern, but the ungetc(3) function bothered me from the start. The character I push back doesn't have to be the character I read? I can't push back EOF, but I can push back a character after reading EOF? I might be able to push back more than one character, but not portably, and I can't even query the pushback depth?

FILE* fp
And did it ever bother you that the "FILE* fp" parameter always comes last (which would have been annoying in and of itself)... except where it comes first? And that you just have to remember which functions are which?

fgets
Why do I have to tell fgets(3) how big the line is, before it gives it to me? How would I know? Sadly, fixed limits are a grand old Unix tradition (see also: getcwd(3)). They didn't even really give them up the second time around: if Plan 9 hadn't been so riddled with fixed-size buffers, I wouldn't have been forced to learn Perl so I could write scripts that could cope with arbitrary-sized data. At least Plan 9 had mechanisms for doing the right thing, even if they weren't widely used; compare Brdline(3) and Brdstr(3) and ask yourself why the former even exists (and why the worse function got the better name).

Even if you're happy with fixed-size buffers, the way fgets(3) leaves the trailing newline in the buffer must annoy you. Especially given the asymmetry with fputs(3), and the lack of anything like Perl's "chomp". Close your eyes and tell me that just thinking about fgets(3) doesn't bring to mind the obligatory next line that overwrites the '\n' with a '\0'! And didn't it break your C-programming heart that you had to pay for the strlen(3)? I wince even now.

This is what happens, Larry, when you don't offer improved APIs (let alone deprecate bad ones).

I know we don't use gets(3) any more, but it's a real textbook example of convenience triumphing over quality. "Let's get rid of the FILE* parameter, because we encourage that to be stdin anyway, and while we're at it, let's get rid of the buffer length — even though we've no idea how large the buffer is, or how long the line is that we're about to copy into said buffer." Surely even in the 1970s, that sounded like a bug rather than a feature.

But hey, at least gets(3) chomped the trailing newline for you.

Why can't most people even name a stdio competitor?
The final sentence of APUE's chapter 5 – "Be aware of the buffering that takes place with this library, as this is the area that generates the most problems and confusion" – is almost funny. About the only thing the "FILE*" part of stdio got convincingly right was that the arguments to fopen(3) were simpler and more memorable than open(2), and not having an equivalent of creat(2) is less confusing than having the pair. I also think fopen(3) specifically helped distance stdio from Unix modes and flags and file descriptors, which was convenient for the small non-Unix machines of the 1980s. It meant a whole generation of us grew up with stdio but without the originally underlying layer. That and stdio's ubiquity across non-Unix platforms via the standard C library left stdio unassailable.

The immortal printf(3) family probably helped stdio too, but I'm not complaining about them. There's a (good) reason why they get emulated in every language's library, sooner or later. Some languages' versions even address most of the problems of the originals, while retaining the advantages.

Python-enabled gdb(1)

2008-12-13T20:25:00.004Z

I'm not a gdb(1) master myself, and usually only turn to it as a last resort, but I've used it enough to know that it's pretty limited, and that trying to extend it with "canned sequences of commands" was about as pleasant as working with make(1).

Tom Tromey's been blogging recently about forthcoming Python support. So instead of GDB's old joke of "canned sequences of commands" (at least they had the decency not to claim it was anything more than it was), you can now use Python. I haven't used it, but it seems impressive, and a huge leap forward.

Here's a handy table of contents if you're interested:

1. Installing a Python-enabled debugger
2. Writing a new gdb command
3. gdb convenience functions
4. Parameters, and extending require
5. The filtering backtrace
6. Auto-loading Python code
7. Pretty printing, part 1
8. Pretty printing, part 2
9. Scripting gdb
10. Wacky stuff
11. The End

It's a shame to see the new features are Python-only. (But at least they're not Guile-only. Does anyone else remember when that was the "official" GNU scripting language? It's rare to see insanity that pure in public.)

I realize that supporting arbitrary languages is a non-trivial problem, but I'd love to see the Free world at least making an effort to solve this problem. Too often we see Python-only or Ruby-only or C-only or Java-only, where a useful aspect of genuine freedom would be "freedom to use whatever language you/your team are most comfortable with". I feel like applications are being used as battlegrounds to wage inter-language wars. Don't get me wrong: just because I'd rather use Ruby right now doesn't mean I won't need to use Python next year, or that I want to stop you using Java in the meantime.

Putting an end to this form of linguistic lock-in would be a huge leap forward for Free software, not just for GDB.

In the meantime, at the cost of learning Python, it looks like advanced GDB users just got a lot of interesting new opportunities.

"New" Xbox Experience

2008-11-28T23:32:00.002Z

Microsoft pushed out a major software update to the Xbox 360 recently. The "New Xbox Experience" (NXE). Basically a new UI, but some added functionality too.

The main new piece of added functionality from my point of view is access to streamed Netflix movies. This would be sweet, except Microsoft expect you to not only have a Netflix subscription but also to have a paid Xbox Live subscription. Thanks, Ballmer, but no thanks. I bought a 360, and I'm already a Netflix subscriber, and I honestly don't see why I owe you anything. Especially since you won't even let me watch crap off youtube on the 360. As far as I can tell, you want me to find some other way to watch video. Fine by me.

The most noticeable change is the new UI. Neither I nor the missus like it. We both admit that it's mainly because we were familiar with the old UI, see no advantages to the new UI, and both wish we didn't have to relearn shit just because Microsoft arbitrarily moved stuff around. We'd already learned to cope with the old UI, and the transition cost seems pretty high for no discernible gain. Personally, I've given up and reverted to the subset of the old UI that's still available via the guide button.

If you've been streaming mp3s to your 360, you'll be disappointed to find that the NXE is every bit as shit as the old experience. There's no caching of the album/artist/song lists, and it's every bit as slow at building the menus, so if you've got a music collection large enough to be worth streaming, you'll still find yourself spending too much time staring at the "please wait" screen. (Which they did waste time updating, of course.)

Tracks within an album are still "helpfully" sorted into alphabetical order, and there still doesn't seem to be any way to get the proper order, so if you were planning on listening to classical music, or anything where one track leads into another, you're still screwed.

And there's still no on-screen keyboard for iTunes-like filtering. Yeah, I know an on-screen keyboard would suck, but it would suck a lot less hard than One Big List. Seriously, dudes, you can't navigate a multi-thousand item list. Not even if it is sorted alphabetically. Given that the best^wbiggest speakers in the house are connected to the 360, I would probably splash out on the little keypad if I thought I could use it to get a sensible interface. Hell, the damn remote, which I did buy, went from dildo proportions with the original Xbox to full-on police baton proportions with the 360. I use about 13 buttons (play, pause, eject, left/right/up/down, OK, info, a/b/x/y) of the 46 buttons on that thing. By my reckoning, those Redmond bastards could have fitted every letter of my alphabet on the remote and still had room to throw in a few accents to keep the French Canadians sweet.

(The 360's "visualizations", by the way, are an interesting contrast with Apple's iTunes ones. Microsoft has some nice variations on the usual themes, their "water-splashed inky line" one being my personal favorite until it starts spinning, but overall their intent seems much harsher. There's a distinct "bad trip" vibe to some of them. I especially hate the Melting Charred Lizard Corpse and the Spinning Klingon Blade Thing and the Big Staring Eye and I'm not even going to mention the Journey Up The Brown Canyon because it scares the poop out of me. What were they thinking? And where was the adult supervision they so desperately needed?)

But the most obnoxious new feature is their crappy avatars. They're more serious than the Wii's, and less creepy-zombie-pseudo-realistic than the PS3's, but they make no sense. Why do I need to create a bloody avatar when I'm about to play a game? A game where my appearance is chosen for me anyway? (And, no, I don't think Halo 3 would be improved if Master Chief were replaced by a little cartoon white guy in glasses and the nerdiest clothes available. I think this is a rip-off of functionality that made some sense on the Wii but makes no sense in its new home.)

While I found creating an avatar a pointless chore, the missus was quite keen. She likes that kind of thing in RPGs. The NXE was a disappointment to her, though, because the choices were so limited. None of the faces look anything like her. None of the clothing looks like anything she'd wear. It's bad enough if you're male and don't dress like some generic TV "cool teenager", but if you're female you might be surprised to find that there's neither a single skirt nor a single dress to choose from, and you don't have control over the clothing's coloring. If you like a design but don't like the colors, well, screw you. Oh, and if you're female, you might assume you get some control over your avatar's body type, but you don't. You get the body of an androgynous teenage boy, which you can make skinny or fat, tall or short, but that's it. If you were planning on any curves that wouldn't look out of place on an androgynous teenage boy, we regret to inform you that won't be possible.

Doubtless you'll be able to purchase all this crap at some point, which only reinforces my personal feeling of "you bastards!".

My original intention was to be a small white ball of healing light (and, thanks to Guilty Spark, I didn't consider this an unrealistic intention). Unfortunately, Microsoft seems to think that Xbox gamers (a) are humanoid – a not unreasonable assumption – and (b) wouldn't look out of place strolling the catwalk at Milan fashion week — which sounds like more of a stretch.

Why am I bothering to mention the NXE at all? Because I think it's interesting in relation to Chen's "The Old New Thing". This, I think, is a glimpse of what Microsoft would be like if their customers weren't corporations. This is what Microsoft is like when their customers will accept any necessary shit just to play their DVDs/games.

Those who wish Microsoft weren't so tied up with backwards compatibility might want to be more careful about what they wish for, in case it should ever come to pass.

tail -F

2008-11-27T17:11:00.002Z

I've been a fan of Unix since I was a little kid with no access to Unix. I'd read books about Unix, borrowed from the "big city" library I'd sometimes get to visit as a treat.

I've used many Unixes since, mainly Linux these days, but I'm old enough to have grown up on commercial Unixes and Plan 9. And although for years now I've found it unbearably frustrating to be stuck on a system that uses some non-GNU variants of commands (Mac OS, mainly, but I sometimes run into undead Solaris zombies), I'll be honest: this GNU stuff is still new to me.

Unlike many of my peers, it wasn't all the fancy extra options that drew me to GNU userspace; it was the absence of fixed-sized buffers. Those old Bell Labs guys loved their fixed-size buffers, and it was a constant pain in my ass that accidentally pushed me into using Perl: I just couldn't trust pre-GNU awk(1) or sed(1) or whatever to cope with "long" lines. (And don't imagine they'd cope gracefully, either. I'm talking buffer overflows, because who could possibly have a line longer than, say, 512 characters?)

One side effect of my non-conversion is that I'm generally ignorant of GNU extensions.

POSIX tail(1) has a -f option ('f' for 'follow') that does its usual job of showing you the last few lines of the file, but then keeps running so it can show you all new content appended to the file. This, as you probably know, is really useful, especially for log files.

My problem is that I keep restarting a program that creates a new log file each time it runs, with the timestamp encoded in the filename. It helpfully creates a symbolic link to the most recent log file, but traditional tail -f opens a file descriptor and, to implement the -f part of its functionality, keeps calling fstat(2) to see if the file pointed to by the file descriptor has changed size, and if it has, it copies the new content to stdout. This, obviously, doesn't work in my case, because I care if the symbolic link changes too, and want to start following the new file.

I finally got sick of "interrupt, up-arrow, return" yesterday, and was about to write a simple script when I realized I should check GNU tail doesn't already have such functionality. And, of course, it does: tail -F. This re-opens the original path if the fstat(2) shows that the file is a different file (recognizable by an inode or device number change).

Why are both useful? Well, you want tail -f if your path might be renamed or unlinked but you still want to read it (remember that an unlinked file continues to exist as long as some process has an open file descriptor for it). And you want tail -F if your path might be renamed or unlinked or replaced by a different symbolic link and you want to switch to reading whatever file (if any) comes along to replace it.

Personally, I think I'm going to start retraining my fingers to use -F because I think that's my more usual use case. YMMV. And apologies to the youngsters who already knew all this, but I'm pretty sure I'm not the only dinosaur still wandering the earth.

Review: "The Old New Thing"

2008-11-24T04:39:00.002Z

If you waste your time reading crap on the internet (as you clearly do), you've probably come across Raymond Chen's blog "The Old New Thing". You may also be aware that he wrote a book with the same title, and, for all I know, it's something all Windows programmers have read.

I, though, am "one of those condescending Unix computer users".

I've used Windows on and off since 1993, but I've only used Windows out of some kind of necessity. It always seemed to me to be badly designed and obscure and cluttered with legacy crap, and I might try to use that as an excuse for not liking Windows, if it weren't for the fact that you could say the same things about Unix. It may be that the real difference is just that I'm much better versed in the bad design and obscurity and clutter of Unix.

I don't actually believe that, but doesn't matter anyway: the fact is, Windows exists, and Windows is huge. One reason I've never been convinced by the argument that "Linux will win the desktop because it costs nothing, and people love not paying for stuff" is that you don't have to strike up too many conversations with strangers to find that the man on the street considers Windows (and MS Office and the like) to cost nothing either. Software either comes with the computer (and they get a new version when they get a new computer), or, if they're really keen, it comes from work or a friend.

People proudly tell me they copy Windows, which I've always found strange on many levels. But that in itself tells you something about Windows' ubiquity and how it's perceived.

Ignoring Windows is like ignoring religion. You may personally think it's a load of shite, but you can't deny its influence, and that influence alone makes it worthy of study.

Although I sometimes read posts on Chen's blog when they're linked to from places I read, and I was aware of the book's existence, I hadn't been curious enough to go out and buy "The Old New Thing", but I saw a copy on a shelf at work, borrowed it, and read it over this past weekend.

I loved it. It's full of random stuff from a perspective quite different from my own, but with an obvious shared heritage of double-frees and memory leaks and performance problems and so forth. There were whole chapters I basically skipped over (chapter 15, "How Window Messages Are Delivered and Retrieved", for example), but the range of the book is sufficiently diverse, and Windows' influence so far-reaching, that there's bound to be something of interest.

Topics range from memory allocation (and problems all the way from 16- through 32- to 64-bit), concurrency primitives, text rendering, debugging, reverse engineering, to why various Windows features/APIs are the way they are.

I liked the debugging tips slipped in here and there; the kinds of things you learn on the job rather than in class. "The poor man's way of identifying memory leaks", for example, talks about just letting a program leak away, and seeing what the heap's full of. And the fact that the caller whose allocation attempt causes your system's out of memory failure is probably responsible (despite your temptation to assume innocence unless proven guilty). And that the former can help you confirm or deny the latter. These are the kinds of things that aren't usually considered academic enough to be written down, but are still bloody useful.

Sure, you and I know that one, and probably most of the other debugging tips in the book (though there were also common Windows API mistakes, and thus fairly meaningless to me), but we also know people who still don't know this stuff, and we can probably still remember the time when we didn't know it either.

If, on the other hand, you're thinking "dude, get better tools", just wait: one of these days you'll be looking at a customer's production system, or looking at a postmortem, or a system too large or new or small to be able to afford better tools.

The Windows-specific stuff is sometimes a bit annoying, but it's only to be expected in a book like this, and it's not too hard to work out what's going on. For example, it seems like Windows uses the terms "brush" and "pen" for something like (but not exactly the same as) Java2D's Paint and Stroke respectively. It's easy enough to work out, and the (surprisingly few) sections that are obviously of no relevance outside Windows itself are easily skimmed over.

The book opens with a chapter of anecdotal examples of the kinds of lessons the Linux desktop still needs to learn. It's interesting at many points in the book, in fact, to contrast the three camps. Chen usually only talks about the Windows camp, for obvious reasons, and their position can seemingly be summarized as "we started this a long time ago, on really crappy hardware, and we know better now, and try to do new stuff in a way that shows we've learned some of these lessons, but our business is selling OS upgrades, and every incompatible change we make costs us a sale, so we go to ridiculous lengths to avoid ever breaking anything, even stuff that only ever worked by coincidence, or that deliberately went out of its way to fiddle with undocumented internals".

Mac OS is pretty much at the opposite end of the scale, with "we know that you can't make an omelet without breaking eggs, so eggs will be broken if we consider it to be in our greater good, and you get about one year of deprecation and the year after that, your code breaks; we're a hardware company, so we know people will buy the new release anyway, even if only because it comes 'free' with your next machine, but actually, they'll probably buy it anyway, because our main customers are individuals, not corporations, and they like shiny new stuff and don't have any DOS applications written in assembler in 1982 that they've long since lost the source to".

Linux? "Nothing really works very well yet, but it's free and Free, and you can fix it yourself, and it doesn't matter anyway because next week we're going to rip out all this half-working stuff and replace it with stuff that fixes a few things that currently don't work while at the same time breaking things that did work; it doesn't matter because we don't have customers, and our non-customers don't have any software they don't have the source to anyway".

Of course, you never hear these philosophies stated explicitly like that. Apple's too secretive, no-one can seriously claim to speak for "Linux", and I've no idea whether Chen is really representative of Microsoft, assuming it's any more possible to be representative (in the non-legal sense) of a company that large than it is to represent "the Linux community". But Chen has a clear voice, and it's an interesting perspective to me, especially presented through the course of a book, rather than in bits and pieces as I read occasional linked-to blog posts.

If you want just one contrast, try this, from "Why does DS_SHELLFONT = DS_FIXEDSYS | DS_SETFONT" (someone spank the editor who changed == to = there!):

... This allowed people to write a single program that got the Windows 2000 look when running on Windows 2000 and got the classic look when running on older systems. (If you make people have to write two versions of their program, one that runs on all systems and one that runs only on the newer system and looks slightly cooler, they will usually not bother writing the second one.)

This, to me, was a revelation. By "people", he means, of course, "Windows programmers". But how interesting that, from my position outside the Windows camp, I actually did a double-take when reading his parenthetical comment. Surely, I thought, he means they won't bother writing the backwards-compatible version? Let the dead bury the dead, and all that? Isn't "looks slightly cooler" an important selling feature, even for Windows (the OS)?

Try to find a Mac application (that's still maintained) that still runs on 10.2, say, and you'll not have much luck. Try to run the latest version of a Linux app on Ubuntu 8.04 (the "long-term" release that some people will be stuck with until 2010-04) without resorting to building it yourself from source... and even then, you'll probably need to build the latest versions of umpteen different libraries it depends on.

I was well aware that Microsoft bend over backwards to keep stuff working (and I was disappointed that Chen doesn't actually go into much detail on the techniques they use, which was something I would have been particularly interested to read about), but I didn't realize the backwards-compatibility-above-all mindset extended to third-party developers too, or is at least perceived to do so.

Going back to some of the other topics, I was interested to hear both that Windows gets the old "pointer change without mouse movement" problem right, and how, and that it causes complaints (presumably from people who haven't suffered systems that get it wrong), and that no-one appreciates the effort much because the related "pointer change when a component scrolls under the mouse" problem needs to be solved per-application, and often isn't. I've always hated programming for systems that won't even let me get those details right, and it's interesting to hear a Windows programmer ("of all people!") saying how much he hates it when apps don't get those details right.

The weird part is that in the final chapter of the book he mentions several "funny stories" that are actually just annoying bugs in Microsoft software, but which he seems to see as cases of user error. It's almost like the last chapter is written by a different person, that depressing guy you always imagine works at Microsoft, cranking out shit code with a shit UI and with no interest in rising to the challenge of writing something that works well under all the awful constraints imposed on it, but who's just interested in getting all the relevant forms filled and boxes checked so he can move on to the next task he also doesn't care about.

Maybe Chen is that guy, at times, and his editor asked for an extra 20 pages Chen didn't want to write? Maybe we're all that guy at times, though most programmers I've known tend to be good at losing tasks in their to-do lists, because there's always more stuff to do than there is time to do it.

What else did I like? I liked the repeated mentions of the need for a holistic view of performance: that making one thing better at the expense of some other thing isn't necessarily a good idea. There are repeated reminders that it's bad to assume that your application is the most important application in the user's day, every day. (You'd hope this wouldn't even need mentioning, but everyone who's ever used software knows it does.)

Speaking of things that shouldn't need mentioning, I was amused to see an analog of something I first came across in a Studs Terkel interview with a Republican ex-governor of a state like Montana. This old Republican was criticizing modern politicians on both sides, complaining that they'd lost the insight of looking at any new legislation as if it had been brought up by the opposition for the opposition's own use. His point being that at some point, anything can and will be used against you. And if you don't like the look of it from the unfriendly end of the barrel, maybe you ought not be handing it out in the first place. Chen's version is more like "what could a virus writer do with this API?" or "if I was a user, how could a developer make a really annoying application with this API?".

One of Chen's code snippets had me mystified. I'll ignore the use of SIZE_T instead of size_t and assume I don't want to know, but there was a bit of C++ that contained this:


// Since this is just a quick test, we're going to be sloppy
using namespace std; //  sloppy

The code contains three lines that would need an explicit "std::", none of which, with the "std::" added, would be as long as the "sloppy" comment. Bizarre. And a missed opportunity to show a handy trick I learned from Martin Dorey. Don't want to keep repeating std::cout (which were two of the three uses of "std::" in Chen's code)? Do this:


  std::ostream& os(std::cout);
  os << blah;

This is nice even in code that's "just a quick test", but it's better still when that "quick test" turns into production code, and you need to refactor so you can pass in an arbitrary stream. Your code is already referring to just "os" (the usual name for an ostream), so all you need to do is turn the local "os" into an extra argument. Job done.

Returning one last time to things I liked, there was an explanation of why NTFS keeps a Unicode case mapping table on disk, which is something I'd wondered about, but obviously never enough to ask. And the weird code I just mentioned was actually in a section titled after the only thing I knowingly quote from a Microsoft developer, Rico Mariani: "a cache with a bad policy is another name for a memory leak".

Would I buy this book? No. It's not a classic, or a reference work I can imagine I'd keep coming back to. But it was an enjoyable read, and an interesting insight into a parallel universe. Borrow.

Nice one, Adobe!

2008-11-18T05:31:00.004Z

I find it interesting that, though I'm actually sincere, the title sounds like sarcasm.

I'm sure you've already heard because it was big news today: Adobe released an alpha 64-bit Flash player today, for Linux (amd64 only), and Solaris (amd64 and sparc). Here's my "added value": I was stupid enough to try it, and can tell you how I got on.

It works.

I'm watching some of the low-quality video crap that clogs up the intertubes in another window right now, even as I write this low-quality text crap which I'll bung into those very same tubes shortly.

I needed to "sudo apt-get autoremove gnash" before Adobe's player would work (the symptom being that I'd click on FlashBlock's icon, and the space reserved for the embedded Flash player would shrink to nothing), but that's fair enough. Thanks to the sheer number of non-functional Flash players I'd tried in earlier attempts, it was quite a hassle to find and remove them all, but that's not Adobe's fault.

I've never had Flash work on 64-bit Linux before. And not for want of trying. I know people who claim they have had working Flash on 64-bit Linux, but I know more who've had as little success as me, and even the "successful" ones will admit to unreliability.

Installation is as simple as a "tar zxf", a "mkdir ~/.mozilla/plugins", and a "mv" of the former file to the latter directory. If you're a Mac or Windows user, this might sound awful, and in a way it is, but the usual instructions for getting Flash working on 64-bit Linux (a) involve sacrifices of rare animals and (b) don't actually work. And you have to realize that if they'd tried anything more fancy, the users of some Linux distribution you've never even heard of would be complaining that there's no package suitable for their homebrew package manager.

As a sign of my gratitude, next time I'm strolling past their headquarters, and the missus goads me by saying "Adobe" out loud, I'll refrain from my usual response of "wankers that they are, cranking out shite".

Provided, that is, I don't encounter Acrobat Reader or any of their installers in the meantime...

Update: it's perhaps worth mentioning that, despite never having had trouble with Linux audio before, I now sometimes have to quit Firefox before RhythmBox can play music. Alternatively, if I use RhythmBox before Firefox, I get no audio in Flash movies.

Watching DVDs on an Xbox 360

2008-11-03T21:30:00.000Z

So I finally got an HDTV, and games look awesome...
That's not a question.

Why do DVDs look shite?
Because they're still playing at 480p. Your TV probably tells you this when it switches mode. The DVD itself is recorded at a higher resolution than that, though.

Okay, so how do I fix it?
You need the 360 to "upscale" to 1080i. The 360 (if it's new enough to have an HDMI port on the back) contains the hardware to do this, but it won't do it via the component cables you're probably using.

But games are already at 1080i; why not DVDs?
Because Microsoft hate you, and want you to buy a custom Microsoft cable for USD50.

Do I have to buy Microsoft's USD50 cable?
No. All you really need is a standard HDMI cable (which shouldn't set you back more than USD5).

HDMI's selling point is that it does audio and video through a single cable, so that's all I need, right?
No. Not unless you want your audio through the TV's crappy speakers.

But my TV has a TOSlink (optical audio) out; can't I use that?
Not necessarily. Not on a Sony Bravia, for example. Audio that came in via HDMI won't come out via TOSlink, just via the TV's crappy speakers.

Why not?
Because Sony hate you too, because they know you're a dirty thief who probably half-inched the telly off them in the first place.

Don't forget that Sony are not only the leading consumer electronics company when it comes to proprietary connectors and formats, they're a recording company as well. Twice the evil!

So what do I do?
You need to use an HDMI cable from the 360's HDMI port to the TV for the video, and also run a TOSlink connection from the usual component cable in the back of your Xbox 360 to your surround sound system, like you were already doing.

That is: you need both the old cable and the new cable plugged into your 360 at the same time. You'll use the audio from the old setup and the video from the new setup.

Can I connect an HDMI cable and the component cable at the same time?
Electrically, yes. But Microsoft hate you, remember, so they put the HDMI port so close to the port for the component cable's proprietary connector that when the component cable's plugged in,it overlaps the HDMI port.

So what do I do?
1. You give Microsoft an extra USD50 for their stupid cable. This is probably the best solution.

2. Alternatively, if you like voiding warranties and begrudge those bastards in Washington USD50 that would be better spent on booze and lottery tickets, you take a screwdriver to your component cable, lever the big gray plastic case off, and plug the cable back in to the 360. Removing the plastic means you now have room to slip the HDMI cable in while the component cable is connected. You can find pictures on the web, but note that the plastic case has changed slightly so it's not quite as easy any more, but it's still pretty easy. The hardest part is that you'll have been holding on so tight you probably flipped the SD/HD switch on the side, and you'll want to flip that back.

Long-term review: Apple Aluminum Keyboards

2008-10-20T00:36:00.002Z

I mentioned Apple's aluminum keyboards earlier, but having recently bought a couple more, I thought that was a fact worthy of mention. The one I had grew on me and my fingers grew lazy enough that I started to get annoyed at having to lift my fingers so high to clear the tops of the front rows of keys on my work keyboard, or the "spare" keyboard I use with the MacBook Pro if I'm planning on doing some work from home.

The day I replaced my work keyboard, someone came into the office talking about cleaning keyboards, prompting me to said that the best feature of Apple's aluminum keyboards is that (so far) they appear impossible to get dirty. They begged to differ and suggested that the keyboard's best feature is that it looks amazing.

He had a point. I probably wouldn't have bought my first one if they didn't look so damn cool.

So, after a few months of use, the aluminum keyboard's feel grew on you to where other keyboards , and the beauty of its appearance doesn't wear off.

Having two (different) keys labeled "delete", one of which is actually backspace, continues to be retarded, and the numeric keypad continues to be useless to me.

(Speaking of which, the only other keyboard-related development I have to report is Microsoft's SideWinder X6 "gaming keyboard". It's not without a numeric keypad, but it does let you detach the numeric keypad to switch sides, and it's not obvious that you have to reattach it. Unfortunately, although the "editing pad" is fine, the esc key is in the wrong place, there's an extra row and extra column of useless keys, and two enormous dials. Plus the thing's almost as deep as it is wide, very clunky-looking, and may or may not require special Windows-only drivers. Microsoft usually says their keyboards and mice work with Mac OS, but they don't with this one. That may just mean that various features you don't actually want won't work, but who knows? And if they're not interested in telling me, I'm not interested in buying one to find out.)

CACM: no longer sucking

2008-10-11T22:45:00.000Z

Communications of the ACM used to be great. When I went to university, I spent nights reading back-issues, and can honestly claim to have read every single issue. In the early days, even the letter pages were great: people were arguing about how to correctly implement semaphores and whether gotos were harmful. And in the main body of the magazine, others would be showing off their new "quick sort" algorithm, introducing a new time-sharing system called "Unix" or a new distributed packet switching system called, of all things, "Ethernet".

And then, somewhere in the 1990s, it turned to shit. All management and business and "IT" and government projects and sociology and just random, well, shit. So although I became a member of the ACM when I left university (mainly so I could get the SIGPLAN proceedings without finding a university library, and so I could avoid being one of those creepy old obviously-not-a-student guys who hangs around university libraries), for most of the past decade, I've "read" CACM in the distance between my letter box and the kitchen bin.

I still read the SIGPLAN conference proceedings, though, because I'm interested in that stuff, it's peer-reviewed (so even when it's crap, it's at least coherent crap written by people with CS degrees), and it gives me an idea of what research is being done in that sub-sub-sub-field.

CACM, though, wasn't even fit to wipe your arse on. (Glossy paper, you see.)

The past few issues, though, have picked up substantially. I didn't comment on the first one, because even a gobshite like Adam Sandler or Will Ferrell might make a good film once in their lives. But we're on the fourth non-shite issue of CACM in four months, and that's better than they managed throughout the entire 1990s. Hell, you'd have to go back to the 1970s for that kind of quality.

Sure, it's not perfect, but it's a huge leap forward. The "news" is now reminiscent of New Scientist, rather than just being a random collection of six-month-old Slashdot stories. The random articles have managed to be mostly decent overviews of mostly sufficiently technical stuff. And the "research highlights" section, well, that's the lesson they learned from Nature/Science, and it's done beautifully. They get someone who knows the subject write a one-page layman's introduction to a paper, followed by the actual paper, that you probably won't fully understand unless it's a field you know. So last month had Mark Moir (senior staff engineer at Sun) introduce/review a paper on using transactional memory in operating systems, followed by the paper; then Hagit Attiya (professor of CS at the Technion) introducing a distributed algorithm for computing the k-th smallest element, followed by the paper.

Other recent-issue articles have included Adam Leventhal's "Flash Storage Memory" on using flash as another level in the storage hierarchy (rather than a straight disk "replacement"), Mark Oskin's "The Revolution Inside the Box", reviewing the last decade of computer architecture research (not all multi-core, but, yeah, plenty of that). Plenty of papers on transactional memory (which it's been pretty hard to avoid lately anyway, but it's nice to have someone picking out just the good bits). And if human interest stories are more your thing, there's been the heartwarming two-part serial about Don Knuth and his massive organ.

This, the new CACM, is a thing of beauty. This is something you'd actually pay money for. (Which is nice, because I've been paying money for it for just over a decade now.)

What's not good? Well, there's some sloppiness; multiple (different) expansions of the acronym XML in the same issue, for example. (And seriously, if it's as mainstream as XML, does expanding the acronym help anyone? If they didn't already know what it meant, the expansion isn't going to clue them in. Typing "XML" into their favorite search engine will, and if they're incapable of doing that themselves, they're a lost cause.) Not all of the articles are of a high-enough quality, with hints of the bad old 1990s "business wank" CACM. There was some article last month from Michael Cusumano (a professor at the MIT Sloan School of Management) that was both irrelevant and factually incorrect; I gave up before coming to his point, assuming he had one.

Back on the bright side, the 2008-10 issue contains the paper "Scene Completion Using Millions of Photographs" by Hays and Efros that's exactly what I want: something interesting, accessible, and well outside my usual sphere; something I wouldn't have come across on my own.

Hopefully the future contains more of the new stuff and ever less of the old stuff.

If you're serious about CS, and interested in having some vague idea of what kinds of things people are looking at in our increasingly wide and diverse field, I can actually recommend CACM for the first time since the 1970s. I finish an issue feeling like I've learned something, rather than feeling like I'm an idiot for paying for some sociologist or MBA to gibber on about stuff that's not true, not falsifiable, or not pertinent.

The only downside to CACM being worth reading again is that it's yet another demand on my copious free time.

Review: "Effective Java", 2nd edition

2008-10-08T18:19:00.001Z

One of the benefits of working at Google that I hadn't heard about until I joined is that you're both actively plied with books when you start, and given annual credit to buy books of your choice (without the hassle of filing expenses).

Any developer who wanted a copy of the second edition of "Effective Java" was offered one when it came out, and I eventually read through it, and I finally found the time to write down a few notes...

The first thing you'll notice is that the second edition is much thicker than the first edition. The last page of actual content in the first edition was page 232; it's page 315 in the second edition. That might not seem like much of an increase, and there's a way to go before it hits the "weighty tome" territory of something like "Mac OS X Internals", but for me those extra pages take "Effective Java" over the threshold from "slim volume" to "book".

In the years from 2001 to 2008, the Java language has changed a lot. Already by page 16 (item 2), we're faced with a bounded wildcard type. Those of us who've been using Java between the two editions have also changed a lot, and a lot of the advice from the first edition, even if you weren't already following it at the time the book came out, has since become the standard idiom that everyone above a basic level of competence uses. As a result, the second edition feels very repetitive.

I really wish there were two variants: "diffs" for those of us who know the first edition and have been following its advice for years, and "from scratch" for those new to Java (or who don't read much). It's not hard to imagine "Effective Java" making a better website than a book, too. Most of the items, especially if you're familiar with the first edition, are things you can grok from the title, with little to add for the expert.

This is in stark contrast to how I remember the first edition. I remember being persuaded of several idioms by the first edition. Some things like "Return zero-length arrays, not nulls" were things we'd already worked out for ourselves, and "Minimize the scope of local variables" has been something that good programmers have understood since the beginning of time, but I remember it feeling really good to hear a voice of authority say "Avoid unnecessary use of checked exceptions". We all knew there was something wrong, but you had to have a certain stature to stand up and say "the emperor has no clothes", and the guy who was the public face of the collections classes had that stature.

The interesting thing reading the second edition is that it's likely to elicit a raised eyebrow now and again, and remind you that, presumably, there are (new) beginners who need to be told of things we've all been doing since the 1990s.

As an aside, an interesting difference between "Effective Java" and "Effective C++" is that my reaction to the latter was to come up with a set of SOPs that mainly boil down to "don't use C++ feature x". Whereas "Effective Java" was more like "use idiom x when using Java feature y". An interesting consequence of which is that I'm much better able to articulate why I do things the way I do in Java than I am in C++.

Anyway, what stood out for me in the second edition of "Effective Java"?

I didn't know about @code and @literal in JavaDoc; I tend to just write as if comments are plain text, since my API documentation isn't likely to be published anyway, and is more useful to me and others if it's readable in the source form, in contrast to Bloch's "generated documentation readability trumps source code readability". When you're the public face of the collections classes, I guess you have a different perspective.

Bloch's claim that AbstractList.removeRange is "of no interest to end users of a List implementation" made me laugh. And not in a good way. I think that specific API design error has singlehandedly caused more O(n^2) behavior in code I've been exposed to than any other, as people fail to find a bulk remove and use a "for" loop instead. Even if you do know about subList().clear(), it's hard to consider that anything but ungainly. Bulk operations seem more fundamental to me than the idea of a view, I guess, and I rarely have use for views other than because I'm trying to removeRange. Personally, I think that was an attempt to introduce an non-Java idiom into Java, and it didn't really work.

Perhaps because of Bloch's background, much of the advice in the book seems overly biased towards public API. This is understandable, but even if one takes the position that all API should be designed as if it were library API, I think it's hard to argue that you should make any more public API than absolutely necessary. YAGNI, after all, and anyone who hasn't realized that it's far easier to add stuff than to take stuff out or fix stuff, well, that person's obviously not a golfer.

Another interesting contrast with C++ is that, whenever you read a book on C++ you're likely to spend most of your time thinking about how the language could be improved, but reading books like this on Java, you're more likely to think about how the libraries could be improved. It's particularly amusing that the "Effective C++" item "Make interfaces easy to use correctly and hard to use incorrectly" (advice valid for any language, and that ties in to my earlier concern about exposing too much API) has an example "pathologically bad" Date class that's not actually as bad as the real java.util.Date.

I've seen Stroustrup actually use the real Java Date and Calendar classes as examples of bad design on a slide, and I felt it was a shame he detracted from his own point by implying that it was Java's fault. As if you can't write utter crap in any language. As if even good programmers don't have days where everything they touch turns to shite.

As if bad design (like good design) isn't universal.

Java programmers have the advantage of a large and readable library. So there's no excuse for not having seen a large amount of code, much of it high-quality, but enough of it low-quality, or old enough to be using idioms that we've since learned don't work out well. So for an experienced Java programmer, "Effective Java" is a quick read, many chapters of which could be replaced with "use common sense" or "copy the good examples, ignore the bad ones". It's sad to see the number of "obvious" points from the first edition that the JDK still falls down on. "Adhere to generally accepted naming conventions" is one that's getting worse with time as more new JDK APIs arrive using the less popular naming conventions and making them more generally accepted (using "X" and "setX" instead of "getX" and "setX", say, which isn't bad in and of itself, but is just one little bit of extra trivia we have to keep around that adds no value, and is too late to retrofit to the entire JDK). "Include failure-capture information in detail messages" is my personal unfavorite, though it's something programmers in all languages get wrong. Why aren't we teaching "how to write a good bug report" and "how to write a good error message" in the schools? Can you really be a productive member of society without these skills?

The synchronization stuff's been usefully updated in the second edition, and is a lot clearer now. It's a shame that stuff like that, which I'd go so far as saying a majority of Java programmers don't properly understand, isn't visibly treated as being any more important than all the fluff about exceptions and serialization and doc comments and (I'm not joking) the performance of string concatenation.

CopyOnWriteArrayList and its suitability for listener lists was new to me (despite dating back to Java 5). I'll have to remember that. It would be nice to have a new Java tutorial, for experienced programmers, that shows the current best idioms, and explains the rationale. Listener lists are a lot harder to get right than you might hope.

Item 69 settled an argument we'd been having at work. Should you call ConcurrentHashMap.get before calling ConcurrentHashMap.putIfAbsent? Our argument was focused on how expensive the constructor for the to-be-inserted object is. The documentation led us to believe that the implementation effectively does a get straight away, which is true, but the documentation is missing the rather important fact that a lock is taken out for the duration of the call to putIfAbsent. When Bloch presents the get-then-putIfAbsent idiom, he gives this as the motivation, and checking the source shows he's right. I've filed Sun bug 6737830 for improvements to the documentation. The evaluation for that bug is interesting, as it argues back in the opposite direction.

It's nice to see CountDownLatch advertised. That's my favorite low-level primitive.

Bloch suggests using System.nanoTime in preference to System.currentTimeMillis for interval timing, because it's more accurate, more precise, and unaffected by system clock changes. I'd been using System.nanoTime in new code, but given that last advantage maybe it's time (ho ho) for a more thorough toilet-cleaning.

The book ends with a bunch of items on serialization, which was rather surprising to me. I haven't written a (deliberately, explicitly) serializable class in the last eight years. I don't know if that's because serialization isn't a mainstream thing, or because it was eight years ago that I last wrote Java (rather than C++) for a living. Maybe just behind some narrow door in all my favorite bars, Java programmers in red Pendleton shirts are getting incredible kicks from squirting serialized objects at each other. Who knows?

Every time I've touched Java serialization I've come away with the feeling that it's sufficiently broken in a sufficient number of ways that I'm better off just avoiding it. Gangrenous appendages eventually fall off of their own accord, don't they?

If you're a Java programmer, you ought to read this book. It's not going to be a life-changing experience for many. (If you find it is, you probably need to ask the best programmer you know to help mentor you.) This isn't as important a book for Java programmers as "Effective C++" is for C++ programmers. It's also not as important a book for Java programmers as are Goetz's or Wadler's recent books.

But you can't not have read it. Get work to buy a copy and hand it round as each of you finishes. You can read through it in a day, and quicker than that if you're familiar with the first edition and you've been writing (and reading) Java in the intervening years.

Desktop Linux suckage: Index

2008-09-25T05:26:00.002Z

Blogger makes it a pain to read a series if you weren't subscribed to the feed as the posts were made, so here's an index/table of contents for use as a jumping-off point:

(Thanks to Eric Bloodworth for suggesting adding an index. I should have thought of it myself after reading Ian Lance Taylor's series on linkers. Hint: the easiest way to read that series is to increment the number in the URL.)

Desktop Linux suckage: where's our Steve Jobs?

2008-09-22T20:44:00.002Z

[This is a long one, but it's the end of the series. Normal service will be resumed shortly.]

The article that prompted me to write these rants, "Why Free Software has poor usability, and how to improve it", failed to mention Linux's lack of a Steve Jobs. This is an important omission. The Linux kernel has its Steve Jobs. The question we have to ask ourselves is whether it's just a coincidence that the Linux kernel is the poster child for successful Free software.

What does the Linux desktop have? The Linux desktop has nothing. No person, not even a committee. Just the GNOME HIG.

I suggest you take a look at the GNOME HIG, in all its barely-changed-since-2001 glory. In particular, compare its breadth, depth, and quality to Apple's or even Microsoft's equivalents. You might think that the Apple and Microsoft guides go too far. Microsoft's one is especially long. But you could hardly argue that developers of GNOME applications are doing such a good job they don't need at least as much help as Mac and Windows developers. At least Apple and Microsoft seem to acknowledge the extent of the problem.

At least Apple and Microsoft seem to understand that developers need help: developers need better languages than C, developers need higher-level toolkits that do ever more of the stupid work that doesn't distinguish one application from another, and developers need documentation that helps them with the stuff that the toolkit can't. (Note the order in which I mention these things. There's significance to it.)

If you hang around Bugzilla enough, you'll start to notice "need direction from the HIG" as a polite way of saying "will not fix". You may as well rely on Santa Claus to buy your wife an anniversary gift as rely on the GNOME HIG being updated to reflect modern application trends.

Ideally, the HIG would be a strong document and there would be libraries that made it easier to write good applications than to write bad ones, and we'd be using a language well-suited both to that library, and to application development in general.

What we actually have is the GNOME HIG, GTK+, and C. Oops.

For the Linux desktop developer, everything is such hard work that it's not hard to see why no-one ever gets round to the all-important polish. Or why no-one has the agility to experiment until they get their UI right, rather than slapping together the first thing they think of, getting it half working, and being too scared to touch the tower of cards lest it topple before their eyes.

Do Mac programmers have the same problem? No. Quite the opposite. Cocoa does all the stupid crap for them, so instead of everyone writing their own code to save and restore window position (et cetera) programmers and users can just take that basic stuff for granted, and spend development time on the stuff relevant to their application. Their language is their weakest link, being Objective C, but since its major problem is that, when Cocoa doesn't already offer what you need, you end up falling back to C, it's obvious that that's a net win compared to the usual GNOME situation. Objective C also recently gained garbage collection. GNOME has a weak toolkit (GTK+) and a hopeless revenant language (C).

There are GTK+ bindings to other languages, but in my experience, they're fragile, incomplete, outdated, and/or unsupported. The three serious contenders for an alternative to C, namely C++, C#, and Java, are probably all too politically contentious to succeed without causing serious upset. (The amusing part being that most of the "C++ is too complicated", "C# is too evil/case-insensitive/Microsoft-encumbered", "Java is too slow" crap is likely to come from people wholly uninvolved in, and probably incapable of, writing software). I'm hoping OpenJDK might improve matters here, but you'll forgive me if I don't risk holding my breath.

I realizing I've been ignoring KDE (based on Qt and C++), but they're just a somewhat better implementation of an orthogonally awful philosophy. "Treat your users like technology fetishists who want to piss away their lives fiddling with myriad options in an effort to tease out a half-decent combination of settings" in contrast to GNOME's "treat your users like irrelevant cretins".

I do try KDE every few years, but it just brings out some kind of visceral disgust in me, without even the morbid curiosity that, say, a dentist's book of pictures of tooth decay and gum disease can bring out.

And I still refuse to believe that "offer a preference for everything anyone can come up with" is a responsible response to the design problem. I'd rather have a bunch of alternative applications each with their own coherent philosophy than a stinking pile of compromise. Though I have to wonder whether experience isn't trying to teach us that a stinking pile of compromise is all we can hope for from a Free desktop.

The fact that the situation is so much better for developers on other platforms is a reason to take issue with the article's title: "Why Free Software has poor usability, and how to improve it". When it says "Free software" it primarily means "Free software on Linux". Free software badly ported to the Mac sucks too, but that's just Linux exporting Linux's problems, not a problem with Free per se. Free software written by Mac developers for the Mac tends to suck a whole lot less than Free software written by Linux developers.

Did the developers of Adium (a Mac IM client) ever make a change so unpopular with their users, and have the cheek to tell their users that they were wrong to want the feature, causing a fork? No. But this actually happened on Linux: Slashdot: Pidgin Controversy Triggers Fork.

Vienna, a Free RSS reader for Mac OS, has a nice UI; admittedly not quite as good as its closed-source counterpart NetNewsWire, but still better than any Linux equivalent. TeXShop is a really nice front-end to TeX. Apple gives out Apple Design Awards to the Free (or at least OSS) Mac software, and not in any kind of sarcastic way. There exist Free applications for the Mac that are good in their own right.

The fact that all Mac developers are using the same high-quality toolkit directly influences Free Mac app quality, as does the better default choice of application implementation language. The fact that Mac developers can't help but be exposed to large numbers of other apps with high-quality UIs, setting a good example ("a higher bar", in the parlance of our times) has a direct influence on Free Mac app quality.

Seriously, suppose you'd only ever used Linux. Just how high would your standards be?

Which isn't to say that all Free Linux apps suck, or all Free Mac apps rock. But it does help to explain why most Free Linux apps are way down the "suck" end.

The fact that I bring up RSS, and the awfulness of Linux desktop RSS readers, brings me to a final concern. Linux desktop applications are so bad that they push people towards web apps. Which is possibly good for the users, but it's not good for the Linux desktop. And, in a way, I don't think a world of only web apps would be good for web apps; today's web apps are as good as they are because they've been striving to compete with desktop apps on desktop apps' terms (irrespective of the locational transparency, availability/backup/disaster recovery, and other "natural" advantages of web apps).

Driving everything onto the web would be one way to claim victory in terms of Linux desktop usability, of course. All you'd need is a browser, and Firefox is already pretty good.

I'd love to believe we can see things through, but I'm not even convinced we're moving in the right direction. Or any direction. Maybe we're just going round in circles. Certainly these arguments, the points made, the examples given, all sound very familiar.

As I said at the beginning of this series, I've had a Linux desktop, on and off, since about 1997. I've had a Unix desktop, in one form or another, since 1993. I even wrote my own window manager in my youth, when I too had more free time than sense. Assuming that Mosaic was an acceptable browser for the web as it was back in 1993, have we really made any progress on the (non-Mac) Unix desktop? As much progress as Apple or Microsoft has made? You could argue that Indigo Magic (SGI's desktop in 1993) or NeXTStep weren't as Free as what we have now, which is true. Or that both of those Unix desktops were way ahead of contemporary versions of Mac OS and Windows, which is also true (if you ignore availability of applications). But still, the main way in which I feel better off in 2008 than 1993 is that I have a bigger display with more colors. And that's not exactly thanks to better software.

Maybe some day, Mark Shuttleworth will turn up to work and say "I'm sick of all this pointless ugly Compiz crap and having to pretend in interviews that I sincerely think Compiz is in any way 'better' than or even remotely related to what Apple has, or even a sensible thing to pay any attention to. I understand now that transparency or wobbliness or shadows isn't what makes a good desktop: good apps make a good desktop, and from now on we're only going to stick apps on the Ubuntu menu if I can honestly stand in front of a non-fanboy audience and say 'this app works, and works well' and not be jeered off stage; in fact, I'm going to start kicking the worst ones off the menu unless they shape up, and we'll roll up our sleeves and fix the least worst, based on the rulings of a benevolent UI dictator".

And even supposing this happens, where would Shuttleworth find himself a suitable dictator? As the Pidgin developers inadvertently demonstrated, it's hard to find a benevolent dictator in a world of anarchic zealots. Maybe as hard as it is to find an honest politician in the real world.

And even assuming we had a way to be sure we were moving in a positive direction, our tools aren't the kind that help us move fast.

Honestly, I see no end in sight. Gruber's article will be as relevant in 2012, Thomas will write the third version of his article in 2014, and I'll still need a Mac as a crutch for my Linux box on the day I retire.

Desktop Linux suckage: 10 years

2008-09-21T06:11:00.001Z

I often mention Joel Spolsky's article Good Software Takes Ten Years. Get Used To It. It's one of the best things he's ever written. One thing Joel doesn't promise in that article (or any other) is that the ten years it takes to create good software will be ten years of fun. He doesn't promise ten years of interesting code or ten years of great new features. Quite the opposite, in fact.

Most of your ten years will be spent on problems with Danish keyboards on Mac OS 10.3, or fonts on the Thai version of Windows 98, or some bug in some IMAP server that this one guy in Kabul sometimes has to connect to, or just making your error messages clearer, more specific, and more focused on helping the user fix their problem.

One of the big things that Joel explicitly states, because he has to, is that contrary to what we leave university thinking, these "hacks" are the most important bits of code we write. They aren't nasty bits of cruft that we should be marching out into the world to do away with, in order to make the world a better place. They're nasty bits of cruft that we're marching out into the world to add, in order to make the world a better place. They make stuff "just work", even though the hardware's broken. They make stuff "just work", even though the software they have to interact with is broken. They make stuff "just work" for most users because making 80% of your users happy is better than waiting for the perfect solution to come along and making no-one happy in the meantime.

The real world is full of lossage. The very phrase "real world" is practically defined by an admission of the existence of lossage. Industrial quantities of the stuff. But the academics mislead us, wanting to reshape us in their own image, and would have us believe that what the world needs is more beautiful theories and, maybe (and only if you can do it without getting your hands dirty) a partial implementation of the beautiful theory. They can only say this with a straight face because they are, at heart, ascetics. That's why they're in universities, for dog's sake. I'm not specifically talking about desktops right now, but their idea of a desktop is probably a single copy of vi running in a single XTerm, where they're editing some 10-line Haskell program with really clever puns in all the function names.

The presence of puns and absence of side-effects is what passes for quality in their world.

By the time we realize these people were lying, or deluded, or naive, or whatever, it's usually too late. We're too busy to give people what they want or need for free any more, so the Linux world has to make do with what our younger selves thought people ought to want. A hundred half-finished flying machines and not one working toilet.

Welcome to paradise. You might want to consider getting constipated.

There's a picture in Apple's HIG, in the section Prioritizing Design Decisions. I'll describe the picture in case the link breaks, or you just can't be bothered to follow it. The picture shows three blocks stacked atop each other. The largest block, on the bottom, is "Minimum Requirements". The next block up, somewhat smaller, is "Features users expect". The top, smallest, block is "Differentiation". The idea being that you start with the big block on the bottom, and work your way up.

Again, no genius. No "rocket science". Just common sense and good engineering. (If you actually read that chapter of Apple's HIG, you'll find that what they're actually talking about is very pedestrian stuff indeed. But at least it's an ethos.)

Problem: Linux desktop apps don't care about "Features users expect" or "Minimum requirements" at all. They may offer some "Differentiation", but not necessarily. Mostly, you just get "The bits that are fun to code".

I said Joel didn't promise the 10 years would be fun. There's something else he didn't promise. He didn't promise that 10 years would be sufficient. Necessary? Yes. Sufficient? Clearly not.

Desktop Linux suckage: a prime example

2008-09-16T04:30:00.001Z

I've said before that, in my experience, filing bugs with Ubuntu seems more effective than filing them upstream, and a while back I filed a minor bug against GDebi. GDebi is a GUI .deb package installer that's pretty cool: it effectively lets you click on a link to a .deb in Firefox and install that program all without touching the command-line. The bug was "Size" field in "Details" tab should show KiB units), an I included a patch, and got it accepted and applied as quickly as any bug relating to the Linux desktop I've ever filed.

If you follow that link now, you can see that someone's just pointed out that, no, when apt-get(1) says "kB" it's not lying, it's completely serious. A quick apt-get install and check of /var/cache/apt/archives suggests they're right, at least as far as reports of download size are concerned. (It's harder to measure installed size, but I've no reason to doubt the claim.)

What I do doubt is the sanity of the person who made this design decision in the first place.

And while I accept that my patch was incorrect, and GDebi now claims file sizes are KiB rather than the kB they actually are, I also doubt the sanity of the person who actually thinks we should display file sizes in kilobytes rather than kibibytes. (Especially considering Nautilus, the GNOME "file manager" does actually use KiB, even if it calls them "KB". Especially considering the recent class-action lawsuits against the drive manufacturers. But I'm preaching to the choir here.)

Any time I try to kid myself that one day we'll have a non-crap Linux desktop, what makes me lose hope is that the Linux desktop isn't the kind of place where stupidity gets fixed or even papered over. There's no-one in charge to tell people to get their act together, and everyone has their own agenda, and half of them ought to be in a padded cell with no access to a keyboard.

At the moment, we have a situation where some applications are using kilobytes, some are using kibibytes, and what these applications appear to be claiming to use may or may not correspond to what they're actually using. And all this breakage surrounds something that's trivial in every way. The home computer operating system I was using in the late 1980s had a library call that you gave a number of bytes to, and it gave you back a string to show the user. Those old guys weren't geniuses, they were just good engineers.

The Linux desktop has neither, and it's also lacking a benevolent dictator to at least herd the monkeys. Or whatever it is one does with monkeys, other than spank them.

It's not like we're talking about something as "hard" as globally respecting a date or time format preference. That's so far in the Linux desktop's future that at this rate, I might actually have retired before it happens.

Desktop Linux suckage: a quick exercise

2008-09-12T20:39:00.001Z

One claim you'll sometimes hear is that, although Mac OS or Windows are better for stuff like word processing, spreadsheets, presentations, reading mail, drawing, painting, cataloging and touching up digital photographs, games, and all those other things that only real people do, Linux is best for hard-core nerdy shit.

Coincidentally, I just so happen to have a hard-core nerdy desire, one that, by this theory, Linux ought to do way better on than Mac OS? Yeah, it's an unscientific sample size, but it still might be instructive, and it's an actual question I want answered right now...

The problem
I want to know how many DIMM slots my motherboard has, and what DIMMs I have fitted.

Apple's solution
On the Mac, I click on the little Apple icon, I click "About this Mac...", I click "More Info..." and System Profiler starts, letting me see any detail about my system's hardware and software, including a convenient detailed display what DIMMs are in which slots. It even lets me save a handy .txt summary for inclusion in bug reports and the like.

Linux's solution
On Ubuntu, on the best-avoided crock that is the "System" menu, I have both "About Ubuntu" and "About GNOME". Neither contains a single piece of useful information. System Monitor, which I always have running because I like visual feedback telling me whether my computer's actually busy, waiting for I/O, or just ignoring me, has recently grown a perfunctory summary of my hardware that's not really useful yet. Maybe it'll improve, but I'd rather see a useful "About" menu item rather than rely on a panel applet that isn't there by default. If I go to someone else's system, I don't want to have to start installing stuff to help answer their simple hardware questions.

So I start a terminal and cat /proc/meminfo, but the information there is too high-level for my present purposes. I tab-complete on "ls", thinking there might be some equivalent of lspci(1), but the promising-looking lshw(1) is another dead-end, telling me only the total amount of memory fitted. Okay, screw guessing about on the command line. I know there is a command, because I've used it before, but I can't remember it, and I was supposed to be using the desktop anyway.

I start Firefox and search the web for "dimm equivalent of lspci" and a few variations before I stumble on "lspci ram" on Yahoo (I never did find anything relevant via Google, even with the search string that works with Yahoo). This gets me a handy Gentoo page (naturally) Detecting your Hardware which suggests hardinfo(1) which, once installed, gives me a GUI which is indeed pretty similar to the Mac's System Profiler. The memory information, though, is /proc/meminfo in a GTK+ text area. The authors seem to have been more interested in benchmarking than hardware inventory. I mutter something about Gentoo users, uninstall, and go back to searching.

The only other GUI I found was sysinfo(1) which was almost like someone's joke entry in a "worst ever Linux software" competition. When it starts, the UI is animated, but doesn't actually show anything that has any meaning. There's a preference dialog, mostly taken up with a preference to turn off the pointless start-up animation. (And don't get too excited by my use of the word "animation". I use it in the loosest possible sense. Think of Windows XP's lame bouncing arrow pointing to the "Start" button.)

sysinfo's status bar constantly shows the date (in some random localized time format that I certainly never asked for). If you expectantly click on "Memory" in the list of options (decorated for some obscure reason with the icon usually reserved for copy-to-clipboard menu items), you see two sliders showing the free/used fraction of "Memory" and "Swap". There's a hopeful-looking disclosure triangle labeled "More details", but that just shows "Cached", "Active", and "Inactive", which I know by now are just the next three lines of /proc/meminfo. Looking to see who was responsible for this travesty, I noticed that the Close button in the about box doesn't work, which I thought was a particularly nice finishing touch to the encounter.

Failure
At this point, I resigned myself to either not knowing what DIMMs I have fitted, or opening the case to take a look.

I don't present this anecdote with any particular interpretation in mind, nor because the specific example is particularly representative, but it should at least serve as a warning to anyone foolish enough to think that "at least Linux is okay for the nerdy stuff". No, it's really crap there too. We have too many half-baked attempts at writing the same few things, and almost nothing that's followed through to a useful "shippable" point.

We're stuck with our 1980s XTerms and our 1970s Emacs/Vim (or our IDEs written by people paid to work on them by commercial vendors) and our compiler, which alongside the Linux kernel is one of our few real successes. Like the Linux kernel, though, many commercial vendors pay people to work full-time on it. We kid ourselves if we think that these are fruits of the "community" of bedroom hackers. Those are the people churning out the the half-baked "utilities" that had more time spent on their about box and their start-up animation than on their usefulness or usability.

On the internet, no-one knows you're 14 year old or a raccoon. Until they use your software.

Postscript
David Bristow reminded me of dmidecode(1) which was the command-line program whose name I couldn't remember that does provide the relevant information though (as David admits) in somewhat hard-core form. Harald Koch explained how to get lshw(1) to do the right thing: "sudo lshw". He also suggests "sudo lshw -short" which provides a much more convenient overview style of output than I'd seen before. To be fair, when not run as root, lshw(1) does say "WARNING: you should run this program as super-user". But it makes the common command-line mistakes of (a) outputting this first rather than last, (b) continuing anyway rather than exiting or insisting on some kind of --partial option before doing the wrong thing, and (c) going on to produce so much output that the warning scrolls off the top of the display or is lost anyway. Sadly, no-one cares enough about command-line usability these days to teach the lost art.

Playing more with lshw(1), it turns out to have a companion GUI called lshw-gtk(1), but the interface is truly awful. You know NeXT's awkward column view interface style that's been slowly dying out in Mac OS because it just doesn't work well? lshw-gtk(1) goes to some effort to recreate that, complete with useless top levels and a requirement that you double-click rather than single click to see a node's children. The Debian package also causes it to hide itself in System > Preferences (surely Administration would make more sense?). Despite these problems, it's still the least worst GUI solution I've seen. The individual panes are really nicely presented if you're persistent enough to navigate all the way down to them. And the "you really need to run this as root" warning is a lot stronger and does actually make you click a button before it'll do the wrong thing. But honestly, my recommendation would be "sudo lshw -short" in the terminal emulator of your choice.

Desktop Linux suckage: Rhythmbox

2008-08-30T20:31:00.001Z

Less important than a web browser, but probably next on the list of mainstream fundamentals is a music player. It's the only other thing I run daily that I haven't already mentioned. Even handy stuff like Wireshark (which gets an exemption from criticism in this series because it's by nerds for nerds) only gets run once a week.

I've said in the past that Rhythmbox is okay, but that was before I was using it as my only mp3 player, rather than as an occasional stand-in. In practice, well, it has a large number of small but annoying flaws. I'm not going to mention them all here. Just a representative sample.

Let's start near the top of the UI: Rhythmbox uses a slider to show how far through the current track you are. This has the annoying consequence that your clicks are interpreted as meaning "move one 'page' in the direction of the mouse", rather than "move to the time corresponding to the location of the mouse". This is completely useless; it might make some sense if clicking to the left of the knob meant "back 10s" and to the right meant "forward 10s", but the time skipped being proportional to the length of the track makes it useless.

I could ignore the fact that a standard slider looks wrong, but it's hard to ignore that it feels wrong. Only, this being the Linux desktop, it gets worse. For some reason, if you actually try to drag the slider, it has a habit of locking up playback. The work-around seems to be to switch to another track, and then switch back, and start playing from the beginning, having learned your lesson. Absolute positioning in the track just doesn't work, so the lesson is "don't touch the slider". Awesome. Anything that works that badly should probably just be left out of the interface until the first version where it actually works. Not having the option of spoiling playback and annoyed by a control that doesn't move how I want it to would be less distressing than just not having any pretend control at all. (GNOME bug 437987 and GNOME bug 445764 may be this.)

An alternative improvement would be to disable the slider, making it evident that it's only useful for showing how far through the track I am. (Funnily enough, I found a bug suggesting that the slider be made worse rather than better: GNOME bug 389170.)

An annoyance that's interesting for what it says about how Linux lowers one's expectations is that there's no visible suggestion that the "media keys" on the keyboard work, and I consider that a problem. The stupidly large toolbar (GNOME bug 343660) and the menu mention the more awkward keyboard equivalents; stuff like alt-space for playback (an annoying difference from iTunes' unmodified space, for no obvious reason, even though unmodified space isn't taken and would make iTunes refugees feel more at home at no cost to anyone else), but there's no suggestion of the media keys.

I realize it's deeply sad that I can't just assume they work, but this is Linux, and it wasn't many releases ago they didn't work. And it's worth bearing in mind that I had to write a custom rc.local script to get my keyboard to work at all with Ubuntu 8.04! (I actually installed KDE and tried amaroK, and note that my media keys don't work there.)

The search field is in the wrong place, too. Where do search fields go? In the top-right of your window. I've made this UI mistake myself in the past, but it's a well established convention now. Firefox uses it (for searching the web, rather than the current document), as does Safari. iTunes uses it. Ubuntu's Add/Remove Applications uses it and Ubuntu's help browser uses it. Microsoft's developer documentation for Vista explicitly says "thou shalt". But Rhythmbox just kind of hides it in the middle of a bunch of other UI clutter.

(GTK+ apps in general make very poor use of empty space; the widgets seem to enforce large amounts of arbitrarily wasted space around themselves, and developers seem, perhaps in reaction to this, to use it only rarely in ways that actually help convey meaning. Mac developers tend to be best at this, perhaps because Apple over-exaggerated in early versions of Mac OS X to get the point across. Good Mac apps tend to have far fewer controls and preferences, of course, which gives developers more room to play with.)

Did I mention that the window manager's stupid "top-left corner" behavior is exacerbated by Rhythmbox not explicitly repositioning itself where it was last? GNOME bug 502827. (Sad, of course, that individual applications need to work around systemic lameness, but remembering your last window position is a good basic feature to have anyway.)

There's advertising in the UI, too. In the latest version, two net music stores have appeared. I thought one advantage of not using iTunes would be to not suffer that crap, but I was wrong. To be fair, judging by the text that disappears before you have chance to read it (and how do you get it back?), Jamendo is actually a "store" for Free music; Magnatune not. But it's unclear, and feels like thin ice that really needs better explanation. Unlike iTunes, the "stores" aren't very store-like, either. They look just like your normal music collection, which isn't a very accessible way to browse unfamiliar music.

But what drives me mad the most (worse than even the slider) is the overall window behavior. When you start Rhythmbox, an icon appears in the top panel, which generally signifies some kind of "background" application; the kind of application whose window you can close without thereby quitting the application, because you can click the panel icon to get the window back. But no, clicking the close icon quits Rhythmbox. No more music. You can click the window's minimize button, but that still wastes space in the list of windows at the bottom of the screen.

You can get better behavior by clicking on the panel icon, funnily enough. If the window is visible, clicking the panel icon (somewhat bizarrely, and with visual feedback to emphasize the idea) minimizes the window "into" the panel. Clicking again re-opens the window, as you'd expect. But if you've just been fiddling in the window, which you probably have if the window's visible, you're focus is on the window and your pointer is in the window. Nowhere near the panel icon. So it's way more convenient to close the window, but closing the window, despite the existence of the panel icon, quits Rhythmbox and stops the music. This is GNOME bug 158168 from 2004, which the developers don't think is a bug.

It would appear from GNOME bug 523072 as if Rhythmbox is equally evil if you're ripping a CD at the time. I can imagine that's even more annoying!

I came up with a patch, ran a patched version for a few weeks, and was much happier. But what's the use of a patch when the developers think the current behavior is not just acceptable, but good? The fanboys who say "patches welcome" whenever someone complains miss the point; patching is necessary but not sufficient. Realistically, you probably have to fork, and get your distribution to package your fork instead. And that's not something you can do on a feature-by-feature basis. You could try to file the bug against your favorite distribution instead, and put the maintenance burden onto the package manager, but that really doesn't seem like a healthy way forward.

Am I really destined to end up running Gentoo? Building everything from source and applying my own little set of patches on my local machine? I think I'd rather buy a Zune.

At this point, I should be honest and point out that I simply don't know what iTunes does if you close its window. Mac OS solves the underlying problem in a more fundamental way: command-h gets any application out of my face in a single keystroke. (My complaint there is that, by default, there's no visual indication that an application is hidden, which I find confusing, especially in combination with the way Mac OS' dock shows some non-running applications too. There's a hidden preference to fix that, though, and it's about the only bit of manual configuration I have to do on a new Mac.) It would be great for Linux to have a similar system-wide fix, but this is Linux, so there's no "system" (hell, there's a whole other desktop I haven't mentioned that's the default in some distributions, and there's a third semi-popular one), and Linux has no Steve Jobs insisting that stuff is not allowed to suck.

[I am reliably informed that iTunes does the right thing. Imagine my surprise.]

So given all those Linuxy lemons, you'd be well advised to have your application know how to make tasty lemonade.

Did I mention that Rhythmbox never remembers its window location? This isn't just a problem if you quit and re-start. It's a problem if you click the panel icon twice to hide and then re-show the main window, because that resets the window position to the top-left. Seriously.

Did I mention that if you quit Rhythmbox (which, as I've said, is easier to do by accident than it ought to be) it forgets what you were playing? I'm not asking for fancy stuff like remembering scroll bar positions and search field contents and multiple selections and the like: just remember which track was playing last, please!

Did I mention that the mis-positioned search field isn't activated if you use control-f (or, if you think of it more like Firefox's "Web Search", control-k)? The former does nothing (unless focus is already in a component that has some default behavior for the keystroke), while the latter is "Play Queue as Side Pane". I'm not sure, but I don't think "Play" is a verb in that sentence.

Did I mention that canceling the search field doesn't keep the currently-playing track visible in the table? I realize the "right" behavior is pretty hard to specify if you want to cover all the special cases, but there's value in doing the right thing in the most common case.

Don't get me wrong: I actually quite like Rhythmbox. It's almost good. And I do think it's a case where a bit of "ronco spray-on usability" could make the difference. I bring this up because I don't want people to think that there isn't low-hanging fruit out there. The little panel icon? It's great. I love its little notifications when the track changes, and I love that I can turn it off if I want, and I love that I turn it off in the place where it appears. All nicely done. I want to use it more, and I want Rhythmbox to stop coming across as if it thinks it's the most important app on my desktop, when 99% of the time that little panel icon is all I need to see. If listening to music were my primary occupation, I wouldn't be running Rhythmbox on Linux; I'd be running the iPod software on an iPod.

But back to my patch. My experience has been that, contrary to what you might imagine, sending patches to the most-upstream source doesn't work well. Perhaps because most of my patches fix things so obviously broken that the only reason the patch is needed is because the developers are either actively hostile towards, or simply indifferent to, their users. The distributions tend to be a better place to go. Sure, they apply patches that make your SSH insecure (Debian) or your Perl slow (RedHat), but at least they get things done.

Anyway, I went to launchpad, Ubuntu's bug database. And I found Bug #38512 in Rhythmbox: "Window close should close, not quit". Reading that bug's comments we learn a great many things. I could summarize the thread, but really it's so educational I think everyone who's interested in the state of the Linux desktop should read the whole thread themselves.

I'll wait.

If, having read that thread, you want to see the alluded-to tantrum by one of the upstream developers, you should read the upstream bug Bug 318629 - Option to minimize to tray on close or minimize (rhythmbox). (Note that a developer calls a user a "dick" before a user calls a developer a "dick".)

Everything that's wrong with the Linux desktop is beautifully illustrated in those two bug reports.

Problem: everyone wants users until they've got them. When they've got them, they want to reshape them into their own image. When that fails, as it always will, they begin to hate their users.

Desktop Linux suckage: an interlude

2008-08-23T20:18:00.003Z

Before you sink too far into despair, let me say: not everything is hopeless.

Firefox is mostly okay. I use Firefox 2 and Firefox 3 on different machines on a daily basis, and I'd be hard-pressed to tell you why you need to upgrade. I can name a few things that are different, but I don't know that they're necessarily better. But Firefox 3 isn't so much worse that I'm cursing the mother that bore it, and that's pretty good for a Linux desktop application major-version upgrade.

The GNOME System Monitor doesn't seem to change much, especially considering how often it's mentioned in the release notes ("now with extra Cairo!"), but it's every bit as "okay, I suppose" as it ever was. I think whoever writes the release notes either needs to concentrate more on stuff real people would care about, or, conversely, has an excellent feel for their audience.

The weather/calendar/date/time/world-clock applet (yes, all those things really have merged into one) is equally, well, "okay".

I use Terminator as my terminal emulator, and it's worth pointing out that it runs better on Linux than on any other platform. Fundamentally, Linux is a good platform to develop for. I use Evergreen as my editor and it, too, runs better on Linux than on any other platform. But, important though they are to me, those are two applications by developers for developers. I'm not presenting them as examples of good Linux desktop applications, suitable for the mainstream. It's okay to expect their users to understand regular expressions and know how to write scripts and all that, and their users are people for whom "Python scripting interface" or "rewritten in Objective MonkeyPoop 7.3" might actually be a feature (the kind of crap you see all too often on the web pages of Linux music players, say).

Terminal emulators and IDEs, despite what we developers might think, aren't core parts of the desktop that everyone from beginner to expert needs to be able to use, and they're not apps it's important to come to a "good enough" consensus on. They're sovereign apps that the people who use them use day-in day-out for years. These people are prepared to devote significant time and effort into learning how to best use them, because it pays off, and they know it. Everyone has their own sovereign application. Yours might be Photoshop. His might be Word. Hers might be Keynote.

I fear for all of those people, but the Linux desktop is years or decades away for them, so maybe it doesn't matter. The people we could usefully reach are the people for whom the web browser is the sovereign application.

Most people wouldn't know a terminal emulator or an IDE if it bit them, and that's as it should be. I don't know how to drive, you don't know how to spacewalk, and that guy creeping up behind you with the knife isn't a hairdresser. These are specialist skills, often requiring specialist tools, corresponding to sovereign applications. We do all know how to open doors, climb stairs, and use toilets. These are non-specialist skills requiring no specialist tools, and correspond to things like calculators and mp3 players.

Notice that it's depth rather than subject area that's important; a calculator isn't a sovereign app, but Mathematica is. An mp3 player isn't a sovereign app, but Logic Pro is.

There's another simple rule of thumb for distinguishing the two kinds of app, and it's this: if Apple or Microsoft will sell you an application separate from their OS, it's a sovereign application. If they give it to you for free (including new versions; the first iPhoto is always free, just to get you hooked), it's not a sovereign application.

Funnily enough, one of the most frequent problems with non-sovereign applications on Linux is that they think they're sovereign applications. More about that in the next post.

Desktop Linux suckage: Evolution

2008-08-22T20:17:00.004Z

Ah, Evolution. Humans are known for allowing hope to triumph over experience, so although I've been public about my dislike of Evolution (the crappy half-finished mailer, not the scientific theory) since the beginning of this blog, I still give it another quick go with each release of Ubuntu.

Assuming the version that shipped with each Ubuntu has been different each time, and you'd certainly be hard-pressed to tell at times, that's at least six versions I've tried. And all have sucked.

I'll gloss over the experience of trying Evolution version crap.useless to save myself the anguish. Evolution continues to be so appalling that the future for webmail would look as bright as ever amongst Linux users, even if the general public weren't webmail-mad. Despite competing in a category of applications that's practically defined by its members' lameness, Evolution is so beyond hope that I won't waste any more time or any more breath on it.

If you think I exaggerate, you don't even need to install and try Evolution. You can learn a lot from the Evolution FAQ:

Why cannot I reset word wrap setting for outgoing mails at 72 characters?
Sorry, the value is hardcoded.

Can I change/personalize the "On <date>, <person> wrote:" string when replying?
Currently you can only change this if you compile from source yourself.

In fact, you can probably get a pretty clear picture by just reading one FAQ question's title: "Does a version for Apple(R,TM,C,whatever) Mac OS(R,TM,C,whatever) exist?" There are two other FAQ entries with the same childishness surrounding "Microsoft", "Windows", "Sun", and "Solaris".

I think that tells you all you need to know.

Don't get me wrong: I think having a vision or an ethos is important, and it's a good and wise decision to not add every single feature anyone on the internet can think of. At the same time, I think it's even more important that you communicate the vision. Especially if one of your philosophical decisions has become a FAQ. That's people's way of telling you they don't understand what the hell you're trying to say. (And to be perfectly clear, though this shouldn't need to be said: that's your fault, not theirs.)

According to their web page "Evolution provides integrated mail, addressbook and calendaring functionality to users of the GNOME desktop". That sounds pretty inclusive to me. That says mainstream to me. That says "we are to the GNOME desktop what Mail.app and Outlook are to Mac OS and Windows". Not "we will give smart-arse answers to perfectly reasonable questions".

Somehow, this "fuck you" attitude manages to carry over to the UI, which would be a neat trick if that were actually useful.

There's Thunderbird, too, but that's like Evolution for people who like more knobs and switches. (Plus the handful of people who actually miss the built-in mailer from late 1990s versions of Netscape.)

I wonder how we got here? Once upon a time, not long ago, Unix was the only place you'd find sensible mailers. They were terminal-based, of course. One of them (mutt) even survives, in some of the more beard-overgrown corners of the internet. Various people had already taken us from terminals to GUIs, giving us nicer (and more) fonts, and eventually giving us HTML mail for amazon and netflix and newegg (and spam). But fast full-text search? Good local caching to cope with high latency/low availability connections? Great editors? Nope. We even took a step backwards on that last one, giving up $EDITOR. Not a problem for the mainstream user who doesn't have a favorite editor, but perhaps one reason why the people who might actually write the code have little inclination to do so. "You want me to use Emacs/Vim to write a program, the use of which would mean that I can't use Emacs/Vim as much? Yeah, sure; I'll get right on that."

As regular readers know, I'm neither an Emacs nor a Vim person, but nevertheless it was actually the crappiness of Evolution's editor, and of Mac OS' Mail's editor (in more recent versions; I think they improved HTML editing at the cost of plain text editing) that made it easy to give up both. If I'm going to have a crappy editing experience, I may as well take the browser's crappy editing experience. For one thing, I have to live with editing so many other things in the browser that I may as well learn its quirks. More importantly, unlike most of the Linux desktop, Firefox actually tends to improve as time passes.

I have yet to become a creationist, but I have abandoned Evolution, and I am happily using webmail for all my mail needs; an idea that I would have considered to be one of the direst of heresies in the 1990s.

Problem: no-one who could write one wants a decent desktop mailer for Linux, so there will never be one.

Desktop Linux suckage: the clipboard

2008-08-21T20:38:00.002Z

X11's equivalent of the clipboard has been broken since I first used X11, back in 1993. 15 years later, things are still as bad as ever they were.

They say hard cases make bad law, and terminal emulators make very hard UI cases. Unfortunately, nerds being nerds, a terminal emulator tends to be the first application written for any Unix GUI. There are two main problems caused by starting with the terminal emulator and generalizing to the other 99% of applications. The more fundamental problem is that terminal emulators expect that most keystrokes can be passed through to the pseudo terminal, including the keystrokes that every other application on your system uses as keyboard equivalents for menu actions.

The X11-specific problem is that XTerm conditioned many long-term Unix users to use the selection instead of the clipboard. (If you're not an X11 user, you probably have no idea what I'm talking about here. Don't worry; we'll get to it.)

The big problem with the X11 clipboard is actually nothing to do with terminal emulators, except in so far as if they'd actually written some real apps instead of guessing what they might be and how they might behave, they probably wouldn't have crippled the clipboard in the way they did.

The easy one first, though. Mac OS uses a modifier key for menu actions (the "command" key) that didn't exist on traditional terminals, cleverly side-stepping the problem. PuTTY on Windows basically does without; a not unreasonable solution. GNOME Terminal uses control and shift (instead of just the "control" key). Terminator uses alt, which used to be popular on Unix, but fell out of favor in Linux times, thanks (I've always assumed) to the influx of Windows users.

As for the second problem, you may or may not know that Mac OS actually has multiple "pasteboards" (as usual, even their terminology is different). Mac OS hides them well enough that real people neither know nor care. Real people using Linux, even if they only use Firefox, get screwed by the old "selection" versus "clipboard" nonsense. Basically, in addition to the usual clipboard with its explicit "copy" and "paste" actions, there's a "selection". To set it you just select some text. To paste it, you press the middle button. (These days, the scroll wheel.) To paste it over existing text (such as in your web browser's location bar)... well, you can't do that. It's roughly that mistake that screws people over.

I see this catch people out at least once a week, and that's amongst X11 users savvy enough to simply shrug, mutter something along the lines of "bloody clipboard", and try again more carefully. As long Linux has no Steve Jobs to stand up and say "this is hurting us, so out it goes", I don't see this getting fixed. Anyone could write the patches to remove X11 selection support. But without a Steve Jobs standing over the relevant projects' maintainers, how could we ever get them accepted? The gatekeepers we have are the XTerm-toting Emacs users who sincerely believe they couldn't live without this shit.

(KDE offers an option to ignore the selection, but I believe it defaults to the old behavior. My work-around is to just never use middle-button paste. It's a lot easier to kick the habit than most old-timers probably imagine, and it makes your muscle memory more portable to other systems.)

This isn't actually the worst part, though. The funny thing about that nonsense is that it's more likely to affect nerds than real people, because real people aren't too likely to come across the selection by accident. In fact, they're likely to give up on Linux before they get that far.

More serious is that there isn't a clipboard in the sense of a central place that stuff gets copied or cut to. The way it works is that it's more like a token that gets handed round. So if you copy in Firefox and paste Rhythmbox, what happened was that at copy-time Firefox said "if anyone wants the contents of the clipboard, they should ask me", and at paste time Rhythmbox asked "who has the clipboard contents?", was told "Firefox", and asked Firefox "could I have the clipboard contents?". Which is all nice and efficient, saving unnecessary copying... until someone copies, quits, and tries to paste. Now the application with the data is no longer running, so it's gone.

This is not how a clipboard should work.

I don't need to tell you that this isn't how a clipboard should work, of course. You have common sense. You're not a paranoid engineer engineering for the worst case (of copying a 2GiB MPEG movie onto the clipboard) and completely ignoring the more common case (of copying 28 characters of text onto the clipboard). Unfortunately, the guys who got here first were paranoid, unrealistic, and broke the system to "protect" it against the possibility of ill-behaved applications.

(Java, despite its Solaris/X11 heritage, comes across as if it believes in the Windows model. So in a Java application, you can't really avoid the copying; the JDK will do it for you. You just don't get any benefit out of it. The lesson toolkit designers seem unwilling to learn, and this applies to modern-day criminals as much as the X11 80s beard crew, is that you can't force applications to behave the way you want. So forget about "all". How do you get 99% to behave well? You give them sensible behavior for free. 99% will take the default behavior, and the other 1% were lost anyway. Fearing and focusing on the 1% makes you about as smart as the TSA, and almost as annoying.)

Problem: we're still living with bad design decisions from the 1980s, and there's neither the will nor the power to get them fixed.

Desktop Linux suckage: Preferences^WAdministration

2008-08-20T20:38:00.001Z

The management of Preferences on the Linux desktop sucks, too. Or do I mean Administration? Although I know enough Linux users that I can find a defender of almost every bad UI choice, Ubuntu's Preferences/Administration is roundly acknowledged to be a clusterfuck. Especially by anyone who's used a Mac, which does to preferences pretty much exactly what search engines (rather than the early 1990s' "directories") did to the web, making it easy and quick to find what you want if you type what you know into a single text field. But even something as basic as Windows' Control Panel would be a step up.

On the Linux desktop, the user has to decide between "Preferences" and "Administration", and then between, say, "Keyboard" and "Keyboard Shortcuts", or "Network" and "Network Tools", or "Appearance", "Screen Resolution" and "Screensaver". They're being compressed slowly; "Appearance" contains what used to be multiple separate craplets, and instead of "just" two keyboard options, we used to have separate "Keyboard Accessibility" and "Keyboard Layout" choices. But there are still way too many choices to make, and each one is an expensive, painful choice that requires closing the newly-opened window and starting again from scratch if you choose wrong; not just hitting a back button. If you're going to make me guess, give me the fewest possible number of chances to guess wrong.

Better still, don't make me guess my way through your needlessly over-complicated UI at all; let me tell you what I know, and take me straight to the right place, or offer a short tightly-focused list of the likely possibilities if there's more than one.

Apple even goes as far as translating Windows terminology into Mac terminology, in an effort to be more helpful to strangers. Too many Linux users, if not developers, actively hate strangers, especially ones from the lands of Mac OS or Windows.

I count 40 choices on Ubuntu 8.04's "Preferences" and "Administration" menu. Even alphabetical order loses its usefulness when you don't know what word they'll have used. Or whether you're even on the correct 20-item menu.

Who thought that arbitrarily splitting "Preferences" and "Administration" was anything but a crutch to support the bad idea of an unworkable overload of piffling little applications? Stuff that doesn't make sense on a menu, and wants to be in a unified System Preferences or Control Panel application, depending on which background you came from. I'm sure someone, at some point, actually thought that lots of tiny programs was a better idea than one big one, but is it not time they admitted that it doesn't actually work out in practice, and we fixed this?

And don't get me started on the utterly useless crap that's the rest of the "System" menu!

Problem: too many cooks, no benevolent dictator.

Desktop Linux suckage: window management

2008-08-15T20:00:00.001Z

If we were talking about Mac OS or Windows, we wouldn't be able to talk about window management. The rest of the world compromised on good-enough solutions years ago, and it hasn't been a problem for them since. On Linux, though, we value our freedom to choose our own window manager more than we value the ability to get useful work done.

I should admit up front to my own past sins in this area; I wrote a window manager (lwm) once, but I stopped maintaining it years ago. I haven't even used it since the end of the 1990s.

At some point I grew up enough to realize that any advantages to be gained through choice of window manager were marginal, and just not worth the cost of being outside the mainstream.

Unfortunately, every few years, the Linux desktop gets a new default window manager. Well, the GNOME Linux desktop, anyway. I wouldn't know about the KDE Linux desktop, or any of the other alternative Linux desktops, but I doubt they're immune to this problem. The fact that none of the choices is as good as Mac OS or Windows' sole choice makes the problem worse, as does the fact that although it's easy to do the first 20%, no-one really wants the 10-year slog to quality, so we start again with a brand new 20% solution.

Nothing really important changes, but the title bar changes from dark blue to light blue, or the close button moves from the left to the right or back again, or the underlying scripting language [you didn't know or care it had] changes from Lisp to Python or Ruby, but never to a generic scripting interface so people can use whatever they like, because those responsible are either not bright enough to realize there's no One True Language, or are too authoritarian to really value any choice but their own. Anyway. That's a separate complaint.

The fact is, that's all you can expect from a change of window manager. Something trivial and unimportant to those who just want to use their computers rather than fiddle with them.

Sure, if you're lucky, there will be one or two minor noticeable improvements, but there will definitely be a whole bunch of regressions (stuff that used to be better and now sucks) that we spend the intervening years until the next regime change arguing about whether the changes are regressions, or whether they're actually improvements, and all the users are just too stupid ("unsophisticated", if we're feeling polite) to understand that we're changing random shit for their own good.

The best part is that, like all software, window managers have bugs. And some window managers are widespread enough, and some applications are caused enough trouble by those bugs, that work-arounds find their way into applications. And you can guess what happens if the window manager ever gets fixed or, more commonly, overthrown.

There was no regime change this year, though one's brewing. First some good news. Some kind soul finally fixed the long-standing problem where a new Firefox window wouldn't get the focus, because of some internal GNOME focus-related turf war no sane user really wants to have to know about. Excellent! I don't know whether it Firefox or GNOME, but someone finally budged, and the desktop is now slightly less annoying for it. In fact, in this respect, it's now back to where it was before the last regime change. Sweet!

That's all the good news I have, though.

Sadly, the current default window manager has a habit of opening every new window in the top-left corner of the display. It's okay for the first couple of windows you open, while there's still blank space on the desktop for it to place windows in without them overlapping, but as soon as a new window would have to overlap an old one, the genius window manager just starts to dump them all on top of each other in the top-left corner.

That's an awesome heuristic. I'm surprised no-one's ever come up with that one before. It does, after all, have the advantage of being zero lines of code. Even if it is pretty much always a bad choice.

This behavior is particularly annoying for me because I have a 30" display, so the top-left corner is on average a long way from where I happen to be. (And even on a 30" display, you can't fit many Firefox windows side-by-side before the screen is full.) If the window manager is going to dump all windows in the same place, how about the center, to be closer on average to my current mouse position?

Searching the web informs me that lots of other people hate this too, and that the fanboy consensus is that you should switch to Compiz (likely beneficiary of the next regime change), which, if you also install and run some special configuration program, has an option to put new windows in the center. A choice which still isn't quite as clever as the Windows or Mac OS heuristics, but would be a damn sight better than dumping everything in the faraway top-left.

The trouble is, Compiz and Java don't get along in a variety of ways (some of them Java's fault, though if there weren't so many X11 window managers, the JDK wouldn't be so full of crufty code to work round their various bugs). And did I mention that Compiz seems to make windows annoyingly hard to grab the edges of, making resizing from the edges hard? And moving from the title bar is hard too, because there you have the opposite problem, where it mistakenly decides that you're trying to resize rather than move? And sometimes clicks on the window frame go straight through to the window beneath? And it causes inadvertent focus-stealing with some apps? And that its wobbly-windowed stupidity makes me apoplectic? (And makes me fear that, yes, it is important to have a paymaster cracking a whip and saying "do useful stuff, not stupid but fun stuff".)

Don't get me wrong, I like the way windows fade out of existence when you close them, I quite like the shadows now they've toned them down, the alt-tab behavior is acceptable, the new window "zoom" effect is okay, and the excessive transparency of title bars is reminiscent of Mac OS 10.1, though I do doubt that the Linux world will learn as quickly as Apple did that it's not such a great idea. (You'd think that one advantage of follow the rest of the world's tail lights would be that you'd see them learn from their mistakes and not blindly make exactly the same ones. You'd be wrong, though.)

Luckily, Compiz is still an option in Ubuntu 8.04, and not yet the default, so at least someone is showing signs of understanding it's not yet of shippable quality. But unless they intend to keep a tight leash on the dog of an interface that's coming our way, a load of things that are working fine will break when the switch comes, and the countdown will start on the next rewrite. That would be par for the course.

Problem: if you keep starting from scratch, you never get far enough to really polish, you waste effort re-doing the bits that were never a problem, and you cause more work (and problems) for everyone else, too.

Desktop Linux suckage: DNS caching

2008-08-10T19:50:00.011Z

Ignoring all the things that I knew 8.04 outright broke for me going in to this, the first new thing I noticed was that Linux coped a lot less well with Comcast's utterly craptastic DNS servers than Mac OS did.

Judging by my quick experiments, about one in ten DNS queries just doesn't come back from Comcast's DNS servers, so the waiting application just sits there, looking back at me with an unblinking stare until it finally times out, and then fails. I'd noticed this problem before, but not much, because I was still doing most non-coding on the Mac.

I didn't investigate why the Mac copes better, but the always-truthful internets support the obvious assumption that Mac OS and Windows XP have local DNS caches. Presumably Apple and Microsoft got sick of lame ISPs making their OSes look bad. I was certainly quite surprised that my Ubuntu box was doing no caching at all by default (though some applications do their own, to work around this at the level they have some control over).

I installed dnsmasq for local caching, and shortly afterwards switched from Comcast to OpenDNS when I saw no sign of Comcast reacting to Kaminsky's widely-publicized DNS flaw. (Read Kaminsky's slides if you haven't already, even if you think you know what they're going to say. There's a lot of interesting stuff in there.)

The consequence of not caching by default is that Ubuntu doesn't handle real-world shitty ISP DNS servers as well as Mac OS and Windows do. I don't know if this is better in other distributions; a quick web search certainly shows that Linux users have a wide variety of DNS caching alternatives to choose from. I don't know why one isn't running by default.

Passing on DNS lossage to me, the end-user, was the most visible problem when I switched to only using Ubuntu, because it was relatively crippling. You get sick of Firefox's "Looking up" status text very fast. This is a problem because DNS is pretty deep magic to most people. Sure, you can explain it's like "a phone book for the internet", but I'm unconvinced that really helps get the idea across, and more to the point, who's out there explaining DNS to the masses? And why should anyone but a sysadmin care about DNS anyway? Especially when "here's a nickel, kid; get yourself a Mac OS or Windows license" would mean they don't have to.

We can fix this. We even have at least three solutions ready-written and raring to go. We just need someone in power to flip the switch.

Problem: the inmates are running the asylum, and even assuming you can get them to accept there's a problem, it's very hard to get them to agree on a solution.

Ignoring for the moment the question of whether Linux has the applications people need, not coping with common ISP problems is the kind of low-level stuff that makes the difference for most people. This is the stuff that keeps people stuck on (or drives the brave ones back to) Mac OS or Windows. And I'm not just talking about people so many of us feel comfortable looking down upon; I'm talking about people like you and me. Not having the time to track down problems like this has kept me from giving up my Mac for months. Demanding all your users have the ability, inclination, and time to solve technical puzzles is setting the bar pretty high. Almost everyone I know (geeks included) lacks one or more of those things.

And before you blame Comcast rather than Ubuntu, on the basis that it's Comcast's DNS that sucks, bear in mind that they have hundreds of thousands of customers you'd like to have as users. If you think it's only Comcast's DNS that sucks, you're sadly mistaken. Also bear in mind that ISP DNS servers are not going to get fixed. Not ever. As long as Mac OS and Windows users are unaffected, there's simply no motivation for them to do so. And none of these users are going to switch to Linux, as long as it "breaks the web" for them.

Desktop Linux suckage: introduction

2008-08-08T03:30:00.021Z

Windows 2000
I remember around the year 2000, or whenever it was that Windows 2000 actually came out, I thought Microsoft were doing pretty well. Much as it pained me to say it, I also thought Windows was the best OS available. At the time, Windows was without question the best place to run Sun's JVM, Netscape was running aground, and IE was picking up steam to become the best web browser of the time.

Microsoft really looked like they were getting their act together. Each release of their OS had been convincingly better than the previous one, and Windows had reached a point where not only could I bear to use it, I actually quite liked using it. Outlook Express was fine for my mailing needs. Windows Media Player was fine for my mp3 needs (I had about five mp3s in those days, and as I recall, mainly used my MiniDisc player, a technology you youngsters probably don't even remember). I used my own editor, though it wasn't called Evergreen in those days, I used Sun's JVM, I used the application I was working on, IE was a nice fast web browser, and thus all my computing needs were covered.

I have always been a man of modest desktop application needs. Right now, I see Firefox, Evergreen, and Terminator. Rhythmbox is lurking, System Monitor is running, and that's it. Oh, there's a 24-hour digital clock in the corner.

But back to the year 2000. I'd been using Linux and Solaris on my desktop for the previous five years. Out of love, rather than because either of them was necessarily a good choice, let alone the best choice. Solaris was slow and, in practice, required over-expensive and under-performing hardware. Linux ran faster (even on the same hardware) but was often unreliable, and didn't have a decent JVM, which meant I couldn't actually get much work done.

As a user, too, the situation on Unix sucked. There was no software to run, decent or otherwise, and even the web browser, an application category invented on Unix, was falling too far behind the actual web. Netscape Navigator was bloating up with useless and ill-conceived crap, rather than actually improving in ways relevant to meeting IE's challenge.

So somewhat reluctantly, when a dual-processor Compaq machine (I told you this was a long time ago) running Windows 2000 appeared, I voluntarily became a Windows user. I was instantly happier and more productive than I'd been for years. I convinced myself that Microsoft had turned a corner, and still think of those days as Microsoft's "golden age", but in retrospect, it looks more like it was a temporary blip caused by the combination of Microsoft's successes and their competition's failures.

The stench of commercial Unix rotting from the inside out had just started to reach my nostrils, Linux still wasn't ready unless you had more free time than sense, and Mac OS was still the crappy 16-bit home computer OS from the 1980s rather than the "Unix for the rest of us" it is today.

What happened next
We all know what happened next. Apple reinvented Mac OS, and the web turned into a dangerous place Windows just wasn't ready for. Commercial Unix died, with the arguable exception of the server room. Linux? Well, Linux just kind of muddled along. It didn't get much better, but it didn't get much worse. It was fine for development work, but a pretty lame desktop. Many Unix lovers (and haters, because it's quite hard to be one without also being the other) settled on a Linux box for development work and a Mac for everything else.

That sums up how I saw the pragmatic developer/user's best choices in 2004, which is when John Gruber wrote Ronco Spray-On Usability, an article recently indirectly slashdotted via Why Free Software has poor usability, and how to improve it.

If you only read one of those articles, read Gruber's. It's the best-written and it's especially interesting because despite being written in 2004, it could have been written yesterday. It's also worth reading his follow-on article from back then, which turns out to say much of what I'm about to say.

Sometimes, you're done
If you've read my reviews of the last few versions of Mac OS, you may have noticed some disenchantment. Not that Mac OS is actively getting worse; it's just no longer getting much better, for me. And the few minor ways in which it's getting worse manage to cancel out the improvements to the extent that, for example, I've stuck with 10.4 on my MacBook Pro at work, despite (or because of) the fact that I've used 10.5 at home. The truth is, 10.3 was pretty much good enough. Good enough to make it hard to add USD120's worth of value.

If you're not familiar with Mac OS, but are familiar with Microsoft Office, you've probably known a similar feeling with Office "upgrades". The few people I know who've claimed to like a version of Word, for example, tend to like the oldest version they came into contact with. Everything since then has seemed superfluous to them; more trouble than it's worth. We know many developers/managers fail to realize that good software takes Joel's ten years, but I think even more have trouble accepting that there does eventually come a day when the software is "done", and most changes you can make from that point will make the product worse for as many people as they make it better.

The arrival of that day presents a major problem for a commercial entity; it's seriously non-trivial to come up with a significantly better way of doing something. Since check-as-you-type spelling checking, for example, what's really made a significant difference to word processing? Word processing is "done" (for now), and Office 2008 is living proof that Microsoft can't afford to accept that. Mac OS, seemingly, is "done" (for now, on the desktop), and Mac OS 10.5 is living proof that Apple can't afford to accept that.

[My dad just called to tell me I'm ignoring Time Machine, which is his new favorite thing. Maybe he's right, maybe he's not. The fact that Mac OS (or Word) is good enough that we can even seriously have this argument is telling. We could hardly be having the same argument about desktop Linux, could we?]

Amusingly, though "done day" would be a time to rejoice rather than worry for Free developers, there's no real sign we're getting anywhere close to that day, despite optimistic pronouncements to the contrary. Sure, Linux has got better since 2004. As a developer, I no longer want to use anything else. OpenJDK even has me actively excited about further improvements in the near future. Life on Linux is sweet for me, as a developer. But as a user?

Ubuntu 8.04
Non-developer progress has been slow, and patchy, and sometimes seems to go backwards. I'm starting to wonder if somewhere between Ubuntu 6.06 and Ubuntu 7.10, we didn't see Linux's "Windows 2000". Ubuntu 8.04 is a drag. So much stuff that used to work no longer works, I may as well be using 64-bit Vista (or whatever it is that's supposedly causing Windows users pain at the moment). I hear people arguing at work about whether Flash works or not. Audio. DVD playing. CD burning. Sleep/hibernation. Apple keyboards.

I've had problems with all of the above, minus audio. Despite audio being one of those in-jokes amongst Linux users, I've never actually had any trouble with it. (Though come to think of it, I've only ever even bothered to try on one of the ten or so Linux machines I've used in the past decade, so maybe that doesn't mean anything.) All of these things (minus Flash) worked in 7.10. All (minus Flash and sleep/hibernation) worked in 7.04. I had high hopes for 8.04. This was obviously going to be the one. A "long-term support" release (where "long-term" means 3 years) coming 6 months after a really strong release. What could possibly go wrong?

I upgraded, and a couple of months later, I stopped using my home Mac. Time, I thought, to make Linux my OS of choice not just for development.

It's interesting to reflect, at this point, upon the fact that I eagerly upgrade my Linux box every six months. Because the new stuff is so great? No. Because the old stuff is so bad. There's always something that's causing me enough pain I want to upgrade to make it go away.

And that would be fine, if upgrading didn't always bring new pain. In the next few posts, I'll talk about how it felt to finally switch to Linux for all my desktop needs.

Making JTable look more at home on Mac OS and GNOME 2

2008-08-07T03:20:00.002Z

Back in 2006, I explained how to make JTable look more at home on Mac OS and GNOME. Since then, Mac OS 10.5 added support for the smaller-sized combo boxes I bemoaned the lack of back then, and I updated the code in Subversion to take advantage of that, but didn't make another post.

At the beginning of July, Rob Ross mailed me to say he'd found a work-around for a problem visible in the screenshot in the original post: the unwanted focus ring around the scroll pane. In his own words:

I discovered that this method would add a Border to the enclosing JScrollPane if the scroll pane's border was null. Well, I had already explicitly set that border to null in an attempt to get rid of it. But when the JTable becomes displayable, this method gets called, and re-installs the default focus border.

So all you have to do in your code is re-set the border on the scroll pane to null, and that focus rectangle goes away!

I was my usual slow-to-answer self, but despite my accidental rudeness, I got another mail from Rob later in July with a follow-up:

I have been informed by some Sun Swing guys that the "canonical" way to handle this is to set an EmptyBorder on the [JScrollPane] when it's created; that way a border will not be added ... and no overriding of configureEnclosingScrollPane() is required for this particular issue, although you would still need to override it to get the other effects you're implementing.

The trouble with the Swing guys' suggestion is that, although you avoid having to subclass JTable, you have to fix every JScrollPane you use to contain a JTable. And as Rob rightly says, I still need to subclass JTable to fix the other problems.

Given that, I've committed Rob's first suggestion. See the latest src/e/gui/ETable.java for the whole thing.

Review: Apple aluminum keyboard

2008-07-30T04:39:00.002Z

I realize keyboards are a personal thing, but I've liked Apple desktop keyboards for years. I liked the one that came with my PowerMac G4, and I loved the different one that came with my PowerMac G5 so much that I bought a bunch of those keyboards for home and work. At the same time, I've never liked Apple laptop keyboards. I didn't like the Titanium PowerBook G4 keyboard, and I don't like the MacBook Pro keyboard (even if the backlighting is kind of cool).

I did like the current MacBook keyboard, though. But it was still a laptop keyboard, and I was a bit dubious when Apple switched their desktop keyboards to be the same. I was mildly disappointed that there's no option to have a keyboard with the editing keypad but without the numeric keypad. (Gosh-darn accountants ruling the world!) And I was especially disappointed by the false dichotomy between wireless/small and wired/large.

The enticing feature, though, was that the new design reversed the fatal flaw of the old one. Gone was the clear plastic well designed to collect crumbs and hair and whatever other detritus your personal desk is most plagued with. In its place: nothing. Keys so low-profile against the metallic base that there's hardly any key there. Given that I've proven unable to stop the missus eating at the computer, and key switches don't work too well when gummed up with chocolate, as my thankfully rarely-used f1 key will attest, I really liked the sound of a keyboard that would stay clean.

Plus it looks cool.

But it also looks like, well, a pretty crap keyboard. I know I praised the MacBook style of keyboard, but I praised it as a laptop keyboard. And we all know that "laptop" is the fancy word for "computer where every component is both crappier and more expensive than its desktop counterpart".

So I was skeptical.

At first, I hated it. And I hated myself for buying it. And I hated Ubuntu for having a kernel that can't cope with it (see Beware Ubuntu 8.04, which I still don't particularly recommend). A guy at work showed me I could disable the eject key (the trick being that it's System > Preferences > Keyboard Shortcuts, and not System > Preferences > Keyboard; did I mention that I hate Ubuntu's crappy preferences/administration recently?). That was an important step because I was regularly hitting the eject key instead of delete. My fingers were still assuming, I think, that the keys had some height to them, which makes it easy to overshoot and hit the next key along, in whatever direction your finger is traveling.

Funnily enough, I don't make that particular mistake any more. I know this because I've since set eject to open a new terminal as a test, which would be annoying, but not as annoying as opening my DVD drive tray, and it just hasn't been a problem at all.

The keyboard takes slightly less space on the desk, but the gosh-darn vestigial numeric keypad still gets in the way of where the mouse ought to be. If they insist on keeping that particular spare prick around, they could at least put it on the left, where only lefties like that Hitler fellow would be inconvenienced by it. The volume-control and eject keys have moved to be over the main part of the keyboard, though, so I still hold out some hope that next time Apple will find their stones and bury it out back, like they did the floppy drive before it, and the parallel printer port before that. But right now, you're gaining very little free space. About a key's width off your old keyboard. If you've got space problems, you'll continue to have them, I'm afraid. (Unless you go for the small keyboard, but then you lose the editing pad and the USB cable. I'm not putting batteries in a device that has no reason to ever move, and I still don't much like editing on a laptop, even if it is a MacBook.)

The feel is okay, but I don't really think it's as good as the keyboard it replaced. Appearance-wise, it's great, and it's holding up well against cruft, though the bare bits of aluminum are getting tarnished. I like the back, play/pause, and forward buttons. They're handy. More useful to me than f7, f8, and f9 ever were. The expose and dashboard buttons and brightness buttons are useless detractions, though, like the corresponding functionality.

Keyboard feel, though, as I said at the beginning, is such a personal thing that I can't usefully comment on it. You're not me, so you'll have to try one yourself. If you like the MacBook keyboard, you'll probably like this. If you don't, you probably won't. It does feel better after a while than it does initially, but I'm told the same is true of hammering nails into your dick, so I won't laugh at you if you go with your initial "this keyboard is kind of crappy and the keys have no travel" reaction. In terms of resistance to dirt, this keyboard is way ahead of its predecessor. I just wish they hadn't changed everything else too.

The real question is what I'll do if I ever stumble across the perfect keyboard. Good feel, easy to keep clean, good-looking, all the useful keys and none of the useless ones... Stockpile, I guess. If only someone could mate the Apple aluminum keyboard with the Happy Hacking Lite2!

Review: "What is Your Dangerous Idea?"

2008-07-27T19:46:00.003Z

When I was a kid, my dad would buy books he thought he ought to read, popular science mostly, read the introduction, and then hand them to me. This let him get back to watching the horse racing, and ensured I'd be quiet until at least the four thirty-five at Ascot.

These days I'm more likely to be "handed" a link to a website he hasn't read. The past couple of years he's sent me a link to www.edge.org, seemingly after Radio 4 has hyped it up. Both times I've followed the link and decided that it wasn't the kind of thing you'd want to read on the web and forgotten about it. Turns out, though, that they've been making dead-tree snapshots too: "What We Believe but Cannot Prove" and "What is Your Dangerous Idea?". I came across the latter in a surprisingly good bookshop in Santa Cruz called, judging by the receipt, Bookshop Santa Cruz. The bookshop was interesting in that it wasn't a Borders/Barnes & Noble, didn't have a built-in coffee shop, and wasn't the wall-to-wall new age crap you might expect given the hippies and acid casualties that litter the streets outside. In fact, the CS books (and I actually mean CS, not "iPod for Cretins" and the like; a book on software transactional memory sticks out in my mind) had print-outs of slashdot reviews poking out of their pages.

That's one of those cases where I'm really not sure if it was done as a joke or meant seriously, but on reflection it's pretty cool either way.

Where chains often have "staff recommendations", there's no sign of any vision; just random books that random people have liked for their own random reasons, and that doesn't help you all much because you're probably not all that much like this particular store's random collection of staff (unless you're an arts student working part-time in a bookstore), even though you may have tastes very similar to one or two of the individuals, and definitely have tastes exactly like some subset of the customers.

This is one reason why netflix recommendations rock and amazon recommendations are mostly spam; netflix does a great job of clustering me with people like me, and amazon offers me 32 different editions of the same damn book. Ironically, netflix doesn't obviously make any money off making awesome recommendations, since they're a subscription service, where amazon could make a ton of money off me if they'd actually advertise stuff I might actually want. I guess what netflix gets out of it is that they give me a strong reason not to switch to Blockbuster, despite the fact that there's a Blockbuster right outside my front door, and they get free word-of-mouth advertising like this.

I don't know if small bookshops are going to survive, and my guess would be they won't, but if they do, it's likely to be because of this "fuck you, man; you don't like my fucking music, get your own fucking cab" style. It's hard to imagine it not being beaten by a website that's "amazon but with netflix-quality recommendations", which may be amazon themselves, should they pull their finger out instead of stagnating, like a sourceforge.net that actually makes money. Personalized recommendations don't scale unless you automate them, and unless you have large amounts of raw data to start with. I realize amazon's database is way dirtier than netflix's, but still...

Anyway, in the here and now, I still tend to buy my books from bookshops that happen to have the kind of thing I like and, getting briefly back to the point, I picked up "What is Your Dangerous Idea?" a few weeks back.

Do you remember watching Star Trek as a kid? Maybe it wasn't Star Trek [TNG] for you; maybe it was reading all of Asimov's short stories? Or maybe you had a more varied diet of Heinlein and Dick, Philip K? But if you're the kind of nerd who I imagine reads my ramblings, there was probably some canonical source of all sci-fi themes you were exposed to at a young age, that effectively spoiled all sci-fi since. You go these days and watch a sci-fi movie, for example, hoping against experience that this one won't suck, and twenty minutes in you've realized which Star Trek episode this story is a retelling of, and from then on, you're mainly there for the lasers, robots, kung-fu, spacecraft, and boobies (or combination thereof) that they're relying on this time round.

You'll note, as an aside, that the new Battlestar Galactica gets an almost perfect score there. There's no kung-fu as such, but there are guns and other militaristic paraphernalia, and that's a common and perfectly acceptable substitute. The fundamental story of Battlestar Galactica is quite simple, and it's a mix of a few of those core sci-fi stories ("what happens to society if there's suddenly very few of us left? what makes humans human? when is a machine alive/human?"), but it's spread out nicely over a long period, has more economics, law, marital [not just martial] drama, and politics than is usual for sci-fi, and has relatively few dud episodes. If only George Lucas had these people's talent, or these people had George Lucas' money, how much quality sci-fi might we have had this last decade?

"What is Your Dangerous Idea?" is like watching the complete Star Trek TNG in fast-forward in that it's a compressed form of a bunch of ideas, but without the detailed exploration of any of them that you'd get from something with a thematic focus. Assuming you read popular science books/magazines (or just sci-fi), you've probably come across a good number of the ideas before, but if you've come across all of the ideas before, you read more than I do. The ideas were grouped by theme, and it was fairly obvious that I, for example, had thought more about traditional sci-fi stuff like impact of technology than about economic or medical stuff.

Two of my favorites were fairly familiar ideas. David Gelernter's "What Are People Well Informed About in the Information Age" worries that people are becoming more ignorant; Daniel Goleman's "Cyberdisinhibition" that our brain's evolution doesn't help it cope with social interaction over the Internet (the difference from the telephone, say, presumably being that it's easier to be part of a group on the net), and Geoffrey Miller's "Runaway Consumerism Explains the Fermi Paradox" that advanced societies that don't reject such things end up investing ever more effort in improving access to fast food and porn (and analogs). Kai Krause worries about what happens to the anthill (society) if all the ants want to be queens (reality-TV stars/rappers/whatever it is the youngsters of today consider the easy way to the top).

It's similar concerns of my own that always leave me baffled when people (nerds) get excited about personalized news. I remember reading in the early 1990s about such a thing from somewhere like MIT, where the over-excited nerd was banging on about how great it would be to only see news stories that interest you. Even then, a non-American would have been likely to find themselves thinking "dude, that's 80% of what's wrong with the US". But it's what we got, one way or another, all of us. I largely gave up on news because I was sick of the stories being factually incorrect or editorially misleading, whenever they covered a topic I knew anything about. I found it hard to believe that was just coincidence, and that all the stories I couldn't easily judge were correct. It's possible, but Sturgeon's Law provides a simpler explanation. So what do I read now? Stuff on subjects I already know about, from the handful of sources that aren't too often incorrect or misleading. People like me, I guess, talking about stuff I already know. Unlike the guy from MIT, I think this is a bad thing. A more acute form of the cause of many of the world's troubles. A deliberate atrophying of our abilities to empathize.

I'm always surprised, for example, when I meet someone who believes in a god. Because people like me don't believe in gods. I never have, and short of serious brain injury, never will. The people I spend my time with, and talk to, and whose books I read (with the exception of Knuth) don't either. So meeting someone who believes in a god is like meeting someone who doesn't have electric lighting. You realize that such people probably exist, but they can't really be many in number, can they?

In their places of worship, they're probably thinking the same things about atheists right now.

Anyway, if you liked Raymond Carver's ability to get across a person's whole life in half a page, and you liked Star Trek as a kid, you might like this book. Don't be deceived by how short the chapters are; apart from the blatantly mad ones, it's easy to let your mind wander and fill in your own sci-fi story for each one. At least if your medication's as strong as mine.

Battlestar Galactica and netflix.com also come highly recommended. Santa Cruz much less so.

Review: "Chances Are: Adventures in Probability"

2008-07-19T20:08:00.003Z

Statistics and probability have a hard life. They're often counter-intuitive (i.e. "hard"), neither the principles, the methods, or the valid/invalid implications are well understood by the general public, they're often abused to add an air of "science" to help snake-oil salesmen and mistruth-peddlers mislead more convincingly, and many of their truths are deeply unsatisfying to humans because they don't give us the simple intelligible insight we crave.

Especially if you're young, I think, and especially if you can program a computer, this can be hard to take. Any "statistical" or "probabilistic" algorithm is naturally suspect, either seen as a cop-out or just impractical because you don't have the amount of data to work with that the Amazons, Googles, and Netflixes of the world do. And I personally as a kid grew up thinking that these "approximations" were just temporary stop-gaps until we really understood, really had the insight.

Now I'm an old man, I accept that there are many situations in which statistics and probability offer us the best answers we can hope for, or the best answers we could want. I'll never know whether this book, had I read it as a teenager, might have offered me a shortcut between there and here, but I can say that it's an enjoyable read. I've never previously been taken through the chronological development of the field, never been shown the wrong turns that were made, and never understood just how recent so much of the field actually is. (I also never knew why "Student's t-test" had such a silly name. The actual reason is pretty silly in its own way, but probably nothing you'd guess.)

This book won't teach you statistics or probability, but it will give you a better appreciation of the field, its history, and its relevance. It's over-written at times, making too much of an effort to be literary to no great advantage, and with some resulting awkwardness, but these are just a handful of sentences in a 300-page book. I also wonder how well the book works if you go in knowing nothing of the subject matter. As ever, there's conflict between telling things in the order in which they happened, the order in which they're easiest to understand, and in this case there's the added dimension of the order which makes for the best story.

As it happens, the missus bought this book for herself. I don't think I'd have bothered to take a closer look in a bookshop, but since it was lying around at home there was nothing to lose, and I'm glad I took the time.

If you're too lazy to read a book but would like to be convinced of the importance of the field, try subscribing to netflix. Marvel at their "viewers like you" ratings, which, once you rate enough movies, "knows" you better than you know yourself, and is significantly more useful than the average rating of all users. The book doesn't talk about clustering algorithms, but it does touch on this kind of distinction. Netflix does this without being able to give you any insight beyond "you are not a beautiful and unique snowflake", and yet no-one could question its usefulness. Sometimes, kid, insight is neither necessary nor possible, and this can be a good thing, not just an unfortunate roadblock on the path to enlightenment. If you won't take my word for it, read the book.

Beware Ubuntu 8.04

2008-05-16T14:53:00.006Z

Being a wary kind of person, I waited a week after the official release before even installing Ubuntu 8.04 on a test machine, but after that seemed to work okay, I went crazy last night and installed it on my main home machine.

Imagine my joy to find that, though the keyboard works well enough for me to log in, I can't type once I'm at the GNOME desktop. No key on my wired aluminum Apple keyboard does anything. (The test machine, of course, had the older white plastic keyboard.) The machine itself is a PC, not a Mac, but it seems someone broke the kernel driver. Launchpad bug #201887 seems related, and showed me the workaround, but I haven't once touched the numeric keypad in the keyboard's whole life, and rebooted to check that the numeric keypad has nothing to do with it by very carefully ensuring that neither my pudgy fingers nor my mouse's cable nor my cat were anywhere near the numeric keypad as I logged in.

Anyway, if I ssh in from another machine, I can use this to fix the problem:


sudo su -
echo 0 > /sys/module/hid/parameters/pb_fnmode

This, I'm told, basically turns off all the dodgy code in the kernel module.

Judging by the bug, they knew about this before the release, but didn't bother to mention it in the release notes. Next time I ask myself why I have more than one computer, remind me that the answer is "so I can search the web for workarounds when one of them is broken".

The second most annoying problem is that sleep ("Suspend") is broken. It worked fine in 7.10 on my desktop (Intel DP35 motherboard with a Core 2 Quad) but won't wake on 8.04; the power light comes on, but nobody's home. Sleep never worked on my other machine, the one I tested on, so I didn't notice this until it was too late. That's an interesting question, actually: do test releases tell you more about crappy old hardware than they do about the newer hardware people are using for their day-to-day work? Or do other people actually buy two of everything, so they can run a properly scientific test?

Other problems? The weather applet has reverted to Fahrenheit, which might be useful for my dead ancestors (but only ones I never met; my grandparents were happily using Celsius before they died). "vbetool", a program I'd never heard of until today, crashes on login; a web search says "vbetool uses lrmi in order to run code from the video BIOS". Which is probably more useful than it sounds, if you like the power-saving states of your display. Speaking of displays, I was reverted to the free driver for NVIDIA cards, and the MP3 codec needs installing again. (In both cases I feel like I've already made my decision there, and shouldn't be made to repeat myself until there's a suitable alternative.) I don't like Firefox 3, either, but I might get used to it (or it might improve when it's no longer a beta; what's a beta version of the web browser doing in a long-term support release of an OS anyway?).

Anything positive to report? The graphs in system monitor (the ones you see if you pop open the detail window) are slightly prettier now, but I never used to look at those anyway. Other than that, you wouldn't know it it's not 7.10. I think I've said before that 7.10's pretty good. I use 6.06LTS at work and really feel a need to upgrade there, but there was no real reason for me to move from 7.10 to 8.04. And, as it turned out, doing so was a mistake. If I could go back in time, I'd stick with 7.10 for now, and I'd probably try testing with a LiveCD on the machine I care about rather than a real upgrade of a different machine before clicking the "Upgrade" button.

Now, if you'll excuse me, it's time to add a line to rc.local so I don't need two machines just to log in...

Generating JVM bytecode 4

2008-04-26T05:57:00.002Z

GCJ lets you compile your Java program to a native executable. This can be handy, but this isn't as handy as you might think in relation to generating JVM bytecode.

A GCJ-compiled program starts quickly, and can be used in places where you can't use a JAR file or a shell script wrapper (the two most common choices for launching Java programs), but when you start to execute your generated bytecode, you'll notice that code runs pretty slowly. About the same speed as most interpreters seem to run. This is not a coincidence.

So that's a shame. If you're not generating bytecode, and just using a tree-walking interpreter, this won't be a problem, and you might still want to consider GCJ. Just watch out for GNU Classpath: there are lots of bugs and unimplemented features (including unimplemented features that just silently do the wrong thing), and performance characteristics can be very different. BigInteger, for example, is way slower at the moment.

One idea I haven't yet investigated is that of using GCJ to make a native binary that uses CNI to start Sun's JVM. I'd use GCJ-compiled code to generate the bytecode and hand over to HotSpot for the actual execution. Given that you'd have to pay for the Sun JVM initialization in the end, and you'd miss out on the benefit of having the compiler "warm up" the JVM for execution of the generated code, I'm not sure this would really gain much over just writing a little C++ JNI-based launcher. Maybe start the Sun JVM on another thread while compiling the bytecodes on the main thread?

The start-up time only matters for scripts that take microseconds to execute, because the JVM soon more than makes up the lost time, but it would be nice to have our cake and eat it.

Generating JVM bytecode 3

2008-04-06T03:14:00.004Z

ASM, gnu.bytecode, and org.mozilla.classfile: a comparison
I explained earlier that I chose ASM over BCEL because the latter has a reputation for being big and slow, while the former makes a big thing of being smaller and faster. Plus ASM has better documentation.

There are other choices, though. You can write your own library, for example. From the various implementations I've seen, I reckon that's about 3000 lines of work. If that end of things interests you, go for it. But it's not obvious that there's anything sufficiently wrong with the existing libraries that you'd really want to write your own.

The only other alternative I'd seriously considered was Per Bothner's gnu.bytecode, written as part of Kawa. Per describes Kawa as "The Kawa Language Framework", but most people who've heard of it probably think of it as a Java implementation of Scheme. (Such a thing is indeed part of Kawa, most of the documentation is about Kawa Scheme, and most people who use Kawa use it as a Scheme implementation, even though it also implements XQuery and Emacs Lisp, so it's easy to see how this comes about.)

So. Why didn't I try gnu.bytecode?

1. It's not packaged and available in Ubuntu's repositories. Despite the fact that nothing I'm the major author of and still use fits this criterion, I mostly shy away from stuff I can't apt-get install. If no-one cares enough to maintain a package in the enormous Debian repositories, my thinking goes, how good can it be? (At the same time, right now, the only GUI application I'm running that's available on packages.ubuntu.com is Firefox. And I'm running all the stuff I usually run. But still, my gut feeling is always that "not in Debian" is a bad thing. ASM is. BCEL is.) Per points out that Kawa is packaged in Fedora.

2. It's easy to fall into the trap of thinking it's part of a Scheme implementation. Because the JavaDoc on the web is collected together, it's not obvious that gnu.bytecode is completely stand-alone. If you download the source and grep the imports, or download the source, delete everything outside gnu/bytecode/ and compile what's left,you can confirm that gnu.bytecode is, indeed, completely stand-alone. But you'd be forgiven (by me, at least) for assuming you'd get all the Scheme junk too. Moreover, it's not obvious that gnu.bytecode is sufficiently complete for compiling arbitrary languages. How do you know, given its bundling, that it's not just whatever was necessary for Scheme? (I have an existence proof for you, now, but more on that later.)

3. It's effectively undocumented. There's no tutorial or even a simple example, most of the classes and methods are undocumented, and those few methods that are "documented" tend to be sufficiently telegraphic that you really need to look at the source to understand what they're trying to tell you. (And a lot of the seemingly obvious methods hide little tricks that you'd do well to examine the source to learn more about, too. More on this, too, later.)

I was in a hurry to dump my old AST-walking interpreter, and ASM seemed fine, so gnu.bytecode really didn't seem like a tempting proposition, given the above. Then, earlier this week, Patrick Wright asked me "why not gnu.bytecode?", and Per Bothner (who was cc:ed) convinced me that point 2 above wasn't really an issue: gnu.bytecode is stand-alone, is intended to be complete, and is intended to be used in other language implementations. I stuck to my "but it's undocumented" guns until Patrick sent me a link to Peter Sestoft's Runtime Code Generation with JVM and CLR, section 2.4 of which is by far the most useful gnu.bytecode documentation I've seen: it's a rough equivalent of the trivial example given by most other bytecode libraries of how to create a class and add a method with a little bit of code in it.

I'd complained in an earlier post that, as far as I know, there's no decent comparison of the bytecode libraries (though Sestoft's paper comes closest, it's pretty old, and isn't really a comparison so much as a couple of data points), so in a moment of foolhardiness I decided that the almost exactly 1000 lines of my language's JvmCodeGenerator.java would be an interesting experiment.

After I'd rewritten my code generator to use gnu.bytecode, I came across org.mozilla.classfile and, either because of a subconscious desire to procrastinate or a desire to be thorough, I rewrote my code generator again.

It's worth pointing out that ASM and gnu.bytecode and org.mozilla.classfile don't really address the same problem. ASM is a bytecode manipulation library, gnu.bytecode is a bytecode generation library that's the lowest-level part of a high-level framework for language implementations, and org.mozilla.classfile is pretty much "the simplest thing that could possibly work", used as the back-end for their Rhino Javascript compiler.

ASM is designed to be equally at home modifying existing classes as writing new ones. This is why it's quite awkward to use for compilation, in comparison to something like gnu.bytecode which was expressly designed for that purpose. If you need to do both, your time is probably better invested in learning ASM. If you only need to compile new classes, gnu.bytecode is worth considering. If you want something small and simple (or you're specifically looking for a GPL-licensed library), org.mozilla.classfile is also worth considering.

The mixed blessing of more checking
Unlike ASM, which mostly signals the fact that you've made a mistake by throwing ArrayIndexOutOfBoundsException or NullPointerException from its bowels, or by letting you output bad/unverifiable code, gnu.bytecode and org.mozilla.classfile make a bit of an effort to keep you on the straight and narrow. Great, you think. And sometimes it is. But at other times I found myself thinking that "mistakes like this were actually easier to understand within the context of bad code". As usual in life, everything's a trade-off, and there's no One True Answer here, either.

Of course, this is one area in which I can't really make a fair comparison, because I've hunted down a lot more bugs in my code generator with ASM, and the bugs in my gnu.bytecode code generator were potentially different kinds of bugs, being errors in my translation between the two APIs rather than the kind of bug you'd see in real life as you tried to write a new code generator. Rewriting again, I made more errors because I was using a lower-level toolkit and because I felt confident doing a lot of the rewrite with search-and-replace. This caused a little bit of trouble because I was moving to a lower-level system, so something that had been a single concept in the higher-level libraries actually needed to be made more explicit.

Naming consistency
In general, gnu.bytecode's naming seems more in line with the JVMS than ASM's. I've been guilty of the mistake of thinking I have better terminology myself often enough, and it never works out unless you can completely isolate people from what lies beneath. And you rarely can, so it always just ends up confusing having to remember which term to use where.

Of course, the fact that gnu.bytecode mostly avoids this trap makes it all the more annoying when it falls right into it, dragging you after it. Calling the constant for the gnu.bytecode.Type representing java.lang.Object "pointer_type", for example, is unfortunate. (It's that kind of unidiomatic language that makes the Sun verifier so unintelligible without reference to the source. Here too, I found pointer_type by searching for java.lang.Object, after my guess "object_type" failed. I might have tried "reference_type", but "pointer_type"?)

The gnu.bytecode "if" instructions are a bit confusing, too. ASM has ifCmp, and gnu.bytecode has a huge range of methods, some with names that make them seem like implementation details rather than things you should be calling. For the common case where you want "ifeq label", gnu.bytecode is slightly more compact, convenient, and obvious, using "code.emitGotoIfEq(label)" instead of "mg.ifCmp(type, GeneratorAdapter.EQ, label)". In the general case, though, where you're implementing all your relational operators with code that's exactly the same apart from the "if" bytecode, it's a pain that you can't just pass a conveniently-named constant as you can with ASM. With gnu.bytecode you either end up passing magic numbers (there's no gnu.bytecode equivalent of ASM's Opcode, which has a named constant for each bytecode) or, as I did, having a secondary switch inside the general-purpose method that invokes the relevant CodeAttr method by name.

Finally, there's no emitNop in gnu.bytecode. You have to use put1(0) instead. The "nop" instruction is bytecode 0, you see. Again, something like ASM's Opcode would have helped here. The source just uses the appropriate magic numbers each time it needs them.

As for org.mozilla.classfile, the interface is way smaller. You deal with a single class (plus a poor-man's enum), and a summary of the public API fits comfortably on my screen. You can emit any instruction you like, and it'll look something like "cfw.add(ByteCode.NOP)". You'd be right to infer from the name "add" that org.mozilla.classfile goes mad for overloading. Personally, I'd rather have gnu.bytecode-like addPushInt, addPushBoolean, addPushLong, addPushDouble, and addPushString than the actual five overloads of addPush, but it's not too bad.

The uniformity of passing ByteCode bytes around in org.mozilla.classfile makes for trivial abstracting-out of common sequences. So in contrast to gnu.bytecode, I had no trouble translating the ASM GeneratorAdapter.EQ example I just mentioned.

One minor drawback of org.mozilla.classfile is that, being lower-level, it's sometimes slightly more verbose than gnu.bytecode, but not all the time. The fact that there's only one class (ClassFileWriter) to deal with, for example, removes some verbiage, because you don't have to deal with Method and CodeAttr objects. (One side effect of this is that this is the only library where you can only generate code for one method at a time. That required minor fiddling in my code generator, because I needed to explicitly delay generating code for function definitions at global scope until I'd finished generating code for code at global scope, where with gnu.bytecode or ASM I could just push a new object representing the new method's code onto a stack. But it makes little difference.)

Labels
Label handling is pretty similar in all three libraries. I preferred ASM's verb "mark" to gnu.bytecode's "define" for the method that fixes a label to the current location, but I preferred gnu.bytecode's "new Label()" to ASM's "mg.newLabel()" for creating a free label, because the latter reads to me as if it ought to mean "new label fixed to the current location". org.mozilla.classfile uses acquireLabel and markLabel. Perfect.

A hierarchy of Types?
In a break from its usual over-designed style, ASM has one class Type that's used to represent primitive types, reference types, and array types alike. gnu.bytecode has a Type class, but it also has ArrayType and ClassType and ObjectType. The last of which isn't what you think. All this is a minor pain. Primarily (in my case) because it means that I can no longer treat "void" uniformly, even though all my other types are ClassType, void is just a Type. So my method that translates my language's types to gnu.bytecode types has to return Type to cope with the "void" case, and almost all callers have to cast that to ClassType to be able to use it. A minor annoyance, but no-one likes seeing casts in their code, and ASM's uniformity was nice and clean.

Unrelated and minor, but one of the few head-scratching gnu.bytecode mistakes I made was that I accidentally created two ClassType instances for the class I was generating. Obviously, stuff added to one isn't visible in the other. It was my fault for coding so late at night, but it did make me wonder about the wisdom of a public constructor on ClassType.

Where gnu.bytecode uses a more complicated solution than ASM, org.mozilla.classfile goes in the opposite direction and uses a simpler solution than ASM. It has no type for types at all, using String instead. This is convenient, but the lack of distinction or automatic conversion between class names and signatures can be annoying at times. Especially because parameter names in the API don't usually make the distinction either, so you'll be relying on your wits (or the JVMS) to work out which you should supply. org.mozilla.classfile has no user-visible abstractions for field references or method references. This, to my mind, actually improves one-off calls to fixed fields/methods. The temptation to cache a field or method reference doesn't exist, and the code reads as clearly as I can imagine. If you keep calling the same method or accessing the same field, though, it's easy to find yourself wishing there were a convenient way to say this just the once. (In the most frequent case in my code, I went for a method that just emits the get/put/invoke in question.)

One other note: org.mozilla.bytecode forces you to pay more attention to return types because you're constructing descriptors manually. This is a price you pay for going low-level.

Cruft
Where the lack of documentation for gnu.bytecode really bites you is when there are a bunch of deprecated alternatives lying around. Sometimes they're literally @deprecated, sometimes they have a comment pointing you in the right direction, but mostly you're just supposed to know.

The Sestoft paper, for example, uses CodeAttr.initCode and then calls getCode, and the reader's left wondering why getCode doesn't just initCode if code is null. It turns out that you're probably much better off calling CodeAttr.startCode, which does this and also creates a Variable instance for each of your method's parameters. (You can get at the Variables by numeric index, so it's hard to imagine when you'd be harmed by this behavior.) Funnily enough, this is documented. The JavaDoc for initCode clearly says most people are better off with getCode. The trouble is that documentation's so sparse that you just don't bother reading it. Where with ASM I worked mainly from the documentation, needing the source pretty much only to work out how to use their verifier (as you saw in an earlier post), with gnu.bytecode I needed to work from the source.

It would be nice to see a clear out of some of the old stuff. It wouldn't solve the documentation problem, but it would help, just by making the forest smaller.

The org.mozilla.classfile API, on the other hand, is pretty lean and mean. It depends on a few other org.mozilla classes for maps with integer keys and/or values that don't require boxing and a weird non-generic variant of ArrayList that keeps the first few elements in fields instead of an array, but swapping these out for their closest JDK equivalents is easy. If you know the JVMS reasonably well, things are pretty self-explanatory, and there's pretty much only one way to do anything.

Debugging information
I've been putting off adding debugging information – local variable tables and line number tables – to my ASM-based code generator for ages. It's way harder than I think it ought to be, unless I'm missing a trick. By contrast, gnu.bytecode has exactly the interface I'd like to see. Local variable tables pretty much added themselves in the translation (I certainly wrote no new code) and line number information required little more than a line per visit method in my code generator's AST node visitor. The resultant line number tables are long and repetitive, but I soon decided that the seemingly obvious optimization (removing any entry that maps a bytecode index to a line number if the previous entry mapped to the same line number) would be invalid in the presence of branches, so long tables is just a fact of life. It would be nice if the disassembler could be easily told to not dump line number tables. It takes an int "flags" parameter, but doesn't currently use it in any way. The disassembler, by the way, is really nice. I've encountered it before and still think it's the best "javap" going; complete and with highly readable output.

It's also a lot easier to deal with "register allocation" using gnu.bytecode. You can pushScope and popScope on the CodeAttr, and local slots allocated during that scope will be freed for reuse. With ASM, I had to manage this myself, and was using a trivial scheme where I never reused slots. This didn't seem to affect HotSpot much in terms of performance, but it probably increased the potential for unwanted heap retention as references hung around in local slots that would never again be accessed.

Turning to org.mozilla.classfile, you get the same simple interface to the line number table, and a seemingly convenient interface to the local variable table, but one that's broken in that it assumes that all variables live from their point of definition to the end of the method. You also get no help when it comes to reusing local slots.

If you want or need stack maps, neither gnu.bytecode or org.mozilla.classfile offer them at the time of writing. They seem to be on the roadmap for gnu.bytecode, but I've no idea about org.mozilla.classfile; judging by the revision history, it's not been actively worked on for years.

Performance
I imagine this is where many people stop skimming and start reading. Which is silly. The stuff above is far more important. Basically, in all aspects of performance, there's pretty much nothing in it between ASM and gnu.bytecode, as far as my totally unscientific tests have shown.

Size-wise, gnu.bytecode initially seems quite large. It's about a 100KiB JAR file for just gnu.bytecode. asm-3.1.jar, by contrast, is 44KiB, and [my modified variant of] org.mozilla.classfile is 23KiB. But adding together all the ASM JAR files I ended up dragging in, as I realized that to get a semi-convenient API, I needed more and more of the "optional" JAR files, I find I was using 160KiB of ASM JAR files, 60KiB more than I'm using of Kawa. If you really want or need small, though, org.mozilla.classfile is your winner. (If you're scratching your head about why anyone would care about this in 2008, that code needs compiling at run-time, and J2ME users don't necessarily have the space to spare. But the main reason I bring it up is that ASM seems to consider its size a feature. BCEL, I'm led to believe, is relatively huge.)

One reason why org.mozilla.classfile is so small is that it doesn't include a disassembler, and it doesn't include a verifier. While you don't necessarily need either of these in your shipping product, you'll want both during development.

Despite doing slightly more, gnu.bytecode compiles slightly faster than ASM. org.mozilla.classfile is slightly faster still. When I say slightly, though, I mean it. If ASM's too slow for you, I seriously doubt that gnu.bytecode's going to change your life. But again, if you're looking at J2ME or the like, you might want to save every scrap you can.

Conclusion
As I said at the beginning, which is best for you depends on what you're doing. The only sensible "conclusion" I can offer is a summary of some of their best features.

ASM's best features:

Only option if you need stack maps.
Best verifier (better than BCEL's, too), though not 100% the same as Sun's.
Most extensive documentation.
Also able to manipulate existing classes; potentially handy if you're not writing a compiler.

gnu.bytecode's best features:

Does several compilation-related tasks for you.
Ties in with even higher-level libraries if you're interested.
Best disassembler (better than javap(1), too).

org.mozilla.classfile's best features:

Very little API to learn, very little need for documentation.
Tiny code size.
MPL1.1/GPLv2 or later.

Generating JVM bytecode 2

2008-03-31T02:51:00.003Z

Constant pools
If you're compiling a language other than Java, you may find the fact that the class file constant pool can only contain JVM primitive types or instances of java.lang.String somewhat frustrating. I wanted a constant pool for my boxed integers and boxed reals.

Disappointingly, the "getstatic", "bipush", "aaload" to load a boxed real constant from my private static final Object[] home-made "constant pool" seems to perform about the same as the "ldc" and "invokestatic" for boxing a JVM double (using something like java.lang.Double).

If you've got a more expensive constructor, though, it's a win. BigInteger constants benefit, and I imagine Pattern constants would too, if you have regular expression literals in your language. I'm not sure why none of the bytecode generation libraries seem to have any support for this. Surely it's a common enough need? Mine is only 100 lines of code including comments, but that still seems a bit weak.

Strange performance problem of the week
I finally tracked down a huge slowdown that had been dogging me for ages now. My most modest test case (not Java) which finally produced little enough bytecode that I could spot the problem by inspection looked like this:


X1 := 0.1;
for (p := 1.0; p > 0.003; p *= 0.98) {
  X2 := 0.1;
  for (y := -1.2; y < 1.2; y += 0.07) {
    for (x := -1.6; x < 1.0; x += 0.03) {
      for (c := 6.0; c < 20.0; ++c) {
        tmp := X2;
      }
    }
  }
}

Running this on Java 6 took about 2s on my machine. Commenting out the definition of X1 (which you'll note is unused) took the run time down to 0.2s. (The types are all effectively java.lang.Double, but that's being inferred by the compiler in this example.)

Running with -Xint increased the run time, but the presence or absence of X1 no longer made a difference.

So why was the unused variable making my program 10x slower? It turned out to be a bug in my code generator. There was no "pop" to get rid of the value of X1 after computing it for its initialization. Leaving that on the operand stack somehow combines with a "sufficiently complicated" method (changing those nested loops to just be one loop that counts to 30,000,000, for example, makes the problem disappear) to upset Hotspot.

I don't really have any particularly useful advice here. I don't know of a tool that spots such mistakes, and I don't know the exact conditions to reproduce the problem (and, since it seems to involve leaving junk on the operand stack, I can't provide a Java example). But if you're having weird performance problems with your generated code, I guess this is one extra avenue to explore.

In my case, Hotspot doesn't appear to compile the method at all. You can see this with the output of +XX:PrintOptoAssembly on a fastdebug build.

Annoyingly vague JVM output of the week
Why is ArrayStoreException so vague? It needs the same battering with the clue stick as those annoying human bug reporters who give you no more to go on than "something's gone wrong". Tell me what type you expected, and what type you saw, damn it! Sun's JVM tells you one of those things, but doesn't make it clear which (and the JavaDoc for the exception doesn't help; I ended up looking at the Hotspot source, which really shouldn't be necessary). jamvm(1) was even less helpful, with no detail message at all. I've sent a patch to the author.

My mistake was the usual gotcha: "anewarray" does what it says on the tin, and creates a new array of the reference type you give it. That is, you don't give it an array type if you want a single-dimensional array. Adding an example to that instruction's section in the JVMS would go a long way.

Building jamvm(1) on Ubuntu
To hack better diagnostics into jamvm(1), I needed to build from source. It took me a while to work out that to get a working VM, you need to configure with --with-classpath-install-dir=/usr (and not /usr/share/classpath). It's clear if you look at the source, but not very clear from the documentation.

Generating JVM bytecode

2008-03-09T23:54:00.006Z

If you've been wondering where the useful code snippets and hints have been hiding the last few months, the good news is they're back in this post. The bad news is that I'll be talking mostly about objectweb.org's ASM all-purpose Java bytecode manipulation and analysis framework, and don't have any significant experience of the alternatives. When I chose ASM, I'd have loved to read a good comparison of the choices, but I didn't find one. And I'm sad to report that I won't be writing one, at least not right now. Maybe if someone points me to a particularly tempting alternative?

objectweb.org's ASM
I chose ASM because it prides itself on being small and fast, and because it's still frequently maintained. I don't honestly know enough about the competition to know what kind of state they're in, and whether the ASM team's comparative benchmarks are fair and representative. ASM is fast, but I don't know that BCEL (the best-known alternative) would be significantly slower. The documentation for both is pretty weak, but ASM's seems better, and the example code for ASM seems more direct and less verbose.

That said, small does not imply simple, and ASM is a little hairy. The authors went a bit pattern-mad, and it's very "designed" in ways that often seem to be to the detriment of convenience and simplicity. (If you think java.io is awkward, you won't like ASM: it's like java.io, only more so.) In addition, the "convenience" layer in the "commons" package isn't high-level enough that you won't need a good understanding of Java bytecode, but it is far enough from the interface presented by the basic "asm" package (which is a pretty direct mapping), and badly-enough documented that, to be honest, it seems more of a hindrance than a help. I've given up fighting it and am starting to use the underlying functionality instead, even to the extent of reverting some of my existing "commons"-using code. (One problem in particular is that, as in a C++ program that uses stdio and iostreams, you need to be careful that each layer knows what's going on. Now imagine that iostreams was badly documented and seemingly incomplete such that you repeatedly found yourself falling back to stdio. That's the ASM situation. It's possibly nothing that good documentation couldn't fix, but that doesn't help us in the here and now.)

Stick to the basic "asm" package, though, and things don't seem too bad. I do have this sneaking feeling there's an even skinnier guy struggling to get out, though. And I for one would like to make his acquaintance. If you see him, let me know.

Documentation
You'll want the ASM3 API documentation and it's probably worth looking at the asm-guide.pdf overview, though it's incomplete and, if you're interested in generating bytecode, will seem too focused on processing existing bytecode (ASM supports both, and you'll notice that they describe it as a "manipulation and analysis framework" rather than the skinny "generation framework" I really wanted).

More importantly, you'll need to have read The Java Virtual Machine Specification (2e) by Lindholm and Yellin, which is also available on dead tree, but is equally outdated there. The web version would be greatly improved by a table of contents with direct links to the individual pages for opcodes starting with each letter of the alphabet, and even more so from having the material from the JVMS maintenance page worked in to the main text. (Those PDFs suggest Sun has the source to the book, so it's a real shame they don't re-issue the HTML version or an all-in-one PDF version.)

The only other book on JVM bytecode I've read was Joshua Engel's "Programming for the Java Virtual Machine" from the late 1990s. My only retained memory was that it was full of mistakes, and re-reading it recently, I can confirm that to be a reasonable summary of the book. You can safely ignore it, unless you want to see the most amusing typo in any book covering compilers. You can be sure I'll be building a "poophole optimizer" in every compiler I write. (That's the funniest error, but it's also the most trivial. Most of the code examples contain at least one error, and there are numerous statements that make you wonder to what extent this "acknowledged expert in the Java virtual machine" knew what he was talking about. He certainly didn't know how to pick technical reviewers, that's for sure.)

You know what I'd love? Someone like Sun's John Rose to go over the JVMS "Compiling for the Java Virtual Machine" chapter and tell us what kind of code we should be generating for the modern HotSpot JVM. Which idioms and bytecode-level optimizations are good, and which idioms we should avoid. Especially balanced against issues like code size, verifiability, and future-proofing.

Verification
The verifier in Sun's JVM may be great for keeping us (and our programs) out of mischief, but it totally sucks in terms of the quality of its diagnostics. Have a look at OpenJDK's "check_code.c" for the details, but suffice it to say that the errors are lacking in clarity, lacking in detail, lacking in context, and don't even manage to use the usual terminology associated with the JVM and its bytecode. Some messages don't even manage to be valid English sentences. "unitialized" indeed.

Normally this isn't a problem because javac(1) is generating the bytecode and although there have been cases where javac(1) has generated unverifiable bytecode, I can only remember it causing me trouble personally in one instance during the last decade-and-a-bit. If you're writing your own bytecode-generating programs, though, you're a lot more likely to see VerifyError.

Free JVMs
There are a bunch of Free JVMs, and for a laugh I told apt-get(1) to install sablevm, kaffe, and jamvm (in that order, because that was the order in which they came to mind). jamvm(1) turned out to be the most useful, despite the fact (or because of the fact) that it doesn't have a verifier. This, combined with the fact that it seemed to be pretty robust, meant that I could often turn a verification error into a run-time error, and that can actually be useful. The other two weren't much use to me. IIRC, sablevm(1) did verify but with diagnostics no more useful than Sun's, and kaffe(1) mostly crashed on unverifiable code. I've kept jamvm(1) installed because it might come in handy again, but the other two are long gone.

BCEL's verifier
Assuming you get far enough to write a .class file, BCEL includes a fancy verifier called JustIce. I link to a page that proclaims itself obsolete because there's a link on that page to the author's Diplomarbeit, which is potentially useful to you. (It's in English.)

Anyway, apt-get(1) libbcel-java and you can run:


bcel_cp=/usr/share/java/bcel-5.2.jar:/your/class/directory
java -cp $bcel_cp org.apache.bcel.verifier.Verifier YourGeneratedClass

JustIce is pretty verbose, which is mildly annoying when it's shouting "EVERYTHING IS OKAY!", but comes in handy when something's broken. Unlike Sun's verifier, you always get a bytecode index, you'll automatically get a disassembly, and you get plain English explanations using the usual JVM bytecode terminology. (Yes, you'll still have to have a rough idea of what you're doing, but I don't see how that's unavoidable. If you're that lost, you should probably be getting someone else to write your bytecode-generating code for you.)

ASM's verifier
ASM also comes with a verifier, and it's less loquacious, but it's under-documented and it took me ages to work out how to make it work on the byte[] from my ClassWriter, despite it being a mere two lines (the ASM 3.1 JavaDoc actually explains this, but earlier versions don't):


  byte[] bytecode = ...
  ClassReader cr = new ClassReader(bytecode);
  CheckClassAdapter.verify(cr, false, new PrintWriter(System.err));

The beauty of this verifier is that you get nice little stack pictures (that show the local slots too). The disassembly is nice and clear, and the explanations are fine too, and are somewhat less wordy than BCEL's.

(Note that, although I show BCEL's verifier being run as an external program and ASM's verifier being run as part of an application, that's only because I'm using ASM and not BCEL. You could equally well run BCEL's verifier as part of your application and ASM's verifier as an external application, and probably should if you're using BCEL.)

It's a shame that ASM's CheckClassAdapter does so little checking for the kind of errors that actually seem likely. I realize error-checking's not free, but ASM feels like it's sacrificed too much. Mostly you'll just see ArrayIndexOutOfBoundsException and NullPointerException exceptions thrown from its bowels. You'd be well advised to "apt-get source asm3" (I didn't work out how to use CheckClassAdapter before reading the source, for example).

Anyway, back to work. I've got bytecode to generate...

.mp3 id3 tags

2008-03-03T05:17:00.004Z

I mentioned recently that I'd switched to Rhythmbox on Linux for my music needs. One thing I didn't mention was the fruitless struggle I'd had trying to persuade it to let me fix some of the metadata. I've since had some success, which I'd like to share.

First things first: if you're using Rhythmbox, you'll probably be tempted by Rhythmbox's UI for editing the odd title or artist. My needs were borderline between that and needing the command line, but it turns out that using Rhythmbox's UI doesn't edit the tags: it just overrides the copy of the tags in Rhythmbox's own XML store.

But if you don't really care about the quality of your tags, you can just use the Rhythmbox UI and forget about all this id3 tag nonsense. If you're happy with that, stop reading now.

I'd tried using id3(1) on the command-line, and that had worked for some .mp3s but not others. For a long time, I suspected that Rhythmbox just wasn't noticing the change. I tried removing Rhythmbox's on-disk XML "database", but it would regenerate it with the same content. I asked strace(1), and it reckoned there wasn't any secret back-up. I looked for duplicates of the offending .mp3 files, but didn't find any. I looked at the tags of one of the offending files, and they looked fine. Yet Rhythmbox was showing something different. Eventually I actually looked at the .mp3 file itself, using less(1), which clearly showed an incorrect id3 tag at the start of the file.

At the start of the file? I thought the whole thing about id3 tags was that they come at the end of the file? Looking at the end of the file, I saw a different id3 tag, with the new data. Wikipedia's id3 article explains that there are two unrelated versions, id3v1 and id3v2. Sadly, their whole "neutral point of view" thing means they don't tell you which to use.

Rhythmbox, then, can use either id3v1 or id3v2 tags, but given both it silently uses the latter.

id3(1) only handles id3v1 tags and completely ignores id3v2 tags (they aren't shown, and you can't delete them).

id3v2(1), despite the name, is able to handle both kinds of id3 tag. It can convert from id3v1 to id3v2, though doing so won't automatically delete the id3v1 tag. It can remove either or both kinds of tag from a file. It can edit the data in id3v2 tags (but not id3v1 tags).

I've duly apt-get removed id3, apt-get installed id3v2, converted and deleted my id3v1 tags, and fixed the mistakes. Rhythmbox was running while I did this, and refreshed its display as things changed on disk, which was nice of it.

In future, I hope Rhythmbox's UI is changed to actually edit tags. Until then, I'll be using id3v2(1) to remove any stray id3v1 tags I happen to run across and fix any errors.

Review: GelaSkins 15" MacBook Pro skin

2008-02-24T01:34:00.004Z

My house may be full of junk, but the stuff that's actually mine is mostly just books and bits of computer. I'll admit that I often choose one product over another for aesthetic reasons (blinkenlights!), but I can't think of anything I own that has fundamentally decorative purpose.

Until, that is, the GelaSkin I bought to cover my work MacBook Pro. This really is a product that doesn't do anything but change the appearance of your laptop. I'm a fan, though.

I'd read about these things ages ago in ars.technica, though at the time they were for iPods. They sounded pretty hard to put on, and they were super-ugly. Not just a little ugly, but really deliberately in-your-face ugly. Expensive mall ugly. Versace furniture ugly. So we kind of got off on the wrong foot. Being labeled "ugly expensive tat" is not a way to find yourself on my shopping list. The ars.technica article was sufficiently effusive in their praise that I visited the web site anyway, and even found that there was one design I actually liked. Even so, I didn't really see the point. Why would one bother? Part of the beauty of the iPods is that they're small and inoffensive. And it looked pretty tricky to line the wrap-around sticker up.

In their coverage of MacWorld 2008 ars.technica again mentioned GelaSkins, and specifically noted that they were showing off their laptop equivalents. Unlike the iPod skins, which wrap around and cover the whole iPod, the laptop "skins" are basically just big stickers that cover the lid of the laptop.

I've never really liked the look of the MacBook Pro. I accept that aluminum is a better laptop material than titanium was; no peeling paint here, and it feels pretty nice and rigid. I just happen not to like the way it looks. When it's open and I'm using it, it's fine, but when it's sat on my desk next to me, I'm not a fan. And being a big fast computer kind of person, laptops spend most of their time with their lids closed, sat on my desk next to me.

I've also never really liked the big glowing Apple logo on the lid. I don't wear t-shirts with logos on them unless they're free, and even then I'm somewhat choosy about what I'll advertise. I feel like I should at least get a discount for carting a big glowing Apple logo around.

So, for USD30, GelaSkins will sell you a big thick sticker to cover the lid of your laptop. It's thick enough that you can't see the Apple light through it, though if you look closely and the light's just right, you can make out the indentation around the edge of the logo.

The sticker also lets you choose how you think your laptop should look. Sexy robots? Check. Over-used pieces of popular fine art? Check. Sexy cartoons? Check. Fractals? Check. (When is this? The 1980s?)

And if, like me, you've always wanted wooden computers, you're in luck. That perversion is catered for too.

It's hard to tell from the web site how the things will actually look in real life. And since I've only tried one, I can only offer anecdote, but the one I have looks pretty good. If the light's right, it actually looks like highly-polished veneer. If the light's wrong, it looks like one of those textured stickers you see on the back of buses in some countries. I was also curious about how it would look given that it doesn't actually extend to the edges or wrap around, but it looks fine to me, and better than I personally imagine it would look if it did extend further.

It was easy to apply, though I'm not sure that wasn't just good luck.

Theoretically it's a "protective skin", but I'm not sure you'd really notice. If you have a habit of keeping metalworking tools on top of your laptop you might find yourself saved a few scratches, but it's hard to see it making much difference in the pocket of a laptop bag. What the sticker does do is make fingerprints and salty sweat marks less obvious. I no longer find myself rubbing the damn laptop clean every time I use it, though the areas to the left and right of the trackpad on the inside remain problematic.

There are wrap-around MacBook Pro skins available from other companies, but I've yet to see one that didn't look about as good as something you did yourself in 2 minutes with a roll of duct tape. The corners in particular are deeply unsatisfactory. I never see the bottom of my laptop, though, so it can look out for itself. The wrap-around skins don't address the inside of the laptop either, so unless you carry your laptop round in a bag of gravel, I don't really get the point.

In summary, although this product doesn't really do anything, I still like it. I can recognize my laptop at a glance, it makes it look better in my opinion than it did out of the box, and I no longer have to see (or show) the big glowing Apple logo. I also don't have to rub sweat marks off the lid on a daily basis. Would I feel silly walking around with my decorated laptop under my arm? Possibly, but I'd feel silly walking around with any kind of laptop anyway (just add bluetooth headset for the complete knob-end experience). Would I feel silly if everyone with a MacBook Pro read this and rushed out to buy one? Only one way to find out...

Review: "A Theory of Fun for Game Design"

2008-02-18T01:36:00.002Z

I'm not much of a gamer. As a kid, I'd play a game until I felt like I knew how you'd go about writing it, and then I'd lose interest. I didn't even care to apply that knowledge, either in using it to "beat" the game (many games in those days just kept getting harder until you failed, and thus only really offered you failure), or in using it to write a clone. I knew I wouldn't play the clone, so I had no motivation to write it.

Before games even progressed to 3D, I had too much other stuff to do, so I kind of missed out on all that. I came back to gaming a couple of years ago, just after the Xbox 360 launch, when I bought an original Xbox. (That sounds perverse, but the 360 launch didn't actually mean there was anything worth playing on the 360 at that time, or that 360s were readily available, or that a handful of specific Xbox games I was interested in were on the 360's compatibility list, or that a sane human with an alternative would want to play an Xbox game on a 360.)

I'd never owned a console before. A computer that isn't end-user programmable? Not really a notion I want to support. It still bothers me, but other than "don't play games", I don't see an alternative. (Running Windows is not an alternative.)

The Xbox was ostensibly for my girlfriend, but through bad choices of games, she lost interest. I was the main user of the Xbox, and I was mainly using it to watch DVDs. Then one weekend I didn't really want to watch a DVD, or read a book, and I didn't want to write code or a blog post either. I wanted something in between, and so I bought a game.

Since then, I've played a bunch of games, found a bunch more that my girlfriend has enjoyed, bought a 360, and bought her a PSP. Gaming fits nicely into a gap on the recreation spectrum, and I've found myself thinking about games quite a lot.

What makes a game good or bad? What aspects of the implementation make/break a game? What trade-offs are visible to the user, how much do they hurt, and could they have been ameliorated? Would games be easier if there were no free amateur game guides on the net, or would they be the same and we'd still be getting "stuck" like in the bad old days? How could difficulty be managed better? Which old ideas still make sense, and which don't? How do you balance realism and fun? Why don't all games encourage playing for speed, or to kill no-one/everyone? If 360 Achievements are so fundamentally pointless and worthless and yet strangely gratifying, is there an analog for serious software ("15 G - Low Ammo: no slide in your presentation used more than 3 bullet points")? What's wrong with the Japanese and why can't they write a game I can enjoy?

Raph Koster's "A Theory of Fun" doesn't touch on the cross-over to "serious" software that interests me (the "achievements" question above), and he's not interested in implementation details either (which aren't very interesting academically, even if implementors can always learn from comparing good and bad implementations). But if you're interested in the more abstract questions, this is a great book. It might not answer many questions, but it does point out aspects and complications you might have overlooked in your own cogitations.

The style of the book is odd. Every facing page is a picture, drawn by the author. Sometimes the picture direct illustrates the accompanying text, sometimes it's more abstract. It's off-putting when you pick the book up, because it's already a small book, and it's not obvious that the pictures really add much. Half-way though, though, I was reminded of Umberto Eco's comment about the start of I promessi sposi, that sometimes text is deliberately there to slow us down. His point was that even skipping or skimming through such text helps us feel the time that's passing in the fictional world. In this book, I realized that the pictures' purpose was, as much as anything, to slow me down. The text was arranged so that no paragraph was ever split across pages, and the "blank" pages were time to think about what you'd just read. (Blank pages wouldn't have worked as well because they have no way to arrest you before you've already turned the page and started on the next.) So, ignore the style; it's a feature, not a bug.

If you're interested in games in the abstract (not even strictly just video games), I'd recommend this as an interesting read.

Mac OS 10.5.2

2008-02-15T06:20:00.002Z

Anyone who cares probably already knows, but Mac OS 10.5.2 is out, and it adds an option (in "Desktop & Screen Saver", rather than "Appearance") to make the menu bar non-translucent, but it doesn't make it look like the 10.4 menu bar; it's a metallic color like the 10.5 title bars, which an ugly hard black bottom edge to it. Honestly, you're better off leaving it translucent. There doesn't seem to be an option to do anything about the transparency of menus themselves, though Apple has somewhat arbitrarily decided to make them non-transparent. I'd have preferred them left alone. (The Help menu still has the ugly shrunken text.)

Time Machine, even if disabled, now adds an icon to the menu bar. Thankfully, unlike the Spotlight icon, you can turn this one off in System Preferences.

Still no Java 6.

For the past month, I've been using 10.4 on a MacBook Pro by day, and 10.5 on a PowerMac G5 by night, and I have to say I honestly have no reason to "upgrade" the MacBook Pro, even though I could if I wanted to. I spent a couple of months with nothing but 10.5, and now I've gone back to a mostly-10.4 situation, and I don't care. There's absolutely nothing compelling in 10.5 to push me over the activation threshold for an update. Sure, if I had to actively click "Cancel" to stop it upgrading itself while I sleep, I'd probably let it go ahead and update. But as long as it's going to take non-zero effort on my part, screw it.

There's exactly one thing I've missed, and I've missed it about once every couple of days. The way they fixed the long-standing bug whereby you weren't able to scroll a window other than the one with the focus? That, it turns out, is the only appreciable difference between 10.4 and 10.5 for me.

Snooze on, my hold-out friends. Snooze on.

I'll wake you when Java 6 arrives, because that will be some solace. Other than that, it's just 10.6. Assuming, that is, Apple's not too busy with the iPhone next time round too.

Rhythmbox

2008-02-04T04:21:00.001Z

I've been in process of "switching" to Linux for roughly forever. Well, since 1998, anyway. I'd been a happy user of Plan 9, IRIX, and Solaris before then (in decreasing order of happiness), as well as being a rather unhappy AIX and SunOS user. I'd seen Linux, but hadn't seen the point. And then in 1998, I found myself with a Sun Ultra 10 on my desk. For reasons that escape me now but which doubtless involved some Solaris decrepitude, I experimented with Linux/SPARC. I remember being shocked at how much faster Linux was than Solaris on the same hardware, but I also remember that I soon gave up and went back to Solaris because the X11 server was unstable.

Still, a seed was planted, and my next desktop box deliberately had all its parts chosen for their known compatibility with Linux (still a necessity then) and I've hardly looked back. RedHat almost put me off, with its insistence that I deal with package dependencies manually, but Debian offered a solution to that, and Ubuntu offered a solution to the sadly typical Debian choice between a repository of broken packages and a repository of packages so stale they could almost ship with Solaris; Ubuntu's six-month cycle offers a good compromise between these two positions, and Ubuntu does a mostly good job of reducing the number of choices Debian expects me to make by offering me reasonable enough default choices. Ten half-working solutions are not better than one mostly-working solution, or, to be honest, one half-working solution.

The only problem is that I live a dual life: developer by day and, well, developer by night. But a different kind of developer. One who has to maintain his own machine and let real people have accounts on it. Because of this, I've been using Macs since 2001. Initially because I wanted a Unix with a nice UI for myself. And although nothing comes close to Mac OS for "nice UI", Linux is still my preferred Unix because I'm a developer. And as a developer, I don't think you can beat Linux. (The obvious exception being if you plan to make money by selling commercial software.) So I've been slowly moving away from Mac OS, and using Linux for more and more things.

RSS was a big sticking point, but I finally gave in to a friend's recommendation of Google Reader. I don't love it (I find its behavior when I scroll or click in an article annoying at times), but it's good enough, and its killer feature is the usual web app advantage of being able to use it from any of my computers, regardless of where I happen to be. Plus it doesn't have ridiculous delusions of being a sovereign app like some of its desktop competition.

RSS, then, is sorted. What's left? At the moment, I use the Mac for Mail and iTunes. I want to keep my mail on my own IMAP server, but I can't stand Evolution or Thunderbird, so I'm sort-of writing a mailer in my copious free time. That's a bit dormant right now. Linux mailers are uniformly awful, and appear to be suffering from an "editor wars" kind of mentality where they're too beholden to their audience of diehards to worry about real people, but it's not hard to imagine that the desktop mailer just isn't going to be very relevant in the future. More than most things, mail lends itself to being a web app because almost everyone's mail is on a machine other than the one right in front of them anyway.

None of this is what I wanted to talk about, though. I wanted to say that I'm divorcing iTunes. I didn't realize how easy it would be, and I haven't seen this spelled out anywhere, so I'm spelling it out here in the hope that it will be useful to someone else.

My situation was that I already had all my music on my Linux box, in exactly the structure that iTunes uses, courtesy of rsync(1). What I thought I wanted was something to convert my iTunes ~/Music directory (and the iTunes database within) into something that a Linux music player would understand. It turns out that there's no need for any such thing. Rhythmbox is perfectly capable of doing the right thing, using the metadata from ~/Music and not duplicating the .mp3 files, and is even capable of checking for changes. So (if you configure it to check for changes, which it doesn't by default) you don't even need to tell it if you add new .mp3 files.

Rhythmbox itself is perfectly adequate. It has the usual GTK+ habit of seeming to take up a lot more screen space than it actually seems to need, and the slider for jumping about within a track doesn't work very well, but the interface is obvious enough and the filtering is fast enough, so it'll do just fine.

Maybe the fact that Rhythmbox can use your iTunes library is so blindingly obvious it doesn't need mentioning, but I didn't know, and I was actively trying to switch to Linux, and had been actively looking for a way to do this. Sometimes you need to explicitly say that something "just works", even if you think your users should feel able to take it for granted.

Someone should write some kind of HOWTO for switching to Linux from the Mac. Maybe I could, presupposing I ever manage to complete the transition!

Things your parents probably don't know about their MacBook

2008-01-17T03:09:00.000Z

I recently had the opportunity to see my parents using their computers. They've been "Mac users" since I gave them a dual G4 and a 23" display. They bought their own MacBook after seeing what a refreshingly unpunishing exercise Mac OS compared to the Windows laptop they'd bought themselves against my advice, before I'd given them the dual G4.

Mac OS has multiple trashes
If you implement reversible deletion, you're faced with a fundamental problem: where does the deleted data go? Apple's simple implementation, where the files in question are moved to a .Trash directory, can still be surprising. For example, suppose my father has a USB flash drive he wants to save to, and is told that the drive is full, he might delete a bunch of files he no longer needs on that drive. But those files have actually just been moved to a .Trash directory on the same device (for speed; doing so means that you don't need to actually move the data, just fiddle the metadata to change the file's parent directory). Unfortunately, because there was already stuff in the .Trash corresponding to the machine's hard disk, the little icon in the Dock was already bulging full, so there's no obvious sign that this is what's happened. And it certainly wasn't my father's expectation: he knew he was deleting this stuff to free up space, and the machine had just told him he couldn't save because the device was full, so "obviously" it knows too.

I'm not sure there is a good answer to this problem. Make deletion non-undoable and you upset a lot of people (not many long-time Unix users, but plenty of real people). I'd be interested to see how people got on with an implementation that copied deleted stuff to a .Trash in their home directory (or, for bonus points, a user-configurable device; people with NFS-mounted home directories, say, might prefer not to keep their trash there for quota reasons). To me, it wouldn't seem unreasonable to treat removable devices differently from non-removable devices. Having deletion undoable only on non-removable devices might be harder to explain than "you can always get files back from the trash", but it does make it easier to answer the question of what and where the trash actually is.

Maybe the interesting question long-term is whether, when all Macs have multiple drives (or partitions) and Time Machine on by default, the concept of trashes will disappear from Mac OS? Will users be expected to use Time Machine to recover their recently-deleted files? It might work better than the current situation, and it might force people who'd otherwise ignore backup to investigate Time Machine...

You can scroll without fiddling with the scroll bar
...Which brings me, in a way, to the question of Apple manuals. Software like iWork comes with pretty thorough printed manuals, and stand-alone copies of Mac OS come with little booklets of varying quality, pointing out some of the more important features without actually going into much detail. The machines themselves come with little booklets, but experience suggests real people don't even notice. They're small, plain, and colorless. They come hidden away in a box that contains a shiny new toy, and they're discarded with said box (or safely stowed away with the box, depending on the individual's personality).

On the one hand, it's understandable, because "how hard can it be?", "how different can it be from the last desktop/laptop I used?", and "all I want to do is watch videos of skateboarding dogs on the web, so anything that isn't obvious isn't worth me worrying about". The trouble is, every ten years or so, something comes along that's both novel and useful, and these people miss out.

For example: you and I know that you can scroll by moving two fingers around on the touchpad. It's even mentioned on pages 21 and 26 of the "Everything Mac" booklet that comes with the MacBook, but the kind of person who reads that probably already knows about the feature from the research they did before buying the product. (The two-finger scrolling is mentioned on Apple's MacBook "Design" page on their web site.) Most users will only read documentation (or search the web) when they run into trouble. Searching the web is a great way to find solutions to problems you know you have, but it's a terrible way to learn about things you haven't imagined. Many people think "difficulty using scroll bars" is the price they pay for having a computer that warms their lap.

I personally think this two-finger scrolling thing is the biggest improvement between my PowerBook G4 of 2001, 666MHz PowerPC powerhouse that it was, and the 2007 MacBook Pro from my employer that I'm using these days. Sure, it's a much faster computer, but it's still a laptop with a low-resolution display and an unpleasant keyboard (funky backlighting notwithstanding) and I still find the least silly way of programming "on" it involves SSH.

While this post was sitting around as a draft, my mum mailed me to thank me for showing her two-finger scrolling. I'd never understood why, despite having more computers in the house than people, my parents still fought over the desktop. Turns out that a good-enough input device reduces demand for the desktop. (It would have been interesting to be able to run the opposite experiment, and give them a 17" MacBook Pro. I doubt it, for their skateboarding-dog-video purposes, or whatever it is old people do with computers.)

Is there a solution to this usability problem? I'm not sure. I'd have suggested a little peel-off sticker, but having just had the job of going round my parents' house removing those "remove before use" stickers from all their Apple power supplies, I'm not sure that's going to work. Maybe as part of the introductory video? It's long struck me as odd that such a thing exists and yet contains no useful content. Whereas the on-line iPhone introductory video was pretty awesome. Short and to the point, yet I came out of it feeling like an iPhone master, and can't claim to have learned much more from the (very good) iPhone manual.

iPhoto isn't part of Mac OS
iPhoto comes with every Mac, but it's actually part of iLife, not Mac OS. So if you do an "erase and install" against your son's advice, you'll lose iPhoto. And you can't just copy across the copy that's on your 10.3 Mac, because a 10.3-era iPhoto won't run on 10.5 (you won't be able to decipher the error, and there won't be much point your son explaining private frameworks, but you'll get the message that it's not going to work).

I think not including iPhoto is unnecessarily tight-fisted. Especially because it's not particularly great, and the fact that it's bundled with a bunch of junk (GarageBand, iDVD, iMovie, and iWeb) strengthens the gut feeling that it's a bit of a scam. I'd bundle iPhoto with Mac OS, unless the lawyers told me I couldn't, in which case I'd make it as cheap as possible. Aperture's the for-pay answer to that particular question. Not making iPhoto part of Mac OS just reduces Mac OS 10.5's already slightly questionable value to real people.

I was shocked to find I knew someone running 10.2 in 2007, but not all that shocked my dad was running 10.3 in the same year, and not at all shocked to find myself running 10.4 in 2008. (I'm running 10.5 on my desktop, but not on my laptop, because it came with 10.4 and by the time I realized I was allowed to run 10.5 on it, I didn't want to risk losing any of my configuration. Which you can take as my official verdict on 10.5: not yet interesting. Literally the only thing I'm missing is the ability to scroll a window without making it the foreground window. Which is the long-overdue fixing of a glaring bug rather than a feature. Bear in mind that I'm running 10.5 at home, so it's not like I've forgotten all its great advantages. There's no "there" there.)

CAPTCHA defeats old people
Finally, in this random collection of little things I noticed, CAPTCHAs don't just defeat bots: they defeat old people too. My mum had been complaining for a while that she sometimes couldn't send mail from one of the big-three webmail systems. What she hadn't managed to articulate until I actually watched her is that the reason is that she can't reliably solve CAPTCHAs. For one thing, her vision's not what it used to be. But I noticed also that because the CAPTCHAs in question don't actually mean anything (they're arbitrary alphanumeric strings) she was having great difficulty with, say, 'z', 'Z', '2', and '7' run into another letter. And I'll be completely honest with you: one of the ones she showed me that she was having trouble with, I couldn't solve either.

I'm not sure what the solution there is, but I do notice that I happen not to use anything that uses CAPTCHAs. I can't remember the last one I saw myself.