Hidden in Plain Sight

In December 1997, Sun Microsystems had just announced its new flagship machine:
a 64-processor symmetric multiprocessor supporting up to 64 gigabytes of memory
and thousands of I/O devices. As with any new machine launch, Sun was working
feverishly on benchmarks to prove the machine’s performance. While the
benchmarks were generally impressive, there was one in particular—an
especially complicated benchmark involving several machines—that was
exhibiting unexpectedly low performance. The benchmark machine—a fully
racked-out behemoth with the maximum configuration of 64 processors—would
occasionally become mysteriously distracted: Benchmark activity would practically
cease, but the operating system kernel remained furiously busy. After some
number of minutes spent on unknown work, the operating system would suddenly
right itself: Benchmark activity would resume at full throttle and run to completion.
Those running the benchmark could see that the machine was on course to break
the world record, but these minutes-long periods of unknown kernel activity
were enough to be the difference between first and worst.