I got to work on part of this system, which IBM calls Autobench, for my senior project at PSU. The system is a highly configurable framework which can download, compile, and run various benchmarks and profilers (for example while compiling a kernel). Its all centrally administered, so IBM can run a battery of tests on a variety of different machines at once.

I think Martin Bligh said that IBM has been using this for a while now, automatically downloading kernels upon release and testing them. The new thing is the regression matrix which is posted to the web. Very cool.