I think that tonight I also got banstyle.nuxx.net working properly again. Over the past two days I did a bunch of extensive testing with spare RAM, Breakin, and a white board, and I think that I may have narrowed down the problem. I believe that the MCEs I was seeing were caused by a combination of a failing DIMM and modules which were the same in part number but not in actual chip content. There may actually be a bad slot there too, but I’m not certain of that.
I’ve winnowed the box down to 6GB of matched, tested RAM and it seems to pass all the tests I’ve thrown at it thus far. With the discovery that ad6 is dying as well I ordered a 3ware 8006-2LP and two Seagate ST3500320AS 500GB disks. Those were fitted into the server and I then dumped the the partitions from ad4 to it and everything seemed to be working fine, but occasionally slowly. Jumpering the board to force the first PCI-X slot to 66MHz (to match the PCI 8006-2LP) and turning on bus mastering for IDE transfers on the PCI slots seems to have sorted this out.
SMART tests and a number of hours of Breakin have shown the disks to be okay, so come Monday morning I’ll attempt to get a good 36 hours of burning in happening. If this all goes good the server will be back in place on Wednesday, with everything moved (shifted?) back over by Thursday evening.
If you are interested, here is a photo of my workbench just after dumping the partitions from one half of the old mirror to the new mirror set. Due to a bug in dump (or UFS) on FreeBSD 7.0 I had 6.3 booting off of an external USB drive, running dump to throw data from disk to another, a partition at a time.
After that photo was taken fstab was edited, everything booted up great, and then the new drives each passed an extended offline SMART test.