Press "Enter" to skip to content

Day: September 14, 2008

3ware 8006-2LP Sucks Under FreeBSD 7.0-RELEASE

Published September 14, 2008

Results from using Bonnie++ on FreeBSD 7.0 with a 3ware controller (twe), gmirror, and just a single local disk.

As mentioned here I got my new server working with a 3ware 8006-2LP and a pair of new 500GB disks. While it was working fine, I noticed that when updating the FreeBSD ports collection that the update would occasionally pause, consuming no CPU, but with the update process having a status of sbwait. I understand this to mean that the process is waiting on a blocked socket.

It turns out that the twe(4) driver is what is known as GIANT-LOCKED, which I believe means that it uses the old SMP locking mechanism in FreeBSD:

twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x8c00-0x8c0f mem 0xfc7ffc00-0xfc7ffc0f,0xfb800000-0xfbffffff irq 28 at device 3.0 on pci1
twe0: [GIANT-LOCKED]
twe0: [ITHREAD]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048

Best I can tell, the result of this is that the disk controller’s driver needs to wait for the kernel to free up other resources and tell the driver that it can go ahead and work before it does things. The result of this tends to be that the driver works well, but there is a lot of latency.

This understanding matches what I observed, which was the aforementioned lengthy pauses when doing things which required a bunch of disk IO. In order to prove this understanding out, I set up a test hard disk running a stock FreeBSD 7.0 amd64 installation from which I could run Bonnie++, a file-based disk benchmarking suite.

In my testing I used the following three scenarios:

· One 120GB IBM Deskstar PATA drive (IC35L120AVVA07) connected to the motherboard booting the OS, listed in the results as banstyle_deskstar.
· Two 500GB Western Digital SATA drives (WD5000AAKS-40TMA0) connected to the motherboard with software RAID 1 via gmirror(8), listed in the results as banstyle_gmirror.
· Two 500GB Seagate SATA drives (ST3500320AS) connected to the 3ware 8006-2LP using the twe(4) driver in hardware RAID 1, listed in the results as banstyle_twe.

The result ended up being that all three configurations are generally around the same speed for throughput, but the 3ware controller had an absurd amount of latency. If one looks at the HTML version of the Bonnie++ output here (or the PNG here or above), one can see that was giving near three SECONDS of latency for random seeks and writes using write(2). This is insane.

The only thing I can think to attribute this to is the GIANT-LOCK in twe(4). I guess this means that I’m going to have to go back to gmirror(8) for software RAID and return the card. How disappointing.

(If anyone reading this disagrees with these findings or wishes to comment on them, please don’t hesitate to do so here or by emailing me directly.)

7 Comments

Salsa’s Fargo

Published September 14, 2008

Salsa's stock photo of their new bike, Fargo. It is a touring bike based around 29" mountain bike wheels. I think I want one.

While waiting for Bonnie++ to run (more on this later — it’s not good) I wanted to post about a new bike recently announced by Salsa, an interesting looking touring bike built around a mountain bike-ish frame and 29″ MTB wheels.

For the last month or so I’ve been kicking around the idea of building a new bike for myself, but each time I find myself stuck with either the idea of another mountain bike or something a bit more touring-ish. I have a mountain bike which fits me quite nicely, so I’ve started to think that I should just be content with what I have and let things be. Then, while looking up info on the new wheel that built, I come across Salsa’s Fargo. This bike looks like it’d do an incredibly good job at longer rides and possible commuting, and maybe even some biking / camping trips.

There’s something I really like about the idea of 29er wheels (29″ / 700c), the ability to use wide mountain bike tires, clearance for fenders with 2.3″ tires, disc brakes, etc.

At an estimated $2k (built up) I don’t know if I’ll be able to afford one, but it’s sure tempting to start thinking about one. The frames are supposed to ship in November and the bikes themselves in February. Trails Edge sells Salsa stuff and is (relatively) local… Maybe I should finally pay them a visit…

Leave a Comment

Black and Shiny

Published September 14, 2008

Set up to polish my boots in the laundry room. One boot is done.

After eating some really nice Skillet Baked Ziti (recipe from America’s Test Kitchen) that Danielle made for dinner I avoided working on my server by polishing my boots. As you can see above or at this close-up of the toes of my boots, they needed it.

Now I get to go back to figuring out why twe(4) in FreeBSD 7.0 seems sluggish. It may just be my perception, so I’m double-checking this by comparing the new 3ware-based array to the old gmirror(8) version. Or, it may be that it’s one of three drivers (the other two are ohci(4) and atkbd(4)) which indicate that they are GIANT-LOCKED, which means that they use the old SMP locking method.

Leave a Comment

Control

Published September 14, 2008

Screen capture from Control of Sam Riley as Ian Curtis, with the Unknown Pleasures album artwork in the background.

Danielle and I finally watched Control (Official Site · IMDB · Wikipedia), which she had received from Netflix last week. While it was a bit slow and (obviously) predictable, I enjoyed it.

I think that tonight I also got banstyle.nuxx.net working properly again. Over the past two days I did a bunch of extensive testing with spare RAM, Breakin, and a white board, and I think that I may have narrowed down the problem. I believe that the MCEs I was seeing were caused by a combination of a failing DIMM and modules which were the same in part number but not in actual chip content. There may actually be a bad slot there too, but I’m not certain of that.

I’ve winnowed the box down to 6GB of matched, tested RAM and it seems to pass all the tests I’ve thrown at it thus far. With the discovery that ad6 is dying as well I ordered a 3ware 8006-2LP and two Seagate ST3500320AS 500GB disks. Those were fitted into the server and I then dumped the the partitions from ad4 to it and everything seemed to be working fine, but occasionally slowly. Jumpering the board to force the first PCI-X slot to 66MHz (to match the PCI 8006-2LP) and turning on bus mastering for IDE transfers on the PCI slots seems to have sorted this out.

SMART tests and a number of hours of Breakin have shown the disks to be okay, so come Monday morning I’ll attempt to get a good 36 hours of burning in happening. If this all goes good the server will be back in place on Wednesday, with everything moved (shifted?) back over by Thursday evening.

If you are interested, here is a photo of my workbench just after dumping the partitions from one half of the old mirror to the new mirror set. Due to a bug in dump (or UFS) on FreeBSD 7.0 I had 6.3 booting off of an external USB drive, running dump to throw data from disk to another, a partition at a time.

After that photo was taken fstab was edited, everything booted up great, and then the new drives each passed an extended offline SMART test.

Leave a Comment