nuxx.net – Page 135 – Making, baking, and (un-)breaking things in Southeast Michigan.

3ware 8006-2LP Sucks Under FreeBSD 7.0-RELEASE

Published September 14, 2008

Results from using Bonnie++ on FreeBSD 7.0 with a 3ware controller (twe), gmirror, and just a single local disk.

As mentioned here I got my new server working with a 3ware 8006-2LP and a pair of new 500GB disks. While it was working fine, I noticed that when updating the FreeBSD ports collection that the update would occasionally pause, consuming no CPU, but with the update process having a status of sbwait. I understand this to mean that the process is waiting on a blocked socket.

It turns out that the twe(4) driver is what is known as GIANT-LOCKED, which I believe means that it uses the old SMP locking mechanism in FreeBSD:

twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x8c00-0x8c0f mem 0xfc7ffc00-0xfc7ffc0f,0xfb800000-0xfbffffff irq 28 at device 3.0 on pci1
twe0: [GIANT-LOCKED]
twe0: [ITHREAD]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048

Best I can tell, the result of this is that the disk controller’s driver needs to wait for the kernel to free up other resources and tell the driver that it can go ahead and work before it does things. The result of this tends to be that the driver works well, but there is a lot of latency.

This understanding matches what I observed, which was the aforementioned lengthy pauses when doing things which required a bunch of disk IO. In order to prove this understanding out, I set up a test hard disk running a stock FreeBSD 7.0 amd64 installation from which I could run Bonnie++, a file-based disk benchmarking suite.

In my testing I used the following three scenarios:

· One 120GB IBM Deskstar PATA drive (IC35L120AVVA07) connected to the motherboard booting the OS, listed in the results as banstyle_deskstar.
· Two 500GB Western Digital SATA drives (WD5000AAKS-40TMA0) connected to the motherboard with software RAID 1 via gmirror(8), listed in the results as banstyle_gmirror.
· Two 500GB Seagate SATA drives (ST3500320AS) connected to the 3ware 8006-2LP using the twe(4) driver in hardware RAID 1, listed in the results as banstyle_twe.

The result ended up being that all three configurations are generally around the same speed for throughput, but the 3ware controller had an absurd amount of latency. If one looks at the HTML version of the Bonnie++ output here (or the PNG here or above), one can see that was giving near three SECONDS of latency for random seeks and writes using write(2). This is insane.

The only thing I can think to attribute this to is the GIANT-LOCK in twe(4). I guess this means that I’m going to have to go back to gmirror(8) for software RAID and return the card. How disappointing.

(If anyone reading this disagrees with these findings or wishes to comment on them, please don’t hesitate to do so here or by emailing me directly.)

7 Comments

Salsa’s Fargo

Published September 14, 2008

Salsa's stock photo of their new bike, Fargo. It is a touring bike based around 29" mountain bike wheels. I think I want one.

While waiting for Bonnie++ to run (more on this later — it’s not good) I wanted to post about a new bike recently announced by Salsa, an interesting looking touring bike built around a mountain bike-ish frame and 29″ MTB wheels.

For the last month or so I’ve been kicking around the idea of building a new bike for myself, but each time I find myself stuck with either the idea of another mountain bike or something a bit more touring-ish. I have a mountain bike which fits me quite nicely, so I’ve started to think that I should just be content with what I have and let things be. Then, while looking up info on the new wheel that built, I come across Salsa’s Fargo. This bike looks like it’d do an incredibly good job at longer rides and possible commuting, and maybe even some biking / camping trips.

There’s something I really like about the idea of 29er wheels (29″ / 700c), the ability to use wide mountain bike tires, clearance for fenders with 2.3″ tires, disc brakes, etc.

At an estimated $2k (built up) I don’t know if I’ll be able to afford one, but it’s sure tempting to start thinking about one. The frames are supposed to ship in November and the bikes themselves in February. Trails Edge sells Salsa stuff and is (relatively) local… Maybe I should finally pay them a visit…

Leave a Comment

Black and Shiny

Published September 14, 2008

Set up to polish my boots in the laundry room. One boot is done.

After eating some really nice Skillet Baked Ziti (recipe from America’s Test Kitchen) that Danielle made for dinner I avoided working on my server by polishing my boots. As you can see above or at this close-up of the toes of my boots, they needed it.

Now I get to go back to figuring out why twe(4) in FreeBSD 7.0 seems sluggish. It may just be my perception, so I’m double-checking this by comparing the new 3ware-based array to the old gmirror(8) version. Or, it may be that it’s one of three drivers (the other two are ohci(4) and atkbd(4)) which indicate that they are GIANT-LOCKED, which means that they use the old SMP locking method.

Leave a Comment

Control

Published September 14, 2008

Danielle and I finally watched Control (Official Site · IMDB · Wikipedia), which she had received from Netflix last week. While it was a bit slow and (obviously) predictable, I enjoyed it.

I think that tonight I also got banstyle.nuxx.net working properly again. Over the past two days I did a bunch of extensive testing with spare RAM, Breakin, and a white board, and I think that I may have narrowed down the problem. I believe that the MCEs I was seeing were caused by a combination of a failing DIMM and modules which were the same in part number but not in actual chip content. There may actually be a bad slot there too, but I’m not certain of that.

I’ve winnowed the box down to 6GB of matched, tested RAM and it seems to pass all the tests I’ve thrown at it thus far. With the discovery that ad6 is dying as well I ordered a 3ware 8006-2LP and two Seagate ST3500320AS 500GB disks. Those were fitted into the server and I then dumped the the partitions from ad4 to it and everything seemed to be working fine, but occasionally slowly. Jumpering the board to force the first PCI-X slot to 66MHz (to match the PCI 8006-2LP) and turning on bus mastering for IDE transfers on the PCI slots seems to have sorted this out.

SMART tests and a number of hours of Breakin have shown the disks to be okay, so come Monday morning I’ll attempt to get a good 36 hours of burning in happening. If this all goes good the server will be back in place on Wednesday, with everything moved (shifted?) back over by Thursday evening.

If you are interested, here is a photo of my workbench just after dumping the partitions from one half of the old mirror to the new mirror set. Due to a bug in dump (or UFS) on FreeBSD 7.0 I had 6.3 booting off of an external USB drive, running dump to throw data from disk to another, a partition at a time.

After that photo was taken fstab was edited, everything booted up great, and then the new drives each passed an extended offline SMART test.

Leave a Comment

Time Machine Network Backup Speedup / Fix

Published September 11, 2008

I just acquired a new external disk enclosure and 750GB disk for hanging off of an AirPort Extreme and using for Time Machine backups of my main machine. From this I currently have ~480GB of data to back up, and for some reason the initial large backup repeatedly fails when I attempt to do it over the network.

The easy way around this is to first do the backup to the drive when it is connected locally and then hang it off of the AirPort Extreme to continue the incremental backups. The problem is that this doesn’t work as one would expect, because when an initial Time Machine backup is made to a local disk the backup ends up in a series of subdirectories, which is a different format from what it is via network.

When the backup is made to a volume hanging off of an AirPort Extreme a .sparsebundle file is created containing the backup; essentially a disk image stored on the network. Therefore, if you make a Time Machine backup locally and then try to use it via an AirPort Extreme the .sparsebundle file will be created on the disk in parallel to the now-useless directory structure.

So, how do you work around this? Easy. Hook the external disk up to the AirPort Extreme then either let the backup fail or cancel it, which will leave the incomplete .sparsebundle file on the disk. Disconnect the drive from the AirPort Extreme, connect it to your Mac, and point Time Machine to that volume. If it finds an appropriate .sparsebundle on the volume (which it will, since it’s already there) it’ll use that instead of creating the aforementioned subdirectory structure.

The backup will then happen quite quickly, and after it completes you can just hang the drive back off of the AirPort Extreme, redirect Time Machine to back up to that network volume, and things will continue via the network.

UPDATE: Since 10.5.5 was applied to my machine I have been unable to use this backup method and have had to resort to making the entire initial backup via network.

Leave a Comment

ad6 is Dying Too!

Published September 11, 2008

Error messages on the console showing that ad6 is actually failing hard. Good thing I ordered replacement disks.

It’s a good thing I received a 3ware 8006-2LP and a pair of Seagate 500GB disks today, because one of the two drives in the mirror set on my new server is just about to fail. To make matters worse, the failing disk is ad6, and ad4 is the one I’d accidently broken the other night, so I’ve been desperately waiting for the disks to finish syncing so that everything would be backed up.

This failed at ~10:00am this morning, which kept me from rebooting the box remotely to run more stress testing and (hopefully) replicating last night’s error.

Now that the data is sync’d I’ll wait for it to finish fscking then I’ll shut it down cleanly and begin running Breakin again.

Leave a Comment

London in November

Published September 11, 2008

This morning Danielle and I booked a vacation to London in November. We’ll be staying at the Holiday Inn Express LONDON-EARL’S COURT from 09-Nov through 17-Nov. Total cost only ended up being $1900-some for the two of us, flying on unrestricted NWA tickets non-stop out of Detroit.

Now to figure out what to do in the UK… We’d like to visit some people from here and #llamasoft, and hopefully some interesting places. Plenty of wandering is in order, too.

Leave a Comment

Esquire Is Good For Something

Published September 10, 2008

The front page e-ink display and PCB from the cover of the October 2008 issue of Esquire magazine.

Yes, I too picked up a copy of the October 2008 issue of Esquire. The magazine itself is going in the trash, but I pulled the front panel apart so that I could poke with an e-paper / Electronic Paper / E-Ink display. I must say, this is a very nice, very high contrast display. If this were put in some more portable, more durable, more cost effective form than the Kindle (and without the data network crap) I could see myself getting one to use for reading.

When the magazine was sitting on the front seat of my car, glancing over at it reminded me of $RANDOM_NEAR_FUTURE_SCIFI where magazines are shown sitting on tables and racks blinking and flashing away. To be honest, I found it as irritating as a banner ad. I hope this isn’t where things actually go tech-wise.

Also, when I arrived home today I noticed a lot of dust and cut marks in the asphalt near my garage door, which makes me believe that something is being done about the sunken area there. What’s strange is that it’s been like this for years without issue. I’m not sure why it’s being fixed now.

Leave a Comment

CPU 2: Machine Check Exception: 4 Bank 4: f61c2001ba080813

Published September 9, 2008

A real, honest, good failure while running Breakin on banstyle.nuxx.net. It points to something being wrong with the second CPU or bank of memory.

In testing my server banstyle.nuxx.net has had its first real set of errors / failures. This is a good thing.

First, last night I started getting SMART warnings about bad blocks on ad6, which is the second hard drive. So today I just went ahead and ordered up a pair of ST3500320AS 500GB disks and a 3ware 8006-2LP, the same as is used in my current server.

Note the sdb errors, which are consistent with the other errors I’d been seeing indicating a bad block on the second hard disk.

Second, I came home today and found my server hung while running Breakin, displaying the error CPU 2: Machine Check Exception: 4 Bank 4: f61c2001ba080813 TSC 2561d00c4ef7 ADDR ce19fd00. So, at least I’ve got some place to look for what else might be the issue.

2 Comments

nuxx.net Is Back Up

Published September 8, 2008

Well, my site, nuxx.net is back up. I have the new server here at home and I’m starting to take a look at it. Hopefully I’ll have some sort of answer soon.

Unfortunately, in looking at it, I both screwed the BIOS and the software RAID array. In the BIOS I tried to backrev the BIOS, only to find out that Tyan (motherboard manufacturer) had changed Flash chips with the particular board I got, and the older BIOS’ don’t support it. Long story short, I was able to downgrade to an older BIOS, but as that older BIOS doesn’t support the new chip type, attempting to upgrade it again simply causes the flashing program to report “Error : Flash part is not supported”.

Beyond that, I was waiting for the server to rebuild the array (after the hard power off of the failure on Sunday morning) and getting impatient, so I decided to disconnect (via software) the inconsistent half of the array, thinking that I could just let it finish building later. This didn’t go so well (for some reason) and I ended up breaking the array. I think it’s back together, but I do worry a bit that something may be lost. We’ll see, I guess.

Mail is back up, things were sync’d over, but there is/will be some quirkyness with the mail received in the last day or two. Expect to see some duplicates. Sorry.

Oh, great: Sep 7 22:37:12 banstyle smartd[863]: Device: /dev/ad6, 1 Currently unreadable (pending) sectors

Right now I’m feeling really frustrated with this whole process and wanting to just put it away and maybe start over later. I just don’t want to leave everyone else’s stuff down.

Leave a Comment

nuxx.net Posts