 |
As mentioned here I got my new server working with a 3ware 8006-2LP and a pair of new 500GB disks. While it was working fine, I noticed that when updating the FreeBSD ports collection that the update would occasionally pause, consuming no CPU, but with the update process having a status of sbwait. I understand this to mean that the process is waiting on a blocked socket.
It turns out that the twe(4) driver is what is known as GIANT-LOCKED, which I believe means that it uses the old SMP locking mechanism in FreeBSD:
twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x8c00-0x8c0f mem 0xfc7ffc00-0xfc7ffc0f,0xfb800000-0xfbffffff irq 28 at device 3.0 on pci1
twe0: [GIANT-LOCKED]
twe0: [ITHREAD]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
Best I can tell, the result of this is that the disk controller’s driver needs to wait for the kernel to free up other resources and tell the driver that it can go ahead and work before it does things. The result of this tends to be that the driver works well, but there is a lot of latency.
This understanding matches what I observed, which was the aforementioned lengthy pauses when doing things which required a bunch of disk IO. In order to prove this understanding out, I set up a test hard disk running a stock FreeBSD 7.0 amd64 installation from which I could run Bonnie++, a file-based disk benchmarking suite.
In my testing I used the following three scenarios:
· One 120GB IBM Deskstar PATA drive (IC35L120AVVA07) connected to the motherboard booting the OS, listed in the results as banstyle_deskstar.
· Two 500GB Western Digital SATA drives (WD5000AAKS-40TMA0) connected to the motherboard with software RAID 1 via gmirror(8), listed in the results as banstyle_gmirror.
· Two 500GB Seagate SATA drives (ST3500320AS) connected to the 3ware 8006-2LP using the twe(4) driver in hardware RAID 1, listed in the results as banstyle_twe.
The result ended up being that all three configurations are generally around the same speed for throughput, but the 3ware controller had an absurd amount of latency. If one looks at the HTML version of the Bonnie++ output here (or the PNG here or above), one can see that was giving near three SECONDS of latency for random seeks and writes using write(2). This is insane.
The only thing I can think to attribute this to is the GIANT-LOCK in twe(4). I guess this means that I’m going to have to go back to gmirror(8) for software RAID and return the card. How disappointing.
(If anyone reading this disagrees with these findings or wishes to comment on them, please don’t hesitate to do so here or by emailing me directly.)
I just got 3ware 8006-2LP. I’ve tried to install it to my supermicro server and disk performance really sucks under FreeBSD 7.1! Did you contact 3ware technical support about this problem?
Maxim Popov: No, sorry. I didn’t bother… I figured that since I was using OSS drivers on what’s technically an unsupported OS that 3ware wouldn’t help me out. The problem is with the driver in FreeBSD, not the card itself.
Another question, did you test it with Write Cache ON or OFF? I know that Write Cache ON kinda dangerous without battery backup but still.
I had it at its defaults, which is OFF. That card was actually being tested in the server this site is hosted on. I ended up just going with FreeBSD’s geom mirroring, since it’s tested to work well.
It’s a bit disappointing, because I’ve got another 8006-2LP which was used under 6.x in my previous server, and it works very well there. It’s too bad the 9000 series is so expensive, because those aren’t GIANT-LOCKED under 7.x.
> It’s a bit disappointing
totally agree with you
> It’s too bad the 9000 series is so expensive, because those aren’t GIANT-LOCKED under 7.x.
hm. it’s kinda expensive for me too… but there is huge saving for 9500S-4LP, you can get it for $175.00 from 3ware website?
Thanks for your blog :)
Another question, if you don’t mind, you mentioned before that you used 8006-2LP under 6.x and it worked very well. I’ve try to boot 6.3 and it showed me the same GIANT-LOCKED for the twe0, which means that I’m gonna get the same performance for 8006-2LP under 6.3/6.4. Am I right? Can you please explain little bit more about “I’ve got another 8006-2LP which was used under 6.x in my previous server, and it works very well there.”
Thank you
To be honest, I’m actually not quite remembering whether it was GIANT-LOCKED or not, now that you mention it. I don’t have a dmesg from that box either, as it’s been wiped and gutted and is now serving second duty as a dev box down in the basement. I’m sorry.