Category: computers

SMART Issues

Published September 16, 2008

When I got home I started running SeaTools, Seagate’s disk diagnostics utility for Windows, on the ad4 which had begun failing earlier. It reported back that it, and the other hard drive, were just fine. However, when booting into FreeBSD after using them I found that both drives were now indicating that Seek_Error_Rate was past threshold. The OS booted very slow, then kicked ad6 out of the mirror set.

I tried connecting the drives to another, standalone SATA controller (some plain old Maxtor bundle-in one) with new SATA cables and same problem.

So, I’m not sure what to do. Here’s every issue I’ve had with the new server and its resolution:

Issue: Server locking up hard, unexpectedly. MCEs on console.
Resolution: Ensure that only matched RAM is used and that all RAM tests good during burn-in.

Issue: Slow performance / absurd latency while using 3ware disk controller.
Resolution: Identified GIANT-LOCK on driver, moved to using software mirroring.

Issue: One of the original two Western Digital disks used, which were part of a gmirror set, has started giving block errors.
Resolution: Replace disks with brand new Seagate pair.

Issue: Both of the new Seagate drives began failing with excessive Seek_Error_Rate within a few hours of each other after extensive burn in.
Resolution: Unsure.

I can’t help but wonder if one of the Seagates beginning to fail was contributing to the latency observed with the 3ware controller, but as neither was throwing SMART errors at the time, so I discount this.

My current thought is that I should order a pair of server-grade disks, burn them in as before (~50 hours of constant activity), copy the data to them, then see if things will keep working. The failed disks and the unwanted 3ware controller will go back to Newegg, and hopefully things will work right.

I don’t know what other option I have besides scrapping the whole idea of moving servers, but I really rather not do that. If anyone else has any ideas, I’d love to hear them…

Leave a Comment

New Hard Disk Is Failing

Published September 16, 2008

root@banstyle:~# smartctl -H /dev/ad4
smartctl version 5.38 [amd64-portbld-freebsd7.0] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
7 Seek_Error_Rate 0x000f 013 012 030 Pre-fail Always FAILING_NOW 38293929828058

root@banstyle:~#

I can’t win. Now one of the brand new hard disks in the server is getting a bunch of seek errors.

Leave a Comment

3ware 8006-2LP Sucks Under FreeBSD 7.0-RELEASE

Published September 14, 2008

Results from using Bonnie++ on FreeBSD 7.0 with a 3ware controller (twe), gmirror, and just a single local disk.

As mentioned here I got my new server working with a 3ware 8006-2LP and a pair of new 500GB disks. While it was working fine, I noticed that when updating the FreeBSD ports collection that the update would occasionally pause, consuming no CPU, but with the update process having a status of sbwait. I understand this to mean that the process is waiting on a blocked socket.

It turns out that the twe(4) driver is what is known as GIANT-LOCKED, which I believe means that it uses the old SMP locking mechanism in FreeBSD:

twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x8c00-0x8c0f mem 0xfc7ffc00-0xfc7ffc0f,0xfb800000-0xfbffffff irq 28 at device 3.0 on pci1
twe0: [GIANT-LOCKED]
twe0: [ITHREAD]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048

Best I can tell, the result of this is that the disk controller’s driver needs to wait for the kernel to free up other resources and tell the driver that it can go ahead and work before it does things. The result of this tends to be that the driver works well, but there is a lot of latency.

This understanding matches what I observed, which was the aforementioned lengthy pauses when doing things which required a bunch of disk IO. In order to prove this understanding out, I set up a test hard disk running a stock FreeBSD 7.0 amd64 installation from which I could run Bonnie++, a file-based disk benchmarking suite.

In my testing I used the following three scenarios:

· One 120GB IBM Deskstar PATA drive (IC35L120AVVA07) connected to the motherboard booting the OS, listed in the results as banstyle_deskstar.
· Two 500GB Western Digital SATA drives (WD5000AAKS-40TMA0) connected to the motherboard with software RAID 1 via gmirror(8), listed in the results as banstyle_gmirror.
· Two 500GB Seagate SATA drives (ST3500320AS) connected to the 3ware 8006-2LP using the twe(4) driver in hardware RAID 1, listed in the results as banstyle_twe.

The result ended up being that all three configurations are generally around the same speed for throughput, but the 3ware controller had an absurd amount of latency. If one looks at the HTML version of the Bonnie++ output here (or the PNG here or above), one can see that was giving near three SECONDS of latency for random seeks and writes using write(2). This is insane.

The only thing I can think to attribute this to is the GIANT-LOCK in twe(4). I guess this means that I’m going to have to go back to gmirror(8) for software RAID and return the card. How disappointing.

(If anyone reading this disagrees with these findings or wishes to comment on them, please don’t hesitate to do so here or by emailing me directly.)

7 Comments

Black and Shiny

Published September 14, 2008

Set up to polish my boots in the laundry room. One boot is done.

After eating some really nice Skillet Baked Ziti (recipe from America’s Test Kitchen) that Danielle made for dinner I avoided working on my server by polishing my boots. As you can see above or at this close-up of the toes of my boots, they needed it.

Now I get to go back to figuring out why twe(4) in FreeBSD 7.0 seems sluggish. It may just be my perception, so I’m double-checking this by comparing the new 3ware-based array to the old gmirror(8) version. Or, it may be that it’s one of three drivers (the other two are ohci(4) and atkbd(4)) which indicate that they are GIANT-LOCKED, which means that they use the old SMP locking method.

Leave a Comment

Control

Published September 14, 2008

Danielle and I finally watched Control (Official Site · IMDB · Wikipedia), which she had received from Netflix last week. While it was a bit slow and (obviously) predictable, I enjoyed it.

I think that tonight I also got banstyle.nuxx.net working properly again. Over the past two days I did a bunch of extensive testing with spare RAM, Breakin, and a white board, and I think that I may have narrowed down the problem. I believe that the MCEs I was seeing were caused by a combination of a failing DIMM and modules which were the same in part number but not in actual chip content. There may actually be a bad slot there too, but I’m not certain of that.

I’ve winnowed the box down to 6GB of matched, tested RAM and it seems to pass all the tests I’ve thrown at it thus far. With the discovery that ad6 is dying as well I ordered a 3ware 8006-2LP and two Seagate ST3500320AS 500GB disks. Those were fitted into the server and I then dumped the the partitions from ad4 to it and everything seemed to be working fine, but occasionally slowly. Jumpering the board to force the first PCI-X slot to 66MHz (to match the PCI 8006-2LP) and turning on bus mastering for IDE transfers on the PCI slots seems to have sorted this out.

SMART tests and a number of hours of Breakin have shown the disks to be okay, so come Monday morning I’ll attempt to get a good 36 hours of burning in happening. If this all goes good the server will be back in place on Wednesday, with everything moved (shifted?) back over by Thursday evening.

If you are interested, here is a photo of my workbench just after dumping the partitions from one half of the old mirror to the new mirror set. Due to a bug in dump (or UFS) on FreeBSD 7.0 I had 6.3 booting off of an external USB drive, running dump to throw data from disk to another, a partition at a time.

After that photo was taken fstab was edited, everything booted up great, and then the new drives each passed an extended offline SMART test.

Leave a Comment

Time Machine Network Backup Speedup / Fix

Published September 11, 2008

I just acquired a new external disk enclosure and 750GB disk for hanging off of an AirPort Extreme and using for Time Machine backups of my main machine. From this I currently have ~480GB of data to back up, and for some reason the initial large backup repeatedly fails when I attempt to do it over the network.

The easy way around this is to first do the backup to the drive when it is connected locally and then hang it off of the AirPort Extreme to continue the incremental backups. The problem is that this doesn’t work as one would expect, because when an initial Time Machine backup is made to a local disk the backup ends up in a series of subdirectories, which is a different format from what it is via network.

When the backup is made to a volume hanging off of an AirPort Extreme a .sparsebundle file is created containing the backup; essentially a disk image stored on the network. Therefore, if you make a Time Machine backup locally and then try to use it via an AirPort Extreme the .sparsebundle file will be created on the disk in parallel to the now-useless directory structure.

So, how do you work around this? Easy. Hook the external disk up to the AirPort Extreme then either let the backup fail or cancel it, which will leave the incomplete .sparsebundle file on the disk. Disconnect the drive from the AirPort Extreme, connect it to your Mac, and point Time Machine to that volume. If it finds an appropriate .sparsebundle on the volume (which it will, since it’s already there) it’ll use that instead of creating the aforementioned subdirectory structure.

The backup will then happen quite quickly, and after it completes you can just hang the drive back off of the AirPort Extreme, redirect Time Machine to back up to that network volume, and things will continue via the network.

UPDATE: Since 10.5.5 was applied to my machine I have been unable to use this backup method and have had to resort to making the entire initial backup via network.

Leave a Comment

ad6 is Dying Too!

Published September 11, 2008

Error messages on the console showing that ad6 is actually failing hard. Good thing I ordered replacement disks.

It’s a good thing I received a 3ware 8006-2LP and a pair of Seagate 500GB disks today, because one of the two drives in the mirror set on my new server is just about to fail. To make matters worse, the failing disk is ad6, and ad4 is the one I’d accidently broken the other night, so I’ve been desperately waiting for the disks to finish syncing so that everything would be backed up.

This failed at ~10:00am this morning, which kept me from rebooting the box remotely to run more stress testing and (hopefully) replicating last night’s error.

Now that the data is sync’d I’ll wait for it to finish fscking then I’ll shut it down cleanly and begin running Breakin again.

Leave a Comment

CPU 2: Machine Check Exception: 4 Bank 4: f61c2001ba080813

Published September 9, 2008

A real, honest, good failure while running Breakin on banstyle.nuxx.net. It points to something being wrong with the second CPU or bank of memory.

In testing my server banstyle.nuxx.net has had its first real set of errors / failures. This is a good thing.

First, last night I started getting SMART warnings about bad blocks on ad6, which is the second hard drive. So today I just went ahead and ordered up a pair of ST3500320AS 500GB disks and a 3ware 8006-2LP, the same as is used in my current server.

Note the sdb errors, which are consistent with the other errors I’d been seeing indicating a bad block on the second hard disk.

Second, I came home today and found my server hung while running Breakin, displaying the error CPU 2: Machine Check Exception: 4 Bank 4: f61c2001ba080813 TSC 2561d00c4ef7 ADDR ce19fd00. So, at least I’ve got some place to look for what else might be the issue.

2 Comments

nuxx.net Is Back Up

Published September 8, 2008

Well, my site, nuxx.net is back up. I have the new server here at home and I’m starting to take a look at it. Hopefully I’ll have some sort of answer soon.

Unfortunately, in looking at it, I both screwed the BIOS and the software RAID array. In the BIOS I tried to backrev the BIOS, only to find out that Tyan (motherboard manufacturer) had changed Flash chips with the particular board I got, and the older BIOS’ don’t support it. Long story short, I was able to downgrade to an older BIOS, but as that older BIOS doesn’t support the new chip type, attempting to upgrade it again simply causes the flashing program to report “Error : Flash part is not supported”.

Beyond that, I was waiting for the server to rebuild the array (after the hard power off of the failure on Sunday morning) and getting impatient, so I decided to disconnect (via software) the inconsistent half of the array, thinking that I could just let it finish building later. This didn’t go so well (for some reason) and I ended up breaking the array. I think it’s back together, but I do worry a bit that something may be lost. We’ll see, I guess.

Mail is back up, things were sync’d over, but there is/will be some quirkyness with the mail received in the last day or two. Expect to see some duplicates. Sorry.

Oh, great: Sep 7 22:37:12 banstyle smartd[863]: Device: /dev/ad6, 1 Currently unreadable (pending) sectors

Right now I’m feeling really frustrated with this whole process and wanting to just put it away and maybe start over later. I just don’t want to leave everyone else’s stuff down.

Leave a Comment

banstyle.nuxx.net: It’s Back

Published September 5, 2008

The display on the VGA output of my new server, banstyle.nuxx.net, after it went down hard at ~01:30 EDT on 05-Sep-2008.

My server, banstyle.nuxx.net is back. In case you didn’t see the LiveJournal post I made about the server being down, know that it went down about 01:30 EDT this morning and didn’t come back up over night. The symptoms were that the machine had an active link to the switch, but the arp cache was aging and the box was generally unreachable and unresponsive. Here’s a Cacti graph showing the outage.

At lunch I drove down to the colo facility, was escorted down to the room, and first noticed that the box was powered up, the network activity LED was blinking, but the disk controller LED was dark. Plugging in a monitor I saw that blinky colored bars overlaid on the normal console, looking like a hardware problem. Perhaps something with the video controller.

The box was rebooted, and as a precaution I went into the BIOS and disabled the bits which redirect video output (text mode only, of course) to the serial port, essentially allowing the whole box to be managed from a terminal. I figure that maybe, possibly, somehow this contributed. After that, I booted the OS back up, did an initial check to be sure everything was okay, started the backed up mail on the old box flushing, and left. Things were a bit slow at first while fsck ran in the background and the mail filtered through, but after that everything seemed good.

So, to be honest, I don’t really know what went wrong. The server is working well again, I guess I’ll just have to keep a close eye on for a while. This is particularly frustrating because it’d been working great for the last four months while I had it at home. If there are any more problems, please bear with me…

For reference, here’s the stuff in /var/log/messages showing that there was nothing between the events noted in last night’s post about SMTP-AUTH and the reboot this morning:

Sep 4 23:05:50 banstyle postfix/smtpd[91552]: sql_select option missing
Sep 4 23:05:50 banstyle postfix/smtpd[91552]: auxpropfunc error no mechanism available
Sep 5 11:57:31 banstyle syslogd: kernel boot file is /boot/kernel/kernel
Sep 5 11:57:31 banstyle kernel: Copyright (c) 1992-2008 The FreeBSD Project.
Sep 5 11:57:31 banstyle kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Sep 5 11:57:31 banstyle kernel: The Regents of the University of California. All rights reserved.
Sep 5 11:57:31 banstyle kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Sep 5 11:57:31 banstyle kernel: FreeBSD 7.0-RELEASE #2: Wed Aug 20 12:57:10 EDT 2008

Leave a Comment