Month: September 2008

Crushed Slug

Published September 20, 2008

Today Danielle and I managed to make it out to Addison Oaks park, where I did two laps on the mountain bike trails. One was on the normal route, the other was on the race route which includes a newly cut section of single track.

The ride went well, but I find it to be a bit more tiring than Stony Creek, because it doesn’t have as lengthy of sections for catching one’s breath.

On Wednesday when riding Addison I ended up with a crushed slug stuck to the top of my shoe. Today I ended up with one stuck on the down tube, so after driving home I decided to take a picture of it. Enjoy! (Click here for the high res version.)

Leave a Comment

Time For A New Printer

Published September 20, 2008

Old parallel port printer cable connecting into the back of my work laptop, a Dell D610.

I guess I really need a new printer. After almost a year of limping along with a failing HP LaserJet 5L at home I’m finding I can’t even convince it to print any more. Today I was able to get it to print a test page while manually guiding the paper deep into the feed mechanism, but then I was unable to print properly from either my Mac or work laptop via lpr (to the JetDirect), or straight from my work laptop via parallel port.

Two pages bearing print did leave the printer eventually when connected via parallel port, but only half of the PDF which I needed to print (a free admission ticket to Addison Oaks for riding the mountain bike trails) was actually render correctly. Oh well. I guess it’s time to go to Kinkos.

Leave a Comment

Good Beer for Good Friends

Published September 19, 2008

A fridge full of beer for when friends visit. There is stuff from Dogfish Head, JW Lees, New Holland, Founders, Magic Hat, and Jolly Pumpkin.

People are visiting tonight and bringing good food, so I stopped and got good beer:

· Dogfish Head Indian Brown Ale
· J.W. Lees Manchester Star Ale
· Founders Breakfast Stout
· Magic Hat’s Circus Boy
· Magic Hat’s 9
· Magic Hat’s Jinx
· Magic Hat’s Participation Lager
· New Holland’s Dragon’s Milk
· Jolly Pumpkin’s Dark Dawn Stout

Not pictured, not purchased tonight, and / or not in the fridge is a quite-aged bottle of vanilla mead made by a friend of mine, some Sparks (for Danielle), and some Kuhnhenn Chocolate Braggot.

Leave a Comment

Busy Weekend

Published September 19, 2008

This weekend looks to be very busy. I’m still at work, don’t know when I’ll be leaving, and likely will have to put in some time on either Saturday evening or early Sunday morning.

The new hard disks for my server are going to be delivered today, so hopefully the wipe of the failing ones (with DBAN) will be complete by the time I arrive home so that I’ll be able to do the dump and restore, check out the install, then get in with more burn-in.

I’d originally planned on riding both the Tour De Troit and the Addison Oaks Fall Classic this Saturday and Sunday (respectively), but I just don’t think I want to schedule things that tightly. So, maybe I’ll get out and ride a bit, but it definitely won’t be anything planned or structured.

Now, to get this stuff at work wrapped up. Thankfully Danielle brought me some really, really yummy lunch from Rangoli Express so that I didn’t have to leave for lunch today. It was really, really, really good.

(No, I’m not neglecting work right now… I’m just waiting for some other folks so I can keep going with stuff that I’m doing.)

Leave a Comment

+12 Hours of Breakin

Published September 18, 2008

Breakin, having run for 12h 28m 33s after swapping RAM around.

Yesterday I ordered a pair of Seagate Barracuda ES.2 ST3500320NS disks to replace the two which failed on Tuesday. Today I called Newegg about my RMA for the old ones and the old controller and was able to get the 15% restocking fee waived for both the controller and drives. Hopefully the drives will arrive tomorrow and I can dump | restore the OS and such, then start Breakin running so that it can thrash the drives for a few days.

Speaking of Breakin, I disconnected the disks from the machine (but left them mostly fitted in the case as to not disrupt airflow) and started Breakin running this morning before I left for work. When I arrived home it was still running, unlike last week when it regularly failed with MCEs. This is good, as I had been unable to get it to run for this long before.

Leave a Comment

SMART Issues

Published September 16, 2008

When I got home I started running SeaTools, Seagate’s disk diagnostics utility for Windows, on the ad4 which had begun failing earlier. It reported back that it, and the other hard drive, were just fine. However, when booting into FreeBSD after using them I found that both drives were now indicating that Seek_Error_Rate was past threshold. The OS booted very slow, then kicked ad6 out of the mirror set.

I tried connecting the drives to another, standalone SATA controller (some plain old Maxtor bundle-in one) with new SATA cables and same problem.

So, I’m not sure what to do. Here’s every issue I’ve had with the new server and its resolution:

Issue: Server locking up hard, unexpectedly. MCEs on console.
Resolution: Ensure that only matched RAM is used and that all RAM tests good during burn-in.

Issue: Slow performance / absurd latency while using 3ware disk controller.
Resolution: Identified GIANT-LOCK on driver, moved to using software mirroring.

Issue: One of the original two Western Digital disks used, which were part of a gmirror set, has started giving block errors.
Resolution: Replace disks with brand new Seagate pair.

Issue: Both of the new Seagate drives began failing with excessive Seek_Error_Rate within a few hours of each other after extensive burn in.
Resolution: Unsure.

I can’t help but wonder if one of the Seagates beginning to fail was contributing to the latency observed with the 3ware controller, but as neither was throwing SMART errors at the time, so I discount this.

My current thought is that I should order a pair of server-grade disks, burn them in as before (~50 hours of constant activity), copy the data to them, then see if things will keep working. The failed disks and the unwanted 3ware controller will go back to Newegg, and hopefully things will work right.

I don’t know what other option I have besides scrapping the whole idea of moving servers, but I really rather not do that. If anyone else has any ideas, I’d love to hear them…

Leave a Comment

New Hard Disk Is Failing

Published September 16, 2008

root@banstyle:~# smartctl -H /dev/ad4
smartctl version 5.38 [amd64-portbld-freebsd7.0] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
7 Seek_Error_Rate 0x000f 013 012 030 Pre-fail Always FAILING_NOW 38293929828058

root@banstyle:~#

I can’t win. Now one of the brand new hard disks in the server is getting a bunch of seek errors.

Leave a Comment

Stony Creek Mountain Bike Videos

Published September 16, 2008

I mounted my old Coolpix 5400 on the handlebar of my bike in order to get some video of bike rides.

Yesterday I decided to go for a bike ride at Stony Creek, but before doing that I grabbed some small pieces of non-slip foam and some cable ties and strapped my old Nikon Coolpix 5400 to the handlebars of the bike. (This is how it looked when riding.)

The resulting videos aren’t great, as being mounted on the handle bars made the video jerky and noisy, with every little movement, bump, and vibration translating into shake, blur, and noise. However, I think they are interesting enough, and in one of them you can even watch me fall.

These videos are only short, couple minute segments of what I feel are some of the more interesting pieces of single track at Stony Creek. Length was limited by the recording time of the camera.

Here’s the videos on YouTube in high res, including the one of me falling (#3):

· The Pines #1
· The Pines #2
· Roller Coaster #1
· Roller Coaster #2
· Roller Coaster #3 & Me Falling
· Roller Coaster #4
· Roller Coaster #5
· Marker 26 to 25
· Back to Parking Lot

The original, and thusly higher quality, MOV files straight out of the camera at https://nuxx.net/videos. Just be warned, they are large:

· stony_creek_15oct2008_pines_1.mov (46.3 MB)
· stony_creek_15oct2008_pines_2.mov (46.4 MB)
· stony_creek_15oct2008_rollercoaster_1.mov (45.9 MB)
· stony_creek_15oct2008_rollercoaster_2.mov (25.5 MB)
· stony_creek_15oct2008_rollercoaster_3.mov (24.8 MB)
· stony_creek_15oct2008_rollercoaster_4.mov (45.7 MB)
· stony_creek_15oct2008_rollercoaster_5.mov (38.8 MB)
· stony_creek_15oct2008_to_parking.mov (46.6 MB)
· stony_creek_15oct2008_up_mount_sheldon_26_to_25 (22.5 MB)

Leave a Comment

More Broken Bike Stuffs

Published September 16, 2008

Today while riding one more go through the Roller Coaster at Stony Creek (after recording videos, to be posted later) I fell pretty hard on the right side, and hurt my shoulder. I was going around a slow-ish switchback and my front wheel skidded to the left and I just went down.

I’d thought the only thing damaged was my shoulder, but after getting back on the bike it didn’t shift right. I just put it on the stand to sort it out so I could ride tomorrow and while it initially appeared to be a cable tension issue, which was wholly possible seeing as I’d just replaced the rear derailleur, that wasn’t it. The bottom of the rear derailleur appears to be cocked inward, and after checking to be sure it wasn’t another broken mounting bolt, I noticed that the derailleur hanger appears to be a bit off. After removing it from the bike and checking it with a straightedge it does appear that I’ve bent it slightly.

When I removed the wheel I also noticed that the rear axel / hub / bearings aren’t turning very smoothly at all. I suspect that when I had to do the half-assed single speed conversion during which the chain would occasionally climb to the next ring there was too much pressure on the bearings, or something. So, I guess I’ll have to fix that before I ride next too. Hopefully it just needs some adjustment and I haven’t destroyed the races.

I need to learn to ride better and stop being so rough on my bike.

Since I’m already buried in working on problems with the server and getting it up and going again I’ll have to put this off for a bit.

Also, in that fall I scraped my leg and bruised my right nipple. I have never before bruised a nipple.

Leave a Comment

3ware 8006-2LP Sucks Under FreeBSD 7.0-RELEASE

Published September 14, 2008

Results from using Bonnie++ on FreeBSD 7.0 with a 3ware controller (twe), gmirror, and just a single local disk.

As mentioned here I got my new server working with a 3ware 8006-2LP and a pair of new 500GB disks. While it was working fine, I noticed that when updating the FreeBSD ports collection that the update would occasionally pause, consuming no CPU, but with the update process having a status of sbwait. I understand this to mean that the process is waiting on a blocked socket.

It turns out that the twe(4) driver is what is known as GIANT-LOCKED, which I believe means that it uses the old SMP locking mechanism in FreeBSD:

twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x8c00-0x8c0f mem 0xfc7ffc00-0xfc7ffc0f,0xfb800000-0xfbffffff irq 28 at device 3.0 on pci1
twe0: [GIANT-LOCKED]
twe0: [ITHREAD]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048

Best I can tell, the result of this is that the disk controller’s driver needs to wait for the kernel to free up other resources and tell the driver that it can go ahead and work before it does things. The result of this tends to be that the driver works well, but there is a lot of latency.

This understanding matches what I observed, which was the aforementioned lengthy pauses when doing things which required a bunch of disk IO. In order to prove this understanding out, I set up a test hard disk running a stock FreeBSD 7.0 amd64 installation from which I could run Bonnie++, a file-based disk benchmarking suite.

In my testing I used the following three scenarios:

· One 120GB IBM Deskstar PATA drive (IC35L120AVVA07) connected to the motherboard booting the OS, listed in the results as banstyle_deskstar.
· Two 500GB Western Digital SATA drives (WD5000AAKS-40TMA0) connected to the motherboard with software RAID 1 via gmirror(8), listed in the results as banstyle_gmirror.
· Two 500GB Seagate SATA drives (ST3500320AS) connected to the 3ware 8006-2LP using the twe(4) driver in hardware RAID 1, listed in the results as banstyle_twe.

The result ended up being that all three configurations are generally around the same speed for throughput, but the 3ware controller had an absurd amount of latency. If one looks at the HTML version of the Bonnie++ output here (or the PNG here or above), one can see that was giving near three SECONDS of latency for random seeks and writes using write(2). This is insane.

The only thing I can think to attribute this to is the GIANT-LOCK in twe(4). I guess this means that I’m going to have to go back to gmirror(8) for software RAID and return the card. How disappointing.

(If anyone reading this disagrees with these findings or wishes to comment on them, please don’t hesitate to do so here or by emailing me directly.)

7 Comments