Press "Enter" to skip to content

nuxx.net Posts

darktrain.nuxx.net Server Issues and Disk Replacement

Published May 21, 2014

My current webserver, darktrain.nuxx.net, has been working well for a couple years, despite needing a proactive (due to bad BIOS chip) motherboard replacement and the normal quirks. This past Saturday morning, about 10am, one of the hard drives failed. Due to the use of a ZFS mirror pool for the root filesystem this shouldn’t have caused any problems, but it did. On top of that, due to not rebooting the server in 600-some days I ran into a few other quirks. Here’s what all happened, in chronological order, to get it running stable again:

  • Second hard disk, /dev/ada1, fails. ZFS throws up on itself and the storage basically falls out from under the OS. As a result, everything not in memory and database-backed websites fail.
  • An OS initiated reboot wouldn’t work (seemed to loop during sync) I powered off the server manually.
  • Upon powering the server up disk performance was really bad until /dev/ada1 was removed from the mirror pool. After this point disks settled out and all was good.
  • Outbound email from server wasn’t working due to DKIM-Milter / OpenDKIM failing to start. This could be bypassed, but this wasn’t a good solution because the MMBA Forum sends a fair bit of email notifications. DKIM-Milter failed to start because OpenSSL had been rebuilt due to Heartbleed  bug, but as I hadn’t restarted it since upgrading OpenSSL I didn’t notice the issue.
  • DKIM-Milter couldn’t be upgraded from Ports because FreeBSD 9.0-RELEASE (which was still running) had been depreciated and Ports intentionally broken on this release.
  • OS upgraded to FreeBSD 9.2-RELEASE-p6 using freebsd-update. DNS and mail broke, but this was fairly easy to fix. Update otherwise went smoothly.
  • Ports updated, OpenDKIM rebuilt, mail working again.
  • Upgraded ZFS on remaining disk with zpool upgrade -a command, then wrote new bootcode to ada0 using gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ada0.

At this point the server was stable and I was able to replace the failed disk. The previous setup was with two Seagate ST1000DM003 disks (the mirror pool) and one Crucial M4 SSD (L2ARC). The biggest difficulty in replacing the disk is not the $54.44 cost of the replacement purchase; it’s setting up time to access the server in the data center. Since there was still one free disk bay in the server, instead of just replacing the one failed disk I decided to put two new ones in. These will then be configured into a three-way mirror pool with the SSD L2ARC. It cost a bit more, but now when the next magnetic disk dies (remember, all parts die eventually) I can drop it from the pool and still have two properly working drives, all without another data center visit.

During lunch today I headed over to the facility housing the server in Southfield (conveniently, only 15-20 minutes from work) and within the span of 12 minutes I’d met the escorts, downed the server, swapped the disks, and brought it back up confirming that they are in place and functional.

After getting the disks back I used hints from the FreeBSD Root on ZFS (Mirror) using GPT article to get the new disks partitioned for swap and boot, then added the /dev/ada1p3 and /dev/ada2p3 partitions to the mirror pool and made sure the L2ARC was working. Now everything’s (essentially) back to functionally normal, hopefully with better reliability than before.

So, what’s next? Probably a FreeBSD 10.0-RELEASE upgrade, and better staying on top of patch levels so I don’t suffer the same fate as last time. Being a whole version upgrade there’ll need to be a good bit more planning and testing than this go around, but so long as I’m doing it less urgently, all should be good.

Leave a Comment

2006 Honda Civic Navigation System GPS Data Viewing

Published May 15, 2014

Back in late 2005 when I purchased my current car, a 2006 Honda Civic EX, I found that the built-in navigation unit could record log files to a PC Card. Knowing nearly nothing about reverse engineering data files I gave up on the idea of using them for anything. Fast forward to a few months ago, and while poking around with GPSBabel for converting some mountain bike trail mapping data I noticed that it supports Honda/Acura Navigation System VP Log File Format (vpl), the format that I’d hoped to interpret all those years ago. The most basic, latitude/longitude parts of the format are documented here in vpl.cc.

This morning I dug out a 512MB compact flash card and PC Card adapter, fitted it in the navigation unit, and used the hidden menu to enable logging. After grabbing the log file and running it through GPSBabel the end result is just what I’d hoped for: easy logging of wherever my car happens to go.

While it’s not terribly interesting to see the routine, boring local trips that I make, I am interested in recording a month’s worth of data and making a heat map, or perhaps visualizing a long trip I may take. This’ll be fun to play with, I only wish I’d noticed the converter sooner.

Leave a Comment

Cutting Coroplast on a Table Saw

Published May 12, 2014

Some of the local mountain bike trails that I work on are in need of more signage, so a few months back I picked up a few sheets of Coroplast and a fellow CRAMBA-IMBA board member printed up some vinyl bicycle decals and directional arrows. This evening I finally got around to cutting the Coroplast, and after trying a few different methods I settled on using a table saw. While I only had a wood cutting blade I found that by using a fence and feeding at just the right speed I was able to make quick cuts and short work of producing 6″ x 4.5″ signs.

Now that this part is done I just have to get the decals applied distribute them as appropriate.

Leave a Comment

Footbeds and Shims for Specialized Pro MTB Shoes

Published May 12, 2014

After a couple rounds of toe numbness and pain on longer rides with my new Specialized Pro MTB shoes (3+ hours, and on 4+ hour hard rides with my older shoes) I stopped by Fraser Bicycle and Fitness intending to pick up a set of higher arch support footbeds. Chris Goddard happened to be there and he checked me over, gave me a bunch of great advise about things to try, and set me up with 2x 1.5mm varus shims for each foot,  blue/moderate support footbeds, and the suggestion that I move my cleats back. While I only had my old shoes with me, these were enough to get a good feel for what I should try.

Upon arriving home I immediately put the footbeds and shims in my new shoes, moved the cleats back a bit (maybe 7mm) and hopped on the trainer to try it out. While I only got in ~10 minutes of spinning, the shoes felt like they fit much better to my feet than they had before. I’m looking forward to a longer ride, but I now feel more like my feet are solidly planted in the shoes when I begin applying force, not that they press into place as I pedal. I hope this translates into even more comfort while riding.

I’m pretty happy on all of my bikes, but these new shoes had me a bit worried. Then with a bit of foot pain during a six-hour endurance race on Saturday using my old shoes, I got a bit more concerned… This seems like a good way forward.

 

Leave a Comment

In The Nick of Time

Published May 7, 2014

It’s a good thing that I decided to give the El Mariachi Ti a quick wash and a once-over before this Saturday’s race. I’d noticed some quirky shifting on the last couple of rides, and apparently this was the cause. I hadn’t planned on replacing the cable and housing this evening, but I was glad to do it now than angrily after dealing with a cable failure (and emergency single speed conversion) mid-race.

I’m really glad it didn’t let go on any of the other rides I’d done recently. I’d been noticing some issues with shifting near the top end of the cassette recently, but after recabling things those quirks seem gone and the bike is ready to ride.

Leave a Comment

Garmin Edge 510 ANT+ Recording Stopped at Lap

Published May 5, 2014

Here’s something odd that happened during a long ride with my Garmin Edge 510 (firmware 2.90). While stopped in River Bends Park to climb over a tree I pressed the Lap button to mark the location so the tree could be removed, but then after that point the unit stopped recording data from either of the ANT+ sensors. Both the HR and Cadence values seemed to be stuck at whatever they were when the Lap button was pressed.

Firmware update 3.00 is out and I’ll try updating to that and see if it comes back, but as the change history simply states “Changes for manufacturing.” I wouldn’t be surprised if it happens again.

UPDATE: A bit of research has shown that it wasn’t actually when the lap button was pressed. It just happens to be right after that.

Here is the first record of the consistent/wrong 142 BPM, 24 RPM data:

<trkpt lat="42.64555980" lon="-83.04996448">
    <ele>160.6</ele>
    <time>2014-05-04T21:32:47Z</time>
    <extensions>
        <gpxdata:hr>142</gpxdata:hr>
        <gpxdata:cadence>24</gpxdata:cadence>
    </extensions>
</trkpt>

Here’s the lap marker, which shows it was earlier:

<gpxdata:lap xmlns="http://www.cluetrust.com/XML/GPXDATA/1/0">
    <index>0</index>
    <startTime>2014-05-04T21:32:09Z</startTime>
</gpxdata:lap>

Close, but definitely earlier.

Leave a Comment

Carbon Fiber Shoes, Squishy Fork, Lubed Cassette

Published May 3, 2014

A rainy, quiet Saturday while Danielle is working gave me a nice opportunity to get some bike stuff done. In preparation for what I hope to be a long ride tomorrow and the Fun Promotions Stoney Creek 6 & 12 Hour Mountain Bike Endurance Race I removed the rigid fork from the El Mariachi Ti and put the original Fox and Arch EX-based wheel back on. While I like the high volume tire and rigid fork, it’s still a bit much for long rides. Thankfully the swap between forks and wheels is easy, taking only about 15 minutes. I imagine I’ll be switching this around a few times this year.

I also had to re-remedy a long-standing problem with the cassette on the El Mariachi Ti: it’s noisy. Last year I was dealing with some odd under-load ticking sounds that were worst mid-cassette and in the little chainring up front, especially when really mashing on the pedals. After a lot of experimentation I was eventually able to reproduce it and found it could be alleviated by lightly greasing the plastic spacers between the cogs. This seems to last for a few months, then it needs to be done again. This should go away once the cassette is replaced, unless the next one has a similar problem…

Finally, I got a pair of Specialized Pro MTB shoes set up and ready to use. At the end of last year I was able to pick them up on clearance at Cycletherapy, since they are the discontinued model with a strap closure, as reviewed here. These are very similar to the Comp model that I purchased in early 2011, except with more venting and a carbon fiber sole. Three years the Comp shoes were starting to get near the end of their usefulness, with holes appearing in the sides of the toes, the sole becoming more flexible than normal, and the insole seriously breaking down.

I appear to have the cleats positioned appropriately, although the pedal body on Eggbeaters seems to hit the lugs a bit. I’m not yet sure if this’ll cause a problem or is a good thing, but if it ends up not working I’ll have to shave a millimeter or two of rubber off of the lugs. Clipping in is a little stiffer than I’m accustomed to, but the solid shoe-against-pedal feeling makes for a very rigid interface. It should also move energy transfer to a larger area, cutting down on the pedal wing / shoe sole transfer that eats through Shoe Shields.

Trying the shoes out on the trainer and in the rain around the parking lot, they are definitely stiffer than the non-carbon fiber composite Comp shoes that I normally wear. They improve on the Comp shoes stiffness to the same degree that the Comp shoes (and my previous Sidis) improved on the Shimano SH-MT40 commuter shoes I’d first started out with. Fit is almost identical to the previous shoes, but with a new, stiff, less slipper-like feel. These should be good.

Leave a Comment

Ideas… Ideas…

Published April 22, 2014

Today is one of those days where I find myself with lots of ideas for things to do swirling around in my head. I just have to parse through them and figure out which ones are feasible, and when I can do them…

So, what am I thinking would be good to do? Here…

  • Take a half-day on Thursday and go ride Poto on a single speed before the Waterloo public meeting with the DNR.
  • Take a half-day on Friday to hike PLRA and gather more GIS data for mapping.
  • Order some wide, hookless carbon fiber rims from Light-Bicycle to build a new set for the El Mariachi Ti. Then sell the existing Arch EX wheels…
  • Hike River Bends and clean up any lingering deadfall. Maybe Saturday?
  • Attend the Stony Creek Trail Day.
  • Write up a new description for Massive Fallout.
  • Write up a document trying to convince more people involved in volunteering to work on MTB trail stuff.

Now, to find the time…

Leave a Comment

Defective REI Venturi Shorts (2014)

Published April 20, 2014

A few days back I picked up a pair of the REI Venturi shorts, as I needed more shorts and these were quite comfortable when tried on. I wore them for much of the day today and they were comfortable, but poorly engineered pockets is causing me to return them.

Upon arriving home from my parents’ Easter dinner, when reaching into my left pocket with my right hand to get my keys, I heard a tearing sound. It turns out that the thread used to stitch along the pockets is not as stretchy as the short material itself, and stretching the edge of the pocket causes the stitching to burst, as seen above. At $69.50/pair I expected better, and I’ll be returning these tomorrow. I’m disappointed in this because they are comfortable and met a need.

Leave a Comment