Archive for the ‘work’ Category.

Consider the Paper Towel Dispenser

One of my first jobs was working at a coffee shop (photo), and part of this involved filling the paper towel dispenser behind the counter. Wanting to be as efficient as possible and not have to refill it frequently I’d do my best to stuff it full. This would result in the first few towels being very difficult to pull out, sometimes tearing off and being a frustration for the person washing their hands. But hey, that’s the price of being sure it’s nice and full and efficient, right?

I’ve been encountering similar overstuffed paper towel dispensers at work, and now that I’ve got a bit more experience I’m seeing how this is a very clear illustration of the administrator/engineer vs. user struggle. On one hand you have the maintenance person (administrator/engineer) filling the paper towel dispenser, doing what they feel is best: making the system low maintenance even if there’s a couple irritations for the end user. On the other hand you have the end user who just wants to quickly dry their hands on a paper towel, frustrated that the dispenser is doesn’t work well.

It’s very easy for those of us who run IT systems to be in a bubble, focusing solely on what we feel is important, not seeing things the way they are actually used. We should all remember to stop, wash our hands, and decide if a torn sheet of paper towel is really what the user was going for.

It’s Time For A Change

Back in 2008, not long after I started mountain biking, The MMBA was looking for a volunteer to help fix its poorly performing website. While fairly new to the mountain biking community I had a good deal of experience with providing reliable small-scale web hosting; lessons learned from running my personal site, nuxx.net and a few other virtual hosts for friends. Through my friends Nick and Marty Shue, whom I had casually known since back in the BBS days, I began leading volunteer work on the site, resolving the main performance issue, migrating it to my server, and eventually expanding its capabilities.

It was the perfect combination of my interests: computers, mountain biking, and providing a useful service to other online users.

With help from numerous people, including Rob Ritzenhein doing Joomla work for the original main site, Nick and Marty working through the migration planning (the photo to the right was taken during the site migration itself) and handling some forum moderation, the transition went smoothly and it’s been stalwart ever since. I was even fortunate enough to work with Jeff Lau who wrote an incredibly impressive Trail Guide whose simplicity and usefulness has no current parallel. Many other volunteers contributed to the content; writing posts, submitting trail guide updates, and generally tweaking content.

While the MMBA has gone through some radical changes, working to redefine itself as the chapters moved into the IMBA Chapter Program, the MMBA website site, particularly the forum and Trail Guide, has remained the top source for information on mountain biking across Michigan.

Through this time I gained tremendous experience learning how to assure reliability of production systems, the political parts of working with others, guiding volunteers as they work on projects they are passionate about, and wrangling problem users in online communities among other things. I also met countless wonderful people, made some great friends, and even managed to make a few people hate me.

After seven years I’ve decided that it’s time for me to move on. Over the next few months I will be working with the MMBA Board of Directors and Byte Productions, LLC of Traverse City to smoothly migrate the site (including the forum) to a new home. Once complete my server will be back to hosting my personal sites and a few small pages for friends, stuff that I consider much less critical. Then I’ll have time to figure out what to do next.

Not Just The Numbers

Recently I was working on a high visibility end user problem with computer performance that ended up having a somewhat-unexpected cause: the laptop’s external power supply.

For months this person had been complaining of serious performance issues with a JavaScript-heavy websites that he needed to get his job done, and despite numerous technicians taking a look at the machine, none were able to fix it. In attempts to resolve the issue he’d been given new hardware, switched from a 32-bit to 64-bit OS, and had his user profile (all settings) reset to defaults more than once, to no avail. By the time the case got to me he was quite frustrated as fault was now being assigned to his data and the websites he was accessing, so I set up a meeting so he could show me the problem and talk through what’s going on. I hoped to get a better understanding of what he was doing and what was occurring to see what I could do for him.

After some cursory remote poking to check the laptop’s capabilities and be sure the it seemed stable we sat down and talked. He showed me what was clearly unacceptable performance, explained how the issue only seems to occur when he’s in the office at his desk, sometimes when at remote sites, and never when he’s connected via VPN. Seeing a nicely bundled set of cables behind his the monitor to which his laptop was connected I asked if he had a another power supply that he used when traveling, and if the one on his desk stays there.

That was it; the one on his desk was the cause. Newer Dell and HP business-class machines both use the same physical power connector and they’ll often charge each other’s devices, but depending on the laptop model, power supply model, and BIOS differences sometimes the  laptop will significantly scale back its performance. This is to save battery, allow charging on a limited supply, or (if you are conspiracy minded) steer people away from the use of third party power supplies. When in the office or at a borrowed desk at a remote site he was using a mismatched power supply, so the laptop would scale back its performance and the job-critical website would be unusable slow. Working from outside of company facilities (via VPN) he’d use the power supply that he carried with him — the one which shipped with the laptop –and performance was as expected.

When troubleshooting complicated problems like this it’s easy to fall into the trap of blaming user behavior, providence (the kind of data being stored), or the big mysterious technical places: bad hardware / software. The numbers. Sometimes one has to step back, sit down, talk to those involved, and look over the whole of the problem. Sometimes it’s as simple tab A being plugged into an incompatible slot C, but without stepping back and taking the user and his/her report into account this can be very hard to find.

Power supply model will even cause power scaling issues within the same brand if a given laptop requires, for example, a 90W supply and it is connected to a 65W supply. There is a POST prompt which warns the user of this, but sometimes users or technicians will see the laptop charging anyway (albeit at a lower rate) and disable it without realizing the consequences.

…to Richmond

It’s been a while since I last rode to my parents house, so with family plans in the afternoon I set out from home just after 11am, hoping to arrive a bit after 1pm. The last time I did this ride it took me just shy of two hours and I was left very winded with sore legs. This time I beat my previous time by twenty minutes and felt like I could have pushed myself harder. Sure, I was on a different bike, in different weather, and dressed differently, but I figure all of that riding earlier this year has actually made a difference.

Here is the ride data uploaded to Strava, if you’re interested. No, I don’t have a wheel sensor on that bike, thus no cadence and GPS-only speed.

Here’s a few more photos that I took during recent rides:

· Start of the east loop at Holdridge, known as Gruber’s Grinder. This is a slow, rough, but fun ride.
· Mushrooms found somewhere along Gruber’s Grinder.
· Ducks in a park in Lake Orion while stopping near the end of a ride with Scott.

Now, back to work for a couple of weeks, then off for another two. Hopefully I’ll be able to get a bunch of autumn riding (read: hopefully cool, dry weather) in during that time.

On the last ride I was wearing typical winter riding wear and on a full suspension 29er with a knobby front tire. This time I was in typical summer jersey and bib shorts on my older Specialized Rockhopper Disc with a 29″ front wheel and two Small Block Eight tires. I doubt the suspension and clothing was responsible for 20 minutes, though.

Résumé Updated for 2012

Updating one’s résumé can be quite a pain especially if done under duress, so I like to periodically update it so that a fairly fresh copy is readily available. This afternoon I put the finishing touches on the most updated version, one which takes into account some changes at work, stuff that I’ve done with CRAMBA and the MMBA, and a few other newly-acquired skills.

If you’d like to see a copy of my resume it can be found at nuxx.net/resume.

MS12-006 and HTTP 408 Errors

I recently worked on an issue where a new Microsoft patch for a security bulletin, MS12-006, was correlated to a web-based application occasionally returning errors. Specifically, while using a web-based version of a reporting package off of a version of Sun (Oracle) Java Web Server the browser would occasionally return a Times New Roman-font “Request Timeout” message when users clicked report-generating links. The folks who support the server working on the issue initially had spent time uninstalling recent patches and found that removing KB2585542 (the patch for MS12-006 on Windows 7 clients) from the client machine would stop the issue from occurring. They then thought it might be a Windows client issue, which is where I got involved.

After some investigation it turns out that the problem was actually with the web server not handling split (fragmented) SSL/TLS records properly. The installation of the patch for MS12-006 causes IE to, in some cases, split SSL/TLS records into multiple requests. Since the server couldn’t appropriately handle these requests it thought the client closed the connection before sending a complete request and returned an 408 Request Timeout message.

This was illustrated by the HTTP logs on the server which show incomplete requests being returned 408 messages, similar to what follows. The bold line illustrates a request that correlates with the issue:

IP       | Username | Date                 | Request Header              | Status | Content | Length
---------+----------+----------------------+-----------------------------+--------+---------+--------

10.0.0.2 | -        | 17/Feb/2012:10:03:25 | "P"                         | 408    |         | 148

10.0.0.2 | -        | 17/Feb/2012:10:01:13 | "POST /TestApplet HTTP/1.1" | 200    |         | 13056

After the installation of the patch for MS12-006 there is a setting which allows this new feature to specifically be used only when called for (“optin mode”, the default), all the time (“enabled for all”), or never (“disabled for all”). Setting the option to “disabled for all” kept the issue from being reproduced, and when set to “enabled for all” it always occurred. (See the Registry Information section of 2643584 for more information on these options and how to set them.) By this I showed that it was this SSL/TLS feature causing the issue communicating with the server. Since this appeared to be an issue only with this particular server the team supporting it is now looking into potential solutions on that side.

While it may be tempting to simply disable this feature on clients, this would handicap one of the security enhancements brought along with the MS12-006 patch. Since this issue isn’t likely to be widespread (else lots of HTTPS sessions would be failing) I believe that the servers should be fixed to be compatible with the client’s standard behavior. Specifically, I believe the issue to actually be with the server not having an RFC2246-compliant TLS implementation and thus it should be fixed.

Hopefully this information will be useful to others who run into such an issue. It’s a bit opaque at first, since it looks to the users like the server is having a problem, but to server like the client is occasionally failing to send proper HTTP requests. Trying to troubleshoot this with Fiddler didn’t help either, because the HTTPS proxying would recreate the request, apparently no longer splitting the SSL/TLS record, which kept the error from appearing.

Here’s a few links which are helpful in understanding this issue, in order of usefulness:

· MS12-006: Vulnerability in SSL/TLS could allow information disclosure: January 10, 2012
· SSL/TLS Record Fragmentation Support – Microsoft blog article describing the issue in reverse, from a few years ago when SChannel didn’t support SSL/TLS Record Fragmentation.
· Microsoft Security Bulletin MS12-006 – Important, Vulnerability in SSL/TLS Could Allow Information Disclosure (2643584)
· HTTP Status Code 408 Documentation
· RFC2246 – See section 6.2.1.

Crucial m4 SSD for Data Processing

I’ve been processing some relatively large sets of data at work lately, and I’m running into disk IO issues after kicking off some of the data processing tasks. As an easy way to alleviate this I picked up a 64GB Crucial m4 SSD and stuffed it into my laptop’s optical drive bay via a $20 Nimitz-branded SATA hard drive adapter acquired from eBay. The drive itself was $99.99 from Micro Center, and since I purchased it myself once I’m done working with it I’ll be able to repurpose it in a personal machine of some sort.

While not a panacea nor excuse for my poor habits of bolting together VBScript and GNU command line utilities, it has helped quite a bit. One typical script (immediately post-reboot, with an empty disk cache) takes ~120.234 seconds to run a task from the hard drive and ~28.400 seconds from the SSD. I’ve seen similar speed improvements across the board, and this really helps when I’m prototyping things and wanting feedback as quickly as possible.

AMB χ1: Complete

Yesterday evening I finished up retrofitting my AMB Mini³ headphone amplifier with the AMB χ1. This is a battery management board which replaces the original charging circuitry and 9VDC NiMH battery resulting in a considerably faster charge and longer runtime. By using a Lithium-ion polymer (LiPo) battery the runtime has been extended to ~25 hours (for my high performance version) while the charging time has been reduced to ~1.5 hours. (AMB.org claims that this is a 10x improvement in charge time and a 3x improvement in run time.)

The photo above shows the rear end of the Mini³ without the end panel, showing the two LiPo packs (the silver / strapping tape pieces) the edge of the χ1 PCB (top), and the bi-color charge/fault indicator LED. Assembling this involved removing a small handful of diodes, regulators, and resistors from the Mini³’s PCB, adding a header in place of the LM7812 voltage regulator, and plugging the χ1 into that. As documented on AMB’s site building the χ1 was pretty straightforward and involved only a handful of medium-size (and easy to solder) surface mount parts. I almost wish the design was all surface mount so I wouldn’t have to deal with as much through hole, but I understand his desire to not go all-SMT in order to keep things easy to assemble.

I use this headphone almost every day at work while listening to music from my iPod, and thus far it’s sounded great. However, due to my weird patterns of not being in the office I haven’t been leaving it plugged in for long enough lately, so whenever I’ve been wanting to use it I must plug it in to use it. Hopefully this new version with a shorter charge time and longer run time will sort out that problem.

Now I want to build something else, but I’m not really sure what. I’m somewhat considering a beefy Class A amp and new monitor speakers for my desk at home, all nicely integrated with a USB DAC and tucked under my desk. I could even build the monitors themselves, but this would end up being a pretty big project and I’m not sure it’d get me enough benefit…

Lunch Time Bleeding Shoulder

One advantage to occasionally working from home is being able to get out for a bike ride at lunch time. Yesterday after a string of meetings I took a quick drive up to River Bends and rode a couple laps during lunch. This worked out wonderfully and was a nice way to break up the day.

At one point while riding I clipped a tree with my shoulder. While it didn’t really hurt (no more than when one normally brushes a tree with a shoulder), a few miles later I noticed blood soaking through my shirt. I imagine this has something to do with how damp my shirt was with sweat, and the wicking material did it’s job and spread moisture nicely. Funny, that.

Historic Case Badges

This evening while cleaning out an old tool bag I found these two old computer case badges for Data Premium and Pack In Tell brand computers. These were two house brands from Computer Warehouse, a retail computer outlet in Troy and Madison Heights where I worked from late 1996 until early 1999. These shops were later known as Thomas Computer Warehouse and then Computer Builders Warehouse. During this time the Computer Warehouse stores worked closely with Inca Computer Company; so much so that I essentially transfered from CW to Inca to work in their R&D department. Once Inca closed up shop (it was basically a brick and mortar chain with grandiose plans opening right in the middle of the dot-com bubble) under I returned to Computer Warehouse for a few months before moving on to the world of corporate IT.

I think I’ve got an Inca badge or two around here somewhere as well.