Press "Enter" to skip to content

Category: computers

It’s Time For A Change

Back in 2008, not long after I started mountain biking, The MMBA was looking for a volunteer to help fix its poorly performing website. While fairly new to the mountain biking community I had a good deal of experience with providing reliable small-scale web hosting; lessons learned from running my personal site, nuxx.net and a few other virtual hosts for friends. Through my friends Nick and Marty Shue, whom I had casually known since back in the BBS days, I began leading volunteer work on the site, resolving the main performance issue, migrating it to my server, and eventually expanding its capabilities.

It was the perfect combination of my interests: computers, mountain biking, and providing a useful service to other online users.

With help from numerous people, including Rob Ritzenhein doing Joomla work for the original main site, Nick and Marty working through the migration planning (the photo to the right was taken during the site migration itself) and handling some forum moderation, the transition went smoothly and it’s been stalwart ever since. I was even fortunate enough to work with Jeff Lau who wrote an incredibly impressive Trail Guide whose simplicity and usefulness has no current parallel. Many other volunteers contributed to the content; writing posts, submitting trail guide updates, and generally tweaking content.

While the MMBA has gone through some radical changes, working to redefine itself as the chapters moved into the IMBA Chapter Program, the MMBA website site, particularly the forum and Trail Guide, has remained the top source for information on mountain biking across Michigan.

Through this time I gained tremendous experience learning how to assure reliability of production systems, the political parts of working with others, guiding volunteers as they work on projects they are passionate about, and wrangling problem users in online communities among other things. I also met countless wonderful people, made some great friends, and even managed to make a few people hate me.

After seven years I’ve decided that it’s time for me to move on. Over the next few months I will be working with the MMBA Board of Directors and Byte Productions, LLC of Traverse City to smoothly migrate the site (including the forum) to a new home. Once complete my server will be back to hosting my personal sites and a few small pages for friends, stuff that I consider much less critical. Then I’ll have time to figure out what to do next.

1 Comment

Bit Field for Internet Explorer 11 Security Protocol Options

In Windows / Internet Explorer the options for different security protocols (eg: TLS 1.2, SSL 3.0, etc) are stored in HKEY_CURRENT_USER\Sofware\Microsoft\Windows\CurrentVersion\Internet Settings|SecureProtocols using a bit field. I wasn’t having much luck finding the specific values documented, only this German TechNet blog post detailing the resulting settings. So, I made my own. Here it is in C format:

/* HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings|SecureProtocols for different SSL/TLS settings. */

#define SSL_2.0 8 /* 000000001000 */
#define SSL_3.0 32 /* 000000100000 */
#define TLS_1.0 128 /* 000010000000 */
#define TLS_1.1 512 /* 001000000000 */
#define TLS_1.2 2014 /* 100000000000 */

This also applies to Internet Explorer 9, but I haven’t checked other versions.

Leave a Comment

Creating a Bridge with a Span Port on OpenBSD 5.6

Since OpenBSD moved the bridge commands into ifconfig(8) I hadn’t been able to find quick info on creating a bridge(4). Presuming you’ve got some em(4) interfaces (for naming purposes) here’s all you need to do:

ifconfig em0 up
ifconfig em1 up
ifconfig em2 up
ifconfig bridge0 create
ifconfig bridge0 add em0
ifconfig bridge0 add em1
ifconfig bridge0 addspan em2
ifconfig bridge0 up

This will result in em0 and em1 bridged together, and a copy of all frames that it sees going out to em2 for monitoring.

After this, if you want it to come up every time you boot the machine, do this:

Create three files, /etc/hostname.em0, /etc/hostname.em1, and /etc/hostname.em2 all containing the single line:

up

Create one file, /etc/hostname.bridge0, containing the following lines:

add em0
add em1
addspan em2
up

Reboot the computer and netstart(8) will use these files to create the bridge for you.

One very important point to note if using this to troubleshoot 802.1x: this sort of bridge cannot be used to troubleshoot 802.1x because it does not pass LLDP frames, nor does it drop/raise the link, which switches commonly use as a trigger for initiating client authentication. For situations like this a tap such as the NetOptics TP-CU3 is much more useful.

Leave a Comment

WinHTTP PAC File Caching Location

When troubleshooting issues with proxy auto-config (PAC) files and WinHTTP on Windows 8.1 you may wish to view the cached PAC files which the WinHTTP Web Proxy Auto-Discovery Service have written to disk. These cached PAC files can be found in c:\Windows\ServiceProfiles\LocalService\winhttp and are named with a nine digit number and a cache extension (eg: 1667635681.cache). There is also a cachev3.dat file which appears to contain the download location of the PAC file, the MIME type of the file, and the download date/time.

This can be discovered by using Process Monitor with a filter of Path contains winhttp when the WinHTTP Web Proxy Auto-Discovery Service is started (net start WinHttpAutoProxySvc).

1 Comment

Fiddler Breaks Local Intranet Zone Assignment in Internet Explorer

Beware: The use of Fiddler to troubleshoot Internet Explorer issues can complicate the use of the Local intranet zone by effectively disabling the intended behavior of the Include all sites that bypass the proxy server setting. KB174360 describes the Local intranet zone as follows:

By default, the Local Intranet zone contains all network connections that were established by using a Universal Naming Convention (UNC) path, and Web sites that bypass the proxy server or have names that do not include periods (for example, http://local), as long as they are not assigned to either the Restricted Sites or Trusted Sites zone.

When Fiddler is enabled, proxy settings on a client machine are changed to direct all traffic to 127.0.0.1 as seen above. This results in no websites automatically matching the Local intranet zone because none bypass the proxy server.

While troubleshooting a suspected issue with custom user agent entries I had Fiddler running, as is my normal practice.  The reported issues was custom user agent strings not being sent, something that won’t happen on IE9 unless Compatibility View is enabled. I’d first thought there was a problem with the Display intranet sites in Compatibility View option not working due to the Local intranet zone assignment not working, but my issue actually turned out to be Fiddler getting in the way by causing no sites to match the Local intranet zone. When Fiddler was disabled and I switched to using an external tap for monitoring, behavior returned to normal.

Fiddler is great for MITMing secure sessions thus making troubleshooting secure websites trivial, so not being able to use it in these circumstances will cause other problems. But there are other ways around that…

Leave a Comment

Fiddler for Chained Proxy Authentication

I recently had an issue where an application which supports proxies but doesn’t prompt for credentials needed to use a proxy server to communicate with the internet. The solution? Use Fiddler and its built in scripting language as a secondary proxy chained to the primary, forcibly sending a valid authentication header to the proxies.

How does one this? Here’s how, which is a bit more detailed writeup than what’s found here at Stack Overflow. Note that this presumes basic familiarity with Fiddler:

  1. Use Fiddler to watch a session that uses Basic authentication to the proxy. Look at one of the requests headers after successful authentication and find the line which contains Proxy-Authorization: Basic. Copy this value to the clipboard. (Example line: Proxy-Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQxMg==)
  2. In Fiddler, click Rules → Customize Rules to open CustomRules.js in an editor.
  3. Locate the function OnBeforeRequst. The line beginning this will read something like static function OnBeforeRequest(oSession: Session) and is line 159 in the rulset that ships with Fiddler v4.4.9.2 (latest as of November 6, 2014).
  4. Below this, add a line as follows, with the secondary part being the string that was copied in step 1. In this example the bolded portion is what we’ve added:
    [...]
    static function OnBeforeRequest(oSession: Session) {
    // Inject a Basic authentication header
    oSession.oRequest["Proxy-Authorization"] = "Basic Yno5eWw1Oldyb25nLmdvLldheSsyNDA=";
    // Sample noRule: Color ASPX requests in RED
    // if (oSession.uriContains(".aspx")) { oSession["ui-color"] = "red"; }
    [...]
  5. Save and close CustomRules.js. Fiddler will now begin using this modified ruleset. You can observe that this header is now included with every request.
  6. Ensure that your application using Fiddler as its proxy. If the application is local to the computer on which Fiddler is running and uses WinINET then this is likely automatic. If not, it will need to be manually pointed to the local proxy. Fiddler can also accept connections from computers elsewhere on a network (Tools → Fiddler Options… → Connections), but configuration of this is beyond the scope of this article. See the Fiddler documentation for more information.
Leave a Comment

Breaking IPv6 on Android OpenVPN via T-Mobile

While getting ready for a trip to DEF CON 22 I wanted to have a VPN set up from my phone and tablet to connect back home. After a little while I had both IPsec and OpenVPN connecting back to the house’s pfSense box and passing IPv4 traffic through the tunnel without issue. But, there was a problem when connecting over the T-Mobile mobile network: the VPN would handle IPv4, but IPv6 was left alone to leak through the carrier.

This can be seen in the screenshot above (link) where IPv4 is passing through my home’s Wide Open West connection, but IPv6 goes through T-Mobile. Such leakage has also been written about here by lxgr, in much greater detail.

By pushing an IPv6 route to the client from OpenVPN I was able to black hole IPv6 on the client and close this leakage. This is done by adding the following to the Advanced configuration section of the OpenVPN server config in pfSense:

push "route-ipv6 ::/128 ::1"

While IPv6 still is configured, all traffic goes to loopback and won’t pass over the mobile network. Connections will then fall back to IPv4, which’ll go via the VPN.

EDIT: I was also reminded that I can turn off IPv6 in the APN settings. This works, but I really do like keeping this at the provider defaults… I like having IPv6 when it is available, I just want data to go via only the connection I prefer.

1 Comment

Garmin Express and Proxy Settings

Garmin has recently moved to using Garmin Express for syncing and updating a number of its devices. I recently had to troubleshoot an issue where it wouldn’t work from within a corporate network that uses proxy servers. This has been widely reported on the Garmin Forums (eg: 1, 2), with the general consensus being that Express doesn’t support proxies. It turns out that this is incorrect; Express does support proxies, but because part of it runs as the LocalSystem Account (NT AUTHORITY\SYSTEM) it typically doesn’t have access to the proxy settings.

First, the cause:

Garmin Express has three main components: a service called Garmin Core Update Service which is Garmin.Cartography.MapUpdate.CoreService.exe running as SYSTEM. The second is a tray applet, ExpressTray.exe, which automatically launches on boot running as the currently logged in user. This in turn launches Express.exe, which is the program’s main user interface. The Garmin Core Update Service handles the network communication with Garmin’s servers — something which would normally use proxy servers — but since the default in Windows is not to have proxy settings for the SYSTEM account, this service doesn’t know how to communicate with the outside world.

Now, a couple workarounds:

The first workaround is to change the Garmin Core Update Service to run as the user who needs to run Garmin Express. This works, but may experience wrinkles long-term. Setting the service to run as a specific user requires that user’s password, when password change time occurs (something fairly common on corporate networks) the service will likely fail to start. Additionally, it changes Garmin’s application architecture and may have other untold consequences such as becoming undone when Express updates itself, keeping Express from properly functioning on multi-user machines (read: tablets), etc.

The second workaround is to use the ProxySettingsPerUser policy setting to make the computer have one set of proxy settings for all accounts, user and SYSTEM alike. This is normally defined by Group Policy, but can be manually set by setting the registry value ProxySettingPerUser in HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings to DWORD 0x0. After changing this setting, resetting the proxy settings in Internet Options may be necessary.

By having one set of proxy settings system-wide, processes running as the SYSTEM account will then be aware of the proxy settings. However, if the corporate network uses some manner of authentication for its proxy servers then communication may still fail as Express may not have access to appropriate credentials.

I do not feel that either of these is a proper solution, neither good long-term or enterprise-wide, but both are usable for an individual attempting to resolve problems with a one-off installation. Ideally I’d like to see Garmin change Express so that network communication is handled as the user running the UI. Additionally, some customizable proxy options (eg: Use System Settings, Manually Specify Proxy, etc) as many other applications offer would make Express‘ internet communication considerably more flexible.

(This post applies to Garmin Express 3.2.4.0 only. Newer versions may change this behavior.)

6 Comments

darktrain.nuxx.net Server Issues and Disk Replacement

My current webserver, darktrain.nuxx.net, has been working well for a couple years, despite needing a proactive (due to bad BIOS chip) motherboard replacement and the normal quirks. This past Saturday morning, about 10am, one of the hard drives failed. Due to the use of a ZFS mirror pool for the root filesystem this shouldn’t have caused any problems, but it did. On top of that, due to not rebooting the server in 600-some days I ran into a few other quirks. Here’s what all happened, in chronological order, to get it running stable again:

  • Second hard disk, /dev/ada1, fails. ZFS throws up on itself and the storage basically falls out from under the OS. As a result, everything not in memory and database-backed websites fail.
  • An OS initiated reboot wouldn’t work (seemed to loop during sync) I powered off the server manually.
  • Upon powering the server up disk performance was really bad until /dev/ada1 was removed from the mirror pool. After this point disks settled out and all was good.
  • Outbound email from server wasn’t working due to DKIM-Milter / OpenDKIM failing to start. This could be bypassed, but this wasn’t a good solution because the MMBA Forum sends a fair bit of email notifications. DKIM-Milter failed to start because OpenSSL had been rebuilt due to Heartbleed  bug, but as I hadn’t restarted it since upgrading OpenSSL I didn’t notice the issue.
  • DKIM-Milter couldn’t be upgraded from Ports because FreeBSD 9.0-RELEASE (which was still running) had been depreciated and Ports intentionally broken on this release.
  • OS upgraded to FreeBSD 9.2-RELEASE-p6 using freebsd-update. DNS and mail broke, but this was fairly easy to fix. Update otherwise went smoothly.
  • Ports updated, OpenDKIM rebuilt, mail working again.
  • Upgraded ZFS on remaining disk with zpool upgrade -a command, then wrote new bootcode to ada0 using gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ada0.

At this point the server was stable and I was able to replace the failed disk. The previous setup was with two Seagate ST1000DM003 disks (the mirror pool) and one Crucial M4 SSD (L2ARC). The biggest difficulty in replacing the disk is not the $54.44 cost of the replacement purchase; it’s setting up time to access the server in the data center. Since there was still one free disk bay in the server, instead of just replacing the one failed disk I decided to put two new ones in. These will then be configured into a three-way mirror pool with the SSD L2ARC. It cost a bit more, but now when the next magnetic disk dies (remember, all parts die eventually) I can drop it from the pool and still have two properly working drives, all without another data center visit.

During lunch today I headed over to the facility housing the server in Southfield (conveniently, only 15-20 minutes from work) and within the span of 12 minutes I’d met the escorts, downed the server, swapped the disks, and brought it back up confirming that they are in place and functional.

After getting the disks back I used hints from the FreeBSD Root on ZFS (Mirror) using GPT article to get the new disks partitioned for swap and boot, then added the /dev/ada1p3 and /dev/ada2p3 partitions to the mirror pool and made sure the L2ARC was working. Now everything’s (essentially) back to functionally normal, hopefully with better reliability than before.

So, what’s next? Probably a FreeBSD 10.0-RELEASE upgrade, and better staying on top of patch levels so I don’t suffer the same fate as last time. Being a whole version upgrade there’ll need to be a good bit more planning and testing than this go around, but so long as I’m doing it less urgently, all should be good.

Leave a Comment

2006 Honda Civic Navigation System GPS Data Viewing

Back in late 2005 when I purchased my current car, a 2006 Honda Civic EX, I found that the built-in navigation unit could record log files to a PC Card. Knowing nearly nothing about reverse engineering data files I gave up on the idea of using them for anything. Fast forward to a few months ago, and while poking around with GPSBabel for converting some mountain bike trail mapping data I noticed that it supports Honda/Acura Navigation System VP Log File Format (vpl), the format that I’d hoped to interpret all those years ago. The most basic, latitude/longitude parts of the format are documented here in vpl.cc.

This morning I dug out a 512MB compact flash card and PC Card adapter, fitted it in the navigation unit, and used the hidden menu to enable logging. After grabbing the log file and running it through GPSBabel the end result is just what I’d hoped for: easy logging of wherever my car happens to go.

While it’s not terribly interesting to see the routine, boring local trips that I make, I am interested in recording a month’s worth of data and making a heat map, or perhaps visualizing a long trip I may take. This’ll be fun to play with, I only wish I’d noticed the converter sooner.

Leave a Comment