Category: computers

My current webserver, darktrain.nuxx.net, has been working well for a couple years, despite needing a proactive (due to bad BIOS chip) motherboard replacement and the normal quirks. This past Saturday morning, about 10am, one of the hard drives failed. Due to the use of a ZFS mirror pool for the root filesystem this shouldn’t have caused any problems, but it did. On top of that, due to not rebooting the server in 600-some days I ran into a few other quirks. Here’s what all happened, in chronological order, to get it running stable again:

• Second hard disk, /dev/ada1, fails. ZFS throws up on itself and the storage basically falls out from under the OS. As a result, everything not in memory and database-backed websites fail.
• An OS initiated reboot wouldn’t work (seemed to loop during sync) I powered off the server manually.
• Upon powering the server up disk performance was really bad until /dev/ada1 was removed from the mirror pool. After this point disks settled out and all was good.
• Outbound email from server wasn’t working due to DKIM-Milter / OpenDKIM failing to start. This could be bypassed, but this wasn’t a good solution because the MMBA Forum sends a fair bit of email notifications. DKIM-Milter failed to start because OpenSSL had been rebuilt due to Heartbleed  bug, but as I hadn’t restarted it since upgrading OpenSSL I didn’t notice the issue.
• DKIM-Milter couldn’t be upgraded from Ports because FreeBSD 9.0-RELEASE (which was still running) had been depreciated and Ports intentionally broken on this release.
• OS upgraded to FreeBSD 9.2-RELEASE-p6 using freebsd-update. DNS and mail broke, but this was fairly easy to fix. Update otherwise went smoothly.
• Ports updated, OpenDKIM rebuilt, mail working again.
• Upgraded ZFS on remaining disk with zpool upgrade -a command, then wrote new bootcode to ada0 using gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ada0.

At this point the server was stable and I was able to replace the failed disk. The previous setup was with two Seagate ST1000DM003 disks (the mirror pool) and one Crucial M4 SSD (L2ARC). The biggest difficulty in replacing the disk is not the $54.44 cost of the replacement purchase; it’s setting up time to access the server in the data center. Since there was still one free disk bay in the server, instead of just replacing the one failed disk I decided to put two new ones in. These will then be configured into a three-way mirror pool with the SSD L2ARC. It cost a bit more, but now when the next magnetic disk dies (remember, all parts die eventually) I can drop it from the pool and still have two properly working drives, all without another data center visit.

During lunch today I headed over to the facility housing the server in Southfield (conveniently, only 15-20 minutes from work) and within the span of 12 minutes I’d met the escorts, downed the server, swapped the disks, and brought it back up confirming that they are in place and functional.

After getting the disks back I used hints from the FreeBSD Root on ZFS (Mirror) using GPT article to get the new disks partitioned for swap and boot, then added the /dev/ada1p3 and /dev/ada2p3 partitions to the mirror pool and made sure the L2ARC was working. Now everything’s (essentially) back to functionally normal, hopefully with better reliability than before.

So, what’s next? Probably a FreeBSD 10.0-RELEASE upgrade, and better staying on top of patch levels so I don’t suffer the same fate as last time. Being a whole version upgrade there’ll need to be a good bit more planning and testing than this go around, but so long as I’m doing it less urgently, all should be good.

Leave a Comment

The Cisco WebEx Connect IM (Instant Message) client has a shortcut key for Change Preferences…. If a user has to go deep into a preferences menu to change something so frequently that it warrants a direct Function key shortcut, the application designers didn’t do their job properly.

Leave a Comment

Today I had to troubleshoot how some files/folders on a share are ending up hidden, so this took some digging into SMB and display filters in Microsoft Network Monitor. Since this wasn’t particularly easy to find I wanted to share it here. This is the filter for displaying when a file or folder is having its hidden attribute set (check box via Properties in Explorer or via attrib +h) over SMB:

SMB.CTransaction2.FileBasicDataBlock.Attributes.Hidden == 0x1

This can be combined with a search through the Description to find specific file or folder names. For example:

SMB.CTransaction2.FileBasicDataBlock.Attributes.Hidden == 0x1
AND
Contains(Property.Description, “handle.exe”)

For SMB2 the filter string is as follows:

SMB2.CSetInfo.FileInfo.FileBasicInformation.FileAttributes.FSSCFileAttribute.Hidden == 0x1

Unfortunately, with SMB2 the file/path info will not be included in the frame shown by the aforementioned filter. This can be identified by looking up the session ID (SMB2.SMB2Header.SessionId == NNNN)  and filtering on that, looking at either the CREATE or CLOSE operations near the beginning and end of each session. So, I also capture the CREATE operations for the path I’m looking for, then manually correlate them (with a bit of filtering) after observing the issue. This results in the SMB2 portion of the filter looking something like this once combined with the related SMB filter:

( SMB.CTransaction2.FileBasicDataBlock.Attributes == 0x1
  AND
  Contains(Property.Description, “file_of_interest.txt”)
)
OR
SMB2.CSetInfo.FileInfo.FileBasicInformation.FileAttributes.FSCCFileAttribute.Hidden == 0x1
OR
( SMB2.SMB2Header.Command == 0x5
  AND
  Contains(SMB2.CCreate.Name, “file_of_interest.txt”)
)

1 Comment

This morning I received the bounce message seen above from a Gmail server (173.194.78.26) saying that my IP has been sending too much unsolicited mail. The amusing part? The IP address being complained about, 74.125.82.53, is one of Google’s devices, and the original message was sent via Google Apps. Thus, Google has rejected a message from its own mail server and bounced the error to an end user.

In the last 30 minutes I’ve received four of these. I wonder when it’ll stop.

Leave a Comment

I like having an IPv6 connection at home, but after running into some weirdness with pfSense 2.1-RELEASE that seems correlated with having the pfSense box as the GIF endpoint / IPv6 router (things would kinda get slow, then eventually sort-of fail) I started looking for other options. I have an Apple AirPort Extreme running in bridged mode which I use to provide wireless access to the LAN, and it has the ability to tunnel out to an IPv6 network, so I decided to set that up†.

However, in the midst of getting this setup I ran into a very frustrating problem: only wireless clients connecting via the AirPort could use IPv6; wired clients wouldn’t work. The problem was my previous observation (and expectation), that turning the AirPort into a bridge would make all ports on the back of the device equal. It does not, and thus I needed to move the wired network connection from the WAN Internet port to one of the three Ethernet ports to have IPv6 work on the wired side.

Contrary to what it says, turning Router Mode to Off (Bridge Mode) doesn’t actually fully bridge; the WAN Internet port remains different from the internal ports. Specifically, the AirPort won’t service Neighbor Discovery Protocol (NDP) requests on the WAN Internet port when in bridged mode. Thus when I had the switch connected to this port wired clients couldn’t get an IPv6 address, couldn’t find the router, and wouldn’t configure IPv6 stack. I suspect that when Apple addressed CVE-2008-2476 via 7.4.1 they did so by wholly blocking NDP on the WAN Internet port and didn’t take Bridge Mode into account.

For reference, here’s the ipconfig output from a Windows 7 client on wired vs. wireless when the wired side was connected to the WAN Internet port:

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . : home.nuxx.net
   Link-local IPv6 Address . . . . . : fe80::3d96:1dd5:728c:fde3%12
   IPv4 Address. . . . . . . . . . . : 192.168.0.162
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.0.1

Wireless LAN adapter Wireless Network Connection:

   Connection-specific DNS Suffix  . : home.nuxx.net
   IPv6 Address. . . . . . . . . . . : 2001:470:1f11:d43:ddca:22b9:af55:639e
   Temporary IPv6 Address. . . . . . : 2001:470:1f11:d43:e82f:53dc:7af4:940b
   Link-local IPv6 Address . . . . . : fe80::ddca:22b9:af55:639e%13
   IPv4 Address. . . . . . . . . . . : 192.168.0.147
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : fe80::216:cbff:fec5:162f%13
                                       192.168.0.1

This can be seen in these two network captures: airport_bridged_lie_wired.pcap and airport_bridged_lie_wireless.pcap. These were recorded on OS X using tcpdump when the respective interfaces were down (either unplugged or with the internal wireless disabled), then brought up, then the capture stopped. Captures were then filtered with icmpv6.type == 133 || icmpv6.type == 134 to show only Router Solicitations (133) and Router Advertisement (134) then exported to these libpcap-format files.

In these captures one can clearly see that the wired connection (c4:2c:03:2e:5d:cf) makes some Router Solicitation requests but doesn’t receive a reply, while the wireless connection (d8:30:62:64:f4:ff) receives a Router Advertisement reply to its Solicitation.

After digging into all of this and initially declaring the problem to be wholly with the wired side I thought maybe there’d be a differentiation between the WAN Internet and Ethernet ports. I then tried moving the wired connection to one of the internal / Ethernet ports I was able to get IPv6 NDP responses to wired clients. Thus, Bridge Mode on the AirPort Extreme isn’t quite as bridged as the selection would imply: the bridge is a lie.

A bug (15716120) has been submitted to Apple on this issue.

—

† With an AirPort Extreme serving as an IPv6 tunnel endpoint behind a pfSense NAT device two settings need to be changed in pfSense to get it working. First, System → Advanced → Networking and check Enable IPv4 NAT encapsulation of IPv6 packets and set the IPv6 router’s IP address. This is needed because it’s not possible to add a NAT forward for IP protocol 41. Second, add an inbound firewall rule allowing TCP/IP Version IPv4 and Protocol IPV6. This will allow the tunnel on the AirPort to work.

The IPv6 tunnel is then configured as per the tunnel provider’s directions, which is well-documented elsewhere.

I don’t like opening up all the IPv6 wireless clients on my network to direct connections from the outside, and Apple seems to have figured that others don’t want this as well. By going into the AirPort Extreme configuration in AirPort Utility, selecting Network then Network Options… and checking Block incoming IPv6 connections, all inbound IPv6 connections are blocked and a Port Settings option becomes available allowing individual ports to be unblocked.

2 Comments

Earlier today when setting up a new Google Group for planning a CRAMBA event I noticed that Google’s reCAPTCHA service has moved from using just scanned book images (info on how this worked) to using house numbers which I suspect are from Google Street View. I imagine that this works well for them because house numbers are inherently human readable and successfully translating them to integers is likely key to their reverse geocoding efforts.

EDIT: Apparently this is old news. Shows how often I use reCAPTCHA… I first noticed it today.

Leave a Comment

Despite some quirky problems, I’ve been using Garmin Edge devices (first a 500, now a 510) for the last couple of years when cycling in order to track and display various statistics. This has generally worked well, but throughout all of this I’d had one overriding problem which wasn’t serious enough to properly dig into until this past weekend: the unit would not always mount (show up in Finder) when I plugged it into my Mac.

The original problem that I’d had with both the units was that, sometimes, plugging the device into a Mac would result in it not mounting, but unplugging it, waiting a few seconds, and plugging it back in would then work. I was content with this for a while and there was no obvious correlation between when it’d happen and wouldn’t, but a few days ago the Garmin Edge 510 stopped mounting at all. I figured nothing was wrong with the Edge 510 because it would mount perfectly fine on a Windows box, so I began looking at the Mac.

In the end the problem has turned out to be VMware Fusion. While I haven’t proved it, it also seems that the upgrade to Fusion 6.0.2 (from 6.0.1) last week changed the problem from sometimes to always and I could not get the Garmin to mount at all. After some thinking and testing I narrowed it down to only occurring when VMware Fusion was running a virtual machine with a USB controller.

VMware has published knowledge base article 1025256 to help one troubleshoot such issues and find workarounds by including quirks definitions in the VMX files, but none of these recommendations worked for my Edge 510, so I opened a support request (#13413345912). I’ve been emailing back and forth with VMware support and the assigned support person seems to be working on it, so hopefully the information I’ve provided them will lead them to developing a proper fix for it. (If/when I receive a fix I’ll update this post.)

In the mean time I’ll just leave the USB controller disconnected from the VM that I have running most frequently. This allows things to work, and as I rarely use USB passthrough it’s a fair trade.

—

The software / hardware versions when replicating the issue are as follows:

Apple iMac: iMac11,3, OS X 10.9 (13A603)

Garmin Edge 510: IC: 1792A-020, FCC ID: IPH-02069, firmware 2.80.

VMware Fusion: 6.0.2 (1398658)

Leave a Comment