Press "Enter" to skip to content

nuxx.net Posts

DIY-ish Under Cabinet Lighting

After the kitchen was redone in our new house, the under-cabinet space (where lights would go) was intentionally left unpopulated; something for me to finish later. The electrician had fitted an outlet per cabinet grouping, and a switch on the wall that toggles those outlets, but I’d asked him to skip the lights because I really wasn’t sure what style or color we wanted, nor how I wanted to wire things. Being low voltage with the outlets and switch already fitted, it would be fairly straightforward later on.

This winter, looking for a project, I decided to finally get this done. After talking with my friend Dan and some investigating, I finally got a solution together.

What I ended up doing was sourcing a handful of parts, flexible PCB LED strips, aluminum enclosures, wire, and wall wart power supplies on Amazon, assembling it, and sticking it to the underside of the cabinets. This has worked out well, so I wanted to share what I did.

Our kitchen has three overhead cabinet groupings, the left-most has 22″ of face, the corner has 37″ of face (2x 10″ side cabinets and 1x 17″ center cabinet), and the right-most is a single 10″ cabinet. I had originally thought of ringing the underside of each cabinet with LED strip lights, but after some consideration I decided to do a single light assembly along the front edge of each cabinet, pointing down, with a wide-angle diffuser.

In the kitchen we have daylight (5000K) colored lights, but as the under-cabinet lighting is likely to be used as more of a night or secondary light, I figured on ~4000K lights. After buying strips of both 4000K and 6000K of to test, we ended up with some strips listed as Cool White 6000K. This matches pretty well whenever the overhead lights are on and it’s bright to cloudy day outside, and likely because of the grey-ish countertops it still looks fine when they are the only lights on.

It would have been possible to use a different type of strip that has multiple color temperature white LEDs and selectable color temperature, but doing this requires a controller external to adjust the light strips, which would have added around $60 just for the controllers, plus another $10 or so in LEDs, a remote control, more space taken up in the cabinets. This setup, with fixed color strips, is just AC-DC adapters plugged into the existing mains outlets, and the LED strips connected to the adapters. If we find that this isn’t good I may eventually go to controllers, but for now this is working nicely.

For the LED strips I chose FCOB LED strips from BTF-LIGHTING, 528 LEDs per meter, in 12V. With the heavy silicone coating over the LED chips it makes for a fairly even light. I chose the 12V variant of the lights because they have cut-points 22.83mm, which made it easy to make maximize the amount of LED strip under each cabinet. If the runs were longer it would have been wise to choose 24V to decrease the amperage, but at 14 watts per meter (W/m), the longest light segment will only take ~1.1 amps (A) so this should be fine. (Amazon Link)

For hookup wire I bought a simple coil of white 20 gauge copper clad aluminum (CCA) wire, again from Amazon. I already had some hookup wire, but I wanted white to blend in nicely with the cabinets. (Amazon Link)

For power supplies I picked up a basic five-pack of 12VDC / 2A wall warts. While I only needed three, the cost for five was $7 more than buying three, and since I’ll have LED strip left over and will likely build some more arrays for elsewhere in the house, I wanted spares. They also happened to come with (very cheap) barrel to screw terminal adapters that worked well enough. (Amazon Link)

Initially I was going to mount the LED strips directly to the cabinets, but Dan convinced me that even while out of sight it would look much better in an enclosure, and he was right. The project feels properly finished this way and it looks tidy. The enclosures also add additional diffusion, making the light look smoother and more even and overall better. For this I chose some 1m long pieces of aluminum channel with a 60° milky white domed cover, then cut it to width for each cabinet section. (Amazon Link)

The light enclosures include screws and mounting clips, but I instead opted for tape, and picked up a 5 yard roll of 3M VHB 4910, which is specifically described as being good for Polycarbonate, Aluminum, and Acrylic/Polyurethane Paint; perfect for sticking these housings to the underside of cabinets. (Amazon Link)

Assembly was done by first determining the width of the housings, so I went with even-inch sizes that are just slightly narrower than the underside of the cabinet. This resulted in five pieces: 1x 22″, 1x 16-1/2″, and 3x 10-1/4″. With the cover snapped into the aluminum channel it all cut clean and easily using a circular miter saw with a fine tooth carbide blade.

I then cut LED strip to the next step shorter than the housing, soldered the supply wire, stuck the strip into the housing via it’s self-adhesive backing, fitted the endcaps, added hot-melt glue for strain relief at the solder area, and snapped the face on. For the corner cabinet lights I wired it in a T arrangement, with the two sides being fed from the center, and then a wire out of the center to head up to the cabinet. This arrangement was because the corner cabinet holds the outlet near the back left side and it minimizes wire distance. It may have also helped with voltage drop on a long-ish strip of LEDs, but I’m not sure the ~30″ total would be enough to actually cause a problem.

A small hole (3/16″) was drilled in the underside of the cabinet to run the wire up to the supply, cabinet undersides cleaned and strips stuck in place with VHB tape, wire was tacked in place with hot melt glue, and screw terminal to barrel adjuster adapters (which came with the power supplies) fitted. A bit of cable tying to tidy the cables in the cabinets and it’s all done.

Total cost for this project was $122.83, and that includes the unused 4000K strip that I bought as a test. There is enough extra left over to do at least another 10′ of enclosed light somewhere else in the house, so I might build some light strips for under my workbench shelves or perhaps where I have music gear in my closet.

Kristen and I are both really happy with how it came out. More than just filling in the room and serving as a night light it actually offers usable additional light when working on the counters, especially when cooking at the stove.

Comments closed

Surprise New Bike Day: 2023 Salsa Warbird C GRX 600 1x

It’s been no secret that for years my favorite drop bar bike was my beloved 2019 Salsa Warbird Carbon 105 700. This bike has been with me on some of my most memorable rides, from remote parts of the UP and Canadian wilderness to silly northern LP bike path routes, from single track where it didn’t quite belong to long summertime wanders on dirt roads.

For years, and especially in 2023 after switching it to 1x, I’ve told numerous folks that there’s nothing I’d change on it and no reason to get anything new; it’s simply excellent. But then in late November while cleaning it up for winter I found a bit of a shock: a small starburst crack in the frame at the top tube/seat tube junction and a couple other hairline cracks on the seat tube. As things wear out and everything eventually fails I can’t say I was devastated, but I really was disappointed. I loved riding that bike and did not want to change anything. I even came to really like the white color.

After finding the crack I rode it a couple more times and things seemed fine. I have a long seatpost which extended well below the crack area, and I didn’t feel or hear anything when riding, but like any crack it’s best to be safe. The warranty for a carbon frame is five years, which was coming up, so I sent photos of the crack to the folks over at the venerable Tree Fort Bikes and Salsa to get things rolling. I wasn’t sure if it was problematic, and — honestly — I really didn’t want to replace the frame if it wasn’t needed, but I wanted to ask. I love how my Warbird looks with the white frame and reflective black vinyl that I added, and I wasn’t exactly keen on a matte black replacement, re-running brake lines, etc. And, it rode great!

After just a handful of questions Craig emailed me with a massive surprise: Salsa is replacing my 2019 Warbird (v4) frame with a complete 2023 Warbird C GRX 1x bike, and instead of being a simple black warranty replacement it’s a nice clay-ish grey color! This was better than I could have possibly hoped for, because not only did it replace the problematic frame, it moved me to Shimano’s GRX drivetrain! And I also wouldn’t have to cut and re-run the brake lines, etc.

Back when my Warbird came out GRX wasn’t available, so it came with the (very good) 105 R7000 road groupset. This is an amazingly good drivetrain, but for rougher roads I switched to the Ultegra RD-RX800 derailleur, which was basically a high-end road derailleur with a clutch, originally intended for cyclocross use. Then in late 2022 I used a few Wolf Tooth Components parts and made it a sorta-hacky yet very functional 1x drivetrain because I wanted to get away from the problems inherent with 2x and riding in poor conditions, notably gunking up a front derailleur.

This setup worked great, but I felt a bit limited by maximum cassette size (40t) and my left brifter had a disconnected shift lever, which would rattle around on chattery roads. Minor, I know, and while I was proud of the semi-hacky drivetrain, improvements such as a full GRX drivetrain would have been nice, but I couldn’t justify it when things worked so well. But suddenly now I had it!

Last week I picked up the bike from Tree Fort, and over some unseasonably rainy, cold, and blah afternoons I shuffled parts around and now it’s ready. My new gravel bike, a 2023 Salsa Warbird C GRX 1x with a few upgrades!

Build

Frame / Fork: 2023 Salsa Warbird Carbon / Salsa Waxwing (Light Grey)
Wheelset: Specialized Roval Terra C
Ratchets: DT Swiss HWTXXX00NSK54S (54T)
Tires: Specialized Pathfinder Pro 2Bliss Ready (700×42, Black Sidewall)
Crank: Shimano FC-RX810-1 (42T, 172.5mm)
Bottom Bracket: Shimano SM-BB72-41B
Cassette:
 Shimano CS-M8000 (11-42)
Right Shift/Brake Levers:
Shimano ST-RX600-R
Left Shift/Brake Levers:
Shimano BL-RX600-L
Brake Calipers:
Shimano BR-RX400
Brake Rotors: Shimano SM-RT64 (160mm)
Brake Pads:
Shimano K05S-RX (Resin)
Chain:
Shimano CN-HG601-11
Rear Derailleur:
Shimano RD-RX812
Bar Tape:
MSW HBT-300 Anti-Slip Gel+ (Black)
Handlebar: Salsa Cowbell Deluxe (44cm)
Headset: Cane Creek Hellbender 70 (IS41/28.6/H9 | IS52/40)
Stem: Thomson Elite X4 (SM-E139 10° X 100mm X 31.8 1-1/8 X4 Black)
Spacers: Generic Aluminum
Stem Cap: MASH Donut 2.0
Seatpost: Thomson Elite (SP-E113SB 27.2 X 410 Setback, Black)
Seatpost Clamp: Salsa Lip Lock
Saddle: Specialized Power Expert (143mm, Black)
Pedals: Shimano PD-M8100
Bottle Cages: Specialized Zee Cage II (2x Left, 2x Right)
Bottle Cage Screws: McMaster-Carr 94500A233 (316 Stainless, M5 x 0.8mm, 20mm)
Front Light: Outbound Lighting Detour
Rear Light: Garmin Varia RTL515
Rear Light Mount: Garmin Varia Seat-post Quarter Turn Mount
Bell: RockBros Bell (Black)
Computer: Garmin Edge 840
Computer Mount: SRAM Quickview Computer Mount
Sensors: Garmin Bike Speed Sensor (Front Wheel), Garmin Cadence Sensor 2 (Crank)
Anti-Rub Tape: McMaster-Carr 76445A764 (Low-Friction UHMW Tape, 0.0115″ Thick, 2″ Wide)
Mounting Hole Plugs: Heyco 2590
Top Tube Bag: Revelate Designs Mag-Tank Bolt-On
Saddle Bag: Lezyne Road Caddy
Frame Pump: Lezyne Sport Drive HP
Derailleur Hanger: 465 / QBP FS2322

Weight

Total weight for the bike, with everything but bottles (including lights, pump, saddle bag+tools, and computer), is 22.54 pounds. Removing the computer/lights/saddle bag/pump brings it down to 20.78 pounds, so I expect that without pedals, cages, or mounts (the usual way of weighing a bike) it’d be in the 19-pound range.

Build Choices:

Crank/Cassette Upgrade: After moving my previous Warbird to 1x a year ago I realized that I like having a 42t front ring. Since the bike came with a 40t ring I wanted to upgrade that, but it turned out that I could get a complete FC-RX810 crankset with a 42t ring for not much more than a stand-alone ring. Swapping from the stock FC-RX610 swap saved 84g while increasing the chainring size, makes mounting a cadence sensor easier, and opens up the possibility of getting a power meter on the bike.

At the same time I ordered an CS-M8000 cassette to replace the stock CS-M5100, which saved another 114g. While I originally was going to get an 11-40 to match my previous Warbird, my friend Ray convinced me that a 42, ending up with an even 1:1 in the lowest gear would be good, and I agreed. Between the two sizes the seven lowest cogs are the same, so typical flat/rolling stuff would feel the same with either, but when I do need a climbing/trail gear it’ll be there.

Chain Drop Protection: I was originally going to fit some chain drop protection, like the Wolf Tooth LoneWolf, but after thinking about how many times I’ve dropped a chain in the past, I opted against it. Mounting this would also require fitting a front derailleur mount which makes bottle cage mounting more fiddly and makes the bike harder to clean. I may still fit this later on, but for now I’m content continuing without.

Rotors: I am generally very fond of Shimano rotors with solid aluminum center carriers as they seem to be harder to bend and have an aluminum core (Ice Technologies) to help with heat dissipation. The bike came with some SM-RT64, and while they are a bit heavier per-rotor (~25g) than others, and just steel, it would have cost a fair bit (~$100) to replace them. I have similar rotors on my fatbike and they’ve been working well, so for now I’m going to stick with these rotors and see how it goes. They can always be upgraded later.

Frame Pump/Saddle Bag: On the previous Warbird I used a somewhat large Specialized saddle bag with a tiny 4″ pump tucked inside next to the tube. This worked well, but I began having problems with the pack Velcro no longer holding, so I also had a releasable cable tie holding it to the saddle.

I’d also never needed — that is tested — the tiny pump in the field, so out of an abundance of caution (and some paranoia) I’d often tuck a second, larger pump in my jersey pocket for long rides. This other pump had been used a few times, so with the bike swap I’ve moved to mounting the beloved (and cheap) Lezyne Sport Drive HP to the frame behind the seat tube bottle cage. I’m wary of road spray causing problems with the pump, but if it does I’ll just start carrying it in my pocket.

By no longer needing room for a pump in the bag I was able to swap to the Lezyne Road Caddy, a small and elegant seat bag that I’ve had on my road bike for a couple years.

Bike Fit: This and my road bike, a custom built Salsa Warroad, are very very similar in geometry, but I’ve had it set up with the bars slightly lower than on the Warbird. Using my favorite stem comparison tool I found that by removing 10mm of spacers below the stem I can get the bar clamp to a nearly identical position on both bikes, so as a bit of an experiment I’m giving this a go, leaving the steerer tube uncut so I can go back if desired.

I’m slightly concerned about the fit when riding more technical trails, and I may have a harder time keeping my forearms near level while on rough surface, but it’s plenty easy to go back if needed.

Bottle Cage Screws: When using Specialized Zee Cages it’s important to have a low profile screw head, else they’ll rub on the bottle and make it hard to insert. The screws which come with the cages are a nice shape, but are a chromed steel that seems to corrode with sweat, sports drink, and road treatment chloride, so I prefer something else. I prefer something like 316 stainless, and I had some 20mm of these laying around from a previous project. For just-bottle-cages this is longer than the needed ~15mm, but the additional mass is across the three standard cages is only 6x 1/4 of the mass of a single screw (1.5x a single screw), or ~5g. It wasn’t worth spending $11+shipping to save that little mass.

Anti-Rub Tape on Head Tube: On my previous Warbird, and on the Warroad, I shortened the front brake hose so it’d take a clean path from the fork to the bar, not touching the head tube. This works, but also gets it the way of the light mount, and makes adjusting spacers difficult because I have to remove the bar from the stem to slide things upward.

The stock hose length on this model rubs the front of the head tube, but I’m not sure I want to shorten it yet. For now I simply put a strip of UHMW PE tape along the front of the head tube, below the Salsa logo, so the hose won’t rub on the frame/paint/carbon. Once I settle on spacers and work a bit more on accessory mounting I may shorten the hose, or I may just leave this alone.

Comments closed

Subaru Outback Cabin Air Filter Airflow Problems

With autumn rolling in the HVAC blower in my Subaru Outback seemed to be acting up. Whenever I’d have the blower at a low speed for basic air circulation it just… didn’t seem to be doing much. It turns out this was caused by the cabin air filter. The heavier filter, the Breathe Easy seen on the right, was restricting flow so much that I’d only get noticeable air movement when the fan was at ~2/3 of maximum or higher.

I’d swapped this filter in over the summer, and at first I didn’t notice, because I normally run the air conditioning (cooling) on a pretty high speed or have the windows down with the blower off. With temperatures dropping I’ll often have fresh air flowing in through the vents, but with the fan at a low speed, just to keep a fresh-air feeling inside the car. On the normal low (one or two bar) setting I just… wasn’t feeling much.

My thoughts first went to problems with an HVAC damper, or perhaps a motor speed controller, but then I remembered the activated charcoal cabin filter I’d fitted and wondered if maybe it was so restricting that the blower wasn’t working right.

Yep, that was the problem. I fitted up a simple Denso paper filter for capturing pollen and everything is working fine again. I’m glad it wasn’t anything more expensive.

Comments closed

Command Line 802.11 Monitor Mode on macOS Sonoma (14.0)

Because it supports monitor mode, a Macbook with the built-in WiFi adapter is one of the simplest ways to grab packets off the air. It’s not the most robust, but often all I need to do is grab data from a couple devices I’m near on a known channel, so fancy antennas and channel hopping and whatnot is overkill; I just need to grab packets. Using the Sniffer built into the Wireless Diagnostics captures in Monitor Mode has been fairly easy for a while, but I was stuck using the GUI.

For a while macOS has had a command line utility called airport to handle all sorts of wireless network manipulation, log gathering, and debugging. It also has a poorly documented command verb sniff, but until the release of macOS Sonoma (14.0) it was only possible to specifying the channel. Not being able to specify the width made it useless for most capturing I’d do in the real world.

Thankfully the airport command now works for channel and width, so now it’s possible to use remotely, in scripts, etc. It’s not well documented, but it works. For example, the following will capture on en0 on 5GHz channel 137 with 80MHz width:

airport en0 sniff 5g137/80

This will capture en1 on 2.4GHz channel 7 at 20MHz width:

airport en0 sniff 2g7/20

Output files end up randomly named in /tmp in pcap format with a name of /tmp/airportSniff??????.cap. They can be opened in Wireshark or your analysis tool of choice.

(I suspect that sniffing from 6GHz WiFi will follow the same pattern, but I don’t have access to a device with such a radio so I’m unable to test. It’d also be pretty nifty to see this somehow built in / better automated via Wireshark… That could be a neat project for later.)

The airport binary can be found at /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport. I link this to ~/bin, with something like the following:

ln -s /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport ~/bin/airport

I keep ~/bin around for personal executable stuff, and it’s been added to my path by putting a line like this in ~/.zshrc:

export PATH=".:$PATH:$HOME/bin"

The airport binary itself has a pretty decent output from --help. It’s light on sniffing examples, but pretty good for other stuff.

Amusingly, this is pretty much the extent of the airport(8) man page; a TODO:

DESCRIPTION
airport manages 802.11 interfaces. airport more information needed here.

Comments closed

Using OpenSSL to Match Certificates and Private Keys

I recently was troubleshooting a problem with network authentication and suspected that the issue was around certificates and private keys not matching on a client. I had a .PEM file for the certificate and a .KEY for the private key, and I wanted to see if they matched.

Thankfully OpenSSL, the Swiss Army Knife of wrangling certs, made it easy. While this isn’t anything particularly secret, it took me a few to figure it out, so I’m re-documenting it here.

To see if the private key matches the certificate, use the following two commands and compare the Modulus section:

openssl x509 -in file.pem -noout -text

openssl rsa -in file.key -noout -text

If they match, the private key matches the certificate. If they don’t, they don’t.

In my case they didn’t match, which was causing the authentication problems. So we then solved what was happening during cert issuance and everything was then good.

Comments closed

Garmin Edge 530 WiFi Connection Weirdness

Today I tried to connect my Garmin Edge 530 running the latest firmware (v9.73) to my home wireless network, and couldn’t get it working. my friend Nick dug up a solution, so I wanted to share it here.

The problem I had is that when trying to find the network to join, either in the Garmin Connect mobile app or right on the device, my WPA2-secured wireless network would be shown as Unsecured and I couldn’t join it. No matter what I tried, on device or in app, switching around network types, names, security, or bands, the Edge 530 always saw it as Unsecured. The one thing I didn’t try was making a non-secured network, but that’s not an option for me.

Turns out you can work around this by using the Garmin Express desktop app then going into the 530, Tools & Content, Utilities, then under Wi-Fi Networks manually adding the network with appropriate security, password, etc.

After saving settings and ejecting the device it joined my wireless network, as confirmed on the device, in the DHCP leases, and on the APs themselves. Now it’ll automatically sync rides whenever I get back home.

Something else odd is the Edge 530 truncates 32-character SSIDs. My network at home is Smart Meter Surveillance Network, which is 32 ASCII characters long. This is the maximum allowed by 802.11, which is 32 octets, or 32 sets of 8 bytes, or 32 ASCII characters. For some reason in much of the Garmin UI it’d drop the last character, truncating to Smart Meter Survellance Networ. Thinking this was the problem I first dug into network name as a problem but eventually found a shorter SSID didn’t help. Also, this isn’t the first device I’ve had with SSID length problems (see Bypassing Reolink SSID Length Limitation); thankfully in this case it only seemed to be a display issue.

Comments closed

How ASUS and a Microsoft Bug Almost Broke Remote Work

A couple of years after it happened I’m sharing this story about the intersection of an OS bug, a network hardware quirk, and a global pandemic. A chain of semi-esoteric things aligned and only caused noticeable problems in a very specific — dare I snarkily say unprecidented — situation.

I found this issue both fascinating and maddening and I hope you will as well. This does not contain code-level details of the bug (I don’t have them), but I’m sharing it both to document this problem and share a little story of what goes on behind the scenes in supporting a big enterprise IT environment.

In March 2020, when Stay Home, Stay Safe began in Michigan, most of my fellow employees began working from home (WFH). For years we’d been building IT systems to allow most people to work from anywhere, and the shift to WFH was going great. Between VPN connections back to the corporate network, lots of things in the cloud, and Azure Active Directory (AAD) to handle Single Sign-On (SSO) for almost every company application, all most folks needed for remote work was their standard laptop and an internet connection. The computing experience of working from home was effectively the same experience as working in the office.

Sure, we had some bumps with home internet connections not being robust enough, but we helped people through those. Mostly we’d find that what someone thought was a good home internet connection — because their phone or video streaming worked fine — wasn’t great for things like moving big files around or video calls. [1] Generally those having network performance issues had fine service from their ISP, but their home routers were old and not up to task. We would recommend upgrading the router, they’d go buy something new, and all would be good.

In early autumn we began receiving reports of users getting the notorious You Can’t Get There From Here (YCGTFH) message when trying to access anything that used AAD for SSO. This generic message is displayed when AAD authentication fails and access is denied. Because so many things were behind AAD SSO this interrupted a lot of work. These computers either were no longer joined to AAD or had an expired token, and it seemed tied to internet access.

Digging into it, there would be errors shown by dsregcmd /status (the AAD CLI utility) and Test-DeviceRegConnectivity.ps1. This script checks internet access as SYSTEM to the AAD public endpoints would fail on all three connection tests, implying the lack of connectivity to Microsoft endpoints was keeping AAD registration (and token refresh) from working properly. But the user still could browse the web and hit those URLs and we hadn’t changed anything internet access-wise since the WFH began.

We found that manually setting a proxy server for the whole of the system (via netsh winhttp set proxy) would allow AAD registration (dsregcmd /join) to succeed and SSO would then work. We also found that restarting the WinHTTP Web Proxy Auto-Discovery Service (WinHttpAutoProxySvc), which handles WPAD, would sometimes fix the problem, but only sometimes.

Even more confusingly sometimes a reboot would fix it. Or, sometimes if a user drove into the office and used the network there, it would work. But not always. [2]

Simply, we had some computers whose SYSTEM account couldn’t access the internet for so long that an AAD token had expired, this broke SSO and users were being told You Can’t Get There From Here.

Typical for a lot of large organizations we have authenticating proxy servers sitting between the client network and the public internet. All requests bound for the internet need to go through them, and these proxies are located by a Proxy Auto-Config (PAC) file that is found either by a direct setting (AutoConfigUrl) or Web Proxy Auto-Discovery (WPAD), via DNS, both of which send the same file. We directly set the PAC file URL on a per-user basis and leave WPAD at its default of enabled. Thus for the end user and things running under their account, WPAD is used, falling back to the PAC file setting if that fails. For the SYSTEM account a direct PAC file setting is not used, relying solely on WPAD to find the path to the internet. [3]

Looking at a network capture when AAD registration would fail instead of the normal chain of events requests we’d see no DNS requests for WPAD, and no PAC file download. Instead we saw the client attempting to resolve the AAD endpoints via DNS, and then would attempt to reach out directly to them, which would be blocked by the company firewall. The proxy was not being used; WPAD wasn’t working. This was weird because every piece worked when tested independently (DNS resolution for WPAD hostnames, invoke-webrequest http://x.x.x.x/wpad.dat, specifying the WPAD PAC file in AutoConfigUrl), but as a whole it just didn’t work.

This went on for quite a while, supported by a Premier case with Microsoft. We could see that WPAD was frequently failing, but struggled with getting a consistent reproduction and going down dead-ends. We bandaged the problem with manual, direct proxy settings and AAD registration. This was mostly fine short-term, but caused overhead for our support folks and was a ticking bomb.

Then one day, thanks to a fortuitous conversation with a very smart lead Microsoft engineer while working another issue I found out about a bug with Microsoft’s WPAD implementation that was just discovered and was being patched in the next round of patches. The description exactly explained our problem and I was elated.

It turned out that if Windows 10 received a blank DHCP option 252, the WinHttpAutoProxySvc service would not query DNS for WPAD, and — the broken part — it would never do so again until the service was restarted. Directly configured PAC files would be used, but WPAD was broken. Here in our environment the SYSTEM account would not have internet access and this meant AAD registration, Test-DeviceRegConnectivity.ps1, the Microsoft Store, and all such internet-needing SYSTEM-level things didn’t work.

Apparently some home router vendors — most notably the hugely-popular ASUS [4] — would send option 252 but leave it blank because they found doing so reduced name resolution requests from clients. This is seen in Wireshark as:
Option: (252) Private/Proxy autodiscovery
    Length: 1
    Private/Proxy autodiscovery: \n

Windows, which has WPAD enabled by default, will try a number of name resolution queries (DNS for wpad/ wpad.local.tld.com/wpad.local.com, NetBIOS, LLMNR, etc) to locate a PAC file server if it does not see a DHCP option 252. Because WinHttpAutoProxySvc looks to DHCP then only tries DNS if that fails, by setting this option but leaving it blank the name resolution steps would not occur. I can only guess as to why these vendors find it desirable, but perhaps they like reducing the load on the built-in DNS forwarders, or they saw it as a security benefit or… who knows. Either way, the result of this blank option and the Windows bug was that WPAD — via DHCP and DNS — didn’t work.

So what is option 252? In WPAD there are multiple discovery mechanisms for finding the PAC file server. Beyond DNS there is also a Dynamic Host Configuration Protocol (DHCP) method where, along with the typical network address settings, the client receives the URL for downloading the PAC file. This is done via option 252, but isn’t widely supported, it’s normally not used, and we don’t use it it either.

While WPAD is core OS function, it unfortunately never left draft RFC status. Implementations have no formal standard to target; it’s just a guideline. Additionally, because it never left draft status, DHCP option 252 also remains unallocated and without a standard, simply part of the Reserved (Private Use) range. So, setting it but leaving it blank is not unacceptable, OS’ should be able to accommodate.

In a network capture I was then able to clearly see this happen, and it was simple to replicate on a test network. And then it finally all came together…

COVID-19 WFH resulted in a bunch of people upgrading their home networks, with lots of them buying new home routers, including the very-popular ASUS brand. If someone booted up their Windows 10 computer and the blank option 252 sending DHCP server was the first thing it saw, WPAD would break, with all the downstream consequences, including our AAD connectivity issues. And if they’d have AAD connectivity issues for long enough — until a token expired — they would start getting You Can’t Get There From Here messages.

If they went in the office or somewhere else which doesn’t set 252 and fully rebooted (which restarts the WinHttpAutoProxySvc service), WPAD would work and AAD registration would work. But if the service never restarted — if they never actually rebooted — WPAD was stuck not doing anything. [2]

Testing a pre-release version of the patch showed it fixed the problem, and then a few weeks later KB4601382 was released, with the detail “Improves the ability of the WinHTTP Web Proxy Auto-Discovery Service to ignore invalid Web Proxy Auto-Discovery Protocol (WPAD) URLs that the Dynamic Host Configuration Protocol (DHCP) server returns.”. We deployed this patch and the reports of AAD registration / You Can’t Get There From Here issues collapsed.

That was it, it was fixed. A popular home router vendor did something weird (but not against standard), the OS implemented something poorly, and people were working for so long in one of those environments that a credential expired, couldn’t be renewed, and they lost access to SSO.


[1] Mobile apps and a lot of modern websites are fairly asynchronous, sending requests in the background while still working nicely, because they are built to be tolerant of the blips that happen while on wireless networks. Video streaming specifically caches (or buffers) the video locally so that hiccups in the network connection don’t make the video pause and stutter. More real-time-ish things like Remote Desktop or video calls or copying files via SMB are considerably more sensitive to poor network connections.

[2] Retrospectively I suspect confusion around what it means to shut down or restart a computer led to many of the reports of reboots/shutdowns/driving into the office fixing the problem or not and the difficulties in getting a reliable reproduction. Different sleep modes, some of which result in the BIOS displaying the POST when waking from sleep even if the OS doesn’t restart, leads some to believe the operating system was restarted when it may not have been. Or other folks believe that closing the lid is “shutting down”.

It was also unfortunately common for users to have a flexible version of “home”. Sometimes home meant where they’d been working for the last six months and had the problem, sometimes it meant a vacation home with a different ISP they’d gone to the day before but failed to mention, sometimes it meant the other side of the world. Teasing this information out was difficult, as many used “home” to mean anywhere but their normally assigned desk. “I’ve only been at home” frequently meant “I continue to be not at the office”.

[3] We use WPAD because Windows 10 requires internet access for a lot of OS level things that run as SYSTEM, including the Microsoft Store and AAD device registration. If proxy servers are used, the SYSTEM account needs to find the proxy servers. Early on in our Windows 10 deployment Microsoft told us either direct internet access or WPAD was required. WPAD via DHCP doesn’t work with most VPN clients, because they configure their virtual adapters directly and not via DHCP. Thus, to have similar connectivity when in the office or remote, DNS for WPAD is the the choice.

[4] I’ve been told multiple brands do this, but ASUS only vendor where I personally observed it.

Comments closed

1x’ing my Salsa Warbird for 2023

Every since getting one of the v4 models back in late 2018, the Salsa Warbird has been one of my absolute favorite bikes. I’ve ridden on all surfaces, and it’s been great, with only one recurring problem: front shifting.

These days bikes ridden anywhere but pavement are almost exclusively rear-shifting-only (1x) — without a front derailleur — and for good reason. Unfortunately, when I bought my Warbird there weren’t many (any?) 1x groups for drop bar use; Shimano hadn’t even announced their excellent gravel-focused GRX group. So, my bike came with the best bang-for-the-buck drivetrain at the time, Shimano 105 R7000. This remains an excellent groupset, and I put it on my new road bike, but a couple things about it were a bit lacking for off-pavement use: lack of a clutch derailleur to keep the chain taut when things got rough, and the front derailleur.

Early on I switched the rear derailleur to a clutched Ultegra RX (RD-RX800-GS), which was basically a preview of GRX, and worked wonderfully. It dropped right in place on the bike and Just Worked to solve chain slap. But, I still had one more problem… The front derailleur.

Salsa Warbird v4 Front Derailleur Cable Route

Located where the rear wheel flings debris, front derailleurs are quite prone to getting gunked up while riding, and (unfortunately) the Warbird exacerbates this by having the shift cable exit the frame at the bottom bracket, behind the seat tube, facing upward. This location collects whatever flings off the tire and is a very hard place to seal a moving cable. Coupled with our gritty, chloride-treated dirt roads, the result has been shift cables and housing corroding and binding far before any other drivetrain parts wear.

This corrosion leads to extra drag on the shift cable needed to shift, up until things bind firmly, after which the sacrificial Main Lever Support L (part number Y63X80010) in the shifter to breaks (exploded view). The overall design of this sacrificial part is great; a removable and cheap plastic bushing on the shift/brake lever blade that presses against the metal piece that handles the shifting. If the cable becomes too tight this $3 piece fails before anything else in the shifter, and replacing it takes seconds. But, the cable and housing also need to be fixed, and COVID-era parts shortages (these supports were unobtainium for about 12 months — I was using 3D printed ones for a while) and it all quickly becomes an irritation.

I’d lived with front shifting quirks for a couple years as most of the time it was fine, but losing the front derailleur was always in the back of my mind. Mountain bikes have this way for years, with a rear cassette that provides all the range needed, and this has become common on new gravel bikes.

After front shifting failed early on in De Ronde van Grampian and I ended up stuck in the big ring, I realized two things: I should get rid of the front derailleur, and I don’t need quite all the range that a 50-34/11-34 2×11 drivetrain offers. I use most of the range, but could lose a little on the top and bottom end of the range and be fine.

The first change was to remove the front derailleur, which was made easy by the front shift housing being split with a barrel adjuster below the handlebar. I was able to pull the cable and remove the housing all the way back to just under the the bar, leaving 2″ stub of housing unobtrusively peeking out of the bar tape. Next time I rewrap the bars I’ll remove this, but for now it’s fine.

Then to finish the conversion I fitted a Wolf Tooth 42t chainring specifically designed with an offset designed for 1x. None of the single-ring chainring bolts that I had around the house fit right, so I grabbed some of Wolftooth’s matching chainring bolts and it was all set. The ring fits the crank perfectly, worked great with the original chain length, and the I could shift across the whole of the cassette without any rear adjustment.

I did two rides on local rolling dirt roads and except for some longer steep climbs things were good. I was feeling good for both rides, so I didn’t mind having to grind out the longer climbs in sub-optimal cadence, but I realized I might need some more range on longer days. I’d also been concerned the left brifter would be flappy and noisy without a shift cable connected, but it’s just the same as before.

Next up was getting more gear range…

Gear Ratios for 105 R7000 50/34 11-34, 42×11-34, and 42×11-40.

The original Shimano 105 R7000 drivetrain, with 50/34 chainrings and an 11-34 cassette, gave a range from 455% to 100%, plenty for all my riding, but with a whole bunch of overlap. While I’d use the lowest ratio gears somewhat frequently, I’d almost never use the highest gear because at typical cadence I’d be moving well above 33 MPH. At those speeds I’m probably coasting down a hill and no longer pedaling.

After changing to a single 42t chainring the available ratios were 382% to 124%. At the high end this is nearly identical to everything but the rarely-used smallest cog, but at the low end it was like I lost the easiest two gears. On a couple test local couple-hour rides I didn’t absolutely need them, but it did require a bit more effort on some of the harder climbs.

With 11-speed mountain bike drivetrains on the decline, I came across Shimano XT 11-40 cassettes on sale and decided to try a bigger cassette on the rear of the bike. I was still on the original cassette and it was nearly worn out, so a change was due. This required a 1.85mm spacer to make the MTB cassette fit on a road freehub, which moved the cogs outboard, but a few limit adjustments and all was good. Now with a range of 382% to 105% it was almost identical to what I’d use on the original 2x setup.

Ultegra RX derailleur and 11-40 Cassette

The larger cassette fit pretty well, but caused some small quirks in spacing with the upper jockey wheel: the B-tension screw had to be cranked in almost all the way to have the gap set properly on biggest cog. It was quite a fiddly adjustment, as a quarter turn too far one way would have the jockey wheel rubbing on the cassette and a quarter turn the other would make shifting slow. This much B-tension also unwrapped the chain from the cassette a fair bit, making me concerned that with less chain on the cassette while pedaling the chain and cassette would wear faster than normal.

At this point I did a test ride and everything worked great when riding. Between the new chain (old was at 0.5%), cassette, and chainring the drivetrain was silent and I didn’t have a bit of trouble, but I still wanted it to be better.

A few days earlier I’d ordered a Wolftooth RoadLink DM, a nifty part that replaces a link in rear derailleurs like mine, repositioning it to handle larger cassettes. This is advertised to improve chainwrap and shifting performance, and immediately after installation the benefits were apparent. On the biggest cog chainwrap is much better, the derailleur no longer looks over-extended, the B-screw doesn’t need to be turned all the way in, and shifting feels more spot on.

Ultegra RX w/ RoadLink DM and 11-40 Cassette

While I could have fitted a Shimano GRX RD-RX812 derailleur instead of using the RoadLink DM on the Ultegra RX, this would have cost $90 more than the RoadLink DM, and I’m not sure it’d be any better.

All done, total cost for this was $265.03 after tax, and with a new cassette, chain, and chainring, I’ve now basically got a new drivetrain.

Wolf Tooth Chainring: $84.95
Wolf Tooth Chainring Bolts: $26.45
Shimano XT Cassette: $95.39
Shimano SLX/105 Chain: $26.49
Wolf Tooth RoadLink DM: $31.75

I’m quite happy with how this all turned out. A problem is solved, worn out things were upgraded, functional parts were kept, and it works great. My Warbird is ready for another year of gravel fun.

Comments closed

BorgBackup Repository on DSM 7.0

A few years back I began using Borg for backing up nuxx.net, sending it home to my Synology DSM 1019+. At the time this was running the 6.x family of DSM and worked great, but it broke after moving to v7.0. Attempts to run Borg would result in this error:

/var/services/homes/borguser/borg: error while loading shared libraries: libz.so.1: failed to map segment from shared object

This appears to be happening because with the upgrade to v7.0 /tmp is mounted noexec.

adminuser@diskstation:/var/services/homes/borguser$ mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec)
adminuser@diskstation:/var/services/homes/borguser$

While a few online solutions (such as this one) propose remounting /tmp with exec, this is a poor solution as it changes the security model for DSM v7.0 and may break in the future during an upgrade. The best solution for this is to create a private temp directory for just borguser and define it as $TMPDIR.

To do this create ~borguser/tmp, ensure it’s owned by your Borg user, and set it to 700:

mkdir ~borguser/tmp
chown borguser:users ~borguser/tmp
chmod 700 ~borguser/tmp

Then create a wrapper script for Borg setting this variable. The result will be Borg using ~borguser/tmp for it’s private temporary directory, leaving /tmp alone, working nicely with the DSM v7.0 security design. I keep mine in ~borguser/.ssh and call it borg.sh. And, be sure it’s executable. Mine is like this:

adminuser@diskstation:/var/services/homes/borguser$ sudo cat .ssh/borg.sh
!/bin/sh
export TMPDIR=$HOME/tmp
/var/services/homes/borguser/borg serve --storage-quota 120G --restrict-to-repository /volume2/Backups/borg
adminuser@diskstation:/var/services/homes/borguser$ sudo ls -als .ssh/borg.sh
4 -rwx------ 1 borguser root 161 Nov 15 06:56 .ssh/borg.sh
adminuser@diskstation:/var/services/homes/borguser$

Finally, change ~borguser/.ssh/authorized_keys limiting the backup user to executing the new script.

command="/var/services/homes/backupuser/.ssh/borg.sh",restrict,from="192.168.0.23" ssh-rsa AAAA[...restofkeygoeshere...] remoteuser@remoteserver.example.com

Comments closed

HOWTO: Apple TV Volume Control of Bose Solo Soundbar Series II

After moving into our new house I we switched our trainer TV’s audio setup to a Bose Solo Soundbar Series II from Costco for $159.99. It’s a cheaper, basic soundbar that connects via TOSLINK, but sounds plenty good for TV/movie watching while pedaling away during the winter.

One of the really nifty features of the Apple TV is its ability to learn the infrared (IR) remote control signals of a another device so its remote can control the volume of receivers and such. This works out wonderfully, as with HDMI ARC, from the one Apple remote we can wake up the Apple TV, wake up the TV (and have it switch to the correct input), and control the volume of the soundbar.

Unfortunately, the learning process just didn’t work with the IR remote that comes with the soundbar. For whatever reason the Apple TV wouldn’t detect the IR signal being sent and the learning would fail.

Our previous setup — an old Yamaha home theater receiver and some Energy surround sound speakers — had no problems with the Apple TV. Once set up it was super convenient, so I wanted this in the new house.

I was able to solve this by buying a cheap universal remote, setting it up to control the soundbar, and then using it to the Apple TV. A bit of hoop jumping, and it cost an extra $10, but to me that’s worth the convenience.

Specifically, I bought this Philips SRP3249B/27 via Amazon, programmed the AUD button with code 2706 (Sound Bar, Bose), then used that to program the Apple TV via SettingsRemotes and DevicesVolume ControlLearn New Device….

When going through the learning process on the Apple TV I noticed an interesting quirk: If I followed the on-screen instructions exactly and held the remote button until the progress bar filled up, I would have to press and release the button on the Apple TV remote repeatedly to keep changing the volume. That is, each press/release turned it up or down just one notch on the soundbar.

By repeatedly pressing the button during the learn process the Apple TV learned something slightly different and then holding the button on its remote resulted in the soundbar continually increasing/decreasing the volume as the button is held.

Either way, it’s now working all from the Apple TV remote and all is good again. It just took an intermediate “universal” remote to bridge the gap.

Comments closed