Press "Enter" to skip to content

nuxx.net Posts

How ASUS and a Microsoft Bug Almost Broke Remote Work

A couple of years after it happened I’m sharing this story about the intersection of an OS bug, a network hardware quirk, and a global pandemic. A chain of semi-esoteric things aligned and only caused noticeable problems in a very specific — dare I snarkily say unprecidented — situation.

I found this issue both fascinating and maddening and I hope you will as well. This does not contain code-level details of the bug (I don’t have them), but I’m sharing it both to document this problem and share a little story of what goes on behind the scenes in supporting a big enterprise IT environment.

In March 2020, when Stay Home, Stay Safe began in Michigan, most of my fellow employees began working from home (WFH). For years we’d been building IT systems to allow most people to work from anywhere, and the shift to WFH was going great. Between VPN connections back to the corporate network, lots of things in the cloud, and Azure Active Directory (AAD) to handle Single Sign-On (SSO) for almost every company application, all most folks needed for remote work was their standard laptop and an internet connection. The computing experience of working from home was effectively the same experience as working in the office.

Sure, we had some bumps with home internet connections not being robust enough, but we helped people through those. Mostly we’d find that what someone thought was a good home internet connection — because their phone or video streaming worked fine — wasn’t great for things like moving big files around or video calls. [1] Generally those having network performance issues had fine service from their ISP, but their home routers were old and not up to task. We would recommend upgrading the router, they’d go buy something new, and all would be good.

In early autumn we began receiving reports of users getting the notorious You Can’t Get There From Here (YCGTFH) message when trying to access anything that used AAD for SSO. This generic message is displayed when AAD authentication fails and access is denied. Because so many things were behind AAD SSO this interrupted a lot of work. These computers either were no longer joined to AAD or had an expired token, and it seemed tied to internet access.

Digging into it, there would be errors shown by dsregcmd /status (the AAD CLI utility) and Test-DeviceRegConnectivity.ps1. This script checks internet access as SYSTEM to the AAD public endpoints would fail on all three connection tests, implying the lack of connectivity to Microsoft endpoints was keeping AAD registration (and token refresh) from working properly. But the user still could browse the web and hit those URLs and we hadn’t changed anything internet access-wise since the WFH began.

We found that manually setting a proxy server for the whole of the system (via netsh winhttp set proxy) would allow AAD registration (dsregcmd /join) to succeed and SSO would then work. We also found that restarting the WinHTTP Web Proxy Auto-Discovery Service (WinHttpAutoProxySvc), which handles WPAD, would sometimes fix the problem, but only sometimes.

Even more confusingly sometimes a reboot would fix it. Or, sometimes if a user drove into the office and used the network there, it would work. But not always. [2]

Simply, we had some computers whose SYSTEM account couldn’t access the internet for so long that an AAD token had expired, this broke SSO and users were being told You Can’t Get There From Here.

Typical for a lot of large organizations we have authenticating proxy servers sitting between the client network and the public internet. All requests bound for the internet need to go through them, and these proxies are located by a Proxy Auto-Config (PAC) file that is found either by a direct setting (AutoConfigUrl) or Web Proxy Auto-Discovery (WPAD), via DNS, both of which send the same file. We directly set the PAC file URL on a per-user basis and leave WPAD at its default of enabled. Thus for the end user and things running under their account, WPAD is used, falling back to the PAC file setting if that fails. For the SYSTEM account a direct PAC file setting is not used, relying solely on WPAD to find the path to the internet. [3]

Looking at a network capture when AAD registration would fail instead of the normal chain of events requests we’d see no DNS requests for WPAD, and no PAC file download. Instead we saw the client attempting to resolve the AAD endpoints via DNS, and then would attempt to reach out directly to them, which would be blocked by the company firewall. The proxy was not being used; WPAD wasn’t working. This was weird because every piece worked when tested independently (DNS resolution for WPAD hostnames, invoke-webrequest http://x.x.x.x/wpad.dat, specifying the WPAD PAC file in AutoConfigUrl), but as a whole it just didn’t work.

This went on for quite a while, supported by a Premier case with Microsoft. We could see that WPAD was frequently failing, but struggled with getting a consistent reproduction and going down dead-ends. We bandaged the problem with manual, direct proxy settings and AAD registration. This was mostly fine short-term, but caused overhead for our support folks and was a ticking bomb.

Then one day, thanks to a fortuitous conversation with a very smart lead Microsoft engineer while working another issue I found out about a bug with Microsoft’s WPAD implementation that was just discovered and was being patched in the next round of patches. The description exactly explained our problem and I was elated.

It turned out that if Windows 10 received a blank DHCP option 252, the WinHttpAutoProxySvc service would not query DNS for WPAD, and — the broken part — it would never do so again until the service was restarted. Directly configured PAC files would be used, but WPAD was broken. Here in our environment the SYSTEM account would not have internet access and this meant AAD registration, Test-DeviceRegConnectivity.ps1, the Microsoft Store, and all such internet-needing SYSTEM-level things didn’t work.

Apparently some home router vendors — most notably the hugely-popular ASUS [4] — would send option 252 but leave it blank because they found doing so reduced name resolution requests from clients. This is seen in Wireshark as:
Option: (252) Private/Proxy autodiscovery
    Length: 1
    Private/Proxy autodiscovery: \n

Windows, which has WPAD enabled by default, will try a number of name resolution queries (DNS for wpad/ wpad.local.tld.com/wpad.local.com, NetBIOS, LLMNR, etc) to locate a PAC file server if it does not see a DHCP option 252. Because WinHttpAutoProxySvc looks to DHCP then only tries DNS if that fails, by setting this option but leaving it blank the name resolution steps would not occur. I can only guess as to why these vendors find it desirable, but perhaps they like reducing the load on the built-in DNS forwarders, or they saw it as a security benefit or… who knows. Either way, the result of this blank option and the Windows bug was that WPAD — via DHCP and DNS — didn’t work.

So what is option 252? In WPAD there are multiple discovery mechanisms for finding the PAC file server. Beyond DNS there is also a Dynamic Host Configuration Protocol (DHCP) method where, along with the typical network address settings, the client receives the URL for downloading the PAC file. This is done via option 252, but isn’t widely supported, it’s normally not used, and we don’t use it it either.

While WPAD is core OS function, it unfortunately never left draft RFC status. Implementations have no formal standard to target; it’s just a guideline. Additionally, because it never left draft status, DHCP option 252 also remains unallocated and without a standard, simply part of the Reserved (Private Use) range. So, setting it but leaving it blank is not unacceptable, OS’ should be able to accommodate.

In a network capture I was then able to clearly see this happen, and it was simple to replicate on a test network. And then it finally all came together…

COVID-19 WFH resulted in a bunch of people upgrading their home networks, with lots of them buying new home routers, including the very-popular ASUS brand. If someone booted up their Windows 10 computer and the blank option 252 sending DHCP server was the first thing it saw, WPAD would break, with all the downstream consequences, including our AAD connectivity issues. And if they’d have AAD connectivity issues for long enough — until a token expired — they would start getting You Can’t Get There From Here messages.

If they went in the office or somewhere else which doesn’t set 252 and fully rebooted (which restarts the WinHttpAutoProxySvc service), WPAD would work and AAD registration would work. But if the service never restarted — if they never actually rebooted — WPAD was stuck not doing anything. [2]

Testing a pre-release version of the patch showed it fixed the problem, and then a few weeks later KB4601382 was released, with the detail “Improves the ability of the WinHTTP Web Proxy Auto-Discovery Service to ignore invalid Web Proxy Auto-Discovery Protocol (WPAD) URLs that the Dynamic Host Configuration Protocol (DHCP) server returns.”. We deployed this patch and the reports of AAD registration / You Can’t Get There From Here issues collapsed.

That was it, it was fixed. A popular home router vendor did something weird (but not against standard), the OS implemented something poorly, and people were working for so long in one of those environments that a credential expired, couldn’t be renewed, and they lost access to SSO.


[1] Mobile apps and a lot of modern websites are fairly asynchronous, sending requests in the background while still working nicely, because they are built to be tolerant of the blips that happen while on wireless networks. Video streaming specifically caches (or buffers) the video locally so that hiccups in the network connection don’t make the video pause and stutter. More real-time-ish things like Remote Desktop or video calls or copying files via SMB are considerably more sensitive to poor network connections.

[2] Retrospectively I suspect confusion around what it means to shut down or restart a computer led to many of the reports of reboots/shutdowns/driving into the office fixing the problem or not and the difficulties in getting a reliable reproduction. Different sleep modes, some of which result in the BIOS displaying the POST when waking from sleep even if the OS doesn’t restart, leads some to believe the operating system was restarted when it may not have been. Or other folks believe that closing the lid is “shutting down”.

It was also unfortunately common for users to have a flexible version of “home”. Sometimes home meant where they’d been working for the last six months and had the problem, sometimes it meant a vacation home with a different ISP they’d gone to the day before but failed to mention, sometimes it meant the other side of the world. Teasing this information out was difficult, as many used “home” to mean anywhere but their normally assigned desk. “I’ve only been at home” frequently meant “I continue to be not at the office”.

[3] We use WPAD because Windows 10 requires internet access for a lot of OS level things that run as SYSTEM, including the Microsoft Store and AAD device registration. If proxy servers are used, the SYSTEM account needs to find the proxy servers. Early on in our Windows 10 deployment Microsoft told us either direct internet access or WPAD was required. WPAD via DHCP doesn’t work with most VPN clients, because they configure their virtual adapters directly and not via DHCP. Thus, to have similar connectivity when in the office or remote, DNS for WPAD is the the choice.

[4] I’ve been told multiple brands do this, but ASUS only vendor where I personally observed it.

Comments closed

1x’ing my Salsa Warbird for 2023

Every since getting one of the v4 models back in late 2018, the Salsa Warbird has been one of my absolute favorite bikes. I’ve ridden on all surfaces, and it’s been great, with only one recurring problem: front shifting.

These days bikes ridden anywhere but pavement are almost exclusively rear-shifting-only (1x) — without a front derailleur — and for good reason. Unfortunately, when I bought my Warbird there weren’t many (any?) 1x groups for drop bar use; Shimano hadn’t even announced their excellent gravel-focused GRX group. So, my bike came with the best bang-for-the-buck drivetrain at the time, Shimano 105 R7000. This remains an excellent groupset, and I put it on my new road bike, but a couple things about it were a bit lacking for off-pavement use: lack of a clutch derailleur to keep the chain taut when things got rough, and the front derailleur.

Early on I switched the rear derailleur to a clutched Ultegra RX (RD-RX800-GS), which was basically a preview of GRX, and worked wonderfully. It dropped right in place on the bike and Just Worked to solve chain slap. But, I still had one more problem… The front derailleur.

Salsa Warbird v4 Front Derailleur Cable Route

Located where the rear wheel flings debris, front derailleurs are quite prone to getting gunked up while riding, and (unfortunately) the Warbird exacerbates this by having the shift cable exit the frame at the bottom bracket, behind the seat tube, facing upward. This location collects whatever flings off the tire and is a very hard place to seal a moving cable. Coupled with our gritty, chloride-treated dirt roads, the result has been shift cables and housing corroding and binding far before any other drivetrain parts wear.

This corrosion leads to extra drag on the shift cable needed to shift, up until things bind firmly, after which the sacrificial Main Lever Support L (part number Y63X80010) in the shifter to breaks (exploded view). The overall design of this sacrificial part is great; a removable and cheap plastic bushing on the shift/brake lever blade that presses against the metal piece that handles the shifting. If the cable becomes too tight this $3 piece fails before anything else in the shifter, and replacing it takes seconds. But, the cable and housing also need to be fixed, and COVID-era parts shortages (these supports were unobtainium for about 12 months — I was using 3D printed ones for a while) and it all quickly becomes an irritation.

I’d lived with front shifting quirks for a couple years as most of the time it was fine, but losing the front derailleur was always in the back of my mind. Mountain bikes have this way for years, with a rear cassette that provides all the range needed, and this has become common on new gravel bikes.

After front shifting failed early on in De Ronde van Grampian and I ended up stuck in the big ring, I realized two things: I should get rid of the front derailleur, and I don’t need quite all the range that a 50-34/11-34 2×11 drivetrain offers. I use most of the range, but could lose a little on the top and bottom end of the range and be fine.

The first change was to remove the front derailleur, which was made easy by the front shift housing being split with a barrel adjuster below the handlebar. I was able to pull the cable and remove the housing all the way back to just under the the bar, leaving 2″ stub of housing unobtrusively peeking out of the bar tape. Next time I rewrap the bars I’ll remove this, but for now it’s fine.

Then to finish the conversion I fitted a Wolf Tooth 42t chainring specifically designed with an offset designed for 1x. None of the single-ring chainring bolts that I had around the house fit right, so I grabbed some of Wolftooth’s matching chainring bolts and it was all set. The ring fits the crank perfectly, worked great with the original chain length, and the I could shift across the whole of the cassette without any rear adjustment.

I did two rides on local rolling dirt roads and except for some longer steep climbs things were good. I was feeling good for both rides, so I didn’t mind having to grind out the longer climbs in sub-optimal cadence, but I realized I might need some more range on longer days. I’d also been concerned the left brifter would be flappy and noisy without a shift cable connected, but it’s just the same as before.

Next up was getting more gear range…

Gear Ratios for 105 R7000 50/34 11-34, 42×11-34, and 42×11-40.

The original Shimano 105 R7000 drivetrain, with 50/34 chainrings and an 11-34 cassette, gave a range from 455% to 100%, plenty for all my riding, but with a whole bunch of overlap. While I’d use the lowest ratio gears somewhat frequently, I’d almost never use the highest gear because at typical cadence I’d be moving well above 33 MPH. At those speeds I’m probably coasting down a hill and no longer pedaling.

After changing to a single 42t chainring the available ratios were 382% to 124%. At the high end this is nearly identical to everything but the rarely-used smallest cog, but at the low end it was like I lost the easiest two gears. On a couple test local couple-hour rides I didn’t absolutely need them, but it did require a bit more effort on some of the harder climbs.

With 11-speed mountain bike drivetrains on the decline, I came across Shimano XT 11-40 cassettes on sale and decided to try a bigger cassette on the rear of the bike. I was still on the original cassette and it was nearly worn out, so a change was due. This required a 1.85mm spacer to make the MTB cassette fit on a road freehub, which moved the cogs outboard, but a few limit adjustments and all was good. Now with a range of 382% to 105% it was almost identical to what I’d use on the original 2x setup.

Ultegra RX derailleur and 11-40 Cassette

The larger cassette fit pretty well, but caused some small quirks in spacing with the upper jockey wheel: the B-tension screw had to be cranked in almost all the way to have the gap set properly on biggest cog. It was quite a fiddly adjustment, as a quarter turn too far one way would have the jockey wheel rubbing on the cassette and a quarter turn the other would make shifting slow. This much B-tension also unwrapped the chain from the cassette a fair bit, making me concerned that with less chain on the cassette while pedaling the chain and cassette would wear faster than normal.

At this point I did a test ride and everything worked great when riding. Between the new chain (old was at 0.5%), cassette, and chainring the drivetrain was silent and I didn’t have a bit of trouble, but I still wanted it to be better.

A few days earlier I’d ordered a Wolftooth RoadLink DM, a nifty part that replaces a link in rear derailleurs like mine, repositioning it to handle larger cassettes. This is advertised to improve chainwrap and shifting performance, and immediately after installation the benefits were apparent. On the biggest cog chainwrap is much better, the derailleur no longer looks over-extended, the B-screw doesn’t need to be turned all the way in, and shifting feels more spot on.

Ultegra RX w/ RoadLink DM and 11-40 Cassette

While I could have fitted a Shimano GRX RD-RX812 derailleur instead of using the RoadLink DM on the Ultegra RX, this would have cost $90 more than the RoadLink DM, and I’m not sure it’d be any better.

All done, total cost for this was $265.03 after tax, and with a new cassette, chain, and chainring, I’ve now basically got a new drivetrain.

Wolf Tooth Chainring: $84.95
Wolf Tooth Chainring Bolts: $26.45
Shimano XT Cassette: $95.39
Shimano SLX/105 Chain: $26.49
Wolf Tooth RoadLink DM: $31.75

I’m quite happy with how this all turned out. A problem is solved, worn out things were upgraded, functional parts were kept, and it works great. My Warbird is ready for another year of gravel fun.

Comments closed

BorgBackup Repository on DSM 7.0

A few years back I began using Borg for backing up nuxx.net, sending it home to my Synology DSM 1019+. At the time this was running the 6.x family of DSM and worked great, but it broke after moving to v7.0. Attempts to run Borg would result in this error:

/var/services/homes/borguser/borg: error while loading shared libraries: libz.so.1: failed to map segment from shared object

This appears to be happening because with the upgrade to v7.0 /tmp is mounted noexec.

adminuser@diskstation:/var/services/homes/borguser$ mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec)
adminuser@diskstation:/var/services/homes/borguser$

While a few online solutions (such as this one) propose remounting /tmp with exec, this is a poor solution as it changes the security model for DSM v7.0 and may break in the future during an upgrade. The best solution for this is to create a private temp directory for just borguser and define it as $TMPDIR.

To do this create ~borguser/tmp, ensure it’s owned by your Borg user, and set it to 700:

mkdir ~borguser/tmp
chown borguser:users ~borguser/tmp
chmod 700 ~borguser/tmp

Then create a wrapper script for Borg setting this variable. The result will be Borg using ~borguser/tmp for it’s private temporary directory, leaving /tmp alone, working nicely with the DSM v7.0 security design. I keep mine in ~borguser/.ssh and call it borg.sh. And, be sure it’s executable. Mine is like this:

adminuser@diskstation:/var/services/homes/borguser$ sudo cat .ssh/borg.sh
!/bin/sh
export TMPDIR=$HOME/tmp
/var/services/homes/borguser/borg serve --storage-quota 120G --restrict-to-repository /volume2/Backups/borg
adminuser@diskstation:/var/services/homes/borguser$ sudo ls -als .ssh/borg.sh
4 -rwx------ 1 borguser root 161 Nov 15 06:56 .ssh/borg.sh
adminuser@diskstation:/var/services/homes/borguser$

Finally, change ~borguser/.ssh/authorized_keys limiting the backup user to executing the new script.

command="/var/services/homes/backupuser/.ssh/borg.sh",restrict,from="192.168.0.23" ssh-rsa AAAA[...restofkeygoeshere...] remoteuser@remoteserver.example.com

Comments closed

HOWTO: Apple TV Volume Control of Bose Solo Soundbar Series II

After moving into our new house I we switched our trainer TV’s audio setup to a Bose Solo Soundbar Series II from Costco for $159.99. It’s a cheaper, basic soundbar that connects via TOSLINK, but sounds plenty good for TV/movie watching while pedaling away during the winter.

One of the really nifty features of the Apple TV is its ability to learn the infrared (IR) remote control signals of a another device so its remote can control the volume of receivers and such. This works out wonderfully, as with HDMI ARC, from the one Apple remote we can wake up the Apple TV, wake up the TV (and have it switch to the correct input), and control the volume of the soundbar.

Unfortunately, the learning process just didn’t work with the IR remote that comes with the soundbar. For whatever reason the Apple TV wouldn’t detect the IR signal being sent and the learning would fail.

Our previous setup — an old Yamaha home theater receiver and some Energy surround sound speakers — had no problems with the Apple TV. Once set up it was super convenient, so I wanted this in the new house.

I was able to solve this by buying a cheap universal remote, setting it up to control the soundbar, and then using it to the Apple TV. A bit of hoop jumping, and it cost an extra $10, but to me that’s worth the convenience.

Specifically, I bought this Philips SRP3249B/27 via Amazon, programmed the AUD button with code 2706 (Sound Bar, Bose), then used that to program the Apple TV via SettingsRemotes and DevicesVolume ControlLearn New Device….

When going through the learning process on the Apple TV I noticed an interesting quirk: If I followed the on-screen instructions exactly and held the remote button until the progress bar filled up, I would have to press and release the button on the Apple TV remote repeatedly to keep changing the volume. That is, each press/release turned it up or down just one notch on the soundbar.

By repeatedly pressing the button during the learn process the Apple TV learned something slightly different and then holding the button on its remote resulted in the soundbar continually increasing/decreasing the volume as the button is held.

Either way, it’s now working all from the Apple TV remote and all is good again. It just took an intermediate “universal” remote to bridge the gap.

Comments closed

My Photo Gallery Has Been Removed

Since 2001 or so I’ve been hosting most of my personal photos in a public gallery at nuxx.net/gallery (archive.org link to one of the first instances).

Today, ~21 years later, I’ve taken it down.

I’ve no single reason why; it’s basically a combination of these things:

  • Two years ago I made the photo gallery static, so nothing new was being added.
  • Keeping the gallery online costs $8/mo for the extra 80GB volume at Linode.
  • I previously used this gallery to share a lot of life online. I no longer do this in the same way.
  • Outside of social media, everything I publish online for the last few years has been via this blog.

The aforementioned static archiving of Gallery and backups with Borg made it easy to move hosting to my NAS at home. Now I can continue accessing the site myself while simplifying my public internet presence.

This feels like a huge shift, but the right one.

Comments closed

Soma Smoothie HP: Brake Safety Issue

Shift cable/brake interference on prototype Soma Smoothie HP Frame (Photo from Soma)

Originally this post was going to be about my new road bike, but instead I’m writing a warning about the new Soma Smoothie HP and a safety issue, recommending against buying it without fully understanding the possible downsides. While the specs of this frame promise to be an excellent modern steel road bike, it has a design problem that can lead to rear brake failure.

Specifically, the location of the brake housing mounts results in the front derailleur cable resting on the brake cable at a mount. Over time movement of this cable can cut into the brake hose/housing, cable clip/tie, etc. If not caught in time this can lead to brake failure.

I purchased one of these frames and have been excitedly building it up, finding this problem late into the build when preparing to run the shift cables and brake lines.

When I brought this up with Soma seeking for a solution, they dismissed the issue, saying they identified this interference during frame development, but that it’s not concerning to them. Stan, with Soma, instead suggested that I rely on the cable tie (as seen above) to mitigate the rub, that rubbing through a brake hose or housing will take a long time, and to add additional material (rubber cable donuts or tape) to the cable if I am worried about it.

I disagree, as any component that’s designed to move should not have unintentional rub, particularly not against a safety-critical system like a brake. A frame should be designed to avoid this; it should not be necessary to bodge in rub protection to stave off cables cutting into other housing.

The photo at the top of this post is of Soma’s prototype build and shows the interference and their reliance on the head of a cable tie for keeping the shift cable in place and away from the brake line. Beyond the safety issue of the front shift cable rubbing on the brake line, the cable does not have a clean path between the upper stop and lower guide, which can lead to shifting issues.

While it would be possible to build this frame up as a 1x (no front derailleur) system to avoid this interference, I want this to be a double-chainring road bike (Shimano 105 R7000) and thus is not an option for me. Or one could go with wireless shifting, but due to cost this isn’t an option for me either.

I really wish Soma had fixed this interference when they first identified it, or at least mentioned it in their documentation so I would have passed on buying the frame. All it would have taken is moving the brake routing to either the centerline (like on this Lynskey) or much further up the downtube (like on my Salsa Vaya) and it’d have been solved.

After I found this a good friend reached out to me, and I’ve since passed the frame on to him. He was looking to build an wireless shifting road bike with fenders, and this will work out perfectly for him. With the electronic shifting he won’t run into this cabling issue.

For those considering this frame, unless you are going with wireless shifting, a 1x setup, or are willing to deal with the compromises of a shift cable rubbing on a brake housing/hose, I suggest that you look elsewhere.

EDIT on 2024-Feb-20: This evening I received email from Stanley at Soma Fabrications about updates to the Smoothie HP frame. It sounds like this will alleviate the concern I mention, so I’m sharing it here:

I just want to update you that early 2023 we replaced the plastic BB mounted cable guide on all our HP frames with one of those guides with two mounting holes (similar to Shimano SP18) which moves the derailleur wire just enough to avoid any contact with the zip tie or brake housing. 

Though we maintain the opinion the possibility of the front derailleur wire cutting thru both the zip tie and housing in real world use is extremely remote with the old set-up, we chose to improve the routing. 

The new production that arrived this month moves the brake guides to the 6 o’clock area which will probably avoid contact no matter what plastic guide is used on the BB.

Stanley Pun, Soma Fabrications, via email on February 20, 2024

Here (PDF) is the full email thread with Soma.

Below are photos showing the interference on my frame. These were taken when the build was nearly complete, while I was planning cable routing. These photos are when I first realized the problem and reached out to Soma.

Comments closed

Upgrading Pi-Hole on Docker on Synology DSM

At home I use Pi-Hole running via Docker on a Synology DS1019+ (details here). Every once in a while I update this, but it seems to be long enough between updates that I forget the exact steps, so I’m documenting them here.

This is the simplest update process I know of. It keeps the configuration the same but updates the container itself. Because the volumes for persistent storage are not changed and the custom environment variables stay the same, all settings are preserved.

  1. Log into DSM.
  2. Launch Docker.
  3. In Docker, update the image:
    • Registry
    • Search for pihole
    • Select pihole/pihole (ensure it is the official image)
    • Click Download
  4. Stop and reset the container:
    • Container
    • Select your Pi-hole container.
    • Stop
    • Reset
    • Start
  5. Wait a few minutes, then log back in to Pi-Hole (eg: http://pi.hole:8081/admin/)
  6. At the bottom of the page observe updated versions. Check here for the latest version of the Docker image, for comparison.

Note: This process presumes you have configured volumes mounted to /etc/dnsmasq.d and /etc/pihole, and have set environment variables INTERFACE, WEB_PORT, WEBPASSWORD, and TZ. Without these your results may vary.

Comments closed

Help Please: 2022 AIDS/LifeCycle Fundraising

Years ago Mark Ferlatte told me about the AIDS/LifeCycle (Wikipedia) charity ride from San Francisco to Los Angeles to raise money for the San Francisco AIDS Foundation and the Los Angeles LGBT Center. Money raised by this event allows the two non-profits to provide free HIV/AIDS medical care, testing, and prevention services. Each rider in the event raises at least $3000, with the event raising over $220,000,000 since it began in 1993.

At 545 miles over seven days, riding everywhere from the Pacific Coast Highway to proper California mountains, this sounded like it a wonderfully fun way to see a new part of the country, really enjoy being outside, and most importantly raise money to directly help save lives. But, it always seemed like such a logistical challenge that fell into the back of my mind as a one-day-in-the-future goal.

Well, here we are, and 2022 is the year!

This year Kristen and I are both signed up for the 2022 AIDS/LifeCycle ride! On June 5th we will set out from Cow Palace in San Francisco to spend the next week pedaling through one of the most beautiful parts of the country with a couple thousand other folks, all of us with one goal: helping those with HIV/AIDS.

This is where we need your help. Specifically, we need money.

Kristen and I each are working to raise least $3000 through donations from people like you. We all give money to trails, trail organizations, and other groups which help make our lives more fun and enjoyable. Please join us in also giving a bit more to save lives, so everyone can have a great life.

Please click one of these links to donate: SteveKristen

(Donations are to a non-profit, and thus tax deductible.)

Thank you for your help, and for making the lives of those with HIV/AIDS better!

Comments closed

Network Capture with Process Name and PID on macOS

There are many times when one wants to see which process is responsible for network traffic on a endpoint. While there are ways to look at which process has an open socket or whatnot, this doesn’t help with UDP, and it’s often quite useful to simply do a recording then later see which process is responsible for creating a packet.

On Windows this is very easy using Pktmon / netsh trace / Network Monitor to do a capture, but on macOS it’s not that straightforward.

Using tcpdump with the -k argument one can print process name and PID to the console. In this example I’m filtering on just the host 8.8.8.8 (Google Public DNS) then running dig in another window to look up dingleberrypie.com:

c0nsumer@myopia ~ % sudo tcpdump -k -i en0 host 8.8.8.8
Password:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:17:58.210959 pid dig.76987 svc BE IP myopia.home.nuxx.net.56248 > dns.google.domain: 10294+ [1au] A? dingleberrypie.com. (47)
09:17:58.292541 IP dns.google.domain > myopia.home.nuxx.net.56248: 10294 1/0/1 A 96.126.107.52 (63)

The bold line is the packet going out to 8.8.8.8, and the pid dig.76987 portion shows that it’s from a dig process, which had process ID 76987.

Unfortunately, due to the non-standard way Apple writes this metadata into files, it’s not viewable in Wireshark. This would be very nice, as it’d make parsing large captures much easier.

For now it seems the best way to do this is to record the capture to a file, then feed it back into tcpdump. Recording this requires using the pktap pseudo interface (see the tcpdump man page about the --interface argument) to ensure this data is saved into the file. The same capture above, writing to a file called out.cap, would be as follows:

sudo tcpdump -i pktap,en0 host 8.8.8.8 -w out.cap

This can then be fed back into tcpdump for parsing/filtering/viewing:

c0nsumer@myopia ~ % tcpdump -k -r out.cap
reading from PCAP-NG file out.cap
09:27:10.964758 (en0, proc dig:77619, svc BE, out, so) IP myopia.home.nuxx.net.52983 > dns.google.domain: 56446+ [1au] A? dingleberrypie.com. (47)
09:27:11.018809 (en0, proc dig:77619, svc BE, in, so) IP dns.google.domain > myopia.home.nuxx.net.52983: 56446 1/0/1 A 96.126.107.52 (63)
c0nsumer@myopia ~ %

In this case it shows both the sending process and the process which received the packet.

When using this for more detailed analysis, I’ll use the macOS tcpdump to grab a very broad capture and then do first-pass filtering before bringing it into Wireshark for more detailed analysis. See the PACKET METADATA FILTER section of the tcpdump man page for details on how to filter on a PID, process name, etc. From the header of this section:

Use packet metadata filter expression to match packets against descriptive information about the packet: interface, process, service type or direction.

Comments closed

Making an Online RAMBA Trails Map

RAMBA Trails Map v1.0 (Default View w/ Epic Loop Layer)

Introduction

Kristen and I have been spending a good deal of time in the Ishpeming and Negaunee area this year, and I’ve made it a personal goal to become more familiar the local trails — both RAMBA-supported and otherwise — and get them documented OpenStreetMap (OSM). Having these trails in OSM provides two big benefits: they appear in other mapping tools (such as OsmAnd, GaiaGPS, Strava, MapMyRide/MapMyRun) and the trail data can be freely used to build other tools.

Official RAMBA Map v6 from 2018

Over the years of making trail maps with OpenStreetMap I’ve mostly produced PDFs for printing, leaving mobile and online mapping to other apps. These work well, but have the big downside of rendering routes with their style. That is, online maps via these tools’ show the routes, but look quite different from print maps, even if all the data for them to display more data (such as colour=* tags on relations) is in OSM.

While these apps work pretty well, and I use them myself routinely for navigation, I got the itch to see if I could make a web-based map that looked more like locally produced print maps than app-based renderings. It seemed like a good project, a good way to learn some basics of modern web development, and maybe make something useful.

What I ended up with was a slippy map of showing the RAMBA trails that uses layers of pre-rendered tiles to show the different official trail routes, placed over a background map. The map viewer is client-side JavaScript that loads static tiles from a basic web server, making this a very simple app to host (just a bunch of static files on a site).

In this post I intend to document the major steps of how I made this map, why I used the tools I did, and share the code to reproduce (and update) this build. Hopefully this’ll allow others to get their head around these map presentation basics, perhaps even reusing this work to make and host another map.

Update OSM Data

Strava Global Heatmap in JOSM

Mostly outside the scope of this article but worth a mention, a significant amount of time was spent ensuring that the RAMBA area trails are accurately listed in OSM. Without good data it would not be possible to go further, as the OSM data is the base data used to create other maps.

By combining information from a bunch of sources, and doing some personal surveying of trails while riding and hiking, I was able to get all of the official RAMBA trails documented, along with numerous other paths and tracks in the area. This building a complete picture of the usable trails in the area.

Information used to get the RAMBA trails in OSM included:

  • My own recorded ride/hiking data, notes, etc.
  • Strava Heatmap data.
  • Official route files (GPXs) from events such as Marji Gesick and Polar Roll.
  • The official RAMBA map.
  • Hand-annotated map from Danny Hill listing local trail names.

These sources were combined in JOSM, cross-referenced, ways drawn and tagged, relations built out, and before long a complete picture of the RAMBA-area trails — official and otherwise — were in OpenStreetMap.

Most importantly, beyond documenting the trail locations, trails were grouped into route relations to show each official route, and then all the official routes were grouped into a superroute for all the RAMBA trails. As of time of writing, relation RAMBA Trails (12425503) is the superroute that aggregates the individual trail routes such as Epic Loop (8467869) and Malton Loop (8468010).

The result of this is accurate trail data that’s easy to query for and style using other tools.

Rendering Tiles with Maperitive

There are myriad ways to render tiles from OSM data, with most of these involving setting up a database server and a toolchain which’ll generate, cache, and serve tiles on demand. For most large data sets this makes a lot of sense, but for a small trail system I really wanted to use static tiles I could serve from a simple webserver.

Eventually I came across Maperitive, a desktop application for Windows that takes GIS data (including OSM), stylizes it with a relatively simple ruleset, and can generate tiles in the standard XYZ format for use elsewhere. It can also be scripted, which meant I could use it as part of an automated workflow to generate new tiles as the OSM data changes. This seemed like a good solution, so I set about writing some rulesets that would reasonably show the RAMBA trail routes and some automation around it all.

After a lot of experimenting I settled on a having separate tile set for each of the official loops, an overview of all trails, and a base map. The base map would always be shown, and a user can toggle between layers which highlight all the trails or individual loops.

After a few iterations of custom rules, I settled on a simplified set based on the Default.mrules file which comes with Maperitive for rendering the base map. The only modification was changing the font to Michael Adams’ Roadgeek 2005 Transport Medium font, as it looks nicer than the default, Verdana. For the overview and route layers I created simple rules based on the the default rendering of highway=path, using the Heavy version of the font. The rule for each trail route (relation) selects the trails in a given relation then colors them accordingly.

Creating these rules took a bit of fiddling, as Maperitive is both a bit of a dead project, not completely documented, and (in the latest Beta) sort-of buggy where sometimes the map display would stop updating. Still, even though I’m not great at making attractive things, I was able to come up with something that worked well enough.

Conveniently, Maperitive also comes with a command line version (Maperitive.Console.exe). After settling on rendering rules and a tile generation script, I used this as part of an automated workflow which downloaded OSM data directly then rendered each of the tile sets.

After tile generation I used a Windows binary of OptiPNG to losslessly compress the tiles, resulting in a ~62% space savings (original: 746MB, optimized: 286MB) which’ll reduce storage and bandwidth overhead.

The Front End

Editing in VS Code

With tiles generated I needed a way to display them. It turns out that OpenLayers was easy to use and it all ran as simple client side application in a browser. By using npm and parcel, with Visual Studio Code for editing, it was quite easy to get the site developed, tested, and bundled up for deployment. The only component I had to add was ol-layerswitcher control, which provides an easy way to toggle between layers.

Prior to this I had very little experience with modern web development, with my exposure to JavaScript pretty much limited to reading others’ code to figure out what it’s doing. After a bit of confusion (and having to accept the hidden complexity of using an application bundler), I was able to focus solely on writing a single main.js file with a basic index.html that together do what I wanted:

  • Run full screen by default.
  • Show all trails by default, with toggles for the defined routes (layers of the map).
  • Show an attractive background map below the routes to show the rest of the area.
  • Offer controls to use geolocation to showing one’s location on the map and reset the view to the original map extents.
  • Look sane on desktop and mobile devices.

This ended up being much easier than I thought, and between the OpenLayers Examples and just some basic programming I was able to get something I’m happy with. Far more time was spent designing the tiles and thinking about what I wanted it to do than writing the code to display it all.

Tile Hosting

The actual map tiles are a number of small PNG files, and a typical session of viewing and panning around the map can result in hundreds of image loads. This was seeming a bit slow when being served from nuxx.net via HTTP/1.1, so I looked into using HTTP/2 to improve performance.

Unfortunately, it was not simple to turn on HTTP/2 here at nuxx.net as I’m using PHP for WordPress, which in turn requires MPM prefork, which precludes mod_http2. I could have set up another web server and such, but for now I’m hosting the tiles in AWS, with the tiles uploaded to an S3 bucket and served via CloudFront.

This should allow for better tile download performance than what I can do from my server. Despite potentially incurring a bit of a financial cost it is a good experiment in hosting tiles in the cloud. I may change this in the future, particularly if it becomes cost prohibitive, but for now it’s working well.

Follow Along At Home

If you would like to generate this same map, start by downloading this ZIP file: ramba_trails_map_code_1.0.zip. It contains the scripts and rules needed to generate the map tiles (ramba.mscript and the .mrules files), the index.html, main.js, and package.json for the OpenLayers-based front end, the .osm file used to generate the first release of the map, and a few batch files that tie it all together.

You will need to download Maperitive (latest beta: link — from this post / mirror), curl, OptiPNG, and the Roadgeek 2005 fonts to generate and optimize the tiles. These scripts may work fine with older/release versions of Maperitive, but betas incorporate a bit of collision detection for text, making things look nicer.

These batch files are included to will help you out, but may need some editing to fit on your environment:

  • fetch_osm.bat: Uses curl to download all OSM data within a bounding box that encompasses the Ishpeming/Negaunee area.
  • generate_tiles.bat: Runs ramba.mscript via Maperitive.Console.exe to generate the tiles.
  • optimize_tiles.bat: Copies the unoptimized tiles from the .../tile_output/raw output directory to the .../tile_output/optimized directory, then runs OptiPNG against the tiles to optimize them in place.

To build the web app you’ll need to install npm, parcel, create a new OpenLayers app as per the directions here. Then install ol-layerswitcher (npm install ol-layerswitcher), replace the default index.html, main.js, and package.json with the ones I provided, and you should be ready to go.

Updating the Map

As you can see, the map is two major pieces: the front end and the tiles. Whenever the map data changes in OSM the tiles can be regenerated to update those layers. The code for the front end web app only needs to change if the storage location changes, features are going to be added, etc.

Conclusion

This map has worked out rather well and I’m happy calling it v1.0. It’s been a great learning experience, and I’ve even managed to produce something useful that didn’t exist before: an interactive map of some of the most rugged single track trails in Michigan; one of my favorite places to ride mountain bikes.

It’s far from perfect, and there are some things I could do differently, but for now, I’m considering it a success. When in Negaunee for vacation last week I successfully used development versions of this map to find my way around, so I know it’s better than nothing.

If you find any quirks in the map data — such as trails with wrong names or in the wrong location — please take a screenshot and show me what’s wrong and email that to steve@nuxx.net. I’ve done my best to ensure the RAMBA trails are accurately mapped, but I’ve certainly missed some things.

Problems

  • No key or other ancillary information (such as logos) as are normally found on print maps.
  • No terrain. While 1m DEM elevation data is available from the USGS, I couldn’t figure out how to use it in Maperitive for generating hillshading.
  • No easy way to add clickable items to show additional info, link to external map apps (eg: for navigation).
  • Maperitive’s text rendering isn’t the best, resulting in goofy looking text at some zoom levels.
  • Long trails only have one label placed on the middle. Trails with one name broken into multiple ways will be labeled numerous times.
  • Due to being run in a browser it’s a sufficient, but not great, mobile experience. Specifically, selecting the geolocation, recenter, and layer controls can be fiddly because they are so small.
  • Does not work offline, but thankfully most of the RAMBA area now has good mobile data coverage.

Things To Investigate

  • Keep an eye on AWS cost and performance.
  • Look at Leaflet for the front end, as it seems a bit more modern.
  • Consider rendering map tiles with TileMill. This will add a lot of complexity both in setup and styling tiles, but once done should allow a lot more flexibility in styling and overcome most of Maperitive’s problems. mapbox/mbutil should work for getting XYZ PNGs out of MBTiles files.
  • Consider using a tile server if I don’t want to deal with discrete files.
  • Look more into using vector tiles with client-side styling. (I passed on this for now, as a GeoJSON file showing each of the route is a large download and had no benefit over raster tiles.)
  • Maperitive should run under Mono, and OptiPNG is available for many platforms, meaning it should be possible to reproduce this build under macOS or Linux. Note that the GUI for Maperitive will not currently run on macOS due to Windows.Forms currently being based on Carbon, which is not available for 64-bit macOS. So while the CLI should work, the GUI version isn’t currently compatible with macOS 11.5 (Big Sur) and higher.
Comments closed