Press "Enter" to skip to content

nuxx.net Posts

Archiving Gallery 2 with HTTrack

Published July 5, 2020

Along with the static copy of the MediaWiki, I’ve been wanting to make a static, archival copy of the Gallery 2 install that I’ve been using to share photos, for 15+ years, at nuxx.net/gallery. Using HTTrack I was able to do so, after a bit of work, resulting in a copy at the same URL and with images accessed using the same paths, from static files.

The result is that I no longer need to run the aging Gallery 2 software, yet links and embedded images that point to my photo gallery did not break.

In the last few years I’ve both seen the traffic drop off, I haven’t posted many new things there, and it seems like the old Internet of pointing people to a personal photo gallery is nearly dead. I believe that blog posts, such as this, with links to specific photos, are where effort should be put. While there is 18+ years of personal history in digital images in my gallery, it doesn’t get used the same way it was 10 years ago.

On the technical side, the relatively-ancient (circa 2008) Gallery 2 has and the ~90GB of data in it has occasionally been a burden. I had to maintain an old copy of PHP just for this app, and this made updating things a pain. While there is a recent project, Gallery the Revival, which aims to update Gallery to newer versions of PHP, this is based around Gallery 3 and a migration to that brings about its own problems, including breaking static links.

I’m still not sure if I want to keep the gallery online but static as it is now, put the web app back up, completely take it off the internet and host it privately at home, or what… but figuring out how to create an archive has given me options.

What follows are my notes on how I used HTTrack, a package specifically designed to mirror websites, to archive nuxx.net’s Photo Gallery. I encountered a few bumps along the way, so this details each and how it was overcome, resulting in the current static copy. To find each of these I’d start HTTrack, let it run for a while, see if it got any errors, fix them, then try again. Eventually I got it to archive cleanly with zero errors:

Gallery Bug 83873

During initial runs, HTTrack finished after ~96MB (out of ~90GB of images) saved, reporting that it was complete. The main portions of the site looked good, but many sub-albums or original-resolution images were zero-byte HTML files on disk and displayed blank in the browser. This was caused by Gallery bug 83873, triggered by using HTTPS on the site. It seems to be fixed by adding the following line just before line 780 in .../modules/core/classes/GallerySession.class:

GalleryCoreApi::requireOnce('modules/core/classes/GalleryTranslator.class');

This error was found by via the following in Apache’s error log:

AH01071: Got error 'PHP message: PHP Fatal error: Class 'GalleryTranslator' not found in /var/www/vhosts/nuxx.net/gallery/modules/core/classes/GallerySession.class on line 780\n', referer: http://nuxx.net/gallery/

Minimize External Links / Footers

To clean things up further, minimizing external links, and make the static copy of the site as simple as possible, I also removed external links in footer by commenting out the external Gallery links and version from the footer, via .../themes/themename/templates/local/theme.tpl and .../themes/themename/templates/local/error.tpl:

<div id="gsFooter">
{*
{g->logoButton type="validation"}
*{g->logoButton type="gallery2"}
*{g->logoButton type="gallery2-version"}
*{g->logoButton type="donate"}
*}
</div>

Remove Details from EXIF/IPTC Plugin

The EXIF/IPTC Plugin for Gallery is excellent because it shows embedded metadata from the original photo, including things like date/time, camera model, location. This presents as a simple Summary view and a lengthier Details view. Unfortunately, when being indexed by HTTrack, selecting of the Details view — done via JavaScript — returns a server error. This shows up in the HTTrack UI as an increasing error count, and server errors as some pages are queried.

To not have a broken link on every page I modified the plugin to remove the Summary and Details view selector so it’d only display Summary, and used the plugin configuration to ensure that every field I wanted was shown in the summary.

To make this change copy .../modules/exif/templates/blocks/ExifInfo.tpl to .../modules/exif/templates/blocks/local/ExifInfo.tpl (to create a local copy, per the Editing Templates doc). Then edit the local copy and comment out lines 43 through 60 so that only the Summary view is displayed:

{* {if ($exif.mode == 'summary')}
* {g->text text="summary"}
* {else}
* <a href="{g->url arg1="controller=exif.SwitchDetailMode"
* arg2="mode=summary" arg3="return=true"}" onclick="return exifSwitchDetailMode({$exif.blockNum},{$item.id},'summary')">
* {g->text text="summary"}
* </a>
* {/if}
* &nbsp;&nbsp;
* {if ($exif.mode == 'detailed')}
* {g->text text="details"}
* {else}
* <a href="{g->url arg1="controller=exif.SwitchDetailMode"
* arg2="mode=detailed" arg3="return=true"}" onclick="return exifSwitchDetailMode({$exif.blockNum},{$item.id},'detailed')">
* {g->text text="details"}
* </a>
* {/if}
*}

Disable Extra Plugins

Finally, I disabled a bunch of plugins which both wouldn’t be useful in a static copy of the site, and cause a number of interconnected links which would make a mirror of the site overly complicated:

  • Search: Can’t search a static site.
  • Google Map Module: Requires a maps API key, which I don’t want to mess with.
  • New Items: There’s nothing new getting posted to a static site.
  • Slideshow: Not needed.

Fix Missing Files

My custom theme, which was based on matrix, linked to some images in the matrix directory which were no longer present in newer versions of the themes, so HTTrack would get 404 errors on these. I copied these files from my custom theme to the .../themes/matrix/images directory to fix this.

Clear Template / Page Cache

After making changes to templates it’s a good idea to clear all the template caches so all pages are rendering with the above changes. While all these steps may be overkill, I do this by going into Site Admin → Performance and setting Guest Users and Registered Users to No acceleration. I then uncheck Enable template caching and click Save. I then click Clear Saved Pages to clear any cached pages, then re-enable template caching and Full acceleration for Guest Users (which HTTrack will be working as).

PANIC! : Too many URLs : >99999

If your Gallery has a lot of images, HTTrack could quit with the error PANIC! : Too many URLs : >99999. Mine did, so I had to run it with the -#L1000000 argument so that it’ll then be limited to 1,000,000 URLs instead of the default 99,999.

Run HTTrack

After all of this, I ran the httrack binary with the security (bandwidth, etc) limits disabled (--disable-security-limits) and used its wizard mode to set up the mirror. The URL to be archived was https://nuxx.net/gallery/, stored in an appropriately named project directory, with no other settings.

CAUTION: Do not disable security limits if you don’t have good controls around the site you are mirroring and the bandwidth between the two. HTTrack has very sane defaults for rate limiting when mirroring that keep its behavior polite, it’s not wise to override these defaults unless you have good control of the source and destination site.

When httrack begins it shows no progress on screen, so I quit with Ctrl-C, switched to the project directory, and ran httrack --continue to allow the mirror to continue and show status info on the screen (the screenshot above). The argument --continue can be used to restart an interrupted mirror, and --update can be used to freshen up a complete mirror.

Alternately, the following command puts this all together, without the wizard:

httrack https://nuxx.net/gallery/ -W -O "/home/username/websites/nuxx.net Photo Gallery" -%v --disable-security-limits -#L1000000

As HTTrack spiders the site it comes across external links and needs to know what to do with them. Because I didn’t specify an action for external links on the command line, it prompts with the question “A link, [linkurl], is located beyond this mirror scope.”. Since I’m not interested in mirroring any external sites (mostly links to recipes or company websites) I answer * which is “Ignore all further links and do not ask any more questions” (text in httrack.c). (I was unable to figure out how to suppress this via a command line option before getting a complete mirror, although it’s likely possible.)

Running from a Dedicated VM

I ran this mirror task from a Linode VM, located in the same region as the VM hosting nuxx.net. This results in all traffic flowing over the Private network, avoiding bandwidth charge.

Because of the ~90GB of images, I set up a Linode 8GB, which has 160GB of disk, 8GB of RAM, and 4 CPUs. This should provide plenty of space for the mirror, with enough resources to allow the tool to work. This VM costs $40/mo (or $0.06/hr), which I find plenty affordable for getting this project done. The mirror took N days to complete, after which I tar’d it up and copied it a few places before deleting the VM.

By having a separate VM I was able to not worry about any dependencies or package problems and delete it after the work is done. All I needed to do on this VM was create a user, put it in the sudoers file, install screen (sudo apt-get install screen) and httrack (sudo apt-get install httrack), and get things running.

Wrapping It All Up

After the mirror was complete I replaced my .../gallery directory with the .../gallery directory from the HTTrack output directory and all was good.

Comments closed

Archiving MediaWiki with mwoffliner and zimdump

Published June 23, 2020

For a number of years on nuxx.net I used MediaWiki to host technical content. The markup language is nearly perfect for this sort of content, but in recent years I haven’t been doing as much of this and maintaining the software became a bit of a hassle. In order to still make the content available but get rid of the actual software, I moved all the content to static HTML files.

These files were created by creating a ZIM file — commonly used for offline copies of a website — and then extracting that file. The extracted files, a static copy of the MediaWiki-based site, was then made available using Apache.

You can get the ZIM file here, or browse the new static pages here.

Here’s the general steps I used to make it happen.

Create ZIM file: mwoffliner --mwUrl="https://nuxx.net/" --adminEmail=steve@nuxx.net --redis="redis://localhost:6379" --mwWikiPath="/w/" --customZimFavicon=favicon-32x32.png

Create HTML Directory from ZIM File: zimpdump -D mw_archive outfile.zim

Note: There are currently issues with zimdump and putting %2f HTML character codes in filenames instead of creating paths. This is openzim/zim-tools issue #68, and will need to be fixed by hand.

Consider using find . -name "*%2f*" to find problems with files, then use rename 's/.{4}(.*)/$1/' * (or so) to fix the filenames after moving them into appropriate subdirectories.

If using Apache (as I am) create .htaccess to set MIME Types Appropriately, turning off the rewrite engine so higher-level redirects don’t affect things:

<FilesMatch "^[^.]+$">
ForceType text/html
</FilesMatch>

RewriteEngine Off

Link to http://sitename.com/outdir/A/Main_Page to get to the original main wiki page. In my case, http://nuxx.net/wiki_archive/A/Main_Page.

 

Comments closed

Seat Tube Angle Makes a Big Difference

Published June 4, 2020

When I started planning the All-City Electric Queen my main goal was to have a hard tail version of my 2016 Specialized Camber. That is, nice suspension, 120mm of travel in the fork, dropper, and a very similar RAD / RAAD. The only dimension I wanted to change was the seat tube angle, as with the 75° tube on the Camber and I need to position a saddle pretty far back on the rails, even with the 10mm setback on the setback Command Post IRCC.

Unfortunately, I think I went too far.

When comparing geometries I saw the Electric Queen has a 71° seat tube and figured that’d be perfect for moving the saddle back, better centering it in the zero-offset Fox Transfer post. What I failed to do was the sort of calculation shown above to illustrate just how far back the saddle would be moved by every degree of seat tube angle change.

It turns out that, at 725mm of saddle height, every degree of seat tube angle change moves the saddle about 12mm. Thus, on my Camber which has a 75° seat tube, 10mm offset post, and the saddle about 15mm behind centered, I should have been looking for a ~73° seat tube angle in order to roughly center the saddle on a straight (0mm offset) seatpost. Going to a 71° seat tube moved the center point a full 48mm — nearly two inches — rearward.

During single track rides on the Electric Queen I’ve ended up with a sore lower back when pushing, I’m not comfortable at higher cadences, and my stomach brushes my thighs when leaned over and seated. All of this is a sign of the saddle being too far back. With the 71° seat tube angle I simply can’t get the saddle far enough forward: it’s center position is 38mm behind where it is in the Camber (with it’s 10mm offset post). There’s just not enough fore/aft room for adjustment on a saddle to make up for this. When just 5mm or 10mm, especially rearward, makes a difference this isn’t good.

This all then brings about the question: did I buy too large of a frame? The answer is: no, I bought a frame with the wrong seat tube angle for my body.

Fitted forward of the bottom bracket — based on stack and reach — the frame fits properly. Where it doesn’t work is from the bottom bracket to the saddle, all due to the 71° seat tube angle. All of the Electric Queen frames have the same seat tube angle, with the only real variances between sizes being stack, reach, and seat tube length.

The problem is that my body shape (shorter legs, likely shorter femurs specifically, longer torso) not being compatible with this frame.

This consistent 71° seat tube angle, from X-Large all the way down to the X-Small frame, makes me wonder how this frame can fit smaller size riders, unless maybe they have exceptionally long femurs. I wear pants with a 30″ inseam — fairly short but not tiny by any stretch — and my legs seem normally proportioned, so if this setback doesn’t work for me there’s likely a whole bunch of other folks it won’t work for either. Or maybe I just have short femurs?

So, where do I go from here?

I’m going to experiment with some other saddles around home (eg: Specialized Phenom) and see if I can make it fit, but it looks like the only choice for getting a comfortably ridable bike will be another frame with a steeper seat tube angle. On first look it seems like the Ritchey Ultra (Large) or Kona Honzo ST (Medium) would work, with the Ultra having a slight edge in with 73.5° seat tube and taking the same size seatpost as the Electric Queen.

Comments closed

All-City Electric Queen: Rugged XC / Trail Hard Trail Bike

Published May 31, 2020

Nearly four years after getting a full suspension XC / trail bike (2016 Specialized Camber) and beginning to really enjoy riding with a bit more travel and a dropper post I started getting the itch to build a similarly spec’d hard tail. Picking up a few particularly good deals (frame/fork), shopping smartly, using some parts I’d collected over the years, and getting some parts from friends, I ended up putting together an All-City Electric Queen.

This steel frame from a QBP brand best known for urban bikes, fitted with a smart choice drivetrain (SRAM GX), higher end wheels (I9 Hydra hubs) and suspension (Fox 34), and reliable brakes (Shimano SLX) is the solid spec build that I was wanting. The frame came complete with a matching rigid fork, and I anticipate switching over to the rigid fork in springtime or when the Fox 34 is out for service.

After the first shakedown ride at Stony Creek I’m pretty happy with the bike. The geometry of the frame itself is a little curious, but it seems to fit me well. Specifically, it has a fairly slack seat tube, tall head tube, tall seat tube, and rather high top tube. This means I have minimal stand-over (akin to the Titus Racer X 29er) and the saddle has to sit quite-forward on the rails, but it seems to work and feels good when pedaling. It also results in the controls (dropper lever and shifter) hitting the top tube when turning the bars, so I’ve had to pad the top tube with some 3M mastic tape.

I suspect this frame would work well for the sort of long-leg/short-torso rider who often finds themselves on a custom frame, but it also works well for me. It also means that a slim top tube bag like the Relevate Designs Tangle would fit along with bottles. Or, it could have been placed almost 1.5″ lower (with a shorter head tube and seat tube) and still been good… But I’m not a frame designer, so I suspect there’s some good but unknown-to-me reason why this was done.

Coming in at 28 pounds it’s not a light bike, but being a cheaper steel frame, 120mm fork with 34mm stanchions, 2.4″ wide quite-knobby tires, dropper post, and aluminum rims I’m okay with it. Going to some carbon bits (eg: bar) I could drop some weight, but I don’t think it’ll matter much, as this is roughly the same weight as the Camber, which I love riding.

Part details are below, and photos of the build showing everything from parts to checking clearances, can be found here: All-City Electric Queen.

Frame: All-City Electric Queen (Large, Lavender/Lime w/ Splatter)
Fork: Fox 34 Step-Cast (Factory, FIT4 damper, Black Upper Tube Finish, 120mm, 51mm offset, 15QR)
Headset: Cane Creek SlamSet (Black, ZS44/EC44/40)
Crankset: SRAM X1 1400 GXP
Bottom Bracket: SRAM GXP (Black)
Chainring: SRAM X-SYNC 2 (32t, steel, Direct Mount, 3mm / Boost)
Derailleur: SRAM GX Eagle
Shifter: SRAM GX Eagle
Shift Cables / Housing: Shimano Bulk
Cassette: SRAM XG-1275
Brakes: Shimano SLX M7100 (Levers: BL-M7100, Calipers: BR-7100)
Front Rotor: SM-RT86-L (203mm)
Rear Rotor: SM-RT86-M (180mm)
Front Brake Adapter: SM-MA-F203P/P (160mm Post to 203mm Post)
Rear Brake Adapter: Shimano SM-MA-R180P/S (IS to 180mm Post)
Stem: Specialized Stout XC, (+6°, 75mm)
Bar: Salsa Salt Flat (750mm)
Wheels: Industry Nine Trail S Hydra 28H (29″)†
Tires: Maxxis Rekon (29 x 2.4″, 3C/EXO/TR)
Seatpost: Fox Transfer Performance Elite (2020, Black, 125mm, 30.9mm, Internal)
Dropper Lever: Wolf Tooth ReMote Light Action (Black, 22.2mm Clamp)
Seatpost Collar: Salsa Lip-Lock (Black, 33.3mm)
Saddle: Specialized Power Expert (143mm)
Pedals: Crank Brothers Eggbeater 3 (Blue, from El Mariachi Ti)
Grips: ESI Extra Chunky (Black)
Bottle Cages: Specialized Zee Cage II (Black Gloss, 1x Left, 1x Right)
Computer: Garmin Edge 530, Garmin Speed and Cadence Sensors (v1)
Bell: Mirracycle Original Incredibell (Black)
Derailleur Hanger: QBP 687
Frame Protection Tape: 3M 2228, McMaster-Carr UHMW PE

Rigid Fork: All-City Electric Queen (Large, Lavender/Lime w/ Splatter)
Crown Race: Cane Creek BAA0009S (40-Series 52/30mm Steel)
Front Brake Adapter: Shimano SM-MA-F203P/S (IS to 203mm Post)

† Per I9, these are built with Sapim Race Straight-Pull spokes, 303mm all around. (For 29er. If these were 27.5 wheels they’d be 285mm.)

Comments closed

High Resolution Strava Global Heatmap in JOSM

Published May 24, 2020

Mapping trails in OpenStreetMap (OSM), using JOSM, is done by overlaying one or more data sources, then hand-drawing the trail locations using this background data as a guide. While not a default in JOSM, with a little know-how and a paid Strava Subscription the Strava Global Heatmap data for this can be used as well.

While there’s a fair bit of info about doing this scattered across some JOSM tickets (eg: #16100), this post is to document how I make this work for me by creating a custom Imagery provider. Because there isn’t an official (and robust) plugin to authenticate JOSM against Strava it’s a little tricky, but overall isn’t bad.

First, you need to sign into Strava, then go to the Global Heatmap and get the values of the three cookies CloudFront-Key-Pair-ID, CloudFront-Policy, and CloudFront-Signature. These values are essentially saved credentials, and by using them in a custom Imagery provider URL, JOSM will use these to access the Global Heatmap data via your account and make it available as an Imagery layer.

The easiest way to get these cookie values is to use the Developer Tools built into your web browser:

  1. Launch your web browser, either Firefox, Chrome, or Edge.
  2. Go to the Strava Global Heatmap, sign in if needed (check the Remember me checkbox), and be sure it’s properly displaying high resolution data.
  3. Load the developer tools for your browser to show cookie details:
    • Firefox: Click the Hamburger Menu → Web Developer → Storage Inspector (or press Shift-F9), expand Cookies and click https://www.strava.com.
    • Chrome: Click the Three Dots Menu → More tools → Developer tools (or press Ctrl-Shift-I), select the Application tab, expand Cookies, then click https://www.strava.com.
    • Edge: Click the Three Dots Menu → More tools → Developer Tools (or press F12), select the Storage tab, expand Cookies, then click heatmap.
  4. Get the value of three cookies, CloudFront-Key-Pair-ID, CloudFront-Policy, and CloudFront-Signature, and make note of them (in Notepad or such). You’ll need them to build the custom URL. Here’s examples of what the three cookies will look like:
    • CloudFront-Key-Pair-ID: VB4DYFIQ7PHQZMAGVMOI
    • CloudFront-Policy: FTUGYikErADOqGmqEnPTaaZ3LCE2QbOJSSsmPD9vdY0hfjSWWHZBgnRai1aQcvUe
      2YtgkG3KdP8IencRaGrS3I5bHAeabQ4I776T7UipDPFzNVh9PBIRvb3DjDqVfssp
      mBA5IrxacQGfB2YmKwegp9OTBfUFkvtEQOlzeXfqeuc34cp408GYXSvp2DsnxKFh
      ofbwygraae3VZqnZxp3vBpz3lBoyPnBEN1djJkxpU1fF_
    • CloudFront-Signature: lrMJG0L6zJilpSdGEnWug0zskcMDD5VVGdSRu~P4OOVp5z8iCMBRBO040wxunEQQ
      p0wVHdaCaTFAVqiM0jC0AApW7HigXjx57nsoEVslOzRuoX9S3g-
      FCePWYNPyZcJ95~6SoGdNRHz-
      b-ZZxk4ERhjzXCNIEE8Gt5uImitcPN3HRlhXexS39g30~5OEv~4FHzXfOOIhQ5X5
      cGUD0SpdYviBE~vkAJCk3TbQMXxlJbyX4OOL5xvVPv5WjzMKTuvEk6Z9m6iFBbZc
      8ov4qS60th0tZ1RM4pNMoIWKoqPS3~UPGISe6Xa8Kq-
      tQPxHlfGciI5uhoTp4b7lPussh52QaN__

(Note: These are not real values, you’ll need to look up your own.)

Next is to add a custom imagery provider for the Strava Global Heatmap to JOSM using a URL that you build with these cookie values:

  1. Edit → Preferences… (F9) → Imagery preferences
  2. Create a new TMS string by replacing the [] sections in the following string with the cookie values from above, without the []s:
    • tms[15]:https://heatmap-external-{switch:a,b,c}.strava.com/tiles-auth/all/hot/{zoom}/{x}/{y}.png?Key-Pair-Id=[CloudFront-Key-Pair-ID]&Policy=[CloudFront-Policy]&Signature=[CloudFront-Signature]
  3. Add a new TMS entry by clicking the +TMS button.
  4. Paste the string from Step 2 into Box 4, Edit generated TMS URL (optional), and then give it a name in Box 5, such as Strava Global Heatmap High-Res.
  5. Click OK twice to return to the main JOSM window, and then try to use this layer over some existing data, just as you would aerial imagery.

If it doesn’t work, double-check your URL and be sure the entered cookie values are right, including the underscores after CloudFront-Policy and CloudFront-Signature. Also be sure you haven’t logged out of Strava, as this will expire the cookies. (If you make changes to the URL in JOSM you will need to delete the existing imagery layer and then re-add it to have the new URL used.)

These cookie values, in particular CloudFront-Signature, will occasionally change as cookies are expired and reset or if you log out of Strava. If things were working and then stopped, you may need to get new cookie values from your browser and update the TMS strings.

By default the TMS URL we started with shows the default heatmap, for all activity types, with Hot coloring. Depending on what other data you are working with it may be useful to show just Ride or Run data, perhaps in different colors. In the TMS URL, the first part after tiles-auth is the type of data, and the second is the color. By using this format, replacing [data] and [color], you can create additional heatmap layers:

tms[15]:https://heatmap-external-{switch:a,b,c}.strava.com/tiles-auth/[data]/[color]/{zoom}/{x}/{y}.png?Key-Pair-Id=[CloudFront-Key-Pair-ID]&Policy=[CloudFront-Policy]&Signature=[CloudFront-Signature]

Valid values for type of data: all, ride, run, water, winter.

Valid values for heatmap color: hot, blue, purple, gray, bluered.

Creating multiple layers, riding-only and running-only, of different colors was extremely useful when mapping Avalanche Preserve Recreation Area in Boyne City as cyclists tend to stick to the mountain bike trails and runners to the hiking trails. While I had orthorectified maps from both the city and TOMMBA, the separate riding and running heatmaps made the routes much clearer. In the example image above (link) I have the Ride data as color Hot and the Run data as color Blue; perfect for illustrating the mountain bike vs. hiking trails.

If you’d like to see this area as an example load the are around changeset 85690641 or a bounding box with the following values and see for yourself:

  • Max Lat: 45.2077126
  • Min Lat: 45.1878747
  • Min Lon: -85.0339222
  • Max Lon: -84.9832392
Comments closed

BorgBackup Repository on Synology DSM 6.2.2

Published January 8, 2020

(UPDATE: With the release of Synology DSM 7.0 this setup will break. It’s easy to fix and I’ve updated added this post describing how to make this system work under the new version.)

Lately I’ve become enamored with BorgBackup (Borg) for backups of remote *NIX servers, so after acquiring a Synology DS1019+ for home I wanted to make it the destination repository for Borg-based backups of nuxx.net. While setting up Borg is usually quite straightforward (a package or stand-alone binary), it’s not so cut and dry on the Synology DiskStation Manager (DSM); the OS which runs on the DS1019+ and most other Synology NAS’.

What follows here are the steps I used to make and the reason for each step. In the end it was fairly simple, but a few of the steps are obtuse and only relevant to DSM.

These steps were written for DSM 6.2.2; I have not checked to see if it applies to other versions. Also, I leave out all details of setting up public key authentication for SSH as this is thoroughly documented elsewhere.

  1. Enable “User Home Service” via Control Panel → User → Advanced → User Home → Enable user home service: This creates a home directory for each user on the machine and thus a place to store .ssh/authorized_keys for the backup user account.
  2. Create a backup user account and make it part of the administrators group: Accounts must be part of administrators in order to log in via SSH. Starting with DSM 6.2.2 non-admin users do not have SSH access.
  3. Change the permissions on the backup user’s home directory to 755: By default users’ home directories have an ACL applied which has too broad of permissions and SSH will refuse to use the key, instead prompting for a password. Home directories are located under /var/services/homes and this can be set via chmod 755 /var/services/homes/backupuser. (See this thread for details.)
  4. Put ~/.ssh/authorized_keys, containing the remote user’s public key, in place under the backup user’s home directory and ensure that the file is set to 700: If permissions are too open, sshd will refuse to use the key.
  5. Test that you can log in remotely with ssh and public key authentication.
  6. Place the borg-linux64 binary (named borg) in the user’s home directory and confirm that it’s executable: Binaries available here.
  7. Create a directory on the NAS to be used the backup destination and give the backup user read and write permissions.
  8. Modify the backup user’s ~/.ssh/authorized_keys to prevent remote interactive logins and restrict how borg is run: This is optional, but a good idea.

    In this example only the borg serve command (the borg repository server) can be run remotely, is restricted to 120GB of disk, in a repository on DSM under the backup directory of /volume2/Backups/borg, and from remote IP of 192.168.0.23:

    command="/var/services/homes/backupuser/borg serve --storage-quota 120G --restrict-to-repository /volume2/Backups/borg",restrict,from="192.168.0.23" ssh-rsa AAAA[...restofkeygoeshere...] remoteuser@remoteserver.example.com

Please note, there are a number of articles about enabling public key authentication for SSH on DSM which mention uncommenting and setting PubkeyAuthentication yes and AuthorizedKeysFile .ssh/authorized_keys in /etc/ssh/sshd_config and restarting sshd. I did not need to do this. The settings, as commented out, are the defaults and thus already set that way (see sshd_config(5) for details).

At this point DSM should allow a remote user, authenticating with a public key and restricted to a particular source IP address, to use the Synology NAS as a BorgBackup repository. For more information about automating backups check out this article about how I use borg for backing up nuxx.net, including a wrapper script that can be run automatically via cron.

Comments closed

Using borg for backing up nuxx.net

Published November 10, 2019

For 10+ years I’ve been backing up the server hosting this website using a combination of rsync and some homegrown scripts to rotate out old backups. This was covered in Time Machine for… FreeBSD? and while it works well and has saved me on some occasions it had downsides.

Notably, the use of rsync’s --link-dest requires a destination filesystem that supports shared inodes, limiting me to Linux/BSD/macOS for backup storage. Also, while the rotation was decent, it wasn’t particularly robust nor fast. Because of how I used rsync to maintain ownership with --numeric-ids it also required more control on the destination than I preferred. And, the backups couldn’t easily be moved between destinations.

In the years since, a few backup packages have come out for doing similar remote backups over ssh, and this weekend I settled on BorgBackup (borg) automated with borgbackup.sh.

Borg and a single wrapper script (that I based on dailybackup.sh) executed from cron simplifies all of this; handling compression, encryption, deduplication, nightly backups, ssh tunneling, and pruning of old backups. More importantly, it removes the dependence on shared inodes for deduplication, eliminating the Linux/BSD/macOS destination requirement. This means my destination could be anything from a Windows box running OpenSSH to a NAS to rsync.net.

Here’s a brief overview of how I have it set up:

  1. Install BorgBackup on both the source and destination (currently Linux nuxx.net server and a Mac at home).
  2. Put a copy of borgbackup.sh on the source, make it executable, and restrict it just to your backup user. Edit the variables near the top as needed for your install, and the paths to back up in the create section.
  3. Create an account on the destination which accepts remote SSH connections via key auth from the backup user on the source.
  4. Restrict the remote ssh connection to running only the borg serve command, to a particular repository, and give it a storage quota:
    restrict,command="borg serve --restrict-to-repository /Volumes/Rusty/borg/servername.nuxx.net” from=“10.0.0.10” ssh-rsa AAAAB3[…snipped…] root@servername.nuxx.net
  5. On the remote server, turn off history (to keep the passphrase from ending up in your history; in bash: set +o history), set the BORG_REPO, BORG_PASSPHRASE, BORG_REMOTE_PATH, and BORG_RSH variables from the top of the script to test what the script defines:
    export BORG_REPO='ssh://user@server.example.com:220/path/to/repo/location/'
    export BORG_PASSPHRASE='PASSPHRASEGOESHERE'
    export BORG_REMOTE_PATH=/usr/local/bin/borg
    export BORG_RSH='ssh -oBatchMode=yes'
  6. From the source, initialize the repo on the destination (this uses the environment variables): borg init --encryption=repokey-blake2
  7. Perform the first backup as a test, backing up just a smallish test directory, such as /var/log: borg create --stats --list ::{hostname}-{now:%Y-%m-%dT%H:%M:%S} /var/log
  8. Remotely list the backups in the repo: borg list
  9. Remotely list the files in the backup: borg list ::backupname
  10. Test restoring a file: borg extract ::backupname var/log/syslog
  11. Back up the encryption key for the repo, just in case: borg key export ssh://user@server.example.com:220/path/to/repo/location/
  12. Define things to exclude in /etc/borg_exclude on the source per the patterns. In my I use shell-style patterns to exclude some cache directories:
    sh:**/cache/*
    sh:**/locks/*
    sh:**/tmp/*
  13. Create the log directory, in the example as /var/log/borg.
  14. Run the backup script. Tail the log file to watch it run, see that it creates a new archive, prunes old backups as needed, etc.
  15. Once this all works, put it in crontab to run every night.
  16. Enjoy your new simpler backups! And read the excellent Borg Documentation on each command to see just how easy restores can be.
Comments closed

Using OsmAnd Offline Maps during Remote Rides

Published October 6, 2019

Many of the popular, long cycling events — or even just solo rides — require a GPS so that one can follow a route in order to navigate. While most of us have bike computers that will show the route, most folks don’t have a GPS with the kind of detailed base maps that become useful for detours, emergencies, or just as an additional tool… Much less something that works when outside of cellular coverage areas.

For this, I recommend you look at offline maps using OsmAnd.

I have (and love) the Garmin Edge 130 as it’s perfect for my riding and training. It’s been great for displaying recording data and following routes from the unmarked 100 mile versions of The Crusher and Barry-Roubaix, to augmenting signage in Marji Gesick and following a route through remote parts of Batchawana River Provincial Park. But occasionally I need a bit more; something to help find a detour around a flooded road, a quick way back to the car in a rainstorm, or just a sanity-check of which branch to take at a fork on the road.

Sure, I could buy a super-high-end bike computer that features detailed base maps, but I rarely need these and don’t want the physically large computer on my bike. So I use my phone, running OsmAnd, loaded with offline maps that display without requiring a data connection. Coupled with putting one’s phone in Airplane mode (turns off the cellular and WiFi radios and saves battery — yet GPS still works) a modern phone will typically get days of battery life while sleeping, with maps and a route quick to access.

Whenever I actually need to see this map I simply pull it out of my pocket or bag, wake it up, get what I need from the map, and carry on. No screwing around with a small touch screen and funky zooming on a Garmin or Wahoo. (I believe that, to date, neither Garmin nor Wahoo have made map perusal as straightforward as it is on a basic mobile phone.)

This is also a great way to get maps on your phone when traveling somewhere costly for cellular data, like during a day-trip to Canada.

Whether you’re doing Marji Gesick or The Crusher or just following a route found on a cycling club’s website, I recommend augmenting your navigation with offline maps in OsmAnd by doing the following:

  1. Visit the OsmAnd site and follow the links to your phone’s store to install the software.
  2. Get a copy of the GPX file on your phone; for Android phones this is often as simple as visiting the download website and downloading the file.
  3. In OsmAnd‘s menu, pick Download Maps and download maps for the regions you want. I personally have Standard Maps, Contour Lines, and Hillshades for Michigan installed all the time, and then if traveling will install for other states/provinces/countries.
  4. Once the offline maps are installed, pick Configure map from the menu, then Map source, and ensure Offline vector maps are selected. The default, OsmAnd (online tiles), is an online map source that requires data.
  5. To display a particular route, pick GPX files…, Add More…, and then browse your phone to add the downloaded GPX file. Ensure this file is enabled (checked), and then go back to the map screen.
  6. You’ll now see the route overlaid on offline maps, easy to zoom in and out of and find your way around. Try turning on Airplane Mode and see that it all still works.

Once I have everything set up, here’s some tips about how I actually use OsmAnd while out on my bike:

  • If in an area with minimal or no cellular service, put the phone in airplane mode so the battery doesn’t get used up while it searches for a signal.
  • Turn off all lock features on my phone (fingerprint / pattern / PIN unlock), so I can wake it up with just a press of the power button.
  • You can use this at the same time as Strava, and Strava generally works in Airplane mode as well.
  • If I’m at a spot where I’m not quite sure which way to turn (based on the single line on my Edge 130), use the map to figure out which branch to turn on. Even the remotest of fire roads and two track are often mapped.
  • If I think I might have missed a route sign (say, in Marji Gesick) I’ll use OsmAnd to see if I’m still on route, and to get back if needed.
  • Look for other nearby roads or trails if I need to detour or shortcut back. I had to do this yesterday during the Founders Fall Fondo as part of the 62 mile Barry-Roubaix route was flooded.
  • Many small, old, sometimes abandoned, yet somewhat navigable roads are still listed in the OpenStreetMap (OSM) data that OsmAnd uses. While these aren’t great for cars and won’t show up on things like Google or Apple Maps, they are often quite useful when on a bike. Look to these for exploring / finding new routes.
  • Many trails open to mountain bikes are included in the OSM data. This includes all CRAMBA trails and most trails in the NTN and RAMBA areas. (Note: Trailforks is a better source for discovering MTB trails themselves.)
  • Periodically update maps to get new OSM data. As roads and trails change, maps get updated.
  • Pay For OsmAnd+ Live ($5.99/year) to get access to more-frequent, automatic map updates and support the open mapping community. 

While this is just a basic overview of using OsmAnd and how I use it when cycling; it does a whole lot more including recording routes, navigation, editing map data, sharing location between contacts, and more. Give the online Help and Features a look for more information.

 

Comments closed

Torklift EcoHitch 2″ Hitch for my 2015 Outback

Published January 7, 2019
IMG_0249.jpg

Back in 2015 when I purchased a Subaru Outback, I ordered it with a OE hitch. At the time it was the best (and thanks to IMBA discount the cheapest) option for mounting a bike rack on the back of the car. Due to load limits on what the vehicle can tow there was only a 1-1/4″ receiver available, and while this fit the 1Up USA rack previously used on my Civic it just didn’t provide the stability of a 2″ receiver.

Fast forward to late 2018 and, out of the blue, I was contacted by Torklift, via Reddit, where they offered to send me their Torklift EcoHitch 2″ receiver hitch in exchange for providing feedback to their designers and engineers. After some email back and forth this was all set up, and a few days ago I received the rack: Torklift 2015-2018 Subaru Outback EcoHitch, part number x7266.

Beyond the 2″ receiver, the biggest difference between the EcoHitch and OE hitches is how they mount. While the EcoHitch sandwiches between the bumper beam and the chassis, held in place by the bumper’s eight fasteners (photo 1, 2), the OE hitch both replaces the bumper beam and mounts inside of the frame rails (photo). While it appears that the OE hitch may be a bit more resistant to very high tongue weights, the Torklift seems plenty solid and is made of thicker material, so I’ve got no concerns with the differences. The other significant difference is that the Torklift EcoHitch doesn’t come with a wiring harness, but only ever having bike racks on my car this doesn’t really affect my usage. When the OE hitch was installed on my Outback this included a wiring harness along with a nifty mount to hold it next to the receiver (photo). I considered adapting this mount to the EcoHitch, but instead took some inspiration from Torklift Central’s (frankly overpriced) Eclipse 4-Flat Plug and fit a neodymium magnet to the cable (photo), then stuck the magnet to the back side of the receiver.

Needing to remove the OE hitch my install was bound to be a bit more complicated than someone who is putting the EcoHitch on a new vehicle, but even this ended up being pretty straightforward.

Whole rack, rear view, with installation instructions and parts bundled to the center.

Whole rack, rear view, with installation instructions and parts bundled to the center.

Unboxing the rack revealed a well-packed (photo), nicely powder coated (photo), solid rack that ships with an instruction manual, the eight nuts (for attaching it to the bumper mount bolts), and a piece of rubber trim gasket to cover the cut edges on the underside of the bumper cover (photo).

Besides the printed instructions (which I wish Torklift posted as a PDF on their website), there are two instructional videos (1, 2) which show the installation on everything from 2015 to 2018 models of the vehicle. These do a good job of showing the basics of the installation, but they gloss over what I think is the most difficult part; removing the bumper cover:

  1. Removing the bumper cover itself works best if you push up on the sides from the bottom, then pull out the at top. Going off of just Torklift’s instruction I first thought to pull straight outward, but this wasn’t working well and — per Subaru — can risk breaking retention clips. Thankfully I have access to Subaru’s service manual which described pushing up from the bottom, and with this technique the cover came off easily.
  2. My vehicle has splash guards installed, which had to be removed before all the bumper clips could be accessed. These guards came off via six screws and four push-lock plastic clips, but I had to use a right-angle screwdriver to fit around the tires. (There is a note in the Torklift instructions mentioning these may need to be removed.)
  3. Three out of the myriad push lock clips used to hold the bumper and splash guards on broke during removal. This is likely due to my Outback being nearly four years old. Thankfully replacements are readily available online and at dealerships and can easily be identified via Subaru Parts Online.

For this installation Torklift instructs that the hitch be fitted, then the bumper beam reinstalled. However, during the install of the OE hitch, the bumper beam and foam energy absorber is removed and discarded. Since my Outback came with the OE hitch installed, I didn’t have these pieces. After consulting with both Torklift and some auto engineer friends we concluded that the Torklift hitch will be fine without the bumper beam.

Despite this, I contacted my local dealer (Sellers Subaru) and to my surprise they gave me discarded/scrap bumper beam from an Outback where they’d recently installed a hitch. It’s maroon, but being hidden the color doesn’t matter. With the purchase of a replacement foam energy absorber ($79.95 MSRP) I was all set part-wise to remove the OE hitch and install the EcoHitch to spec. While this likely wasn’t needed, it makes me feel better about the overall install. (Photos 1, 2)

Removing the OE hitch was a bit of a hassle, as getting to two of the hitch bolts required removing the muffler and heat shield. (Another plus for the EcoHitch.) Thankfully none of the bolts were seized and removal this went smoothly. Before long I also had the OE rack completely removed, exhaust re-fitted, and it was time to install the EcoHitch.

Foam cushion over the bumper beam, and hitch installed.

Foam cushion over the bumper beam, and hitch installed.

At this point — just after bumper cover removal on a normal install — installation is very straightforward: folding a thin metal bumper cover mounting tab up against the body, bolting the hitch and bumper back on, torquing some nuts, and cutting a notch out of the bumper cover. After this it’s just a matter of putting the the bumper cover, tail lights, splash guards, and some plastic covers back in place.

From the OE hitch I already had a notch cut in my bumper cover, but it was pretty easy to cut the wider notch required by the EcoHitch. The OE hitch uses a longer notch than the EcoHitch, and thankfully Torklift provides enough trim to cover the extra-large notch present after a bumper’s been cut for both (photo). The cutting itself went easily, as the plastic is pretty soft and can either be scored and folded or cut with anything from a rotary cutter to a jigsaw.

During installation one of the bumper cover mounting tabs is folded out of the way and the mounting hole cut out of the bumper cover during the notching, which eliminates one of the points where the bumper mounts to the chassis. While initially concerning, this part of the cover rests on the top of the receiver after everything’s put back together, so this lack of retention point isn’t a problem.

With everything put back together I’m currently quite happy. The TorkLift EcoHitch looks good and sits nice and close to the underside of the bumper; I don’t think it could go much higher. While the OE hitch had 14.5″ of ground clearance, this drops to 13.5″ with the EcoHitch, an acceptable change for adding 0.75″ to the receiver’s height. This hitch is a great choice for either new installs on an Outback or retrofitting an OE hitch to a 2″ receiver.

I particularly like how the powder coating has a thick, textured look to it, and the end of the receiver has a nice TORKLIFT logo. The whole unit looks like it’ll hold up nicely to Michigan winters (and road salt). Now to wait for a new, matching black 1UP USA 2″ Super Duty Single rack to arrive!

Click here or on the large photo above to see a complete album of photos from the install.

Leave a Comment

Pi-hole and NBC Sports Gold

Published December 23, 2018

I use Pi-hole to block some ads on my home network, but I also pay for a NBC Sports Gold Cycling Pass to easily watch cycling events. Unfortunately, the default block list for Pi-hole will keep NBC Sports Gold from working, breaking it in a somewhat odd way: pages load, but the list of events doesn’t display.

So, how do you make it work? Easy! Whitelist these three sites:

  • experience.tinypass.com
  • geo.nbcsports.com
  • nbc.demdex.net
Leave a Comment