Category: computers

At home I use Pi-Hole running via Docker on a Synology DS1019+ (details here). Every once in a while I update this, but it seems to be long enough between updates that I forget the exact steps, so I’m documenting them here.

This is the simplest update process I know of. It keeps the configuration the same but updates the container itself. Because the volumes for persistent storage are not changed and the custom environment variables stay the same, all settings are preserved.

1. Log into DSM.
2. Launch Docker.
3. In Docker, update the image:
   • Registry
   • Search for pihole
   • Select pihole/pihole (ensure it is the official image)
   • Click Download
4. Stop and reset the container:
   • Container
   • Select your Pi-hole container.
   • Stop
   • Reset
   • Start
5. Wait a few minutes, then log back in to Pi-Hole (eg: http://pi.hole:8081/admin/)
6. At the bottom of the page observe updated versions. Check here for the latest version of the Docker image, for comparison.

Note: This process presumes you have configured volumes mounted to /etc/dnsmasq.d and /etc/pihole, and have set environment variables INTERFACE, WEB_PORT, WEBPASSWORD, and TZ. Without these your results may vary.

There are many times when one wants to see which process is responsible for network traffic on a endpoint. While there are ways to look at which process has an open socket or whatnot, this doesn’t help with UDP, and it’s often quite useful to simply do a recording then later see which process is responsible for creating a packet.

On Windows this is very easy using Pktmon / netsh trace / Network Monitor to do a capture, but on macOS it’s not that straightforward.

Using tcpdump with the -k argument one can print process name and PID to the console. In this example I’m filtering on just the host 8.8.8.8 (Google Public DNS) then running dig in another window to look up dingleberrypie.com:

c0nsumer@myopia ~ % sudo tcpdump -k -i en0 host 8.8.8.8
Password:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:17:58.210959 pid dig.76987 svc BE IP myopia.home.nuxx.net.56248 > dns.google.domain: 10294+ [1au] A? dingleberrypie.com. (47)
09:17:58.292541 IP dns.google.domain > myopia.home.nuxx.net.56248: 10294 1/0/1 A 96.126.107.52 (63)

The bold line is the packet going out to 8.8.8.8, and the pid dig.76987 portion shows that it’s from a dig process, which had process ID 76987.

Unfortunately, due to the non-standard way Apple writes this metadata into files, it’s not viewable in Wireshark. This would be very nice, as it’d make parsing large captures much easier.

For now it seems the best way to do this is to record the capture to a file, then feed it back into tcpdump. Recording this requires using the pktap pseudo interface (see the tcpdump man page about the --interface argument) to ensure this data is saved into the file. The same capture above, writing to a file called out.cap, would be as follows:

sudo tcpdump -i pktap,en0 host 8.8.8.8 -w out.cap

This can then be fed back into tcpdump for parsing/filtering/viewing:

c0nsumer@myopia ~ % tcpdump -k -r out.cap
reading from PCAP-NG file out.cap
09:27:10.964758 (en0, proc dig:77619, svc BE, out, so) IP myopia.home.nuxx.net.52983 > dns.google.domain: 56446+ [1au] A? dingleberrypie.com. (47)
09:27:11.018809 (en0, proc dig:77619, svc BE, in, so) IP dns.google.domain > myopia.home.nuxx.net.52983: 56446 1/0/1 A 96.126.107.52 (63)
c0nsumer@myopia ~ %

In this case it shows both the sending process and the process which received the packet.

When using this for more detailed analysis, I’ll use the macOS tcpdump to grab a very broad capture and then do first-pass filtering before bringing it into Wireshark for more detailed analysis. See the PACKET METADATA FILTER section of the tcpdump man page for details on how to filter on a PID, process name, etc. From the header of this section:

Use packet metadata filter expression to match packets against descriptive information about the packet: interface, process, service type or direction.

About a week back I did a round of updates at home, including updating the Pi-hole container (running in Docker on a Synology DS1019+) to the latest version, v4.2.2. Not long after this I noticed that backups to Backblaze, via Arq running on my main Mac, were stuck with a Caching existing backup metadata (this may take a while) message.

Since it said it might take a while I gave it a few days, but after a week it was likely something was wrong. Turns out it wasn’t caused by any of my updates, but instead by two versions of the the block list HOSTS (v3.5.3 and v3.6.0) — the default block list in Pi-hole — in turn caused by the Polish block list KAD.

How’d I figure it out? Here goes:

First, a wee bit of digging led to this Reddit thread on /r/Arqbackup, and a quick look at Pi-hole showed that yes, f000.backblazeb2.com is being blocked over and over.

Whitelisting this site allowed backups resume working. But… why?

I then disabled the whitelist entry and updated gravity in Pi-hole (pulling down and compiling a new copy of the blocklists) and everything kept working. So, this seems like a block list might have been the source of the problem.

I only use two block lists, one the Pi-hole default and the other from the COVID-19 Cyber Thread Coalition. Taking a quick look through the current versions (1 · 2) didn’t show anything blocking this site as of this morning, which seemed rational as the blocklist update fixed things. Local DNS for this client is via Pi-hole, which in turn points to my firewall, which is running Unbound to handle all resolution itself. So, it shouldn’t have been caused by a DNS provider blocking things.

Pi-hole automatically updates gravity every Sunday early in the morning, which would about correlate with the Arq problems starting. So maybe this is it? With the last Gravity updates happening on 2021-Apr-04 and 2021-Mar-28 we’ve got a window to look for f000.backblazeb2.com in blocklists.

The COVID-19 Cyber Threat Coalition domain blocklist was updated this morning, and doesn’t have any obvious version control, so I skipped over this one for now. The second, the Pi-hole default HOSTS, is hosted in GitHub and has regular releases. So let’s look through there…

Grabbing the last four, v3.5.2, v3.5.3, v3.6.0, and v3.6.1 spanned the last 18 days, which should cover the window during which this broke. A quick unzip and grep showed f000.backblazeb2.com and www.f000.backblazeb2.com in the fakenews, gambling + social, gambling + porn, and social categories in versions 3.5.3 and 3.6.0, but not anything before nor after.

There we go; the reason for the block and it’s all within the observed timeframe. This isn’t a hostname one would normally want to block, as it’s part of BackBlaze’s CDN (PDF). Sounds like an overzealous addition to a blocklist got sucked up into the HOSTS list.

Looking further through the grep output, this was part of the .../KADhosts/hosts file from the KAD list. It turns out that f000.backblazeb2.com was added to the KAD list on 2021-Mar-26 and then removed on 2021-Apr-01. HOSTS pulled from KAD for v3.5.3 on 2021-Mar-28 and v3.6.0 2021-Mar-31, which caused it to inherit the block in those versions.

Quite an interesting chain, eh? A Polish ad blocking group makes a change that ends up in the default list for one of the most common DIY adblockers, which in turn breaks access to a fairly common CDN, in turn breaking data backups. It’s dependencies all the way down…

It’s now fixed, and everything would have resolved itself had I waited until Sunday, but at least now I know why.

Consider the following situation: You have a web app from a vendor and during a security scan it crashes. The web app is running over HTTPS with your certificates, but neither the scanning tool nor web app offer sufficient logging to see exactly which request caused the crash.

Because you can’t decrypt HTTPS without access to a client key log file (or making a bunch of TLS changes), and the client is a security scanning tool, Wireshark is not an option to see the triggering request. Fiddler is also likely out, as that’d require the security scanner to trust a new root cert. So what can you do? Stick something else in the way to proxy the connection, logging all the requests!

Having access to the private certificates for the server this is quite easy: set up nginx as a proxy. The only wrinkle is that getting access to all of the request headers requires Lua, so you’ll need to ensure your nginx install supports this. On macOS this was easy using Homebrew to install nginx from denji’s GitHub repository (the default nginx doesn’t support Lua):

brew tap denji/nginx
brew install nginx-full --with-lua-module --with-set-misc-module

This configuration uses the web app’s certificates in nginx to proxy requests it receives to your main site, logging the client IP, request, headers, body, and request status to intercept.log. Requests are broken out by line to make for easy visual reading. You may wish to move this all on to one line to make parsing easy:

events {
}

http {
    log_format custom 'Time: $time_local'
                      '
'
                      'Remote Addr: $remote_addr'
                      '
'
                      'Request: $request'
                      '
'
                      'Request Headers: $request_headers'
                      '
'
                      'Body: $request_body'
                      '
'
                      'Status: $status'
                      '
'
                      '-----';

    server {
        listen 443 ssl;
        server_name example.com;
        access_log /path/to/intercept.log custom;
        ssl_certificate /path/to/cert.pem;
        ssl_certificate_key /path/to/privkey.pem;

        location / {
            proxy_pass https://example.com;
            proxy_set_header Accept-Encoding ''; 
            set_by_lua_block $request_headers {
                local h = ngx.req.get_headers()
                local request_headers_all = ""
                for k, v in pairs(h) do
                    request_headers_all = request_headers_all .. ""..k..": "..v..";"
                end
                return request_headers_all
            }
        }
    }
}

To put this in place, ensure that requests from the scanner go to nginx instead of the web app and then nginx will forward and log the requests. There are a few ways you could do this:

• Run nginx on the same server as the web app, move the web app to listen to another port for HTTPS, and set proxy_pass to the other port: proxy_pass https://example.com:4430
• Run nginx on a new server, change the DNS records for the site to point to the new server, and point nginx to the old server by IP: proxy_pass https://192.168.10.10
• If the scanner tool’s name resolution can be adjusted, such as via a HOSTS file or custom configuration, point it to the nginx proxy for the site name.

To test you can use a web browser on a client computer and a HOSTS file to point the original hostname nginx. To get the screenshot above I ran nginx on iMac running macOS, then in a Windows VM I changed the HOSTS file to map nuxx.net to the iMac’s IP. Firefox on the Windows VM then sent requests for nuxx.net to nginx on macOS which logged and proxied the requests out to the real nuxx.net.

I was recently asked to look at a family member’s iPad because it was no longer sending email. Turns out that it had been set up to use an additional email account that steals copies of all their outgoing mail. Unfortunately, they didn’t notice until the attacker’s system stopped working and the iPad started showing an error message. Besides the irritating (or worse) spam they saw, their stolen emails could have been used for anything from spear phishing to accessing one’s online accounts, impersonating them, phishing others, delivering targeted spam, fake news / propaganda, etc.

So how did this get set up?

Apparently at some point this person installed the My Accurate Forecast app [1]. Included in this app was a Profile — or a set of settings for Apple devices — that added a second email account with address lazaroburst@my.minbox.email. This account was also set as the outgoing server for their Hotmail (Outlook.com) account.

This person would then have seen all messages in this account, with notifications just like their normal Hotmail email. Worse, everything they sent, from any email account, went to the attacker first. As it’s a separate email account, all the normal spam and malware protections from a normal email provider don’t apply… It’s a firehose of junk straight to their mailbox, with outgoing mail theft frosting on top.

This is bad because not only does it end up with them getting more spam, it allows the attacker to know exactly what they sent and to whom, and to modify those messages before delivering them to the intended recipients.

I think this was likely generated based on geolocated advertising, but it’s possible this individual was specifically targeted. The signed Profile had a name of “WEATHER ALERTS” a description of “Tap ‘Install’ above to get your local radar forecasts and weather alerts in 48062”, showing its intent to deceive; trying to make the normal Profile installation security alert — which is supposed to warn the user of a change to important settings — look like part of an application install.

I’m unsure when this first got installed, but judging by the the Profile signing certificate expiring on December 8, 2016 it was likely within a year or two prior. (Unfortunately I didn’t check the issuance date before deleting the profile.) The Profile which made these changes was signed by secure5g.com, an “advertising” company which has ties to minbox.email (the Unsubscribe link at the bottom of the page is a generic link to a minbox.email page).

A post from June 2018 on Medium, Unwanted Profiles Pop Up in iOS Devices, Inviting Spam and Malware, reports the same problem almost two and a half years ago. Curiously, the handful of other posts I read about this (ref: 1, 2) didn’t mention (or maybe didn’t notice) the outgoing server change? Perhaps because they only noticed before things broke, or maybe this iPad somehow ended up different? (It does seem that at least one other app: Daily Bible Verse, included similar email hijacking.)

Cleaning this up these settings was easy, just a matter of removing the malicious Profile, outgoing mail account, and setting the Hotmail account back to using the appropriate servers. But, who knows what damage was done with the theft of the sent mail and receipt of spammy stuff.

[1] The My Accurate Forecast website still shows screenshots of the app, but does not link to any app stores. It also no longer appears in the Apple App Store, implying that it’s been pulled out.

I purchased a Reolink E1 Zoom camera for occasional around the house use. It turns out that my SSID, Smart Meter Surveillance Network is too long for their setup app. While the standard is 32 octets (32 ASCII characters) — and my SSID is exactly this — some things, such as the Reolink app, only accept 31 characters. In this case it pulled the SSID from my phone (the network in use) and then truncated it. †

So, I set out to find a workaround, and I did.

During setup the Reolink app walks you through scanning a serial number QR code on the camera, prompts for the wireless network info, and then generates a QR code and displays it on the mobile device’s screen. The camera is then pointed at the screen, this QR code is read, and the camera configures its WiFi settings based on the code.

I figured that maybe if I generated a new QR code with the correct info I’d be able to configure the camera with a longer SSID and it turns out that worked.

After a couple minutes of generating codes I found the configuration QR code is text, formatted as follows, with #### as the last four characters of the camera’s serial number:

<QR><S>ssid</S><P>password</P><C>####</C></QR>

Using the first free online QR code generator I could find, I created a new QR code with containing the following text:

<QR><S>Smart Meter Surveillance Network</S><P>notmyrealpassword</P><C>M77L</C></QR>

I reset the camera, had it scan the new QR code, and it connected to the wireless network. It worked! The camera was now on the wireless network and I was able to connect to it in the app.

There did seem to be a bit of quirkyness in the app, possibly because of the long SSID. It’s working fine with the desktop app, so all is good. It’s also really nice to now have a way of reconfiguring the camera without having to install and use their app.

† The standard maximum for SSIDs is 32 octets, or 32 ASCII characters. It appears some companies treat this as 31 characters, reserving the 32nd for the string termination character. Sort-of makes me wonder how I’ve been able to use this one for so long… It was fine with my old Apple AirPorts and I’ve had it running this way for couple years on Ubiquiti UniFi. Although it looks like the UniFi v6 UI now refuses to save changes with this SSID, so I guess I’m going to have to change it…