Category: computers

Timeouts with PHP as FastCGI, phpBB + reCAPTCHA + DNSBL, Apache 2.2, mod_fcgi, and FastCgiExternalServer

Published April 16, 2012

Sunday evening after migrating the MMBA Forum to a new webserver I received email from a user claiming that they were unable to sign up for the forum, receiving an 500 Internal Server Error some time after clicking submit. The problem ended up being the signup page taking longer than expected to run and timing out and was resolved by increasing the timeout by adding -idle-timeout 60 to the FastCgiExternalServer line in the vhost’s config.

More specifically, I’d just moved from an older server running lighttpd to a new one using the venerable Apache HTTP Server v2.2. Both setups had per-vhost FastCGI setups pointing to PHP instances running as the user who owned the vhost, which helps ensure that compromised PHP apps affect only files/sites owned by that the user^†.

For example, lighttpd would be set up something like this:

fastcgi.server = ( ".php" =>
  ( "socket" => "/var/run/php-fastcgi/username/username-php-fastcgi.sock",
    "check-local" => "disable",
    "broken-scriptfilename" => "enable"
  )
)

Apache uses something like this:

FastCgiExternalServer /var/run/php-fastcgi/vhosts/example.com -socket /var/run/php-fastcgi/users/username/username-php-fastcgi.sock
AddHandler php-fastcgi .php
Action php-fastcgi /php-fastcgi
Alias /php-fastcgi /var/run/php-fastcgi/vhosts/example.com

During the forum signup, to help cut down on the number of spammy accounts created, there are both reCAPTCHA and DNS Blacklist checks that occur before the account creation actually happens. These were taking longer than the default 30 second timeout, causing the FastCGI interface to time out and close the connection, resulting in log entries such as this:

[Sun Apr 15 20:00:09 2012] [error] [client 192.168.0.2] FastCGI: comm with server "/var/run/php-fastcgi/vhosts/mmba.org" aborted: idle timeout (30 sec)

This led me to increase the FastCgiExternalServer timeout in mod_fastcgi by adding -idle-timeout 60, doubling it from its default, as follows:

FastCgiExternalServer /var/run/php-fastcgi/vhosts/example.com -socket /var/run/php-fastcgi/users/username/username-php-fastcgi.sock -idle-timeout 60
AddHandler php-fastcgi .php
Action php-fastcgi /php-fastcgi
Alias /php-fastcgi /var/run/php-fastcgi/vhosts/example.com

The problem then went away.

I’m not exactly sure why this cropped up with the move to Apache, but I suspect that on lighttpd there was a considerably longer default timeout. This can be set in the lighttpd config by setting idle-timeout, but I wasn’t able to easily figure out what the default is. It’s possible I’ll have to further tune this further in the future, but at least I now know why the problem was occurring.

^† Yes, I know this isn’t a perfect solution, but it’s been proven to work when sites are compromised by automatic tools that attempt to change/delete all they can. In each case that I’ve experienced the damage has typically been limited to content in that user’s home directory. This would not be good mitigation against something which attempted privilege escalation once on the box, went after the httpd itself, etc.

1 Comment

MMBA Forum Posts during 2012 Iceman Registration

Published March 1, 2012

The image above (click to embiggen) shows the one hour moving average statistics for posts (replies) and topics (new threads) on the MMBA Forum. This morning registration for Iceman opened on USA Cycling‘s website and as a result of the site collapsing under the load many people visited the forum to complain, commiserate, and generally discuss the situation. Most of the traffic seemed to be in this thread, although I don’t keep per-thread statistics beyond what’s already exposed to the end users.

Leave a Comment

Android Phone as BitLocker Recovery Key

Published February 28, 2012

I recently ran into an issue where I needed my BitLocker recovery key, but found that I’d left it at home. This locked me out of my machine for a few hours and was a bit of a hassle. It got me wondering if maybe I should instead keep said key on my phone instead, as it does a pretty good job serving as a USB mass storage device.

It turns out that this is not such a keen idea, as it doesn’t work out very well. On my Nexus One (and presumably other Android phones), when connected to a computer they prompt the user to switch the microSD storage over to USB Mass Storage. Once selected this takes a few seconds. Whenever the phone is disconnected from the USB host it automatically presents the card back to the phone, and only prompts the user once a host is detected.

With BitLocker, one becomes aware of needing this key when the boot is interrupted and the user is prompted to insert a USB mass storage device with the key present. The computer is then rebooted, finds the key, and continues to boot. With an Android device as the USB mass storage device this reboot resets the phone. I’ve found that it’s typically not possible to get the phone into mass storage mode between POST and OS load, particularly as the machines I’ve tried it on seem to reset USB after POST.

I think I’ll just have to stick with a USB flash drive for this.

Leave a Comment

Shattered Hard Drive Platter

Published February 24, 2012

To collect neodymium magnets for rebuilding the not-currently-great drip tray to kegerator / kegged beer cooler I started disassembling more old hard drives, attempting to salvage the magnets. When disassembling one drive I decided it’d be fun to bend the platter, but I (stupidly) didn’t pay attention to the platter material before doing so. The result? Tiny bits of broken glass and amazingly thin foil spraying all over the basement floor. Whoops.

At least it was easy to clean up.

I had little luck collecting useful magnets from the drives because some were too small, some drives only had one, and many of them shattered as I removed them. Instead I’ll just buy a bunch of tiny ones on eBay or Amazon and epoxy them on instead. It’ll be a lot easier.

1 Comment

iPod Battery Replaced

Published February 24, 2012

Roughly six years after buying my black fifth generation iPod I finally replaced the battery. After years of working well it was starting to fail, providing me with only a few hours of music listening while at work, way down from the original 10-15 that I was getting when it was new.

The replacement battery was part number BIPOD550MV30 from OWC, a very reliable online Apple reseller who has been around for years, and cost just over $20 shipped. While I could have purchased a cheaper battery via eBayI wanted to be sure to get one from a reliable place, so if I had problems with it there wouldn’t be an issue returning it.

The battery is now installed and charging, so hopefully by morning it’ll be full and ready to go.

Leave a Comment

MS12-006 and HTTP 408 Errors

Published February 20, 2012

I recently worked on an issue where a new Microsoft patch for a security bulletin, MS12-006, was correlated to a web-based application occasionally returning errors. Specifically, while using a web-based version of a reporting package off of a version of Sun (Oracle) Java Web Server the browser would occasionally return a Times New Roman-font “Request Timeout” message when users clicked report-generating links. The folks who support the server working on the issue initially had spent time uninstalling recent patches and found that removing KB2585542 (the patch for MS12-006 on Windows 7 clients) from the client machine would stop the issue from occurring. They then thought it might be a Windows client issue, which is where I got involved.

After some investigation it turns out that the problem was actually with the web server not handling split (fragmented) SSL/TLS records properly. The installation of the patch for MS12-006 causes IE to, in some cases, split SSL/TLS records into multiple requests. Since the server couldn’t appropriately handle these requests it thought the client closed the connection before sending a complete request and returned an 408 Request Timeout message.

This was illustrated by the HTTP logs on the server which show incomplete requests being returned 408 messages, similar to what follows. The bold line illustrates a request that correlates with the issue:

IP       | Username | Date                 | Request Header              | Status | Content | Length

---------+----------+----------------------+-----------------------------+--------+---------+--------

10.0.0.2 | - | 17/Feb/2012:10:03:25 | "P" | 408 | | 148

10.0.0.2 | -        | 17/Feb/2012:10:01:13 | "POST /TestApplet HTTP/1.1" | 200    |         | 13056

After the installation of the patch for MS12-006 there is a setting which allows this new feature to specifically be used only when called for (“optin mode”, the default), all the time (“enabled for all”), or never (“disabled for all”). Setting the option to “disabled for all” kept the issue from being reproduced, and when set to “enabled for all” it always occurred. (See the Registry Information section of 2643584 for more information on these options and how to set them.) By this I showed that it was this SSL/TLS feature causing the issue communicating with the server. Since this appeared to be an issue only with this particular server the team supporting it is now looking into potential solutions on that side.

While it may be tempting to simply disable this feature on clients, this would handicap one of the security enhancements brought along with the MS12-006 patch. Since this issue isn’t likely to be widespread (else lots of HTTPS sessions would be failing) I believe that the servers should be fixed to be compatible with the client’s standard behavior. Specifically, I believe the issue to actually be with the server not having an RFC2246-compliant TLS implementation and thus it should be fixed.

Hopefully this information will be useful to others who run into such an issue. It’s a bit opaque at first, since it looks to the users like the server is having a problem, but to server like the client is occasionally failing to send proper HTTP requests. Trying to troubleshoot this with Fiddler didn’t help either, because the HTTPS proxying would recreate the request, apparently no longer splitting the SSL/TLS record, which kept the error from appearing.

Here’s a few links which are helpful in understanding this issue, in order of usefulness:

· MS12-006: Vulnerability in SSL/TLS could allow information disclosure: January 10, 2012
· SSL/TLS Record Fragmentation Support – Microsoft blog article describing the issue in reverse, from a few years ago when SChannel didn’t support SSL/TLS Record Fragmentation.
· Microsoft Security Bulletin MS12-006 – Important, Vulnerability in SSL/TLS Could Allow Information Disclosure (2643584)
· HTTP Status Code 408 Documentation
· RFC2246 – See section 6.2.1.

1 Comment

Burroughs Mainframe Cards

Published February 17, 2012

Years ago I got the chance to help empty out a friend’s neighbor’s basement after they passed away. This basement was stuffed with old computers and electronics, and I acquired a number of interesting items from there, many of which are now displayed around the house. (Core memory, old test equipment, notebooks, etc.)

The photos from this excursion can be found here, if you’re interested.

One of the items I’d acquired was a milk crate full of old cards from a Burroughs mainframe. Each is a bunch of discrete components built into what’s essentially a logical unit, designed to perform some function. These days a small integrated circuit would be used to perform the same (or multiple) functions.

I’ve given many of these items away, as in my excitement I ended up with many more items than needed. Everything from large boxes of valves to old, uncalibrated oscilloscopes have found homes with friends who had a better use for them than I did. A little while back I was contacted by a guy named William Donzelli who was interested in the history of the Burroughs mainframe and the cards from it. I didn’t know much about it and couldn’t answer many questions, so a couple days back I sent off this box containing the cards to him. Hopefully he’ll find them enlightening.

1 Comment

Crucial m4 SSD for Data Processing

Published February 14, 2012

I’ve been processing some relatively large sets of data at work lately, and I’m running into disk IO issues after kicking off some of the data processing tasks. As an easy way to alleviate this I picked up a 64GB Crucial m4 SSD and stuffed it into my laptop’s optical drive bay via a $20 Nimitz-branded SATA hard drive adapter acquired from eBay. The drive itself was $99.99 from Micro Center, and since I purchased it myself once I’m done working with it I’ll be able to repurpose it in a personal machine of some sort.

While not a panacea nor excuse for my poor habits of bolting together VBScript and GNU command line utilities, it has helped quite a bit. One typical script (immediately post-reboot, with an empty disk cache) takes ~120.234 seconds to run a task from the hard drive and ~28.400 seconds from the SSD. I’ve seen similar speed improvements across the board, and this really helps when I’m prototyping things and wanting feedback as quickly as possible.

Leave a Comment

Vector Symbols for Maps

Published December 21, 2011

Ever been drawing a map and found yourself needing vector copies of symbols? Well, thanks to the great state of Minnesota there is a font containing most of these symbols freely available. On the state’s page Recreation Symbols Extension for ArcView one can download recreate.ttf, a 126-character TrueType font containing indicators for everything from XC ski trail difficulty to lighthouse locations and from winter camping sites to RV dump locations.

To make using this font easier I’ve created a spreadsheet-based key listing each of the symbols next to their corresponding Unicode character. This can be found here: Minnesota DNR Recreation Symbol True Type Font Key.xlsx

There’s a few other nifty things that Minnesota has provided to the general GIS community, including Road Symbols. The rest can be found here.

UPDATE: Here is another set of symbols provided by the National Park Service (NPS). These are much more up to date and provided as both PDF and .AI files.

2 Comments

Plastic Bottles, Foil, Visine, and a Shoe

Published November 23, 2011

Oh, the things that one finds when looking for a route for some new trail. There were three or four of these, an empty bottle of Visine, and one shoe are all laying within 15′ of each other next to a hiker / game trail leading from some apartments to the main two track. If I’m able to route a trail as hoped then another piece of return trail will pass through this area and make for a small segment ending directly across from the current trail start.

I came across this while looking over a place for a new segment of trail after finishing up some tweaking of the newest segment of trail in the park. There were a few poorly designed corners to sort out, some overhanging brush to trim, and some weirdly rough/lumpy trail surface to smooth. It was a fun 3.5 hours of work on a very nice autumn afternoon. Now, to ride it and see what else needs adjusting.

1 Comment