After a bunch of frustration I found an interesting quirk in how Google Talk uses Proxy Autoconfiguration (PAC) files (Additional Documentation). There are two MIME types which can be set for PAC files when serving them, application/x-javascript-config and application/x-ns-proxy-autoconfig, and both work with most browsers, but only the latter works properly with Google Talk.
It turns out that if the MIME type is application/x-javascript-config for the PAC file and the browser does not look directly to the PAC file itself, Google Talk will fail to connect. Looking deeper at it in a network capture (partially pictured above) I’d see the client closing the connection before it was done downloading the PAC file and thus the client would never receive a complete PAC file. Inspecting the TCP stream via Wireshark would show the data simply ending part-way through the PAC file. This resulted in a mysterious failure to connect with no useful error returned from Google Talk. If I simply changed the MIME type back to application/x-ns-proxy-autoconfig for .pac files and restarted Apache, Google Talk would then download the whole PAC file and login fine.
In both of these cases I am serving a typical PAC file off of Apache using RewriteRule ^/$ /internet.pac [R] to ensure that all requests to http://server.local/ go to http://server.local/internet.pac via an HTTP 302. Using a DirectoryIndex directive specifying internet.pac as the index file for the site also resulted in the same issue. In both cases the client was configured to look to http://server.local/ for a PAC file.
Complicating things further I found that if instead of relying on a RewriteRule or DirectoryIndex I instead pointed the config directly to the PAC file itself (in this case http://server.local/internet.pac) then the problem would not occur; Google Talk would download the entire PAC file (confirmed in a network capture) and sign in successfully regardless of MIME type.
For purposes of this testing I used the latest released version of Google Talk, 1.0.0.104, running on Windows 7. Google Talk was also set to Detect proxy automatically which reads the system’s proxy settings. In this case it is the PAC file setting is defined in either Internet Explorer or Control Panel under (Internet Options → Connections → LAN Settings → Use automatic configuration script).