Networking Problem
I’m having a bit of a ssh / networking problem. It’s really confusing me.
If you would like to read about it and hopefully help me solve it, please do. I’ll break sections of info up with hr’s.
First off, there are three machines which I’ll mention here:
· jawbreaker.dreamhost.com at 208.113.141.20, known as jawbreaker, and is shared box at a managed hosting facility.
· home.nuxx.net at 68.60.175.29, known as home, has no packet filter rules loaded, and is connected directly to a Comcast cable modem.
· rez.nuxx.net at 204.11.33.41, known as rez, is colocated in Troy, MI, and is connected directly to a live, unfiltered connection.
I will also occasionally connect from work, which is via a tunnel through an HTTP proxy which has no inbound connection.
Here are the symptoms I’m seeing:
· I can freely SSH from work to jawbreaker, home, and rez.
· I can freely SSH from rez to home and jawbreaker.
· I can freely SSH from jawbreaker to home and rez.
· I can freely SSH from home to rez.
· I cannot SSH from home to jawbreaker.
· Other protocols, such as HTTP, HTTPS, POP3, SMTP all work between home and jawbreaker just fine.
I first noticed the problem when running a script to use rsync tunneled through ssh to back up all of my users on dreamhost. This was running successfully overnight, but after finishing one user at around 03:51 EDT on 16-Jul-2006 things stopped working.
I have made no changes, and I was told by support over at DreamHost that there is no entry in the hosts.deny for the IP of home, 68.60.175.29.
The issue occurs whether “TCP window increasing” on home is disabled (sysctl -w net.inet.tcp.rfc3390=0) or not.
TCP window scaling is disabled on jawbreaker:
jawbreaker:~> cat /proc/sys/net/ipv4/tcp_default_win_scale
0
jawbreaker:~>
I checked these things because of this reported Debian bug which doesn’t seem especially related, but I figured I’d look anyway.
This issue is reproducable when trying to connect to multiple user accounts on jawbreaker and from multiple machines at home, both home itself all machines to which it provides internet access via NAT. Note that all detailed troubleshooting was done directly from home itself with all NAT and packet filtering disabled.
Now the output I see from ssh -vvv c0nsumer@jawbreaker.dreamhost.com is as follows:
-bash-3.1$ ssh -vvv c0nsumer@jawbreaker.dreamhost.com
OpenSSH_4.3, OpenSSL 0.9.7g 11 Apr 2005
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to jawbreaker.dreamhost.com [208.113.141.20] port 22.
debug1: Connection established.
debug1: identity file /home/c0nsumer/.ssh/identity type -1
debug1: identity file /home/c0nsumer/.ssh/id_rsa type -1
debug1: identity file /home/c0nsumer/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host
· All SSH keys on both ends have been removed from appropriate ~/.ssh directories and multiple usernames have been tried.
Attempting to connct to a completely non-existant username on the server fails as well. The user is never prompted to accept the new host key:
-bash-3.1$ ssh testuserwhichshouldntexistatall@jawbreaker.dreamhost.com
ssh_exchange_identification: Connection closed by remote host
-bash-3.1$
· home is running OpenBSD 3.9 without pf and with the shipping kernel.
· jawbreaker is running: Linux jawbreaker 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+++opt+c6+gr2b-v6.192 #1 SMP Wed Dec 14 17:06:16 PST 2005 i686 GNU/Linux.
Here is a traceroute from home to jawbreaker:
# traceroute jawbreaker.dreamhost.com
traceroute to jawbreaker.dreamhost.com (208.113.141.20), 64 hops max, 40 byte packets
1 * * *
2 ge-2-1-ur03.macomb.mi.michigan.comcast.net (68.86.120.117) 16.132 ms 11.747 ms 11.594 ms
3 te-9-1-ur02.sterlinghub.mi.michigan.comcast.net (68.87.190.254) 19.257 ms 12.22 ms 11.844 ms
4 te-9-4-ur03.sterlinghub.mi.michigan.comcast.net (68.87.191.2) 16.701 ms 12.409 ms 26.386 ms
5 te-9-1-ur02.warren1.mi.michigan.comcast.net (68.87.191.6) 16.41 ms 12.951 ms 38.499 ms
6 te-9-4-ur03.warren1.mi.michigan.comcast.net (68.87.191.10) 30.742 ms 12.651 ms 12.459 ms
7 te-9-1-ur02.royaloak.mi.michigan.comcast.net (68.87.191.14) 29.955 ms 13.359 ms 13.726 ms
8 te-9-2-ur03.royaloak.mi.michigan.comcast.net (68.87.191.18) 18.200 ms 36.832 ms 14.443 ms
9 te-9-1-ur02.rochesterhlls.mi.michigan.comcast.net (68.87.191.22) 18.975 ms 13.789 ms 13.674 ms
10 te-9-1-ur02.auburnhills.mi.michigan.comcast.net (68.87.191.26) 21.968 ms 13.650 ms 14.102 ms
11 te-9-1-ar02.pontiac.mi.michigan.comcast.net (68.87.191.30) 19.882 ms 14.50 ms 14.155 ms
12 pos-6-1-ar01.pontiac.mi.michigan.comcast.net (68.87.191.165) 30.622 ms 22.374 ms 12.877 ms
13 12.116.16.25 (12.116.16.25) 26.569 ms 20.704 ms 19.463 ms
14 tbr2032901.cgcil.ip.att.net (12.123.4.238) 70.520 ms 66.947 ms 79.192 ms
15 tbr2-cl7.sl9mo.ip.att.net (12.122.10.46) 74.259 ms 93.523 ms 68.248 ms
16 tbr2-cl21.la2ca.ip.att.net (12.122.10.14) 72.703 ms 68.30 ms 80.93 ms
17 gar1-p3100.lsnca.ip.att.net (12.123.199.229) 71.31 ms 67.786 ms 84.23 ms
18 12.119.138.66 (12.119.138.66) 70.535 ms 67.61 ms 119.277 ms
19 border1.po1-bbnet1.ext1a.lax.pnap.net (216.52.255.31) 86.158 ms 66.103 ms 69.94 ms
20 newdream-1.border1.ext1a.lax.pnap.net (216.52.220.78) 72.42 ms 65.867 ms 67.859 ms
21 jawbreaker.dreamhost.com (208.113.141.20) 99.638 ms 67.68 ms 66.781 ms
Here is a traceroute from jawbreaker to home:
jawbreaker:~> traceroute home.nuxx.net
traceroute to home.nuxx.net (68.60.175.29), 30 hops max, 38 byte packets
1 gw-208-113-128-1 (208.113.128.1) 0.374 ms 0.330 ms 0.286 ms
2 border1.g3-5.newdream-1.ext1a.lax.pnap.net (216.52.220.77) 0.272 ms 0.229 ms 0.268 ms
3 core3.t2-2-bbnet2.lax.pnap.net (216.52.255.67) 95.864 ms 3.780 ms 202.082 ms
4 12.119.138.65 (12.119.138.65) 0.352 ms 0.419 ms 0.336 ms
5 tbr2-p013101.la2ca.ip.att.net (12.123.199.230) 48.256 ms 48.507 ms 48.592 ms
6 tbr2-cl21.sl9mo.ip.att.net (12.122.10.13) 47.738 ms 48.064 ms 48.390 ms
7 tbr2-cl7.cgcil.ip.att.net (12.122.10.45) 48.313 ms 47.864 ms 47.980 ms
8 gar4-p390.cgcil.ip.att.net (12.123.6.14) 47.141 ms 46.832 ms 47.465 ms
9 12.118.239.42 (12.118.239.42) 53.263 ms 53.205 ms 53.169 ms
10 pos-2-1-ar02.pontiac.mi.michigan.comcast.net (68.87.191.166) 52.850 ms 52.797 ms 52.795 ms
11 te-9-2-ur02.auburnhills.mi.michigan.comcast.net (68.87.191.29) 53.859 ms 53.768 ms 53.816 ms
12 te-9-2-ur02.rochesterhlls.mi.michigan.comcast.net (68.87.191.25) 54.144 ms 53.967 ms 53.914 ms
13 te-9-1-ur03.royaloak.mi.michigan.comcast.net (68.87.191.21) 54.695 ms 54.884 ms 54.742 ms
14 te-9-2-ur02.royaloak.mi.michigan.comcast.net (68.87.191.17) -1531.895 ms 54.341 ms 54.341 ms
15 te-9-1-ur03.warren1.mi.michigan.comcast.net (68.87.191.13) 54.607 ms 54.566 ms 54.593 ms
16 te-9-4-ur02.warren1.mi.michigan.comcast.net (68.87.191.9) 54.405 ms 54.275 ms 54.252 ms
17 te-9-1-ur03.sterlinghub.mi.michigan.comcast.net (68.87.191.5) 54.275 ms 54.201 ms 54.265 ms
18 te-9-4-ur02.sterlinghub.mi.michigan.comcast.net (68.87.191.1) 54.296 ms 54.277 ms 54.413 ms
19 te-9-1-ur03.macomb.mi.michigan.comcast.net (68.87.190.253) 54.825 ms 54.893 ms 54.856 ms
20 * * *
21 c-68-60-175-29.hsd1.mi.comcast.net (68.60.175.29) 75.717 ms 62.029 ms 62.405 ms
Here is the tcptraceroute output connecting from home to jawbreaker via port 22 (SSH):
# tcptraceroute jawbreaker.dreamhost.com 22
Selected device fxp0, address 68.60.175.29, port 11734 for outgoing packets
Tracing the path to jawbreaker.dreamhost.com (208.113.141.20) on TCP port 22, 30 hops max
1 * * *
2 ge-2-1-ur03.macomb.mi.michigan.comcast.net (68.86.120.117) 13.499 ms 10.856 ms 24.718 ms
3 te-9-1-ur02.sterlinghub.mi.michigan.comcast.net (68.87.190.254) 35.414 ms 12.855 ms 24.947 ms
4 te-9-4-ur03.sterlinghub.mi.michigan.comcast.net (68.87.191.2) 12.240 ms 12.372 ms 62.391 ms
5 te-9-1-ur02.warren1.mi.michigan.comcast.net (68.87.191.6) 13.717 ms 13.378 ms 15.417 ms
6 te-9-4-ur03.warren1.mi.michigan.comcast.net (68.87.191.10) 23.959 ms 12.563 ms 12.578 ms
7 te-9-1-ur02.royaloak.mi.michigan.comcast.net (68.87.191.14) 37.649 ms 12.883 ms 26.065 ms
8 te-9-2-ur03.royaloak.mi.michigan.comcast.net (68.87.191.18) 35.619 ms 12.960 ms 26.129 ms
9 te-9-1-ur02.rochesterhlls.mi.michigan.comcast.net (68.87.191.22) 13.028 ms 12.907 ms 18.709 ms
10 te-9-1-ur02.auburnhills.mi.michigan.comcast.net (68.87.191.26) 14.315 ms 13.827 ms 13.773 ms
11 te-9-1-ar02.pontiac.mi.michigan.comcast.net (68.87.191.30) 14.012 ms 14.944 ms 12.863 ms
12 pos-6-1-ar01.pontiac.mi.michigan.comcast.net (68.87.191.165) 16.173 ms 12.604 ms 18.535 ms
13 12.116.16.25 (12.116.16.25) 32.703 ms 22.959 ms 20.496 ms
14 tbr2032901.cgcil.ip.att.net (12.123.4.238) 67.933 ms 79.263 ms 66.734 ms
15 tbr2-cl7.sl9mo.ip.att.net (12.122.10.46) 76.101 ms 82.298 ms 66.611 ms
16 tbr2-cl21.la2ca.ip.att.net (12.122.10.14) 84.126 ms 79.689 ms 69.245 ms
17 gar1-p3100.lsnca.ip.att.net (12.123.199.229) 66.972 ms 78.704 ms 66.304 ms
18 12.119.138.66 (12.119.138.66) 98.005 ms 164.422 ms 194.164 ms
19 border1.po1-bbnet1.ext1a.lax.pnap.net (216.52.255.31) 67.800 ms 67.176 ms 79.373 ms
20 newdream-1.border1.ext1a.lax.pnap.net (216.52.220.78) 66.434 ms 90.888 ms 67.048 ms
21 jawbreaker.dreamhost.com (208.113.141.20) [open] 90.653 ms 67.222 ms 83.555 ms
The next steps I’m going to try are to replace the OpenBSD machine home with a Windows XP box and try SSHing out. If I can then connect, it’s a problem with OpenBSD on home and it’s interaction with jawbreaker. If it does not work, it’s likely either a Comcast or AT&T problem. Those should be fun to sort out. :\
Problem is somewhat solved. I swapped interfaces on the firewall forcing an IP change. My old IP was 68.60.175.29 and my new one is 68.43.131.193. And now things work. So, either it’s odd Comcast dealings, or it really was something being blocked at the ISP. Either way, it’s really screwey. Maybe I’ll dig into it some more tomorrow. For now I need sleep.
UPDATE: Problem kinda solved. Check below the cut for details.
try enabling another user account on jawbreaker with shell access and see if you have the same problem with that account going from home to jawbreaker.
I do. I’ve been using katester’s account as a test as well.
For what it’s worth, the script had just sync’d both my account (c0nsumer) and kate’s (katester), and when it went to try another (lisette) that’s when the problem happened. I’d imagine whatever change is causing the issue (at DH’s end or somewhere in between) took place within the window of the established session which was used for syncing Kate’s account.
I’ll make this clearer in the post.
Sorry, didn’t mean for that reply to come by so harshly. It’s not meant that way. :)
not harsh at all, no worries. Have you tried it with a complete new created account? just make one to test this problem with. passing this on to a friend to mull over, since I’m not seeing anything totally obvious here otherwise…
Because it doesn’t even connect enough to get the host’s key (~/.ssh/known_hosts is never appended with the server’s stuff, the user is never prompted to accept the new host key) I don’t think this matters.
See here:
-bash-3.1$ ssh testuserwhichshouldntexistatall@jawbreaker.dreamhost.com
ssh_exchange_identification: Connection closed by remote host
-bash-3.1$
Hmm… I don’t see anything that jumps out at me (being the aforementioned friend).
I did try the following, and apparently get further in the connecting process than you do:
[ jarehart@blackbox 02:56pm ttyp4
~ ] $ \ssh -v jarehart@jawbreaker.dreamhost.com
OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3, SSH protocols 1.5/2.0, OpenSSL 0x0090603f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: ssh_connect: needpriv 0
debug1: Connecting to jawbreaker.dreamhost.com [208.113.141.20] port 22.
debug1: Connection established.
debug1: identity file /home/jarehart/.ssh/id_rsa type 1
debug1: identity file /home/jarehart/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_3.8.1p1 Debian-8.sarge.4
debug1: match: OpenSSH_3.8.1p1 Debian-8.sarge.4 pat OpenSSH*
Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: dh_gen_key: priv key bits set: 143/256
debug1: bits set: 1056/2048
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
The authenticity of host 'jawbreaker.dreamhost.com (208.113.141.20)' can't be established.
RSA key fingerprint is bc:f9:b5:55:4b:2c:07:d7:42:8c:00:2a:8d:f0:2c:de.
Are you sure you want to continue connecting (yes/no)? no
Host key verification failed.
debug1: Calling cleanup 0x8063aac(0x0)
And my next suggestion would be exactly what you have planned, to try a different machine/OS at home and see what happens. If it comes to yelling at Comcast/AT&T I don’t envy you, and in fact may get to do something similar with Grande (several of my friends who have shell accounts on my home machine have extreme lag but only during evenings, and only via ssh; ping and traceroute are unaffected).
Thanks for taking a look at it. Just quickly, an XP machine directly connected to the cable modem can connect just fine using PuTTY. While NAT’d behind the OpenBSD box it exhibits the same problems.
So, it looks like the problem is OpenBSD. Now to figure out A) how to resolve it, and B) what changed. B may not be by doing, or it may be something automatically tuned on OpenBSD (unlikely). Or it may be AT&T. Or Comcast. Or… I don’t know.
:\
Have you tried forcing v2 connections using the -2 flag? v1 is notoriously insecure, so dreamhost has probably either turned it off or not even compiled it in. And your client could be configured to either try v1 first, or only use v1.
Turns out it is OpenBSD. Not OpenBSD’s version of SSH, but packets originating from it.
Now’s where it gets fun. :\
Hmm… That makes me think of ECN and PMTU discovery. Have you checked on the relevant toggles controlling those at home and Dreamhost?
Should be simple. Just assume it is a bug in OBSD, and annoy Theo until it’s fixed ;)
Arguing with Theo will take longer than just snail mailing all the data on 500 3.5 floppy disks to dreamhost.
Oh, also, (this wasn’t in my original post) the rsync script forces v2.
ssh -vvv c0nsumer@jawbreaker.dreamhost.comOpenSSH_4.3, OpenSSL 0.9.7g 11 Apr 2005
Kinda old version of OpenSSL to be compiling OpenSSH against there… OpenSSH was released 1 Feb 2006, but they’re up to OpenSSL 0.9.8b as of May. And you’re getting problems in the identification phase… so how about trying the latest OpenSSH compiled against the latest OpenSSL libs?
what’s more interesting, is that I tried to ssh into my Dreamhost account (on cognac) from Linux behind a Linksys/DD-WRT NAT:
OpenSSH_4.3p1, OpenSSL 0.9.7g 11 Apr 2005
It pauses right at the identity point where it stops for you for 15-30 seconds, and then:
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_3.8.1p1 Deb ian-8.sarge.4
debug1: match: OpenSSH_3.8.1p1 Debian-8.sarge.4 pat OpenSSH_3.*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug2: fd 3 setting O_NONBLOCK
and more stuff before prompting for a password.
(Jawbreaker is apparently running the same version of SSH)
Thanks for the idea, but the SSH version is not it. If it were, clients behind the OpenBSD box would be able to connect still.
The problem is with SSH traffic which either originates with or traverses the OpenBSD box.
Huh. I missed that bit. Tried running a sniffer to see what’s going on when you try to ssh jawbreaker vs when you ssh rez? Since it’s dying before it gives you the remote protocol version/remote software version, this would let you see some of what’s going on (if there’s any incoming packets disappearing into the Land of Null or being NAKed or whatever).
Check the main post. A new IP fixed it. Not sure why, but I’ll dig into it more tomorrow. It’s a stupid fix, though… Ah well. At least it’s working.