SSH – Sol's blog

NoMachine ssh host key update

May 042018

I recently ran into an issue where I needed to update the SSH host keys on a machine which was running NoMachine. In the process, I came across the following annoying error which was not at all indicative of the actual issue

-- NX SERVER START: -c /usr/bin/nxserver - ORIG_COMMAND=
-- NX SERVER START:  - ORIG_COMMAND=                    
Info: Using fds #4 and #3 for communication with nxnode.
HELLO NXSERVER - Version 3.2.0-74-SVN OS (GPL, using backend: not detected)
NX> 105 hello NXCLIENT - Version 3.2.0
NX> 134 Accepted protocol: 3.2.0
NX> 105 SET SHELL_MODE SHELL                            
NX> 105 SET AUTH_MODE PASSWORD  
NX> 105 login
NX> 101 User: awesomeuser
NX> 102 Password:
Info: Closing connection to slave with pid 6424.
NX> 404 ERROR: wrong password or login          
NX> 999 Bye

I verified repeatedly that other methods for logging in were working properly. This problem was due to us using the SSH protocol for NoMachine logins. Unfortunately, this error is also not at all useful because it doesn’t show the actual problem. The problem was that the “nxserver” user had a known_hosts file containing the old ssh host key and for some reason it wouldn’t allow me to accept the new one. I simply removed the old key.

# rm /var/lib/nxserver/home/.ssh/known_hosts
rm: remove regular file `/var/lib/nxserver/home/.ssh/known_hosts'? y

Once the old key was removed, I was once again able to login successfully. Curiously, the known_hosts file was not regenerated. I’m not exactly sure how it wound up there to begin with since this setup was created before I arrived.

Openssh ignore global known hosts

Linux, SSH No Responses »

Oct 252011

I use Bcfg2 to create and synchronize the /etc/ssh/ssh_known_hosts file across all the machines I manage. The result of this is that the known_hosts file actually contains useful information.

The one case where this bites me is when I want to boot from a live CD and image the drive on the machine itself. Booting into the live CD and starting sshd creates new keys which gives me this ugly message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
69:38:ba:80:93:b8:2a:29:ec:b3:65:e2:40:da:78:54.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /etc/ssh/ssh_known_hosts:153
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Permission denied (publickey,keyboard-interactive).

I don’t want to go to the trouble of editing the global known_hosts file since it actually contains correct information (and someone may want to use that before bcfg2 runs again). Therefore, I just want to temporarily disable checking of the file. I found a cool little option for ssh to do just that. It’s called GlobalKnownHostsFile and we can set it to /dev/null to temporarily turn off the feature.

ssh -o GlobalKnownHostsFile=/dev/null

You will probably want to use this in conjunction with the UserKnownHostsFile option so that the client doesn’t save the temporary key to your ~/.ssh/known_hosts.

Debugging OpenSSH GSSAPIAuthentication

Kerberos, Linux, SSH 2 Responses »

Aug 172011

This is just a quick post to show how I go about debugging problems with GSSAPIAuthentication. You want to debug both the server side and the client side, so the first thing to do is start a new instance of the openssh server in the foreground on a different port.

# `which sshd` -o "GSSAPIAuthentication yes" -d -D -p 2222
debug1: sshd version OpenSSH_5.3p1 Debian-3ubuntu7
debug1: read PEM private key done: type RSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-D'
debug1: rexec_argv[3]='-p'
debug1: rexec_argv[4]='2222'
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.

This will start up the ssh server listening on port 2222 with debugging turned on. Then you need to try connecting to this instance from the client that is unable to connect.

$ ssh -o "GSSAPIAuthentication yes" -vvv -p 2222 server.example.com

This will output a ton of information on both the server and the client which should help you figure out why you are unable to login using GSSAPIAuthentication. Some common pittfalls to keep in mind

Make sure you have GSSAPIAuthentication turned on either globally or for the user trying to login (this is done for you in the examples above, so if things work then this may be your problem).
Make sure you have created a host principal for the ssh server and have added it to that machine’s /etc/krb5.keytab

You can test this by logging into the ssh server and running klist -k.

# klist -k
	Keytab name: WRFILE:/etc/krb5.keytab
	KVNO Principal
	---- --------------------------------------------------------------------------
	   2 host/server.example.com@EXAMPLE.COM
	   2 host/server.example.com@EXAMPLE.COM
	   2 host/server.example.com@EXAMPLE.COM
	   2 host/server.example.com@EXAMPLE.COM

If none of these steps turn up anything useful, check the kdc logs for errors.

Please note that the environment referred to above is using MIT Kerberos. I would expect the methods for debugging other software to be similar, but I cannot guarantee that the kerberos-related commands will be the same.

Multiplexing multiple sessions over ssh

Linux, SSH No Responses »

Nov 142010

I have recently started using ssh multiplexing and thought I’d share this technique with everyone. You will find this technique especially useful in cases where the initial connection negotiation takes longer than expected. You may also find it useful if the remote host forces password authentication and you get tired of typing your password repeatedly.

The first thing you need to do is copy the following lines to your ~/.ssh/config (or the global ssh_config):

Host *
    ControlMaster auto
    ControlPath /tmp/%r@%h:%p

After doing this, go ahead and test that things are working:
solj@abbysplaything $ ssh -f pjacosta-desktop sleep 60 solj@abbysplaything $ ls -l /tmp/solj* srw------- 1 solj solj 0 Nov 14 13:44 /tmp/solj@pjacosta-desktop:22

Here you can see that a socket has been created for my user which can be reused by any additional ssh connections which go to the same user/host/port combination. Not only does this bypass future negotiations, but it also prevents opening additional connections unnecessarily. This has also really helped out at work because now when I log into a remote machine with an extremely high load, I can simply use the existing connection if I need to open up multiple sessions.

I haven’t found any downsides to using this sort of multiplexing and it certainly has some upside. This feature doesn’t appear to be very popular or publicized, but I think that it provides really useful functionality.