A while ago, our Capistrano-based deployment procedure stopped working for some of the developers on the project. Sometimes it would simply “hang” (or at least stall for a very long time before being killed), sometimes it would exit with a Net::SSH::AuthenticationFailed exception when it attempted to connect to the remote servers.

The strange thing was that these developers had no trouble opening ordinary ssh connections to the exact same machines using the exact same remote user name. The problem only seemed to manifest itself when Capistrano was in the driving seat.

I spent quite a while comparing the versions of ssh, OpenSSL, Capistrano, net-ssh found on the various machines, looking for some difference that could explain the behavior. Nothing suspicious turned up. It wasn’t until I took a look at the net-ssh source code that the culprit was found: when net-ssh is parsing the ssh_config files (which it looks for in ~/.ssh/ssh_config, /etc/ssh_config, or /etc/ssh/ssh_config) it interprets the HostbasedAuthentication entry slightly differently than plain-vanilla OpenSSH: If any HostbasedAuthentication entry is found then this method becomes the *only* allowed authentication method going forward; not just one to try. In our case, this method failed on some machines due to network configuration issues and with no other authentication method to fall back on (it should have used public key), net-ssh simply gave up. Once we removed the HostbasedAuthentication entry from the configuration file, everything started working again.

We never found out whether the problem was introduced by an upgrade of net-ssh, the OS (and hence possibly configuration files), a change in the network configuration, or quantum mechanic effects.