87

Connection to one of my servers using ssh takes more than 20 seconds to initiate.

This is not related to LAN or WAN conditions, since connection to itself takes the same (ssh localhost). After connection is finally establised, it is super fast to interract with the server.

Using -vvv shows that the connection is stuck after saying "pledge: network". At this point, authentication (here using key) is already done, as visible here :

...
debug1: Authentication succeeded (publickey).
Authenticated to myserver.mydomain.com ([xx.xx.xx.xx]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network

(...stuck here for 15 to 25 seconds...)

debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
...

Server is Ubuntu 16.04. It already happened to me in the past with another server (was Ubuntu 12.04) , nerver found the solution and the problem disapeared after a while...

sshd_config is the default one provided by Ubuntu.

So far I have tried :

  • using -o GSSAPIAuthentication=no in the ssh command
  • using password instead of a key
  • using UsePrivilegeSeparation no instead of yes, in sshd_config
M-Jack
  • 1,436

15 Answers15

62

This is probably an issue with D-Bus and systemd. If the dbus service is restarted for some reason, you will also need to restart systemd-logind.

You can check if this is the issue by opening the ssh daemon log (on Ubuntu it should be /var/log/auth.log) and check if it has these lines:

sshd[2721]: pam_systemd(sshd:session): Failed to create session: Connection timed out

If yes, just restart systemd-logind service:

systemctl restart systemd-logind

I had this same issue on CentOS 7, because the messagebus was restarted (which is how the D-Bus service is called on CentOS).

Strahinja Kustudic
  • 910
37

found the answer :

changed UsePAM from yes to no in sshd_config file

After restarting the ssh service, the connection is now immediate to the server. On this server, PAM is linked to ldap, so that is probably the reason, even if here I am connecting with a user declared on the server itself, not an LDAP one.

Well, this is more a way to bypass the issue, not really a solution... I have other servers set up the same way that are not having this issue.

Hope this may help someone...

M-Jack
  • 1,436
23

This happened on two of my Fedora 25 servers, and was due to lots of failed SSH login attempts.

(The common suggestions of using GSSAPIAuthentication=no and UseDNS=no, or restarting systemd-logind, made no difference.)

On these servers, /etc/pam.d/postlogin contains:

session     optional      pam_lastlog.so silent noupdate showfailed

The man page for pam_lastlog explains that the showfailed option will:

Display number of failed login attempts and the date of the last failed attempt from btmp.

On these servers, the /var/log/btmp files were enormous due to many failed login attempts. The btmp log files weren't being rotated either.

I installed the logrotate package to ensure the log files will be rotated in future. (On Fedora, the configuration that ships with logrotate handles the rotation of /var/log/btmp.)

I also deleted the enormous btmp log files; as soon as I did this, connecting to the servers was instantaneous again.

Richard Fearn
  • 361
19

On Ubuntu 16+ every time I have seen ssh -v XXX@YYY stalling at pledge: network it can be fixed by following the instructions I found here A comprehensive guide to fixing slow SSH logins. Specifically, an optional PAM module that does not appear to be needed is causing the delay.

In /etc/pam.d/common-session on the machine you see slow logins for (ie. the server). Comment out the line session optional pam_systemd.so. That should immediately fix the problem.

This avoids having to completely shut down PAM which cripples login with passwords.

Jonathan Gutow
  • 291
7

The problem for me (Ubuntu 19.10) was that my:

/etc/pam.d/sshd

# Print the message of the day upon successful login.
# This includes a dynamically generated part from /run/motd.dynamic
# and a static (admin-editable) part from /etc/motd.
session    optional     pam_motd.so  motd=/run/motd.dynamic
session    optional     pam_motd.so noupdate

Commenting the motd settings got me right in.

Walter
  • 243
6

For me this issue is caused by large (hundreds of MBs) btmp file. This file logs login attempts. When people are trying to brute force your password this file can be big and cause delays in the "pledge: network" phase.

Try to clear log file

echo "" > /var/log/btmp

and see if it helps.

Tamas Foldi
  • 103
  • 2
Marek Nagy
  • 81
3

For me the first clue was provided by

UseDNS no

to the /etc/ssh/sshd_config and then of course service ssh restart (on our Debian/Jessie server).

before:

ssh git@git.*****.de true  0.03s user 0.01s system 0% cpu 13.440 total
ssh git@git.*****.de true  0.03s user 0.01s system 0% cpu 20.990 total
ssh git@git.*****.de true  0.03s user 0.02s system 0% cpu 31.114 total
ssh git@git.*****.de true  0.03s user 0.01s system 0% cpu 25.898 total

after:

ssh git@git.*****.de true  0.03s user 0.02s system 5% cpu 0.832 total
ssh git@git.*****.de true  0.03s user 0.01s system 7% cpu 0.523 total
ssh git@git.*****.de true  0.03s user 0.01s system 7% cpu 0.574 total

This revealed that my DNS configuration was wrong (I had a typo in the DNS address). After fixing the IP and restoring the setting UseDNS yes everything worked fine.

tamasgal
  • 195
2

In my case the reason was a crashed rsyslogd. I found this out because there were no more log-messages in e.g. /var/log/syslog or /var/log/mail.log

So service rsyslog restart resolved the problem for us.

randomcontrol
  • 260
1

In my case, it is because there are too many logs. You can test if you are in the case, by issuing this command:

sudo journalctl --list-boots

If it takes a while to give results, and give many lines of the result, then, you are in.

To truncate the logs, do this:

sudo journalctl --vacuum-time 2d

It will delete logs which are older than two days.

hongquan
  • 111
1

In my case it was a firewall problem after upgrade to debian 11.

Solved by adding:

iptables -t filter -A INPUT -i lo -j ACCEPT

At the beggining of the firewall script.

Ziprasidone
  • 11
1

I was also having issues with SSH connections from some clients getting stuck at pledge: network, and it turned out to be due to an incorrectly set MTU value on the server's primary network interface (ip a | grep mtu). I reset the MTU to the default value of 1500 and the issues went away (the method for setting your MTU is going to depend on your particular distro, e.g., whether or not you are using NetworkManager).

jayhendren
  • 1,074
0

In my case the reason was a crashed rsyslogd. I found this out because there were no more log entries in /var/log/secure

So I Restarted service rsyslog restart resolved the problem for us.

AVL SubbaRao
  • 11
0

I got the same issue and we had configured the SSSD service for AD login.

Stopping the SSSD service fixed the issue for me.

Reni
  • 1
0

If you are using NIS/YP, make sure nscd is running. In my case, I was getting the following when ssh in.

systemd-logind[30061]: do_ypcall: clnt_call: RPC: Unable to send; errno = Operation not permitted

According to https://github.com/systemd/systemd/issues/7074, nscd should be running to ensure smooth operation.

After I started nscd (which was not running since I upgraded Ubuntu) incoming ssh sessions are much faster.

stex
  • 1
0

I noticed the following line in my debug feedback:

Control socket connect(/var/lib/jenkins/.ssh/USER@HOST:22): Permission denied

Which was a file that was owned by root:root while I'm jenkins. Removing this file resolved my issues.

Ambidex
  • 141