10

Using Rocky 9.4, OpenSSH 8.7p1, I have an RSA keypair which is not accepted unless sshd is in debug mode. (Thankfully I have an old pair that works).

I have unset SSH_AUTH_SOCK and verified that no agent is involved. I have moved ~/.ssh/config to avoid that complication. I have verified that ~/.ssh/ is mode 0700, id_rsa and authorized_keys are 0400, known_hosts is 0600. I have confirmed that the correct public key is in authorized_keys, and verified using 'ssh-keygen -l'.

When sshd is running as a service and I run 'ssh -avv -i ~/.ssh/id_rsa localhost', I see:

Offering public key:  (with the correct filename and SHA256 fingerprint)
we sent a publickey packet, wait for reply
Authentications that can continue:  publickey, ...
we did not send a packet, disable method
Next authentication method: keyboard-interactive

... and I am prompted for my regular password.

/var/log/secure shows:

Connection from 127.0.0.1 ...
Failed publickey for (user) and shows the same correct SHA256 code.

I found the advice that I should stop the sshd service, and run manually in debug mode. I opened a second session as root and ran:

systemctl stop sshd
/usr/sbin/sshd -D -ddd -e

When I repeat the 'ssh -avv -i ~/.ssh/id_rsa localhost' command in the original user window, I see:

Offering public key:  (with the correct filename and SHA256 fingerprint)
we sent a publickey packet, wait for reply
Server accepts key:   (with the correct filename and SHA256 fingerprint)
Enter passphrase for key '(filename)':  (I type passphrase)
Authenticated to 127.0.0.1 (via proxy) using "publickey".

... and I've successfully logged in using the same key.

I was hoping sshd -D -ddd would help me figure out the issue, but it happily accepts my key.

Update 2024-04-30

Taking @mircea-vutcovici suggestion, setting LogLevel: DEBUG for sshd generated different sequences of messages when trying the different keys: the failed try included a message that authorized_keys was not readable despite all keys being in the same file and the same sshd instance!

My homedir is an NFS share and SELinux is being enforced, so did some more searching using @grawity suggestion: ls -LZ produced system_u:object_r:nfs_t:s0 which is a generic SELinux label. Apparently this is a problem if the NFS mount precedes SELinux policy load patchwork: selinux: make labeled NFS work when mounted before policy load. This issue was fixed in 2023, and should? have made it into Rocky 9.4 (mid-2024). That said, /home is mounted with NFS vers=3 FWIW.

I eventually turned up a useful incantation serverfault: Wrong SELinux labels on NFS homes? : setsebool -P use_nfs_home_dirs on. I can now use both keys, but still cannot understand why they were treated differently given both public keys are in the same file and both private keys have same permissions in same folder.

Just bumped into a similar question from a 13 years ago: public key authentication fails ONLY when sshd is daemon. Apparently SELinux is always the issue :-) I'm going to credit @grawity for an answer that sent me into a good rabbit hole.

ljcorsa
  • 103

2 Answers2

15

My wild guess is that you have SELinux enabled and the authorized_keys file has the wrong label, so it is readable by 'sshd' started from within your interactive user session, but not readable by 'sshd' started from 'init', as they run in different SELinux contexts.

That is to say, starting sshd manually with -d affects other things besides just "debug mode". (To minimize the differences, debug mode should instead be enabled through sshd_config, as in Mircea Vutcovici's post.)

Inspect the file with ls -lZ, then run restorecon -v on it (which I think is the correct way of fixing it?), then ls -lZ it again.

grawity
  • 17,092
11

Make sure you have your public key in ~/.ssh/authorized_keys file of the remote user. Each line should have one single key.

Edit /etc/ssh/sshd_config file, then change the line:

#LogLevel INFO

to something like:

LogLevel DEBUG

The possible values are: QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG, DEBUG1, DEBUG2, and DEBUG3.

Then restart the sshd service:

systemctl status sshd.service
systemctl restart sshd.service
systemctl status sshd.service

If you need more details you can strace sshd while running as daemon:

sudo strace -s300 -fyyTttp $(pgrep -f 'sshd -D') -o strace-sshd-$HOSTNAME-$(date +%F_%H%M%S).txt`

This will generate a trace file in the local folder. Check to see what is happening in the trace file.

Mircea Vutcovici
  • 19,011