ssh: "Access denied by PAM account configuration" for one non-root user but not another

Question

On a VM I am initializing I am able to log in as one non-root user (admin) but not another (tbbscraper) over SSH with public key authentication. The only error message I can find in any log file is

Sep 18 17:21:04 [REDACTED] sshd[18942]: fatal: Access denied for user tbbscraper by PAM account configuration [preauth]

On the client side, the syndrome is

$ ssh -v -i [REDACTED] tbbscraper@[REDACTED]
...
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: [REDACTED]
debug1: Authentications that can continue: publickey
debug1: Trying private key: [REDACTED]
debug1: read PEM private key done: type RSA
Connection closed by [REDACTED]

Changing 'tbbscraper' to 'admin' allows a successful login: debug1: Authentication succeeded (publickey). appears instead of the "Connection closed" message.

This doesn't seem to be a permissions problem...

# for x in admin tbbscraper
> do ls -adl /home/$x /home/$x/.ssh /home/$x/.ssh/authorized_keys
> done
drwxr-xr-x 3 admin admin 4096 Sep 18 17:19 /home/admin
drwx------ 2 admin admin 4096 Sep 18 16:53 /home/admin/.ssh
-rw------- 1 admin admin  398 Sep 18 17:19 /home/admin/.ssh/authorized_keys
drwxr-xr-x 3 tbbscraper tbbscraper 4096 Sep 18 17:18 /home/tbbscraper
drwx------ 2 tbbscraper tbbscraper 4096 Sep 18 17:18 /home/tbbscraper/.ssh
-rw------- 1 tbbscraper tbbscraper  398 Sep 18 17:18 /home/tbbscraper/.ssh/authorized_keys

# cmp /home/{admin,tbbscraper}/.ssh/authorized_keys ; echo $?
0

... nor a PAM-level access control problem ...

# egrep -v '^(#|$)' /etc/security/*.conf
#

... so none of the existing answers to similar questions would seem to apply. The only other piece of evidence I've got is:

root@[REDACTED] # su - admin
admin@[REDACTED] $

but

root@[REDACTED] # su - tbbscraper
su: Authentication failure
(Ignored)
tbbscraper@[REDACTED] $

which suggests some larger-scale PAM issue, but I can't find anything obviously wrong with the stuff in /etc/pam.d. Any ideas?

The VM is an EC2 instance, OS is Debian 7.1 (Amazon's off-the-shelf AMI).

Answer 1

After all that, it turns out to have been a one-character typo in /etc/shadow. Spot the difference:

admin:!:15891:0:99999:7:::
tbbscraper:!::15966:0:99999:7:::

That's right, there are two colons after the exclamation point on the tbbscraper line. That shoves all the fields over one and makes PAM think that the account expired on January 8, 1970.

Answer 2

Thank you for posting your question. I was getting the same error, but my problem wasn't related to the shadow file. I found my fix and wanted to post an answer as well for anyone else Googling this error. This serverfault question comes up first.

Try checking the /etc/security/access.conf !

We're using Active Directory for authentication, but I needed to login in as a local, non-AD user (jenkins). My boss had originally setup the box with these line in the /etc/security/access.conf:

+:root:ALL
-:ALL:ALL

I changed it to the following and logins now work; I didn't even need to restart any services.

+:jenkins:ALL
+:root:ALL
-:ALL:ALL

Answer 3

In my case I was renaming local CentOS 6 users, and forgot to rename them in /etc/shadow (who are password-less key-authenticated, didn't pop up in my mind), so the records for the new usernames were just absent in /etc/shadow. In /var/log/secure it was giving me unix_chkpwd error and Access denied by PAM:

    unix_chkpwd[12345]: could not obtain user info (user2)
    sshd[12354]: fatal: Access denied for user user2 by PAM account configuration

Answer 4

Had the same error message. Shutted down the sshd and restarted it in debug mode

    /usr/sbin/sshd -ddd

this indicated the reason:

    debug3: User autossh not allowed because account is locked
            ...
    input_userauth_request: invalid user <username> [preauth]

Checked account:

    passwd -S <username>

which showed that the account was locked (flag "L") Un-locked the account by setting a new password:

    passwd <username>

Done.

Answer 5

I got the same problem this morning but the server authenticates users against Active Directory. Turns out the user's domain password had expired.

Answer 6

I had the same issue and none of suggested options worked. But I found in one of forums (https://ubuntuforums.org/showthread.php?t=1960510) a "workaround" which worked perfectly.

Edit /etc/ssh/sshd_config and set

UsePAM no

While it's probably not the real solution, because something is definitely wrong with my machine (yesterday it was working fine!), this one at least works.

Answer 7

The accepted answer helped me find my error. The accepted answer show and extra colon between the 'encrypted password' field and the 'date of last password change' field. See man shadow

user1:!:15891:0:99999:7:::
user2:!::15966:0:99999:7:::

Where the OP's problem shifted the fields by one, mine were in the correct place, but there was an extra field at the end.

user1:<removed-to-shorten-the-line>:19744:0:99999:7:::
user2:<removed-to-shorten-the-line>:19745:0:99999:7::::

Once corrected on the target machine, the ssh session worked as expected.

The usermod utility was not as immediately helpful since the shadow file was in a system image that is pushed to hundreds of nodes. (I could have used the usermod in a chroot environment, but sometimes it is quicker to make the change by hand in the image.)

However, on a node running the "broken" image the error can be detected using the passwd utility.

the problem showed up with this output:

passwd -S user2
user2 P

Once /etc/shadow was corrected:

passwd -S user2
user2 P 01/01/2023 0 99999 7 -1

Answer 8

In my case it was junk hitting ''/etc/tcb/USER/shadow'' after ext4 rootfs corruption in "interesting" conditions; it looked pretty texty so wasn't spotted during initial examination (can't reinstall the node right now but will have to).