Yrurtj

There is an authentication failure when I'm trying to enter password for any of my accounts after an update using yum. My authentication got broken, while everything else works.

The updates were mostly from pre-existing AWS sources, but I also included an external one. I would like to know what happened, if there is a way how to fix it and perhaps roll back the changes.

I can still log in using ssh (via the pem file) to my AWS EC2 instance, but that's all I can do. I can't login via ssh root@ipaddress using my old password. Changing it on Security Credentials page -> Password doesn't help either. I found a similar problem.

My /var/log/secure:

sshd[1184]: Server listening on 0.0.0.0 port 22.

sshd[1184]: Server listening on :: port 22.

runuser: pam_unix(runuser:session): session opened for user mongod by (uid=0)

runuser: pam_unix(runuser:session): session closed for user mongod

sshd[1576]: Accepted publickey for ec2-user from xxx.xxx.xxx.xxx port xxxxx ssh2: RSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx

sshd[1576]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)

sudo: pam_unix(sudo:auth): authentication failure; logname=ec2-user uid=500 euid=0 tty=/dev/pts/0 ruser=ec2-user rhost=  user=ec2-user

sudo: ec2-user : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/su -

su: pam_unix(su:auth): authentication failure; logname=ec2-user uid=500 euid=0 tty=pts/0 ruser=ec2-user rhost=  user=root

sudo: pam_unix(sudo:auth): authentication failure; logname=ec2-user uid=500 euid=0 tty=/dev/pts/0 ruser=ec2-user rhost=  user=ec2-user

sudo: ec2-user : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/ls

sshd[1184]: Received signal 15; terminating.

sshd[1576]: pam_unix(sshd:session): session closed for user c2-user

My /etc/ssh/sshd_config:

#   $OpenBSD: ssh_config,v 1.28 2013/09/16 11:35:43 sthen Exp $



# This is the ssh client system-wide configuration file.  See

# ssh_config(5) for more information.  This file provides defaults for

# users, and the values can be changed in per-user configuration files

# or on the command line.



# Configuration data is parsed as follows:

#  1. command line options

#  2. user-specific file

#  3. system-wide file

# Any configuration value is only changed the first time it is set.

# Thus, host-specific definitions should be at the beginning of the

# configuration file, and defaults at the end.



# Site-wide defaults for some commonly used options.  For a comprehensive

# list of available options, their meanings and defaults, please see the

# ssh_config(5) man page.



# Host *

#   ForwardAgent no

#   ForwardX11 no

#   RhostsRSAAuthentication no

#   RSAAuthentication yes

#   PasswordAuthentication yes

#   HostbasedAuthentication no

#   GSSAPIAuthentication no

#   GSSAPIDelegateCredentials no

#   GSSAPIKeyExchange no

#   GSSAPITrustDNS no

#   BatchMode no

#   CheckHostIP yes

#   AddressFamily any

#   ConnectTimeout 0

#   StrictHostKeyChecking ask

#   IdentityFile ~/.ssh/identity

#   IdentityFile ~/.ssh/id_rsa

#   IdentityFile ~/.ssh/id_dsa

#   Port 22

#   Protocol 2,1

#   Cipher 3des

#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc

#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160

#   EscapeChar ~

#   Tunnel no

#   TunnelDevice any:any

#   PermitLocalCommand no

#   VisualHostKey no

#   ProxyCommand ssh -q -W %h:%p gateway.example.com

#   RekeyLimit 1G 1h

#

# Uncomment this if you want to use .local domain

# Host *.local

#   CheckHostIP no



Host *

    GSSAPIAuthentication yes

# If this option is set to yes then remote X11 clients will have full access

# to the original X11 display. As virtually no X11 client supports the untrusted

# mode correctly we set this to yes.

    ForwardX11Trusted yes

# Send locale-related environment variables

    SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES

    SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT

    SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE

    SendEnv XMODIFIERS

# Don't show actual hostnames in .ssh/known_hosts

    HashKnownHosts yes

Calling ls -l /etc/passwd /etc/shadow /etc/sudoers results in this:

-rw-r--r-- 1 root  root 1402 Oct  7 10:58 /etc/passwd

---------- 1 root  root  838 Oct  7 10:58 /etc/shadow

-r--r----- 1 root wheel 2299 Oct  7 10:58 /etc/sudoers

My OS config:

NAME="Amazon Linux AMI"

VERSION="2015.09"

ID="amzn"

ID_LIKE="rhel fedora"

VERSION_ID="2015.09"

PRETTY_NAME="Amazon Linux AMI 2015.09"

ANSI_COLOR="0;33"

CPE_NAME="cpe:/o:amazon:linux:2015.09:ga"

HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

Amazon Linux AMI release 2015.09

Command aws ec2 get-password-data --instance-id XXXXXXXXXX results in (not sure if it's applicable for non-Windows instance):

{

    "InstanceId": "XXXXXXXXXX", 

    "Timestamp": "2015-10-12T21:02:51.000Z", 

    "PasswordData": ""

}

Eventually I gained back the lost access to root account by launching a new instance and mounting the volume from the existing one.

sudo mount /dev/sdf /mnt/my-data

In sudo vi /mnt/my-data/etc/ssh/sshd_config I changed the following:

PermitRootLogin yes

#PermitRootLogin forced-commands-only

Then I put the volume back to the original instance and logged in as root through ssh.

sudo ssh -i /key.pem root@ec2-xx-xx-xxx-xxx.xxxx.compute.amazonaws.com

While logged in as root I ran passwd username to change password for each user.