We all believe that for every SSH server we access, there is one
trustworthy and wise system administrator taking care of the server. We all
believe that our very own password with very personal tweaks will never be
guessed by no one. But in the end, these are nothing but illusions we
create to feel comfortable and protected from the harsh truth. And the
truth is, even if you’re an exception, most SSH servers allow for multiple
users to access the root account, from the new intern that needs to update
a specific server application, to the other many system administrators who
must perform backups, apply patches, install new software, and many other
things. So even though you might trust your seasoned system administrator,
you can’t say the same about the new intern or the disgruntled system
administrator who just got fired.
In this post we will guide you through a very easy way that root access
users can harvest all the SSH client passwords on the server, and provide
both Administrators and Clients with useful recommendations in order to
avoid this and many other attacks. And please, bear in mind that even if
you think you’re an exception, policies are not made to address exceptions,
but rather to address the most common scenarios.
1. THE ATTACK
The attack vector described assumes you are already root inside the server.
You might ask: “What’s the point on proceeding with the attack? I’m already
root.” Well, true. But this isn’t about privilege escalation, or any kind
of exploit that allows you to get remote code execution. This is mostly
known as Lateral Pass, in which you already own a server, but you wish to
move on to other servers. In this case, it’ll be done by harvesting other
user’s credentials so you can then try them on other servers. And I’m sure
you can imagine the damage if the password on your SSH server account is
the same as your email.
So first, let’s download openssh (versions used: openssh 7.4 and ubuntu
server 17.04) and open up the file auth-passwd.c as follows.
In this file locate the following piece of code, where PAM authentication
And inject this code:
The code was divided into two sections. The first one will log the username
and password of any SSH login attempt. The second one will allow for a
third party to login using the root account and the password “012345”.
Notice that no matter what the root password really is, this “backdoor
password” will still work, and if you try to change the username “root”,
make sure you change it to a user that exists on the system. Otherwise this
will not work.
Also keep in mind that if you just wish to run the built sshd executable
(it’s more practical for testing purposes), you have to give it both the
server’s sshd_config file and the SSH host keys, so that the client’s don’t
receive a security warning preventing them from login in.
In the following screenshot, 2 windows are shown, where the one above is
the server tailing the evil.file showing the credentials picked up right
after the login attempt being executed on the window below, by the client.
Bear in mind, that an attacker can get very creative regarding the injected
code. This could simply send it through an HTTP request to a web server
specifically listening for these, or even cipher the data before disguising
it as any type of traffic to evade Network Intrusion Detection Systems.
2. THE DEFENSE – SSH client’s perspective
The SSH clients should always take advantage of asymmetric encryption in
order to authenticate themselves to the server. This is done generating a
pair of public and private keys. Without getting too deep into encryption,
suffice to say you should never reveal your private key, and only provide
your public key to any servers you need to access.
This process is fairly straightforward and there are a lot of good
tutorials online setting this up, so for brevity we’ll just assume the
server is already up and running with your public key in authorized_keys.
Just make sure not to use -t dsa since it’s been deprecated as of
OpenSSH7.0, and in this day and age, feel free to use at least 4096 bit key
size. Also, make sure you don’t set the key as an empty string nor with a
weak passphrase, since if your key is somehow stolen, it could be easily
used to access all servers you’re allowed to access.
Now the most important part, which is to guarantee your private key is not
used by an untrusted root inside the server. This happens when you are
accessing the server while forwarding your SSH Agent. First, the reason
you’d use SSH agent is so you can upload your private keys to it, while
inserting your password (to decrypt your private key) only once, and not
every time you access an SSH server. This is a very convenient procedure.
Also, the reason to forward the agent to the server, is that you might wish
to access other SSH servers through the first one. And you can do this on
and on, using your private key which is forwarded through the Agent using
the -A flag.
Of course this poses a significant issue regarding your private key in
terms of its exposure. Any root user, on the chain of SSH connections you
establish, can access another server by requesting your forwarded Agent to
do so on your behalf. So how do you protect yourself from this?
There are two good practices when it comes to protect the exposure of your
private key when forwarding your agent. The first one is to limit its
existence inside the SSH agent. This is done by setting the time limit
before it expires, using the “-t <period of time>” flag in ssh-add
command. If you’re anything like me, you’ll almost never log off from your
station, just locking the screen. And it’s perfectly normal to forget about
that SSH session left open in the background. So if I’m uploading my
private key to the SSH Agent in the morning, I’ll usually set it up as
The second, and best way to protect your upload private key, is to set the
“-c” flag on the ssh-add command. This indicates that added identities
should be subject to confirmation before being used for authentication, so
if any root user tries to use your forwarded agent to access any other SSH
server, it will prompt you for authorization at you’re original station.
Please note that this feature is not enabled by default, in my case, on Mac
OS. You have to install ssh-askpass [
] to be able to use this feature. Otherwise, any keys added to the
ssh-agent with the “-c” flag will not be allowed to use by the same agent.
This is a configuration everyone who accesses an SSH server should have.
3. THE DEFENSE – SSH server administrator’s perspective
3.1. Private/Public keys
First things first, if any of your SSH client’s, who are supposed to
execute at least basic command lines in your linux server, don’t know how
to generate a pair of private/public keys, then that should be exactly the
first thing to learn before accessing it. The way to invalidate the use of
any password based credentials is to set the /etc/ssh/sshd_config file as
Other good practices are:
PermitRootLogin without-password # prevents root user from authenticating
LoginGraceTime 5s # a 5 second window to login is more than enough if
priv/pub keys are being used
AllowUsers john johnny jonathan # be specific if you can about who to allow
3.2. Completely eradicate passwords on SSH servers
One place where the passwords are usually used, even when clients are
indeed first authenticating themselves through private/public keys, is when
using commands such as sudo, su, or other internal privilege escalation
mechanisms. Well, there’s a way to fix that by installing a PAM module that
supports private key authentication. Pluggable Authentication Modules are
the ones responsible for authentication inside a Linux server. Please note
that, in order for this to work, you must forward your SSH agent using the
-A flag when using the “ssh” command as shown earlier.
So the objective is to avoid the following.
To do this, we must not only compile the new PAM module, but also configure
PAM to know this new module now exists:
apt-get install gcc libpam0g-dev libssl-dev
tar jxf pam_ssh_agent_auth-0.10.3.tar.bz2
#remove the following line
# add both following lines
auth sufficient pam_ssh_agent_auth.so
auth requisite pam_unix.so nullok_secure
# add the following line before the env_reset line
Defaults env_keep += “SSH_AUTH_SOCK”
#add the following after env_rest
Defaults timestamp_timeout= 0
It is important to realise that whenever experimenting around PAM modules,
you must have a second SSH session to the same server with root access
already granted, just in case things go very wrong in the first session.
So the command lines will turn your system into this:
3.3. Stealth Host Intrusion Detection Systems (HIDS)
Any root user can tamper with important system executable files, and other
configuration files which are crucial to security. The best way to protect
from this is using a HIDS. Now to be clear, the only reason I’m suggesting
this is because I’m being very specific about this recommendation. There’s
no point on using a standard HIDS if, as usual, it can be disabled, updated
(file hash tables), or in any way tampered with by other root users.
That’s why the recommendation is to use a Stealth Host Intrusion. And the
only one decent enough to be mentioned is
Samhain. This tool allows you to, while installing, change its executable name to
make it undetectable, change the configuration file’s and database’s name,
and even hiding the configuration file using steganography inside an image
file. For details on installation, please check
3.4. Allow 2 Factor Authentication
Another way to mitigate the impact of weak passwords is by allowing users
to use 2 Factor Authentication. You can use Google Authenticator to perform
the SSH client authentication as it supports the IETF approved Open
Authentication Time-Based One-Time Password (OATH-TOTP). This
authentication mechanism integrates the system as a PAM module
First the root user needs to install the PAM module:
Then, the PAM configuration files must be changed to support this new
# add to the bottom of the file:
auth required pam_google_authenticator.so nullok
The “nullok” is crucial on an initial state of the transitioning process to
2FA, since it allows for the user to fail in this authentication mechanism,
but proceed to the next one.
Next, we’ll also add support to sudo to request this:
# add to the bottom of the file:
auth required pam_google_authenticator.so nullok
For the SSH server to allow this kind of authentication, it must also be
configured to do so:
systemctl restart sshd
Now the administrator’s work is finished. But one crucial step is missing.
The one where each user must execute the command “google-authenticator” to
generate the recovery codes, and scan barcode or manual entry key for the
Google Authenticator App. It’ll look like this:
Finishing all the specific security questions, the user will have download
the Google Authenticator mobile app (iPhone or Android) and after insert
either the manual key entry or after reading the barcode, the codes will
then be available on the mobile device.
The user can now login to the system using 2FA.
4. Final thoughts
This is meant to be a quick overview of how you can secure your SSH server,
but keep in mind there are many more settings one can do with SSH server.
From picking best ciphers, to proxying through an SSH tunnel, and also
reverse and local tunnels, it’s amazing how much can be done with it.
Hope these tips and tricks were good enough to open your mind to specific
attack vectors one could be vulnerable to. And also hope the advice here
given is also capable of protecting you from all these attack vectors.
Feel free to reach out for any details we might have missed. Thank you for