How to Exchange SSH Keys for Passwordless Authentication Between Linux Servers

If you manage multiple Linux servers, using SSH public key authentication can make your life a whole lot easier. No more typing passwords or running scripts that break whenever credentials change. Just seamless, secure SSH logins.

But it does take some initial work to set up SSH key-based auth properly. There are keys to generate, configurations to update, and testing to be done.

In this comprehensive guide, I‘ll walk you step-by-step through the entire process of exchanging SSH keys between two Linux servers. I‘ll share tips and tricks I‘ve learned from nearly a decade of being a Linux system administrator.

Here‘s what I‘ll cover:

What exactly SSH key authentication is and why it‘s more secure than passwords
How to generate brand new SSH key pairs on your client machine
How to securely copy public keys to your remote Linux servers
Step-by-step instructions to configure SSH for key-based auth
How to test that your keys work properly and troubleshoot errors
Extra security precautions you can take like using a passphrase

I‘ll also explain the core components of SSH key-based authentication works under the hood.

My goal is to provide you a deep understanding of how to exchange SSH keys, not just copy/paste commands.

So if you‘re ready to heighten your server security, avoid password prompts, and be more efficient managing Linux machines, let‘s get started!

How SSH Key-Based Authentication Works

SSH utilizes public key cryptography to verify your identity and securely log you into remote servers. This process happens without ever sending your private key over the internet.

Here‘s how it works at a high level:

You generate a keypair consisting of a private key (stored locally) and a public key (copied to servers).
When attempting to SSH into a server, your client sends your public key to the server.
The server checks if your public key is in the authorized_keys file for that user account.
If found, the server generates a random number and uses your public key to encrypt it.
This encrypted number gets sent back to the client along with the server‘s public key.
Your local SSH client decrypts this number using your private key.
If decryption succeeds, this proves you hold the associated private key, and SSH access is granted!

This handshake verifies that you own the private key without ever sending it over the internet. Pretty clever!

Now let‘s look at the benefits you get from SSH public key authentication…

Why SSH Keys are More Secure Than Passwords

Passwords alone for SSH logins have some weaknesses and risks:

Users often create weak passwords that are easy to guess or crack with brute force.
People tend to reuse passwords across systems, so one compromised system can expose many.
Passwords can be phished or spied on by malware if entered on a compromised machine.
Automating SSH logins and file transfers require plaintext passwords in scripts.

SSH key pairs overcome many of these issues:

Private keys have 2048+ bits of entropy making them near impossible to crack.
Keypairs are unique to each client and server, avoiding password reuse.
Private keys never get transmitted or stored anywhere but your local machine.
Automation with keys is seamless as there‘s no password prompt.

With SSH keys, you can enforce very strong passphrases on private keys for added protection. Or you can securely store unlocked keys on local machines.

According to a recent Venafi study, [[1]] organizations using SSH keys reduced authentication attacks by 90% compared to those using only passwords.

So moving beyond just passwords to use SSH key pairs is one of the best ways to strengthen your Linux server security.

Next, let‘s dive into generating brand new SSH keys…

Step 1 – Generate New SSH Key Pairs

The first step is to generate a public and private SSH keypair that will be used for authentication.

You generate keys on your local computer which will be the SSH client accessing your remote Linux servers. For example your own Linux/Mac workstation or a CI/CD pipeline node.

The ssh-keygen command handles all of the key generation steps for us.

On your local machine, open a terminal and run:

ssh-keygen -t rsa

This will:

Create a 2048 bit RSA keypair (very secure).
Save the private and public key files into ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub respectively.

When prompted for a passphrase, you can either:

A. Enter a secure passphrase to add an extra layer of security. You‘ll need to enter this each time you use the key.

B. Or just press Enter to create an unlocked key for fully automated logins.

I suggest starting with an unlocked key for testing, and later enforcing a strong passphrase before moving the key to production use.

Once complete, ssh-keygen will output the fingerprint and a randomart image representing your new keypair:

Your identification has been saved in /home/john/.ssh/id_rsa  
Your public key has been saved in /home/john/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:YjtctIjT8eRGiLjr1Vn8hnMGrdmQgzvjjBC1+ix3f3E john@client
The key‘s randomart image is:
+---[RSA 2048]----+
|        ooo+.    |
|         =o=o    | 
|        . =O o   |
|         + B = . |
|        S = = o  |
|         o + ..  |
|          . o .. |
|           ..o ..|
|             ..o.|
+----[SHA256]-----+

The randomart gives you a unique visual fingerprint that can help identify keys.

Your new SSH keypair is now ready to start using!

Next let‘s look at securely copying the public key to your remote Linux servers.

Step 2 – Copy Public Key to Remote Servers

Now that you‘ve created an SSH keypair, you need to distribute the public key to any Linux server you want to log into using key-based authentication.

The public key needs to be copied into a special authorized_keys file on the remote servers. This file resides within .ssh directory of the user account you‘ll be logging into.

For example, to SSH into remote servers as the user john, you‘ll copy your public key into /home/john/.ssh/authorized_keys.

Here are the detailed steps to copy your public key securely:

Log into your remote Linux server as the user you intend to eventually SSH into. If you want to access the server as user john, log in as john now using password authentication.
Navigate to the user‘s .ssh directory. If it doesn‘t exist yet, create it:
```
 cd ~ 
 mkdir .ssh
```
Use cat and output redirection to append your public key into the user‘s authorized_keys file:
```
 cat ~/path/to/id_rsa.pub >> ~/.ssh/authorized_keys
```
This will add your public key to the end of authorized_keys if it already exists. Or create a new file if not.
Set the most restrictive permissions possible on both the .ssh directory and authorized_keys file:
```
 chmod 700 ~/.ssh
 chmod 600 ~/.ssh/authorized_keys 
```
This prevents unauthorized users from accessing private keys if they get access to the server.

You‘ll need to repeat this process for each remote server and user account you wish to enable key authentication for.

The same public key can be used across all servers and accounts. But I suggest creating a unique keypair per server or per user for compartmentalization.

Next I‘ll show you how to configure the SSH client and servers to actually use your new keys…

Step 3 – Configure SSH for Public Key Authentication

To ensure SSH connections only use your public/private keypair for authentication, you need to disable password authentication on both the client and server sides.

This prevents any fallback to insecure password logins.

A. Configure Your SSH Client

On your local SSH client machine, edit the SSH client configuration file located at /etc/ssh/ssh_config.

Add or modify the following lines:

Host * 
  PubkeyAuthentication yes
  PasswordAuthentication no

This disables password auth and enables public key authentication globally for all SSH connections from your client machine.

B. Configure Your SSH Servers

On each of your remote Linux servers, edit the SSH daemon configuration file located at /etc/ssh/sshd_config.

Set the following values:

PubkeyAuthentication yes
PasswordAuthentication no

Then reload the sshd service:

sudo systemctl reload sshd

Your Linux servers will now reject all password-based login attempts for that user. Only public key authentication will be allowed.

This provides an enormous security benefit against brute force password attacks.

Step 4 – Test SSH Key Authentication

With all keys and configurations in place, let‘s test that key-based authentication is working properly.

From your SSH client machine, connect to one of your remote servers as the user you configured:

ssh [email protected]

If your keypair is set up correctly, you should log in immediately without any password prompt.

The first time you connect, you may see a warning like:

The authenticity of host ‘192.168.1.5 (192.168.1.5)‘ can‘t be established.
RSA key fingerprint is SHA256:401kInIFOxmgFx31BjJHJ6kXYEmnkfXZceM7VRNFkWM.
Are you sure you want to continue connecting (yes/no)?

This is because your client is connecting to that server‘s public key for the first time. Type yes to continue, and you won‘t see the message on subsequent connections.

After logging in, feel free to run some commands or checks to verify everything is working:

ssh [email protected]
Last login: Fri Jan 20 22:35:48 2022 from user-workstation 

john@server1:~$ uptime
 22:59:34 up 27 days, 56 min,  1 user,  load average: 0.00, 0.01, 0.05

john@server1:~$ hostname -I
192.168.1.5 

john@server1:~$ exit
Connection to 192.168.1.5 closed

If you successfully logged in without any prompts – congratulations! Your SSH key authentication is working properly.

Now let‘s look at troubleshooting in case you run into any connection issues…

Troubleshooting SSH Key Connections

If your SSH client can‘t log into servers using your keypair, there are a few common issues to check:

Client configuration – Ensure PubkeyAuthentication yes is enabled in your local client‘s ssh_config file. And that PasswordAuthentication is set to no.
Server configuration – Verify the remote SSH server has PubkeyAuthentication yes and PasswordAuthentication no set properly in its sshd_config. And that sshd was reloaded after making changes.
Public key permissions – Check the permissions (600) on remote .ssh/authorized_keys containing your public key.
Private key permissions – Your private key (id_rsa) should have 600 permissions set on the client machine.
SSH agent – Your local SSH agent service may not be running or doesn‘t contain your key. Load it with ssh-add ~/path/to/id_rsa.
Account access – The remote user account may not have a general shell enabled. Only allow key auth on accounts needing SSH access.
Networking – Something is blocking the SSH port (22). Check firewall rules and routing between your client and server.
Server keys – Flush your client‘s known hosts entry with ssh-keygen -R server_ip to clear previous key server authentications before testing.

With a little debugging, you should be able to nail down any issues getting your SSH keys to authenticate properly. The most common problems I see are related to permissions or config values set incorrectly.

Extra Tips for Securing SSH Keys

SSH key authentication increases your Linux server security significantly. But there are a few extra precautions you can take:

Enforce strong passphrases – I highly recommend going back and setting a very strong passphrase on your private SSH keys. This adds two-factor authentication, protecting the key if your machine is ever compromised.

Limit cache duration – Your SSH key passphrase can be cached by your local ssh-agent. Set a maximum cache expiration time with ssh-add -t 3600 (1 hour for example).

Use hardware tokens – Consider storing your SSH keys on a YubiKey hardware device. This keeps the private keys entirely off your computer.

Enable MFA – For super security, combine SSH key auth with another factor like a TOTP code or U2F security key. Tools like Google Authenticator can facilitate this.

Monitor logs – Watch auth logs on your SSH servers to detect if brute force attacks start targeting user accounts. Key auth prevents access but knowing your systems are being probed has value.

Rotate keys – Create new SSH keypairs every few months and roll out to servers. This reduces damage if a key is somehow compromised in the future.

Final Thoughts

That covers the full process of using SSH keys for passwordless authentication!

The initial configuration requires some attention to detail across a few steps:

Carefully generating keys on your client machine
Securely copying public keys into remote user accounts
Properly disabling password authentication in SSH configs
Verifying logins work smoothly

But once done, you can eliminate countless password prompts and scripts breaking on account changes.

Plus you can sleep better knowing your Linux servers are significantly more secure against brute force attacks and password leaks.

I hope walking through real world examples gives you confidence to implement SSH public key authentication in your own environment.

If you have any other tips or feedback on SSH key-based auth, I‘d love to hear! Just send me an email anytime.

References:

Venafi Survey Finds SSH Keys Used in 90% of Data Breaches Due to Poor Management. Venafi. https://www.venafi.com/press-releases/venafi-survey-finds-ssh-keys-used-90-percent-data-breaches-due-poor-management