Managing Multiple SSH Keys on Linux: A Beginner’s Complete Guide

What Are SSH Keys and Why Do You Need Multiple Ones?

SSH (Secure Shell) keys are like digital keys that allow you to securely connect to remote servers and services without typing passwords every time. Think of them as a more secure alternative to passwords - they come in pairs: a public key (which you share) and a private key (which you keep secret).

You might need multiple SSH keys when you:

  • Work with different GitHub accounts (personal and work)
  • Connect to various AWS instances
  • Access multiple servers or cloud services
  • Separate keys for security reasons (different keys for different purposes)

Step 1: Understanding the SSH Directory Structure

Before we create keys, let’s understand where SSH stores everything on your Linux system:

1
2
3

# Navigate to your SSH directory
cd ~/.ssh
ls -la

You’ll typically see files like:

  • id_rsa and id_rsa.pub (your default private and public keys)
  • config (configuration file we’ll create)
  • known_hosts (stores server fingerprints)

Step 2: Creating Your First SSH Key Pair

Let’s start by creating SSH keys for different services. We’ll begin with a GitHub key:

1
2
3
4
5
6

# Create a new SSH key for GitHub
ssh-keygen -t rsa -b 4096 -C "[email protected]" -f ~/.ssh/id_rsa_github

# When prompted:
# - Enter a secure passphrase (recommended) or press Enter for no passphrase
# - Confirm the passphrase

The -f flag specifies the filename, making it easy to identify which key is for which service.

Step 3: Creating Additional Keys for Different Services

Now let’s create keys for other services:

1
2
3
4
5
6
7
8

# For AWS
ssh-keygen -t rsa -b 4096 -C "aws-access" -f ~/.ssh/id_rsa_aws

# For a work GitHub account
ssh-keygen -t rsa -b 4096 -C "[email protected]" -f ~/.ssh/id_rsa_github_work

# For a personal server
ssh-keygen -t rsa -b 4096 -C "personal-server" -f ~/.ssh/id_rsa_personal_server

After creating these keys, your ~/.ssh directory will contain:

1
2
3
4
5
6
7
8

id_rsa_github (private key)
id_rsa_github.pub (public key)
id_rsa_aws (private key)
id_rsa_aws.pub (public key)
id_rsa_github_work (private key)
id_rsa_github_work.pub (public key)
id_rsa_personal_server (private key)
id_rsa_personal_server.pub (public key)

Step 4: Setting Proper Permissions

SSH is very strict about file permissions. Let’s set them correctly:

1
2
3
4
5
6
7
8

# Set permissions for private keys (readable only by you)
chmod 600 ~/.ssh/id_rsa_*

# Set permissions for public keys
chmod 644 ~/.ssh/id_rsa_*.pub

# Set directory permissions
chmod 700 ~/.ssh

Step 5: Creating the SSH Config File

This is where the magic happens. The SSH config file tells your system which key to use for which service:

1
2

# Create or edit the config file
nano ~/.ssh/config

Add the following configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# GitHub Personal Account
Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_rsa_github
    IdentitiesOnly yes

# GitHub Work Account
Host github-work
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_rsa_github_work
    IdentitiesOnly yes

# AWS Instance
Host aws-server
    HostName your-aws-instance-ip-or-domain
    User ec2-user
    IdentityFile ~/.ssh/id_rsa_aws
    Port 22

# Personal Server
Host personal-server
    HostName your-server-ip-or-domain
    User your-username
    IdentityFile ~/.ssh/id_rsa_personal_server
    Port 22

Step 6: Adding Public Keys to Services

Now you need to add your public keys to the respective services:

For GitHub:

1
2

# Display your GitHub public key
cat ~/.ssh/id_rsa_github.pub

Copy this output and add it to your GitHub account at Settings → SSH and GPG keys → New SSH key.

For AWS:

1
2

# Display your AWS public key
cat ~/.ssh/id_rsa_aws.pub

Add this to your AWS EC2 Key Pairs or directly to the server’s ~/.ssh/authorized_keys file.

For your personal server:

1
2

# Copy the public key to your server
ssh-copy-id -i ~/.ssh/id_rsa_personal_server.pub username@your-server-ip

Step 7: Using Your SSH Keys

Now that everything is set up, here’s how to use your different keys:

Connecting to GitHub:

1
2
3
4
5

# Personal GitHub (uses default github.com host)
git clone [email protected]:username/repository.git

# Work GitHub (uses custom github-work host)
git clone git@github-work:company/repository.git

Connecting to servers:

1
2
3
4
5
6
7
8

# Connect to AWS instance
ssh aws-server

# Connect to personal server
ssh personal-server

# Or connect directly using the key
ssh -i ~/.ssh/id_rsa_aws ec2-user@your-aws-ip

Step 8: Managing the SSH Agent

The SSH agent manages your keys in memory, so you don’t have to enter passphrases repeatedly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Start the SSH agent
eval "$(ssh-agent -s)"

# Add your keys to the agent
ssh-add ~/.ssh/id_rsa_github
ssh-add ~/.ssh/id_rsa_aws
ssh-add ~/.ssh/id_rsa_github_work
ssh-add ~/.ssh/id_rsa_personal_server

# List loaded keys
ssh-add -l

# Remove all keys from agent
ssh-add -D

Step 9: Testing Your Connections

Always test your SSH connections to ensure everything works:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Test GitHub personal
ssh -T [email protected]

# Test GitHub work
ssh -T git@github-work

# Test AWS server
ssh aws-server

# Test with verbose output for troubleshooting
ssh -v aws-server

Step 10: Troubleshooting Common Issues

Issue 1: “Permission denied (publickey)”

Solution: Check if your public key is added to the server and your private key has correct permissions:

1

chmod 600 ~/.ssh/id_rsa_your_key

Issue 2: Wrong key being used

Solution: Add IdentitiesOnly yes in your SSH config to prevent SSH from trying all keys.

Issue 3: Can’t connect to GitHub with work account

Solution: Make sure you’re using the correct host alias:

1
2
3
4
5

# Wrong
git clone [email protected]:company/repo.git

# Correct
git clone git@github-work:company/repo.git

Step 11: Advanced Tips and Best Practices

Organizing Your Keys

Create a naming convention that makes sense:

  • id_rsa_github_personal
  • id_rsa_github_work
  • id_rsa_aws_production
  • id_rsa_aws_development

Backup Your Keys

1
2
3

# Create a secure backup of your SSH directory
tar -czf ssh_backup.tar.gz ~/.ssh
# Store this backup in a secure location

Use Different Key Types

For enhanced security, consider using Ed25519 keys:

1

ssh-keygen -t ed25519 -C "[email protected]" -f ~/.ssh/id_ed25519_github

Automate SSH Agent Loading

Add this to your ~/.bashrc or ~/.zshrc:

1
2
3
4
5
6

# Auto-start SSH agent
if [ -z "$SSH_AUTH_SOCK" ]; then
    eval "$(ssh-agent -s)"
    ssh-add ~/.ssh/id_rsa_github
    ssh-add ~/.ssh/id_rsa_aws
fi

Step 12: Quick Reference Commands

Here’s a handy reference for daily use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# List all keys in SSH directory
ls -la ~/.ssh/

# Test connection with specific key
ssh -i ~/.ssh/id_rsa_specific_key user@host

# Add key to SSH agent
ssh-add ~/.ssh/id_rsa_key_name

# View SSH config
cat ~/.ssh/config

# Test GitHub connection
ssh -T [email protected]

# Clone with specific SSH config host
git clone git@custom-host:user/repo.git

Conclusion

You now have a complete setup for managing multiple SSH keys on Linux! This setup allows you to:

  • Seamlessly switch between different GitHub accounts
  • Connect to various AWS instances with different keys
  • Maintain security by using separate keys for different purposes
  • Easily manage and organize your SSH connections

Remember to keep your private keys secure, use strong passphrases, and regularly backup your SSH configuration. With this setup, you’ll never have to worry about SSH key conflicts or security issues when working with multiple services.

The key to success with multiple SSH keys is organization and consistent naming conventions. As you add more services and servers, simply follow the same pattern: create the key pair, add the configuration to your SSH config file, and test the connection.