Understanding and using ssh correctly

Everything you always wanted to know about ssh, ssh keys, the passphrase and ssh agent, but were afraid to ask At least everything you need to know in order to work efficiently, without getting bored to death

A quick introduction

ssh is a program for connecting securely to a remote server and for executing commands on this server

More precisely ssh is an SSH client using the SSH protocol

We assume below that you have a my_login account on the remote remote_server computer, and you know your password

This page will also show some examples using the LSCE, IPSL Mésocentre ESPRI and TGCC servers

Instead of a password, you can also use a set of private and public keys and a passphrase

This is the only kind of authentication that will work if you need to use the IPSL Mésocentre ESPRI (aka ciclad and climserv)!

Many programs are said to work over ssh when they implicitly use the ssh protocol to securely transfer data from one server to another: scp (copy remote directories and files), rsync (synchronize remote directories and files), ...

Some history: before you were born, and the world and internet were a safer place, people used less secure programs like telnet, rlogin, rsh, ftp, ...

Using ssh

Standard usage

The following will work in a Linux terminal, but can also work in a terminal on a Mac or on a Windows 10 computer On Windows 10, ssh is directly available in a Windows Powershell, a Windows Terminal or the old cmd, but the most user-friendly way to use ssh is to use PuTTY

ssh [options] [my_login@]remote_server

If your login is the same on the local and remote computer, you can omit the optional my_login@ part:
e.g. simply use ssh ssh1.lsce.ipsl.fr instead of ssh my_login@ssh1.lsce.ipsl.fr

The first time you connect to a new server, ssh will ask if you are sure of what you are doing, and then store some unique information about the remote server in the known_hosts file (details).

PS C:\Users\my_login> ssh ciclad.ipsl.jussieu.fr
The authenticity of host 'ciclad.ipsl.jussieu.fr (134.157.176.129)' can't be established.
RSA key fingerprint is SHA256:n6wFvMaJuyInd0LNhp78dfMd04Dr751lEekcU7X2UfU.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ciclad.ipsl.jussieu.fr,134.157.176.129' (RSA) to the list of known hosts.
my_login@ciclad.ipsl.jussieu.fr: Permission denied (publickey,hostbased).

ssh will automatically check this security information each time you connect to the same server, and warn you if something seems wrong.

Most common options

-X: enable X11 forwarding. This option will allow you to use graphical programs on the remote server

If -X does not work, use -Y instead (Enable trusted X11 forwarding)

More details in the Using an X server to display graphics section

-A: enable agent forwarding. This is useful when you use ssh keys and an ssh agent

-t command: this option allows you to execute a specific command on the remote server (without displaying the output of the initial ssh). We use this mostly to chain ssh connections, when we want to automatically go through a specific gateway server to access another server
e.g. ssh -A -X my_login@ssh1.lsce.ipsl.fr -t ssh -A -X obelix

-v: verbose mode. Use this option only when you can't connect, or things don't seem to work correctly. Analyzing the verbose output when you start ssh should allow you, or the system administrators, to find out what is wrong

Connecting to the LSCE servers, IPSL servers, TGCC, ...

There are several ways to use ssh to connect to the LSCE obelixNN servers (more details about the available LSCE servers)

If your computer is on the LSCE ethernet/wired network:

Go to the server with the smallest load:
ssh -A -X my_LSCE_login@obelix
or ssh -A -X my_LSCE_login@obelix.lsce.ipsl.fr

Go to a specific obelix (possibly because you have some running processes on this server that you want to monitor with top, or terminate with kill)
ssh -A -X my_LSCE_login@obelix4

If your computer is outside LSCE, or on the LSCE WiFi network, you have to:

Ask your advisor to send a mail to help-lsce, and request an access to the ssh1 server

Go first through the ssh1 gateway server
ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X obelix

If you want to connect to the IPSL servers:

Connecting to ciclad:
ssh -A -X my_ciclad_login@ciclad.ipsl.jussieu.fr

More details

If you want to connect to the the TGCC servers:

Connecting to irene:

Note: you have to go trough ssh1, even if you are on the LSCE network!

ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X my_TGCC_login@irene-ccrt.ccc.cea.fr

The TGCC connection details may vary, depending on your login type

If you have to use ssh regularly (with the appropriate options), you should define the following aliases in the ~/.bashrc configuration file of your local Linux account, or properly configure and use PuTTY on Windows

# Connecting to LSCE from a computer on the LSCE network
alias obelix='ssh -A -X my_LSCE_login@obelix'
# Connecting to LSCE from outside the LSCE network
alias sobelix='ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X obelix'
# Connecting to ciclad @ IPSL
alias ciclad='ssh -A -X my_ciclad_login@ciclad.ipsl.jussieu.fr'
# Connnecting to irene @ TGCC
alias sirene='ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X my_TGCC_login@irene-ccrt.ccc.cea.fr'

If your connection shell is tcsh instead of bash, use the appropriate alias syntax in your ~/.cshrc configuration file,
e.g. alias obelix 'ssh -A -X my_LSCE_login@obelix'

Using an X server to display graphics

A terminal can be used to display text information
e.g. the output of ls and top, the vi editor, etc...
but also to start programs that will open new (graphical) windows outside of the initial terminal
e.g. evince to display pdf files, eog to display png/jpg images, the emacs editor, ferret, etc...

If you want to use ssh to start graphical programs on a remote server, you need to:

Use ssh -X (or ssh -Y if -X does not work) to connect to the remote server

-X: enable X11 forwarding

-Y: enable trusted X11 forwarding (low security, but you trust the remote server)

Using the -X/-Y option will automatically define the DISPLAY environment variable that is required by graphical programs on the remote server to determine where to display the graphical windows.
DISPLAY will not be defined if you forget to use -X/-Y

Example:

my_login@my_local_computer:~$ echo $DISPLAY
localhost:0.0
my_login@my_local_computer:~$ ssh ssh1.lsce.ipsl.fr
Last login: Wed Jul 8 14:45:31 2020 from [...some address...]
[my_login@ssh1 ~]$ echo $DISPLAY
DISPLAY: Undefined variable.
[my_login@ssh1 ~]$ logout
Connection to ssh1.lsce.ipsl.fr closed.
my_login@my_local_computer:~$ ssh -X ssh1.lsce.ipsl.fr
[my_login@ssh1 ~]$ echo $DISPLAY
localhost:43.0

And have a local X server running!
An X server is basically a program running on your local computer that understands the X Windows System protocol used by the remote server to display graphics

Linux computer: nothing to do, an X server is already running !

Windows: install, configure and launch VcXsrv

Mac:

Configuration files

ssh will store all its configuration text files in a .ssh sub-directory of your home directory

Linux: ~/.ssh/ directory

Windows: C:\Users\your_windows_login\.ssh

Mac:

You will find (some of) the following text files:

known_hosts: the text file were ssh stores one line of security information about each server you have connected to from this computer
e.g. ciclad.ipsl.jussieu.fr,134.157.176.253 ssh-rsa AAAAB3NzaC1y[a long identifier...]

config: an optional configuration text file, e.g.

# Empty lines and lines starting with '#' are "comments"
# More details => man ssh_config
ServerAliveInterval=120
ServerAliveCountMax=90

ssh keys related information

authorized_keys: the public key(s) of the account(s) authorized to connect to this account.

the private (and possibly the public) ssh key(s) used on this account

A recommended ssh client for Windows

PuTTY is a convenient and user-friendly ssh client for Windows

Solving common problems

You want to start a graphical program on a remote server, but get a Can't open display: [NO VALUE DISPLAYED HERE] error

$ xterm &
$ xterm: Xt error: Can't open display:
xterm: DISPLAY is not set
$ echo $DISPLAY

The DISPLAY variable is probably not defined because you have not specified the -X (or -Y) option when connecting to the remote server. See Using an X server to display graphics

You want to start a graphical program on a remote server, but get a Can't open display: localhost:[SOME VALUE] error

$ xterm &
$ connect localhost port 6000: Connection refused
xterm: Xt error: Can't open display: localhost:12.0

The DISPLAY variable is defined correctly, but you probably don't have a local X server running. See Using an X server to display graphics

Other types of errors: remember that you can run ssh in verbose mode to help you determine what is wrong (-v option)

Copying files between servers/computers

Sometimes you just need to copy files from one remote server (or your desktop) to the other. The files can be securely copied over ssh with the scp command

Note: if you work with big data files, you should keep the files were they are instead of duplicating them, and move the data processing (your scripts, etc...) to the server where the files are located (e.g. the ciclad server at IPSL)

Copying files with scp

Note: the following will work in a Linux terminal, but can also work in a terminal on a Mac or on a Windows 10 computer (scp is directly available in Windows Powershell, Windows Terminal or the old cmd, but it is not the most user-friendly way to use scp on Windows) If you have a Windows computer, it is much easier to use WinSCP for copying files

scp [options] local_path_or_file(s) [my_login@]remote_server:remote_path
or scp [options] [my_login@]remote_server:remote_path_or_file(s) local_path

If your login is the same on the local and remote computer, you can omit the optional my_login@ part

If you are copying files from a remote server to the current local directory, you can use . instead of the full path of the local directory:

$ cd /some/path
$ scp -p ssh1.lsce.ipsl.fr:/some/remote/path/scatter_regress_example.py .
scatter_regress_example.py 100% 4988 134.6KB/s 00:00

if you have problems using wildcards to specify the files you want to copy, you can use quotes around the path specification:

$ scp -p ssh1.lsce.ipsl.fr:'/some/remote/path/matplotlib/plot_lat_test.*' .
plot_lat_test.eps 100% 43KB 1.0MB/s 00:00
plot_lat_test.pdf 100% 20KB 853.8KB/s 00:00
plot_lat_test.png 100% 77KB 1.5MB/s 00:00

Most common options:

-p: preserves modification times, access times, and modes from the original file.
This option is very useful if you want the copied file(s) to have the same date/time as the original file(s). Otherwise, the time will be the time when you copy the file(s)

-r: recursively copy entire directories.
You have to use this option if the source location is a directory. scp -r will copy the complete content of the directory (including sub-directories)

A recommended graphical scp client for Windows

WinSCP is a convenient and user-friendly scp client for Windows

Synchronizing directories

In some cases, you may want to synchronize the content of directories:

because you are creating a backup

because you have lots of files, possibly (very) big, and you don't want to start copying everything again if the copy fails due to temporary network problems

...

In that case, you should use the rsync command, that will only copy files that are not already in the destination (and that have not changed since the previous copy).

rsync has lots of complex options and rules, and should be used carefully if you do not want to lose files. This page does not cover this topic. Use man rsync or ask somebody

Using ssh keys

What are ssh keys and why use them?

ssh keys are a combination of two specific (and unique) text files, the private key file and the public key file, linked by a special kind of password called the passphrase, that can be used instead of a standard password to connect securely from one server to another server

ssh keys have to be configured properly (a few easy steps), and are very convenient because:

They usually don't expire!
You don't have to change them (except in some extra secure computing centers like TGCC) and you can keep them for years

They don't depend on the accounts and the passwords of the servers where you use them

You can (and should!) use the same set of ssh keys on several servers, and you can then connect to these servers just using the same passphrase, rather than memorizing different passwords
e.g. if you have your private key on account_A of server_A and install the matching public key on account_B of server_B, etc... you can then use ssh on account_A@server_A to access account_B@server_B, account_C@server_C, ... with the same passphrase !

You can give your public key to somebody and then access their account using your own passphrase (no need to know the password of the other person)

The IPSL Mésocentre ESPRI servers can only be accessed with a public key and passphrase (the password is not used)

By default, ssh will ask you to type your passphrase each time you connect to a server, but you can use an ssh agent to securely store your passphrase for you
Once you have typed your passphrase in the ssh agent, you can connect to all the servers that have your public key without having to type your passphrase!

scp (and WinSCP) and the tools using ssh on your local computer will not ask your passphrase, if they find the passphrase in a running ssh agent on the local computer

if you use the -A option (agent forwarding), the remote server will also know (securely) your passphrase, and you will not have to type the passphrase when using ssh, scp and tools running over ssh on the remote server(s)

the local ssh agent is terminated when you log out of your local computer (or reboot it)

Generating ssh keys

Some common sense advice

Generate only one pair of private/public keys and use the same pair of keys everywhere!
Put differently, do not generate a different pair of key on each computer/server you use (even if you always use the same passphrase)!

Do not use an empty passphrase!
If you do that, somebody gaining access to your private key will be able to access all the accounts where you have installed your public key

Keep a backup of your your keys outside of the computer where they were generated

Useful if you erase or overwrite the keys by mistake, or if you move to another lab and use a new computer/account, but still need to access the accounts where you have installed your public key...

If you have not used an empty passphrase, and have not saved the passphrase in a file with the keys, the keys can't be used easily by somebody else to gain access to your accounts

Do not forget your passphrase!
Do not write your passphrase on a postit taped to your computer. When you create your keys and type your passphrase, choose something that you will be able to remember during several years

Generating keys in a terminal

Remember that if you already have a pair of keys, you probably don't want to generate a new pair, unless you have been asked to, or have lost one of the keys, or forgotten your passphrase. If you generate a new pair of keys, you will probably have to replace the old keys that you were using on all the remote servers

There are several ways to generate pairs of ssh keys with ssh-keygen. The following one is the one recommended for opening an account on IPSL Mésocentre ESPRI. If you open an account on ciclad, but already have a public key, just send your existing public key!

Type ssh-keygen -t rsa -b 4096

Accept the default path and key name

Do not specify an empty passphrase!

This will generate two text key files in a sub-directory of your account (~/.ssh/ on Linux, C:\Users\my_login\.ssh\ on Windows 10):

The private key, that has to be readable only by you: id_rsa

> cd ~/.ssh
> ls -l id_dsa
-rw------- 1 my_login my_group some_date id_rsa
> cat id_rsa
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,906569054A4C58A28AD23CBA28771EDE
C/Aacy+qcSWIG56eWc3XQhm2oyfAVKFKVm54pwoCmIZ5nmLx/8kV8XcDcMHxoWIz
xgc3cPwxNczIS/i4A0AOk3uI8JiT8RVLELVbn+B5T0ewbvMjln4Ec/7W9+aNe/NF
[ lots of literally cryptic lines ]
v/rj1Ze/PEQ+nVX3dh3FB1TaL/aNm48PBP9WQQXm011PY6isZJklyWANGJ6jtOf9
-----END RSA PRIVATE KEY-----

The public key: id_rsa.pub
This is the information that you can share. Note that the my_login@my_machine at the end of the line is just some information about who generated the keys, and where, and can be removed or replaced by something more informative

> cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQ [ lots of cryptic characters ] 8WPbpreOOrIbNw== my_login@my_machine

Generating or importing keys with PuTTY on a Windows computer

Read the Importing or creating ssh keys with PuTTYgen section

Installing ssh keys

Using the keys

Using an ssh agent

More...

If you want to know more (options, etc...), check the man(ual) page on Linux: man ssh

Editing remote text files with emacs (implicit scp): /user@server:/path/file

Use the tramp lines from the recommended .emacs file

Digging ssh tunnels when using ipython notebooks

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]