Understanding and using ssh correctly

• ssh is a program for connecting securely to a remote server and for executing commands on this server
  • More precisely ssh is an SSH client using the SSH protocol

• We assume below that you have a my_login account on the remote remote_server computer, and that you know your password
  • This page will also show some examples using the LSCE, IPSL Mésocentre ESPRI and TGCC servers

• Instead of a password, you can also use a set of private and public keys and a passphrase
  • This is the only kind of authentication that will work if you need to use the IPSL Mésocentre ESPRI (aka ciclad and climserv)!

• Many programs are said to work over ssh when they implicitly use the ssh protocol to securely transfer data from one server to another: scp (copy remote directories and files), rsync (synchronize remote directories and files), …

• Some history: before you were born, and the world and internet were a safer place, people used less secure programs like telnet, rlogin, rsh, ftp, …

• ssh [options] [my_login@]remote_server
  • If your login is the same on the local and remote computer, you can omit the optional my_login@ part:
    e.g. simply use ssh ssh1.lsce.ipsl.fr instead of ssh my_login@ssh1.lsce.ipsl.fr
  • The first time you connect to a new server, ssh will ask if you are sure of what you are doing, and then store some unique information about the remote server in the known_hosts file (details).
    

    PS C:\Users\my_login> ssh ciclad.ipsl.jussieu.fr
    The authenticity of host 'ciclad.ipsl.jussieu.fr (134.157.176.129)' can't be established.
    RSA key fingerprint is SHA256:n6wFvMaJuyInd0LNhp78dfMd04Dr751lEekcU7X2UfU.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'ciclad.ipsl.jussieu.fr,134.157.176.129' (RSA) to the list of known hosts.
    my_login@ciclad.ipsl.jussieu.fr: Permission denied (publickey,hostbased).

    
    ssh will automatically check this security information each time you connect to the same server, and warn you if something seems wrong.

• -X or -Y options if you will need to use graphical programs on the remote server
  • All the Details in the Using an X server to display graphics section

• -A: enable agent forwarding. This is useful when you use ssh keys and an ssh agent

• -t command: this option allows you to execute a specific command on the remote server (without displaying the output of the initial ssh). We use this mostly to chain ssh connections, when we want to automatically go through a specific gateway server to access another server
  e.g. ssh -A -X my_login@ssh1.lsce.ipsl.fr -t ssh -A -X obelix

• -v: verbose mode. Use this option only when you can't connect, or things don't seem to work correctly. Analyzing the verbose output when you start ssh should allow you, or the system administrators, to find out what is wrong

There are several ways to use ssh to connect to the LSCE obelixNN servers (more details about the available LSCE servers)

• If your computer is on the LSCE ethernet/wired network:
  • Go to the server with the smallest load:
    ssh -A -X my_LSCE_login@obelix
    or ssh -A -X my_LSCE_login@obelix.lsce.ipsl.fr
  • Go to a specific obelixNN (possibly because you have some running processes on this server that you want to monitor with top, or terminate with kill)
    e.g. ssh -A -X my_LSCE_login@obelix4
• If your computer is outside LSCE, or on the LSCE WiFi network, you have to:
  • Ask your advisor to send a mail to help-lsce, and request an access to the ssh1 server
  • Connect through the ssh1 gateway server, i.e you first use ssh to connect to the ssh1 gateway, and then use ssh to go to an obelix server
    These two ssh commands can be conveniently chained with the '-t' option!
    ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X obelix

If you want to connect to the IPSL servers (only possible with ssh keys!):

• Requesting an account at IPSL
• Connecting to spirit1:
  • ssh -A -X my_meso_login@spirit1.ipsl.fr
  • Depending on what you need to do, you can also use spirit2, spiritx1 or spiritx2
• More details
• Note: the ciclad server may still be accessible when you read this page, but its usage has been deprecated in favor of the spirit servers

If you want to use the TGCC computers (e.g. irene):

• Note: you have to go trough the ssh1 LSCE gateway to access the TGCC, even if you are on the LSCE wired network!
• ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X my_TGCC_login@some_tgcc_login_node
  • Once you are on a TGCC login node (e.g. irene), you can get lots of information by typing <node>.info (e.g. irene.info)
• more TGCC connection details

If you have to use ssh regularly (with the appropriate options), you should define convenient shell aliases, or add PuTTY connection profiles on Windows

Define the following aliases in the ~/.bashrc configuration file of your local Linux account

# Connecting to LSCE from a computer on the LSCE network
alias obelix='ssh -A -X my_LSCE_login@obelix'

# Connecting to LSCE from outside the LSCE network
alias sobelix='ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X obelix'

# Connecting to ciclad @ IPSL
alias ciclad='ssh -A -X my_ciclad_login@ciclad.ipsl.jussieu.fr'

# Connnecting to irene @ TGCC
alias sirene='ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X my_TGCC_login@irene-ccrt.ccc.cea.fr' 

If your connection shell is tcsh instead of bash, use the appropriate alias syntax in your ~/.cshrc configuration file,
e.g. alias obelix 'ssh -A -X my_LSCE_login@obelix'

A terminal can be used to display text information
e.g. the output of ls and top, the vi editor, etc…
but also to start programs that will open new (graphical) windows outside of the initial terminal
e.g. evince to display pdf files, eog to display png/jpg images, the emacs editor, ferret, etc…

If you want to use ssh to start graphical programs on a remote server, you need to:

• Use ssh -X (or ssh -Y if -X does not work) to connect to the remote server
  • -X: enable X11 forwarding
  • -Y: enable trusted X11 forwarding (low security, but you trust the remote server)
  • Using the -X/-Y option will automatically define the DISPLAY environment variable that is required by graphical programs on the remote server to determine where to display the graphical windows.
    DISPLAY will not be defined if you forget to use -X/-Y
    
  • Example:
    

    my_login@my_local_computer:~$ echo $DISPLAY
    localhost:0.0
    
    my_login@my_local_computer:~$ ssh ssh1.lsce.ipsl.fr
    Last login: Wed Jul  8 14:45:31 2020 from [...some address...]
    [my_login@ssh1 ~]$ echo $DISPLAY
    DISPLAY: Undefined variable.
    [my_login@ssh1 ~]$ logout
    Connection to ssh1.lsce.ipsl.fr closed.
    
    my_login@my_local_computer:~$ ssh -X ssh1.lsce.ipsl.fr
    [my_login@ssh1 ~]$ echo $DISPLAY
    localhost:43.0

• And have a local X server running!

ssh will store all its configuration files in a .ssh sub-directory of your home directory. The configuration files are in a text format.

• Linux: ~/.ssh/ directory
• Windows: C:\Users\your_windows_login\.ssh directory
• Mac: /Users/your_mac_login/.ssh directory (should be the same path as ~/.ssh/)

You will find (some of) the following text files:

• known_hosts: the text file were ssh stores one line of security information about each server you have connected to from this computer
  e.g. ciclad.ipsl.jussieu.fr,134.157.176.253 ssh-rsa AAAAB3NzaC1y[a long identifier…]

• config: an optional configuration text file, e.g.
  

  # Empty lines and lines starting with '#' are "comments"
  # More details => man ssh_config
  
  ServerAliveInterval=120
  ServerAliveCountMax=90

• ssh keys related information:
  • authorized_keys: the public key(s) of the account(s) authorized to connect to this account.
  • the private (and probably the public) ssh key(s) used on this account
    • e.g. id_ed25519 and id_ed25519.pub files

PuTTY is a convenient and user-friendly ssh client for Windows

A Terminal window will open a local Linux-like shell session on the Mac, where you can use ssh to connect to another server, or other standard Linux commands

• Built-in: Terminal application (available in /Applications/Utilities)
  • Terminal User Guide
• Recommended: the iTerm2 application
  • Improved Favorite Sessions settings

• X server related errors

• Other types of errors: remember that you can run ssh in verbose mode to help you determine what is wrong (-v option)

Sometimes you need to copy files from one remote server (or your desktop) to the other. The files can be securely copied over ssh with the scp command

Note: you should keep the big data files were their original version is instead of duplicating them, and move the data processing (your scripts, etc…) to the server where the original files are located (e.g. the ciclad server at IPSL)

• scp [options] local_path_or_file(s) [my_login@]remote_server:remote_path
  or scp [options] [my_login@]remote_server:remote_path_or_file(s) local_path
  • If your login is the same on the local and remote computer, you can omit the optional my_login@ part
  • If you are copying files from a remote server to the current local directory, you can use . instead of the full path of the local directory:
    

    $ cd /some/path
    $ scp -p ssh1.lsce.ipsl.fr:/some/remote/path/scatter_regress_example.py .
    scatter_regress_example.py                    100% 4988   134.6KB/s   00:00

  • if you need to use wildcards to specify the files you want to copy, you can use quotes around the path specification:
    

    $ scp -p ssh1.lsce.ipsl.fr:'/some/remote/path/matplotlib/plot_lat_test.*' .
    plot_lat_test.eps                             100%   43KB   1.0MB/s   00:00
    plot_lat_test.pdf                             100%   20KB 853.8KB/s   00:00
    plot_lat_test.png                             100%   77KB   1.5MB/s   00:00

• Most common options:
  • -p: preserves modification times, access times, and modes from the original file.
    This option is very useful if you want the copied file(s) to have the same date/time as the original file(s). Otherwise, the time will be the time when you copy the file(s)…
  • -r: recursively copy entire directories.
    You have to use this option if the source location is a directory. scp -r will copy the complete content of the directory (including sub-directories)

WinSCP is a convenient and user-friendly scp client for Windows

In some cases, you may want to mirror the content of directories:

• because you are creating a backup
• because you have lots of files, possibly (very) big, and you don't want to start copying everything again if the copy fails for some reason (e.g. temporary network problems if you transfer data between servers)
• mirroring means that, after running rsync, the SRC (Source) and DST (Destination) directories will have the exact same content (files in DST and not in SRC will be deleted)

rsync is a convenient Linux command that can be used for mirroring a directory hierarchy to another location on the same computer (e.g. a removable disk), or a remote Linux machine, over ssh.

Mirroring means, in the rsync case, that we will only copy new or changed files. The first copy may take some time, but will be much faster afterwards, when only a few files have been created/changed and have to be copied. Or, if a copy is interrupted, the files already copied will not be copied a second time.

Note: use man rync to get all the details and options

Local usage: rsync [OPTIONS] SRC DEST

Remote usage (i.e. the SRC and DEST directories are on different Linux machines):

         Pull: rsync [OPTIONS] [USER@]HOST:SRC... DEST
         Push: rsync [OPTIONS] SRC... [USER@]HOST:DEST

-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
        -r, --recursive             recurse into directories
        -l, --links                 copy symlinks as symlinks
        -p, --perms                 preserve permissions
        -t, --times                 preserve modification times
        -g, --group                 preserve group
        -o, --owner                 preserve owner (super-user only)
            --devices               preserve device files (super-user only)
            --specials              preserve special files
        -D                          same as --devices --specials


-v, --verbose               increase verbosity
-z, --compress              compress file data during the transfer
-W, --whole-file            copy files whole (w/o delta-xfer algorithm)
-C, --cvs-exclude           auto-ignore files in the same way CVS does
             RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state .nse_depinfo *~ #* .#* ,*  _$*
             *$  *.old  *.bak  *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln
             core .svn/ .git/ .hg/ .bzr/

--delete                delete extraneous files from dest dirs

-n, --dry-run               perform a trial run with no changes made
-c, --checksum              skip based on checksum, not mod-time & size

• We have an existing source /mnt/h/test/ directory that we would like to mirror to another disk or destination, as /mnt/i/test/.
  i.e we want to recursively copy the content of test/ to a test/ directory somewhere else
  
  The examples below are on the same machine, but we could do a copy/mirror over ssh by just adding [USER@]HOST: in front of the Source or Destination directory
  
  
  • Show what would be done, but do not do it (yet)
    Note that we have a trailing / after the source directory, and NO / after the destination directory
    rsync --dry-run -avW -C /mnt/h/test/ /mnt/i/test
    • Note: using -C makes sure that files and directories considered as temporary will not be copied.
      Do NOT use the -C option if you really want to copy all the files!
  • Same as above, without the verbose mode. Probably nothing will be displayed on the screen
    rsync --dry-run -aW -C /mnt/h/test/ /mnt/i/test
  • Actually copy the files, without displaying anything
    rsync -aW -C /mnt/h/test/ /mnt/i/test
  • Actually mirror the content of the source directory.
    Warning! Warning! the --delete option will make sure that files present in the destination directory, but not in the source directory will be deleted! Be careful, make some tests and use the --dry-run option before using this
    rsync -aW -C --delete /mnt/h/test/ /mnt/i/test

ssh keys are a combination of two specific (and unique) text files, the private key file and the public key file, linked by a special kind of password called the passphrase, that can be used instead of a standard password to connect securely from one server to another server

ssh keys have to be configured properly (a few easy steps), and are very convenient because:

• Contrary to passwords, they usually don't expire!
  You don't have to change ssh keys (except in some extra secure computing centers like TGCC) and you can keep them for years

• They don't depend on the accounts and the passwords of the servers where you use them
  • You can (and should!) use the same set of ssh keys on several servers: you can then use the same passphrase to access all these servers, rather than having to memorize different passwords
    e.g. if you have your private key on account_A of server_A and install the matching public key on account_B of server_B, etc… you can then use ssh on account_A@server_A to access account_B@server_B, account_C@server_C, … with the same passphrase !
  • You can give your public key to somebody and then access their account using your own passphrase (no need to know the password of the other person)

• The IPSL Mésocentre ESPRI servers can only be accessed with a public key and passphrase (the password is not used)

• By default, ssh will ask you to type your passphrase each time you connect to a server, but you can use an ssh agent to securely store your passphrase for you
  Once you have typed your passphrase in the ssh agent, you can connect to all the servers that have your public key without having to type your passphrase!
  • scp (and WinSCP on Windows) and the tools using ssh on your local computer will not ask your passphrase, if they find the passphrase in a running ssh agent on the local computer
    • if you use the -A option (agent forwarding), the remote server will also know (securely) your passphrase, and you will not have to type the passphrase when using ssh, scp and tools running over ssh on the remote server(s)
  • the local ssh agent is terminated when you log out of your local computer (or reboot the computer)

• Generate only one pair of private/public keys and use the same pair of keys everywhere!
  Put differently, do not generate a different pair of key on each computer/server you use (even if you always use the same passphrase)!

• Do not use an empty passphrase!
  If you do that, somebody gaining access to your private key will be able to access all the accounts where you have installed your public key… You obviously do not want that, right?

• Keep a backup of your your keys outside of the computer where they were generated
  • Useful if you erase or overwrite the keys by mistake, or if you move to another lab and use a new computer/account, but still need to access the accounts where you have installed your public key…
  • If you have not used an empty passphrase, and have not saved the passphrase in the same directory as the keys, the keys can't be used (easily) by somebody else to gain access to your accounts

• Do not forget your passphrase!
  • Do not write your passphrase on a post-it taped to your computer
  • When you create your keys and type your passphrase, choose something that you will be able to easily remember during several years. It can even be a long (but easy to remember!) sentence!
    • Easy to remember passphrase example: “I love working at LSCE!”

If you already have a pair of ssh keys, you probably don't want to generate a new pair, unless you have been asked to (e.g. because an old encryption type like DSA has been deprecated), or you have lost one of the keys, or forgotten your passphrase. If you generate a new pair of keys, you will have to replace the old keys that you were using on all your desktops/laptops, and all the remote servers

There are several ways to generate pairs of ssh keys with ssh-keygen. The following one is the one recommended for opening an account on IPSL Mésocentre ESPRI. If you open an account on spirit, but already have a public key, just send your existing public key!

• Type ssh-keygen -t ed25519
  • Accept the default path and key name
  • Do not specify an empty passphrase!
  • Note: ssh-keygen -t ed25519 will also work on Windows! But then you will still have to convert the generated private key with PuTTYgen
• This will generate two text key files in the ssh configuration directory:
  • The private key: id_ed25519
    • Note: on a Linux computer, the private key has to be readable only by you, otherwise ssh will not work

    •  > cd ~/.ssh
       > ls -l id_ed25519
      -rw------- 1 my_login my_group some_date id_ed25519
       > cat id_ed25519
      -----BEGIN OPENSSH PRIVATE KEY-----
      b3BlbnNzaC1rZXktdjEAAAAACmFlczI1Ni1jdHIAAAAGYmNyeXB0AAAAGAAAABC7W9+Eu7
      [ lots of literally cryptic lines ]
      cG7sHta/m1cOGM8ej7yD8ejCRMKGX1pEqGx/8=
      -----END OPENSSH PRIVATE KEY-----

  • The public key: id_ed25519.pub
    • This is the key that you can share, or that you have to send when opening an account on IPSL Mésocentre ESPRI.
      • Note that the my_login@my_machine string at the end of the line is just some information about who generated the keys, and where, and can be removed or replaced by something more informative

    •  > cat id_ed25519.pub
      ssh-ed25519 AAAAC3NzaC1lZDI1NT [ lots of cryptic characters ] Frx8rRFKthpmqRdkXl my_login@my_machine

Read Converting existing ssh keys with PuTTYgen, or Creating ssh keys with PuTTYgen

An ssh agent running on your computer will securely store your passphrase and supply it to applications that use ssh on your computer to connect to remote Linux servers

• start the agent when you open a new session on your computer
• enter your ssh passphrase(s). An agent can store multiple passphrases associating multiple pairs of private and public ssh keys, but you can also use the same public key on different remote Linux servers
• the ssh agent will store your passphrase until you exit your session, or restart your computer

• On Windows, we recommend using Pageant/PuTTY as an ssh agent, because:
  • Pageant/PuTTY also offers a very easy and convenient way to define profiles to connect to your favorite servers
  • Some programs that use ssh to transfer files will automatically use the keys stored in Pageant: WinSCP, emacs, …

• It is also possible (but less convenient) to use the Windows built-in ssh-agent and ssh commands!
  • Note that the agent service is not activated by default and you will get the following error when you try to use ssh-add

    • C: > ssh-add
      Error connecting to agent: No such file or directory

  • It is necessary to first activate the agent Windows service (in an elevated PowerShell, i.e with Administrator privileges) as explained in the User key generation section

    • C: > Get-Service ssh-agent
      Status   Name               DisplayName
      ------   ----               -----------
      Stopped  ssh-agent          OpenSSH Authentication Agent
      
      C: > Get-Service ssh-agent | Set-Service -StartupType Automatic
      
      C: > Start-Service ssh-agent
      
      C: > Get-Service ssh-agent
      Status   Name               DisplayName
      ------   ----               -----------
      Running  ssh-agent          OpenSSH Authentication Agent
      
      C: > ssh-add
      Enter passphrase for C:\Users\your_login/.ssh/id_dsa: XXXX_Type_Your_Passphrase_Here_XXXX
      Identity added: C:\Users\your_login/.ssh/id_dsa
      Identity added: C:\Users\your_login/.ssh/id_ed25519
      
      C: > ssh-add -l
      1024 SHA256:/vC3Ma6s9Wj[Some_Summary_Info_About_The_Key]c1Q4 (DSA)
      256 SHA256:8BGKU+zBnJXH[Some_Summary_Info_About_The_Key]2Al8 jypeter@obelix5 (ED25519)

The ssh-agent process will be automatically started, if a user runs ssh or ssh-add!

Check the launchd documentation if you need more details

$ launchctl list | grep ssh
      7240	0	com.openssh.ssh-agent

It seems that, once used, the passphrase will be automatically stored in the Keychain for future sessions

• If you want to know more (options, etc…), check the man(ual) page on Linux: man ssh
• Editing remote text files with emacs (and automatic scp): check the emacs section of the Text editors
• Digging ssh tunnels when using ipython notebooks

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]