User Tools

Site Tools


other:ssh

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
other:ssh [2020/08/31 13:41]
jypeter [Connecting to the LSCE servers, IPSL servers, TGCC, ...]
other:ssh [2023/05/03 08:32] (current)
jypeter [Windows ssh agent] Improved
Line 13: Line 13:
     * More precisely ''​ssh''​ is an //SSH client// using the //SSH protocol//     * More precisely ''​ssh''​ is an //SSH client// using the //SSH protocol//
  
-  * We assume below that you have a ''​my_login''​ account on the remote ''​remote_server''​ computer, and you know your password+  * We assume below that you have a ''​my_login''​ account on the remote ''​remote_server''​ computer, and that you know your password
     * This page will also show some examples using the [[https://​intranet.lsce.ipsl.fr/​informatique/​fr/​inter.php|LSCE]],​ [[https://​mesocentre.ipsl.fr/​|IPSL Mésocentre ESPRI]] and [[https://​intranet.lsce.ipsl.fr/​informatique/​en/​tgcc.php|TGCC]] servers     * This page will also show some examples using the [[https://​intranet.lsce.ipsl.fr/​informatique/​fr/​inter.php|LSCE]],​ [[https://​mesocentre.ipsl.fr/​|IPSL Mésocentre ESPRI]] and [[https://​intranet.lsce.ipsl.fr/​informatique/​en/​tgcc.php|TGCC]] servers
  
Line 30: Line 30:
   * The following will work in a **Linux** terminal, but can also work in a //​terminal//​ on a **Mac** or on a **Windows 10** computer   * The following will work in a **Linux** terminal, but can also work in a //​terminal//​ on a **Mac** or on a **Windows 10** computer
  
-  * On **Windows 10**, ''​ssh''​ is directly available in a ''​Windows Powershell'',​ a [[https://​www.microsoft.com/​store/​productId/​9N0DX20HK701|Windows Terminal]] or the old ''​cmd'',​ but the most user-friendly way to use ''​ssh''​ is to use [[other:​putty_conf|PuTTY]]+  * On **Windows 10**, ''​ssh''​ is directly available in a ''​Windows Powershell'',​ a [[other:win10apps#​windows_terminal|Windows Terminal]] or the old ''​cmd'',​ but the most user-friendly way to use ''​ssh''​ is to use [[other:​putty_conf|PuTTY]] 
 + 
 +  * On a **Mac**, ''​ssh''​ is directly available in the built-in Apple [[other:​ssh#​a_recommended_terminal_for_mac|Terminal application]]
  
 </​WRAP>​ </​WRAP>​
Line 45: Line 47:
 ==== Most common options ==== ==== Most common options ====
  
-  * ''​-X''​: **enable //X11 forwarding//​**. This option ​will allow you to **use graphical programs on the remote server** +  * ''​-X'' ​or ''​-Y''​ options if you will need to **use graphical programs on the remote server** 
-    * If ''​-X''​ does not work, use ''​-Y''​ instead (Enable //trusted// X11 forwarding) +    * All the Details ​in the [[other:​ssh#​using_an_x_server_to_display_graphics|Using an X server to display graphics]] section
-    * More details ​in the [[other:​ssh#​using_an_x_server_to_display_graphics|Using an X server to display graphics]] section+
  
   * ''​-A'':​ **enable //agent forwarding//​**. This is useful when you use [[other:​ssh#​using_ssh_keys|ssh keys and an ssh agent]]   * ''​-A'':​ **enable //agent forwarding//​**. This is useful when you use [[other:​ssh#​using_ssh_keys|ssh keys and an ssh agent]]
Line 55: Line 56:
   * ''​-v'':​ **verbose mode**. Use this option only when you can't connect, or things don't seem to work correctly. Analyzing the verbose output when you start ''​ssh''​ should allow you, or the [[other:​newppl:​starting#​getting_help_from_the_lsce_system_administrators|system administrators]],​ to find out what is wrong   * ''​-v'':​ **verbose mode**. Use this option only when you can't connect, or things don't seem to work correctly. Analyzing the verbose output when you start ''​ssh''​ should allow you, or the [[other:​newppl:​starting#​getting_help_from_the_lsce_system_administrators|system administrators]],​ to find out what is wrong
  
-==== Connecting to the LSCE servers, IPSL servers, TGCC, ... ====+==== Connecting to servers commonly used by LSCE users ====
  
-There are several ways to use ssh to connect to the LSCE ''​obelixNN''​ servers (more details about the [[other:​newppl:​starting#​which_linux_servers_should_you_use)|available LSCE servers]])+=== LSCE servers === 
 + 
 +There are several ways to use ''​ssh'' ​to connect to the LSCE ''​obelixNN''​ servers (more details about the [[other:​newppl:​starting#​which_linux_servers_should_you_use)|available LSCE servers]])
  
   * If your computer is **on the LSCE ethernet/​wired network**:   * If your computer is **on the LSCE ethernet/​wired network**:
     * Go to the server with the smallest [[other:​newppl:​starting#​determining_the_load_of_a_linux_server|load]]:​\\ ''​ssh -A -X my_LSCE_login@obelix''​\\ or ''​ssh -A -X my_LSCE_login@obelix.lsce.ipsl.fr''​     * Go to the server with the smallest [[other:​newppl:​starting#​determining_the_load_of_a_linux_server|load]]:​\\ ''​ssh -A -X my_LSCE_login@obelix''​\\ or ''​ssh -A -X my_LSCE_login@obelix.lsce.ipsl.fr''​
-    * Go to a specific ''​obelix''​ (possibly because you have some running processes on this server that you want to monitor with [[other:​newppl:​starting#​determining_the_load_of_a_linux_server|top]],​ or terminate with ''​kill''​)\\ ''​ssh -A -X my_LSCE_login@obelix4''​+    * Go to a specific ''​obelixNN''​ (possibly because you have some running processes on this server that you want to monitor with [[other:​newppl:​starting#​determining_the_load_of_a_linux_server|top]],​ or terminate with ''​kill''​)\\ ​e.g. ''​ssh -A -X my_LSCE_login@obelix4''​
   * If your computer is **outside LSCE**, or **on the LSCE WiFi network**, you have to:   * If your computer is **outside LSCE**, or **on the LSCE WiFi network**, you have to:
     * Ask your advisor to send a mail to [[other:​newppl:​starting#​getting_help_from_the_lsce_system_administrators|help-lsce]],​ and request an access to the ''​ssh1''​ server     * Ask your advisor to send a mail to [[other:​newppl:​starting#​getting_help_from_the_lsce_system_administrators|help-lsce]],​ and request an access to the ''​ssh1''​ server
-    * Go first through the ''​ssh1''​ gateway server\\ ''​ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X obelix''​+    * Connect ​through the ''​ssh1''​ gateway ​server, i.e you first use ''​ssh''​ to connect to the ''​ssh1''​ gateway, and then use ''​ssh''​ to go to an ''​obelix'' ​server\\ These two ''​ssh''​ commands can be [[other:​ssh#​most_common_options|conveniently chained with the '​-t'​ option]]!\\ ''​ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X obelix''​ 
 + 
 +=== IPSL servers === 
 + 
 +If you want to connect to the [[https://​documentations.ipsl.fr/​spirit/​spirit_clusters/​head_nodes.html|IPSL servers]] (only possible with [[other:​ssh#​using_ssh_keys|ssh keys]]!): 
 +  * Connecting to ''​spirit1'':​ 
 +    * ''​ssh -A -X my_meso_login@spirit1.ipsl.fr''​ 
 +    * Depending on what you need to do, you can also use ''​spirit2'',​ ''​spiritx1''​ or ''​spiritx2''​ 
 +  * [[https://​documentations.ipsl.fr/​spirit/​spirit_clusters/​head_nodes.html|More details]] 
 +  * Note: the ''​ciclad''​ server may still be accessible when you read this page, but its usage has been deprecated in favor of the ''​spirit''​ servers 
 + 
 +=== TGCC (super)computers === 
 + 
 +If you want to use the [[https://​www-hpc.cea.fr/​tgcc-public/​en/​html/​tgcc-public.html|TGCC computers]] (e.g. ''​irene''​):​ 
 + 
 +  * Note: you have to go //trough// the ''​ssh1''​ LSCE gateway to access the TGCC, even if you are on the LSCE wired network! 
 +  * ''​ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X my_TGCC_login@some_tgcc_login_node''​ 
 +    * Once you are on a TGCC login node (e.g. ''​irene''​),​ you can get lots of information by typing ''<​node>​.info''​ (e.g. ''​irene.info''​) 
 +  * [[https://​intranet.lsce.ipsl.fr/​informatique/​en/​tgcc.php|more TGCC connection details]] 
 + 
 +=== IDRIS (super)computers === 
 + 
 +FIXME 
 + 
 + 
 +==== Using shell aliases shortcuts to connect to the servers ====
  
-If you want to connect to the IPSL servers: +If you have to use ''​ssh'' ​regularly (with the appropriate options), you should define convenient ​//shell aliases//, or [[other:​putty_conf#​adding_more_connection_profiles|add PuTTY connection profiles]] on Windows
-  * Connecting ​to ''​ciclad'':​\\ ​''​ssh ​-A -X my_ciclad_login@ciclad.ipsl.jussieu.fr''​ +
-  * [[https://documentations.ipsl.fr/MESO_User/Quick_start.html|More details]]+
  
-If you want to connect to the the TGCC servers: +=== bash shell users ===
-  * Connecting to ''​irene'':​ +
-    * Note: you have to go trough ssh1, even if you are on the LSCE network! +
-    * ''​ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X my_TGCC_login@irene-ccrt.ccc.cea.fr''​ +
-    * The [[https://​intranet.lsce.ipsl.fr/​informatique/​en/​tgcc.php|TGCC connection details]] may vary, depending on your login type+
  
-If you have to use ''​ssh''​ regularly (with the appropriate options), you should define ​the following aliases in the ''​~/​.bashrc''​ configuration file of your local Linux account, or [[other:​putty_conf|properly configure and use PuTTY]] on Windows+Define ​the following aliases in the ''​~/​.bashrc''​ configuration file of your local Linux account
  
 <​code>​ <​code>​
Line 91: Line 113:
 alias sirene='​ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X my_TGCC_login@irene-ccrt.ccc.cea.fr' ​ alias sirene='​ssh -A -X my_LSCE_login@ssh1.lsce.ipsl.fr -t ssh -A -X my_TGCC_login@irene-ccrt.ccc.cea.fr' ​
 </​code>​ </​code>​
 +
 +=== tcsh shell users ===
  
 If your connection shell is ''​tcsh''​ instead of ''​bash'',​ use the appropriate alias syntax in your ''​~/​.cshrc''​ configuration file,\\ e.g. ''​alias obelix 'ssh -A -X my_LSCE_login@obelix'​ ''​ If your connection shell is ''​tcsh''​ instead of ''​bash'',​ use the appropriate alias syntax in your ''​~/​.cshrc''​ configuration file,\\ e.g. ''​alias obelix 'ssh -A -X my_LSCE_login@obelix'​ ''​
Line 100: Line 124:
 If you want to **use ''​ssh''​ to start graphical programs on a remote server**, you need to: If you want to **use ''​ssh''​ to start graphical programs on a remote server**, you need to:
  
-  * **use ''​ssh -X''​** (or ''​ssh -Y''​ if ''​-X''​ does not work) to connect to the remote server+  * **Use ''​ssh -X''​** (or ''​ssh -Y''​ if ''​-X''​ does not work) to connect to the remote server
     * ''​-X'':​ **enable //X11 forwarding//​**     * ''​-X'':​ **enable //X11 forwarding//​**
     * ''​-Y'':​ enable //trusted// X11 forwarding (low security, but you //trust// the remote server)     * ''​-Y'':​ enable //trusted// X11 forwarding (low security, but you //trust// the remote server)
-    * Using the ''​-X''/''​-Y''​ option will automatically define the ''​DISPLAY''​ environment variable that is required by graphical programs on the remote server to determine where to display the graphical windows.\\ ''​DISPLAY''​ will not be defined if you forget to use ''​-X''/''​-Y''​\\ ​+    * Using the ''​-X''/''​-Y''​ option will **automatically** define the ''​DISPLAY''​ environment variable that is required by graphical programs on the remote server to determine where to display the graphical windows.\\ ''​DISPLAY''​ will **not** be defined if you forget to use ''​-X''/''​-Y''​\\ ​
     * Example:\\ <​code>​my_login@my_local_computer:​~$ echo $DISPLAY     * Example:\\ <​code>​my_login@my_local_computer:​~$ echo $DISPLAY
 localhost:​0.0 localhost:​0.0
Line 117: Line 141:
 [my_login@ssh1 ~]$ echo $DISPLAY [my_login@ssh1 ~]$ echo $DISPLAY
 localhost:​43.0</​code>​ localhost:​43.0</​code>​
-  * and **have a local //X server// running**!\\ An //X server// is basically a program running on your computer that understands the [[https://​en.wikipedia.org/​wiki/​X_Window_System_core_protocol|X Windows System protocol]] used by the remote ​server ​to display graphics +  * **And** [[other:x_conf|have a local X server running]]!
-      * Linux computer: nothing to do, an //X server// is already ​running ​+
-      * Windows: [[other:​win10wsl#​installing_an_x_server|install,​ configure and launch VcXsrv]] +
-      * Mac: FIXME+
  
 ==== Configuration files ==== ==== Configuration files ====
  
-''​ssh''​ will store all its **configuration ​text files** in the ''​~/.ssh/''​ directory ​(''​C:​\Users\your_windows_login\.ssh''​ on Windows 10)+''​ssh''​ will store all its **configuration files** in ''​.ssh'' ​sub-directory of your //​home// ​directory. ​The configuration files are in a //text// format.
  
 +  * **Linux**: ''​~/​.ssh/''​ directory
 +  * **Windows**:​ ''​C:​\Users\your_windows_login\.ssh''​ directory
 +  * **Mac**: ''/​Users/​your_mac_login/​.ssh''​ directory (should be the same path as ''​~/​.ssh/''​)
 +
 +You will find (some of) the following text files:
   * ''​known_hosts'':​ the text file were ''​ssh''​ stores one line of security information about each server you have connected to from this computer\\ e.g. ''​ciclad.ipsl.jussieu.fr,​134.157.176.253 ssh-rsa AAAAB3NzaC1y[a long identifier...]''​   * ''​known_hosts'':​ the text file were ''​ssh''​ stores one line of security information about each server you have connected to from this computer\\ e.g. ''​ciclad.ipsl.jussieu.fr,​134.157.176.253 ssh-rsa AAAAB3NzaC1y[a long identifier...]''​
  
Line 134: Line 160:
 ServerAliveCountMax=90</​code>​ ServerAliveCountMax=90</​code>​
  
-  * ''​authorized_keys'':​ the //​public ​keys// of the accounts ​authorized to connect to //this// account +  ​* [[#​using_ssh_keys|ssh keys]] related information:​ 
- +    ​* ''​authorized_keys'':​ the //​public ​key(s)// of the account(s) ​authorized to connect to //this// account. 
-  ​* the private and public //ssh keys// used on this account+    * the **//private// (and probably the //public//) //ssh key(s)//** used on this account 
 +      * e.g. ''​id_ed25519''​ and ''​id_ed25519.pub''​ files
  
 ==== A recommended ssh client for Windows ==== ==== A recommended ssh client for Windows ====
Line 142: Line 169:
 [[other:​putty_conf|PuTTY]] is a convenient and user-friendly //ssh client// for Windows [[other:​putty_conf|PuTTY]] is a convenient and user-friendly //ssh client// for Windows
  
-==== Solving common problems ==== 
  
-  * You want to start a graphical program on a remote server, but get a ''​Can'​t open display: //[NO VALUE DISPLAYED HERE]//''​ error\\ <​code>​$ xterm & +==== A recommended terminal for Mac====
-$ xterm: Xt error: Can't open display: +
-xterm: DISPLAY is not set +
-$ echo $DISPLAY+
  
 +A ''​Terminal''​ window will open a local Linux-like shell session on the Mac, where you can use ''​ssh''​ to connect to another server, or other standard Linux commands
  
-</​code>​\\ The ''​DISPLAY''​ variable is probably not defined because you have not specified the ''​-X''​ (or ''​-Y''​) ​option when connecting to the remote serverSee [[other:ssh#​using_an_x_server_to_display_graphics|Using an X server to display graphics]]+  * Built-in: ​''​Terminal'' ​application ​(available in ''​/​Applications/​Utilities''​) 
 +    * [[https://​support.apple.com/​guide/​terminal/​welcome/​mac|Terminal User Guide]] 
 +  * Recommended:​ the [[https://​iterm2.com|iTerm2]] application 
 +    * Improved //Favorite Sessions// settings 
 + 
 +==== Solving common problems ====
  
-  * You want to start a graphical program on a remote server, but get a ''​Can'​t open display: localhost://​[SOME VALUE]//''​ error\\ <​code>​$ xterm & +  * [[other:x_conf#troubleshooting|X server ​related errors]]
-$ connect localhost port 6000: Connection refused +
-xterm: Xt error: Can't open display: localhost:​12.0</​code>​\\ The ''​DISPLAY''​ variable is defined correctly, but you probably don't have a local //X server// running. See [[other:ssh#using_an_x_server_to_display_graphics|Using an X server ​to display graphics]]+
  
   * Other types of errors: remember that you can run ''​ssh''​ in **verbose** mode to help you determine what is wrong (''​-v''​ option)   * Other types of errors: remember that you can run ''​ssh''​ in **verbose** mode to help you determine what is wrong (''​-v''​ option)
Line 160: Line 187:
 ===== Copying files between servers/​computers ===== ===== Copying files between servers/​computers =====
  
-Sometimes you just need to copy files from one remote server (or your desktop) to the other. The files can be securely copied //over ssh// with the ''​scp''​ command+Sometimes you need to copy files from one remote server (or your desktop) to the other. The files can be securely copied //over ssh// with the ''​scp''​ command
  
-Note: if you work with big data files, ​you should keep the files were they are instead of duplicating them, and move the data processing (your scripts, etc...) to the server where the files are located (e.g. the ''​ciclad''​ server at IPSL)+Note: you should ​**keep the big data files were their original version is** instead of duplicating them, and move the data processing (your scripts, etc...) to the server where the original ​files are located (e.g. the ''​ciclad''​ server at IPSL)
  
 ==== Copying files with scp ==== ==== Copying files with scp ====
  
 <WRAP center round tip 60%> <WRAP center round tip 60%>
-Note: the following will work in a **Linux** terminal, but can also work in a //​terminal//​ on a **Mac** or on a **Windows 10** computer (''​scp''​ is directly available in ''​Windows Powershell'', ​''​Windows Terminal'' ​or the old ''​cmd'',​ but it is not the most user-friendly way to use ''​ssh''​ on Windows)+Note: the following will work in a **Linux** terminal, but can also work in a //​terminal//​ on a **Mac** or on a **Windows 10** computer (''​scp''​ is directly available in ''​Windows Powershell'', ​[[https://​www.microsoft.com/​store/​productId/​9N0DX20HK701|Windows Terminal]] or the old ''​cmd'',​ but it is not the most user-friendly way to use ''​scp''​ on Windows)
  
 If you have a Windows computer, it is much easier to use [[other:​win10apps#​winscp|WinSCP]] for copying files If you have a Windows computer, it is much easier to use [[other:​win10apps#​winscp|WinSCP]] for copying files
 </​WRAP>​ </​WRAP>​
  
-  * **''​scp [options] local_path_or_file [my_login@]remote_server:​remote_path''​**\\ or **''​scp [options] ​ [my_login@]remote_server:​remote_path_or_file local_path''​**+  * **''​scp [options] local_path_or_file(s) [my_login@]remote_server:​remote_path''​**\\ or **''​scp [options] ​ [my_login@]remote_server:​remote_path_or_file(s) local_path''​**
     * If your login is the same on the local and remote computer, you can omit the optional ''​my_login@''​ part     * If your login is the same on the local and remote computer, you can omit the optional ''​my_login@''​ part
 +    * If you are copying files from a remote server to the current local directory, you can use ''​.''​ instead of the full path of the local directory:​\\ <​code>​$ cd /some/path
 +$ scp -p ssh1.lsce.ipsl.fr:/​some/​remote/​path/​scatter_regress_example.py .
 +scatter_regress_example.py ​                   100% 4988   ​134.6KB/​s ​  ​00:​00</​code>​
 +    * if you need to use [[https://​www.tecmint.com/​use-wildcards-to-match-filenames-in-linux/​|wildcards]] to specify the files you want to copy, you can use quotes around the path specification:​\\ <​code>​$ scp -p ssh1.lsce.ipsl.fr:'/​some/​remote/​path/​matplotlib/​plot_lat_test.*'​ .
 +plot_lat_test.eps ​                            ​100% ​  ​43KB ​  ​1.0MB/​s ​  00:00
 +plot_lat_test.pdf ​                            ​100% ​  20KB 853.8KB/​s ​  00:00
 +plot_lat_test.png ​                            ​100% ​  ​77KB ​  ​1.5MB/​s ​  ​00:​00</​code>​
  
   * Most common options:   * Most common options:
-    * ''​-p'':​ **preserves modification times**, access times, and modes from the original file.\\ This option is **very useful** if you want the copied file(s) to have the same date/time as the original file(s). Otherwise, the time will be the time when you copy the file(s)+    * ''​-p'':​ **preserves modification times**, access times, and modes from the original file.\\ This option is **very useful** if you want the copied file(s) to have the same date/time as the original file(s). Otherwise, the time will be the time when you copy the file(s)...
     * ''​-r'':​ **recursively** copy entire directories.\\ **You have to use this option if the source location is a directory**. ''​scp -r''​ will copy the complete content of the directory (including sub-directories)     * ''​-r'':​ **recursively** copy entire directories.\\ **You have to use this option if the source location is a directory**. ''​scp -r''​ will copy the complete content of the directory (including sub-directories)
  
Line 183: Line 217:
 [[other:​win10apps#​winscp|WinSCP]] is a convenient and user-friendly //scp client// for Windows [[other:​win10apps#​winscp|WinSCP]] is a convenient and user-friendly //scp client// for Windows
  
-==== Synchronizing directories ​====+==== A recommended graphical scp client for Mac ====
  
-In some cases, you may want to synchronize ​the content of directories:​+FIXME 
 + 
 +==== Mirroring directories with rsync ==== 
 + 
 +In some cases, you may want to **mirror ​the content of directories**:
   * because you are creating a backup   * because you are creating a backup
-  * because you have lots of files, possibly (very) big, and you don't want to start copying everything again if the copy fails due to temporary network problems +  * because you have lots of files, possibly (very) big, and you don't want to start copying everything again if the copy fails for some reason (e.g. temporary network problems ​if you transfer data between servers) 
-  * ...+  * //​mirroring//​ means that, after running ''​rsync'',​ the ''​SRC''​ (//​Source//​) and ''​DST''​ (//​Destination//​) directories will have the exact same content (**files in ''​DST''​ and not in ''​SRC''​ will be deleted**)
  
-In that case, you should use the ''​rsync''​ command, that will only copy files that are not already in the destination ​(and that have not changed since the previous copy).+''​rsync'' ​is a convenient **Linux** ​command that can be used for //​mirroring//​ a directory hierarchy to another location on the same computer ​(e.g. a removable disk), or a remote Linux machine, //over ssh//.
  
-''​rsync'' ​has lots of complex options and rulesand **should be used carefully** if you do not want to lose files. ​This page does not cover this topicUse ''​man rsync''​ or ask somebody+//​Mirroring//​ means, in the ''​rsync'' ​casethat we will **only copy new or changed files**. The first //copy// may take some time, but will be much faster afterwards, when only a few files have been created/​changed and have to be copiedOr, if a copy is interrupted,​ the files already copied will not be copied a second time.
  
 +<WRAP center round important 60%>
 +**Warning!** <wrap em>It is easy to lose files with ''​rsync''​ if you use the wrong syntax or options!</​wrap>​
 +
 +Mirroring //no files// by mistake, to a place where there are files, when using the ''​%%--%%delete''​ option, means that existing files or whole directory hierarchies will be deleted!
 +
 +  * Be sure to understand the basic options, and use ''​%%--%%dry-run''​ (simulate what would be done) and ''​-v''​ (verbose) before performing the actual mirroring
 +  * <wrap em>​Having a trailing ''/''​ or not behind a directory name makes a difference!</​wrap>​
 +</​WRAP>​
 +
 +=== Basic rsync syntax ===
 +
 +Note: use ''​man rync''​ to get all the details and options
 +
 +Local usage: ''​rsync [OPTIONS] SRC DEST''​
 +
 +Remote usage (i.e. the ''​SRC''​ and ''​DEST''​ directories are on different Linux machines):
 +
 +<​code> ​        Pull: rsync [OPTIONS] [USER@]HOST:​SRC... DEST
 +         Push: rsync [OPTIONS] SRC... [USER@]HOST:​DEST</​code>​
 +
 +<​code>​
 +-a, --archive ​              ​archive mode; equals -rlptgoD (no -H,-A,-X)
 +        -r, --recursive ​            ​recurse into directories
 +        -l, --links ​                copy symlinks as symlinks
 +        -p, --perms ​                ​preserve permissions
 +        -t, --times ​                ​preserve modification times
 +        -g, --group ​                ​preserve group
 +        -o, --owner ​                ​preserve owner (super-user only)
 +            --devices ​              ​preserve device files (super-user only)
 +            --specials ​             preserve special files
 +        -D                          same as --devices --specials
 +
 +
 +-v, --verbose ​              ​increase verbosity
 +-z, --compress ​             compress file data during the transfer
 +-W, --whole-file ​           copy files whole (w/o delta-xfer algorithm)
 +-C, --cvs-exclude ​          ​auto-ignore files in the same way CVS does
 +             RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state .nse_depinfo *~ #* .#* ,*  _$*
 +             ​*$ ​ *.old  *.bak  *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln
 +             core .svn/ .git/ .hg/ .bzr/
 +
 +--delete ​               delete extraneous files from dest dirs
 +
 +-n, --dry-run ​              ​perform a trial run with no changes made
 +-c, --checksum ​             skip based on checksum, not mod-time & size
 +</​code>​
 +
 +=== Examples ===
 +
 +  * We have an existing //source// ''/​mnt/​h/​test/''​ directory that we would like to //mirror// to another disk or //​destination//,​ as ''/​mnt/​i/​test/''​.\\ i.e we want to recursively copy the content of ''​test/''​ to a ''​test/''​ directory //somewhere else//\\ \\ The examples below are on the same machine, but we could do a copy/mirror //over ssh// by just adding ''​[USER@]HOST:''​ in front of the //Source// or //​Destination//​ directory\\ \\ 
 +    * Show what would be done, but do not do it (yet)\\ <wrap em>Note that we have a trailing ''/''​ after the source directory, and NO ''/''​ after the destination directory</​wrap>​\\ ''​rsync %%--%%dry-run -avW -C /​mnt/​h/​test/​ /​mnt/​i/​test''​
 +      * Note: using ''​-C''​ makes sure  that files and directories considered as temporary will not be copied.\\ Do NOT use the ''​-C''​ option if you really want to copy all the files!
 +    * Same as above, without the //verbose// mode. Probably nothing will be displayed on the screen\\ ''​rsync %%--%%dry-run -aW -C /​mnt/​h/​test/​ /​mnt/​i/​test''​
 +    * Actually **copy** the files, without displaying anything\\ ''​rsync -aW -C /​mnt/​h/​test/​ /​mnt/​i/​test''​
 +    * Actually **mirror** the content of the //source// directory.\\ **Warning! Warning!** the ''​%%--%%delete''​ option will make sure that files present in the destination directory, but not in the source directory will be deleted! Be careful, make some tests and use the ''​%%--%%dry-run''​ option before using this\\ ''​rsync -aW -C %%--%%delete /​mnt/​h/​test/​ /​mnt/​i/​test''​
 ===== Using ssh keys ===== ===== Using ssh keys =====
  
 ==== What are ssh keys and why use them? ==== ==== What are ssh keys and why use them? ====
  
-//ssh keys// are a combination of two specific (and unique) text files, **the private key** file and **the public key** file, linked by a special kind of password called **the passphrase**,​ that can be used instead of a standard password to connect securely from one server to another server+//ssh keys// are a combination of two specific (and unique) ​**text files**, **the //private// key** file and **the //public// key** file, linked by a special kind of password called **the passphrase**,​ that can be used instead of a standard password to connect securely from one server to another server
  
 ssh keys have to be configured properly (a few easy steps), and are **very convenient** because: ssh keys have to be configured properly (a few easy steps), and are **very convenient** because:
  
-  * **They** ​usually ​**don't expire!**\\ You don't have to change ​them (except in some extra secure computing centers like TGCC) and you can keep them for years+  * Contrary to passwords, ​**they usually don't expire!**\\ You don't have to change ​ssh keys (except in some extra secure computing centers like TGCC) and you can keep them for years 
   * **They don't depend on the accounts and the passwords of the servers where you use them**   * **They don't depend on the accounts and the passwords of the servers where you use them**
-    * You can (and should!) use the same set of ssh keys on several servers, and you can then connect to these servers just using the same passphrase, rather than memorizing ​different passwords\\ e.g. if you have your private key on ''​account_A''​ of ''​server_A''​ and install the matching public key on ''​account_B''​ of ''​server_B'',​ etc... you can then use ''​ssh''​ to access ''​account_B@server_B'',​ ''​account_C@server_C'',​ ... with the same passphrase ! +    * You can (and should!) use the same set of ssh keys on several serversyou can then use the **same** passphrase ​to access all these servers, rather than having to memorize ​different passwords\\ e.g. if you have your //private// key on ''​account_A''​ of ''​server_A''​ and install the matching ​//public// key on ''​account_B''​ of ''​server_B'',​ etc... you can then use ''​ssh''​ on ''​account_A@server_A''​ to access ''​account_B@server_B'',​ ''​account_C@server_C'',​ ... with the **same** passphrase ! 
-    * You can give your public key to somebody and then access their account using your own passphrase (no need to know the password of the other person) +    ​* **You can give your public key** to somebody and then access their account using your own passphrase (no need to know the password of the other person) 
-  * The [[https://​mesocentre.ipsl.fr/​|IPSL Mésocentre ESPRI]] servers can only be accessed with a public key and passphrase (the password is not used) + 
-  * By default, ''​ssh''​ will ask you to type your passphrase each time you connect to a server, but **you can use an //ssh agent// to securely store your passphrase for you**\\ Once you have typed your passphrase in the ssh agent, you can connect to all the servers that have your public key without having to type your passphrase! +  * The [[https://​mesocentre.ipsl.fr/​|IPSL Mésocentre ESPRI]] servers can **only** be accessed with a public key and passphrase (the password is not used) 
-    * ''​scp''​ (and ''​WinSCP''​) and the tools using ''​ssh''​ on your local computer will not ask your passphrase, if they find the passphrase in a running ​local ssh agent + 
-      * if you use the ''​-A''​ option ([[other:​ssh#​most_common_options|agent forwarding]]),​ the remote will also //know// (securely) your passphrase, and you will not have to type the passphrase when using ''​ssh'',​ ''​scp''​ and tools running //over ssh// on the remote server(s) +  * By default, ''​ssh''​ will ask you to type your passphrase each time you connect to a server, but **you can [[other:​ssh#​using_an_ssh_agent|use an ssh agent]] to securely store your passphrase for you**\\ Once you have typed your passphrase in the //ssh agent//, you can connect to all the servers that have your public key without having to type your passphrase! 
-    * the ssh agent is terminated when you log out of your local computer (or reboot ​it)+    * ''​scp''​ (and [[other:​win10apps#​winscp|WinSCP]] on Windows) and the tools using ''​ssh''​ on your local computer will not ask your passphrase, if they find the passphrase in a running ​//ssh agent// on the local computer 
 +      * if you use the ''​-A''​ option ([[other:​ssh#​most_common_options|agent forwarding]]),​ the remote ​server ​will also //know// (securely) your passphrase, and you will not have to type the passphrase when using ''​ssh'',​ ''​scp''​ and tools running //over ssh// on the remote server(s) 
 +    * the local //ssh agent// is terminated when you log out of your local computer (or reboot ​the computer)
 ==== Generating ssh keys ==== ==== Generating ssh keys ====
  
 === Some common sense advice === === Some common sense advice ===
  
 +  * **Generate only one pair of private/​public keys and use the same pair of keys everywhere!**\\ Put differently,​ do not generate a different pair of key on each computer/​server you use (even if you always use the same passphrase)!
 +
 +  * <wrap em>Do not use an empty passphrase!</​wrap>​\\ If you do that, somebody gaining access to your private key will be able to access all the accounts where you have installed your public key... You obviously do not want that, right?
  
-  * **Generate only one pair of private/​public keys and use the same pair of keys everywhere!**\\ Put differently,​ do not generate a different pair of key on each computer/​server you use! 
-  * <wrap em>Do not use an empty passphrase!</​wrap>​\\ If you do that, somebody gaining access to your private key will be able to access all the accounts where you have installed your public key 
   * **Keep a backup of your your keys outside of the computer where they were generated**   * **Keep a backup of your your keys outside of the computer where they were generated**
-    * Useful if you erase or overwrite the keys by mistake, or if you move to another lab and use a new computer/​account,​ but still need to access the accounts where you have installed your public key +    * Useful if you erase or overwrite the keys by mistake, or if you move to another lab and use a new computer/​account,​ but still need to access the accounts where you have installed your public key... 
-    * The keys can't be used easily by somebody else to gain access to your accounts ​if you have not used an empty passhrase +    * If you have not used an empty passphrase, and have not saved the passphrase in the same directory as the keys, the keys can't be used (easilyby somebody else to gain access to your accounts ​
-  * **Do not forget your passphrase!**\\ Do not write your passphrase on a postit taped to your computer. When you create your keys and type your passphrase, choose something that you will be able to remember during several years+
  
-=== Generating ​keys in terminal ===+  * **Do not forget your passphrase!** 
 +    * Do not write your passphrase on a post-it taped to your computer 
 +    * When you create your keys and type your passphrase, choose something that you will be able to easily remember during several years. It can even be long (but easy to remember!) sentence! 
 +      * Easy to remember passphrase example: "//I love working at LSCE!//"​
  
-Remember that if you already have a pair of keys, you probably don't want to generate ​new pair, unless you have been asked to, or have lost one of the keys, or forgotten your passphrase. If you generate a new pair of keys, you will probably have to replace the old keys that you were using on all the remote servers+=== Generating ​keys in terminal (Linux and Mac) ===
  
-There are several ways to generate pairs of ssh keys with ''​ssh-keygen'​'​. ​The following one is the one recommended for opening ​an account on [[https://mesocentre.ipsl.fr/account-opening/|IPSL Mésocentre ESPRI]]. If you open an account on cicladbut already ​have a public keyjust send the existing key!+If you already have a pair of ssh keys, you probably don't want to generate a new pair, unless you have been asked to (e.g. because ​an old encryption type like //DSA// has been deprecated),​ or you have lost one of the keys, or forgotten your passphrase. If you generate a new pair of keysyou will have to replace the old keys that you were using on all your desktops/​laptopsand all the remote servers
  
-  ​* Type ''​ssh-keygen -t rsa -b 4096''​+There are several ways to generate pairs of ssh keys with ''​ssh-keygen''​. The following one is the one recommended for opening an account on [[https://​mesocentre.ipsl.fr/​account-opening/​|IPSL Mésocentre ESPRI]]. If you open an account on ''​spirit'',​ but already have a public key, just **send your existing public key**! 
 + 
 +  ​* Type ''​ssh-keygen ​ -t ed25519''​
     * Accept the default path and key name     * Accept the default path and key name
     * <wrap em>Do not specify an empty passphrase!</​wrap>​     * <wrap em>Do not specify an empty passphrase!</​wrap>​
-  ​* This will generate two text //key// files in a sub-directory of your account (''​~/​.ssh''​ on Linux, ''​C:\Users\my_login/​.ssh/''​ on Windows 10)+    * Note: ''​ssh-keygen -t ed25519''​ will also work on Windows! But then you will still have to [[other:​putty_conf#​converting_existing_ssh_keys_with_puttygen|convert the generated private key with PuTTYgen]] 
-    * The private key, that has to be readable only by you''​id_rsa''​\\ <​code>​ > cd ~/.ssh +  ​* This will generate two text //key// files in the [[other:ssh#​configuration_files|ssh configuration directory]]
- > ls -l id_dsa +    * The **//private// key**: ''​id_ed25519''​ 
--rw------- 1 my_login my_group some_date ​id_rsa +      * Note: on a Linux computerthe private key has to be readable only by you, otherwise ​''​ssh'' ​will not work 
- > cat id_rsa +      * <​code>​ > cd ~/.ssh 
------BEGIN ​RSA PRIVATE KEY----- + > ls -l id_ed25519 
-Proc-Type: 4,​ENCRYPTED +-rw------- 1 my_login my_group some_date ​id_ed25519 
-DEK-Info: AES-128-CBC,​906569054A4C58A28AD23CBA28771EDE + > cat id_ed25519 
- +-----BEGIN ​OPENSSH ​PRIVATE KEY----- 
-C/Aacy+qcSWIG56eWc3XQhm2oyfAVKFKVm54pwoCmIZ5nmLx/​8kV8XcDcMHxoWIz +b3BlbnNzaC1rZXktdjEAAAAACmFlczI1Ni1jdHIAAAAGYmNyeXB0AAAAGAAAABC7W9+Eu7
-xgc3cPwxNczIS/​i4A0AOk3uI8JiT8RVLELVbn+B5T0ewbvMjln4Ec/​7W9+aNe/​NF+
 [ lots of literally cryptic lines ] [ lots of literally cryptic lines ]
-v/rj1Ze/PEQ+nVX3dh3FB1TaL/​aNm48PBP9WQQXm011PY6isZJklyWANGJ6jtOf9 +cG7sHta/m1cOGM8ej7yD8ejCRMKGX1pEqGx/8= 
------END ​RSA PRIVATE KEY-----</​code>​ +-----END ​OPENSSH ​PRIVATE KEY-----</​code>​ 
-    * The public key: ''​id_rsa.pub''​\\ This is the information ​that you can share. Note that the ''​my_login@my_machine''​ at the end of the line is just some information about who generated the keys, and where, and can be removed or replaced by something more informative\\ <​code>​ > cat id_rsa.pub +    * The **//public// key**: ''​id_ed25519**.pub**''​ 
-ssh-rsa AAAAB3NzaC1yc2EAAAADAQ ​[ lots of cryptic characters ] 8WPbpreOOrIbNw== ​my_login@my_machine</​code>​+      * This is the //​key// ​that **you can share**, or that you have to send when opening an account on [[https://​mesocentre.ipsl.fr/​account-opening/​|IPSL Mésocentre ESPRI]]. 
 +        * Note that the ''​my_login@my_machine'' ​string ​at the end of the line is just some information about who generated the keys, and where, and can be removed or replaced by something more informative 
 +      * <​code>​ > cat id_ed25519.pub 
 +ssh-ed25519 AAAAC3NzaC1lZDI1NT ​[ lots of cryptic characters ] Frx8rRFKthpmqRdkXl ​my_login@my_machine</​code>​
  
-=== Generating or importing keys with PuTTY on a Windows ​computer ​===+=== Generating or importing keys with PuTTY (Windows===
  
-Read the [[other:​putty_conf#​importing_or_creating_ssh_keys_with_puttygen|Importing ​or creating ​ssh keys with PuTTYgen]] ​section+Read [[other:​putty_conf#​converting_existing_ssh_keys_with_puttygen|Converting existing ssh keys with PuTTYgen]], ​or [[other:​putty_conf#​creating_ssh_keys_with_puttygen|Creating ​ssh keys with PuTTYgen]]
  
 ==== Installing ssh keys ==== ==== Installing ssh keys ====
Line 259: Line 362:
  
  
-==== Using an ssh agent ====+===== Using an ssh agent =====
  
 +An //ssh agent// running on your computer will securely store your passphrase and supply it to applications that use ''​ssh''​ on your computer to connect to remote Linux servers
  
 +  * start the //agent// when you open a new session on your computer
 +  * enter your //ssh passphrase//​(s). An agent can store multiple passphrases associating multiple pairs of private and public ssh keys, but you can also use the same public key on different remote Linux servers
 +  * the //ssh agent// will store your passphrase until you exit your session, or restart your computer ​
 +
 +==== Linux ssh agent ====
 +
 +FIXME
 +
 +==== Windows ssh agent ====
 +
 +  * On Windows, we recommend using [[other:​putty_conf#​using_the_private_key_in_pageant|Pageant/​PuTTY]] as an //ssh agent//, because:
 +    * [[other:​putty_conf#​launching_putty_pageant|Pageant/​PuTTY]] also offers a very easy and convenient way to define profiles to connect to your favorite servers
 +    * Some programs that use ''​ssh''​ to transfer files will automatically use the keys stored in ''​Pageant'':​ [[other:​win10apps#​winscp|WinSCP]],​ [[other:​emacs_doc|emacs]],​ ...
 +
 +  * It is also possible (but less convenient) to use the Windows built-in ''​ssh-agent''​ and ''​ssh''​ commands!
 +    * Note that the //agent service// is not activated by default and you will get the following error when you try to use ''​ssh-add''​
 +      * <​code>​C:​ > ssh-add
 +Error connecting to agent: No such file or directory</​code>​
 +    * It is necessary to first **activate the agent Windows //​service//​** (in an **//​elevated//​ PowerShell**,​ i.e with //​Administrator//​ privileges) as explained in the [[https://​learn.microsoft.com/​en-us/​windows-server/​administration/​openssh/​openssh_keymanagement#​user-key-generation|User key generation]] section
 +      * <​code>​C:​ > Get-Service ssh-agent
 +Status ​  ​Name ​              ​DisplayName
 +------ ​  ​---- ​              ​-----------
 +Stopped ​ ssh-agent ​         OpenSSH Authentication Agent
 +
 +C: > Get-Service ssh-agent | Set-Service -StartupType Automatic
 +
 +C: > Start-Service ssh-agent
 +
 +C: > Get-Service ssh-agent
 +Status ​  ​Name ​              ​DisplayName
 +------ ​  ​---- ​              ​-----------
 +Running ​ ssh-agent ​         OpenSSH Authentication Agent
 +
 +C: > ssh-add
 +Enter passphrase for C:​\Users\your_login/​.ssh/​id_dsa:​ XXXX_Type_Your_Passphrase_Here_XXXX
 +Identity added: C:​\Users\your_login/​.ssh/​id_dsa
 +Identity added: C:​\Users\your_login/​.ssh/​id_ed25519
 +
 +C: > ssh-add -l
 +1024 SHA256:/​vC3Ma6s9Wj[Some_Summary_Info_About_The_Key]c1Q4 (DSA)
 +256 SHA256:​8BGKU+zBnJXH[Some_Summary_Info_About_The_Key]2Al8 jypeter@obelix5 (ED25519)</​code>​
 +
 +==== Mac ssh agent ====
 +
 +<WRAP center round tip 60%>
 +Nothing to configure!
 +</​WRAP>​
  
 +The ''​ssh-agent''​ process will be automatically started, if a user runs ''​ssh''​ or ''​ssh-add''​!
  
 +Check the [[https://​support.apple.com/​guide/​terminal/​script-management-with-launchd-apdc6c1077b-5d5d-4d35-9c19-60f2397b2369/​mac|launchd documentation]] if you need more details
 +<​code>​$ launchctl list | grep ssh
 +      7240 0 com.openssh.ssh-agent</​code>​
 +      ​
 +It seems that, once used, the passphrase will be automatically stored in the [[https://​support.apple.com/​guide/​keychain-access/​what-is-keychain-access-kyca1083/​mac|Keychain]] for future sessions
 ===== More... ===== ===== More... =====
  
-  * If you want to know more (options, etc...), check the //man(ual) page// on Linux: ''​man ssh''​ +  * If you want to **know more** (options, etc...), check the //man(ual) page// on Linux: ''​man ssh''​ 
-  * emacs+  * Editing **remote text files** with emacs (and automatic ''​scp''​):​ check the ''​emacs''​ section of the [[other:​newppl:​starting#​text_editors|Text editors]] 
 +  * Digging **ssh tunnels** when using //ipython notebooks//
  
 /* standard page footer */ /* standard page footer */
other/ssh.1598881318.txt.gz · Last modified: 2020/08/31 13:41 by jypeter