PMIP3 wiki

User Tools

Site Tools

You are here: index » other » available_servers
other:available_servers

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
other:available_servers [2026/08/07 17:48] – created, with the Common sense section jypeterother:available_servers [2026/08/25 16:19] (current) – [IPSL servers] Added some IPSL content jypeter
Line 4: Line 4:
 ===== Common sense reminders ===== ===== Common sense reminders =====
  
-  * You can do what you want on **your** local laptop/desktop (except get viruses)+  * You can do what you want on **your local laptop/desktop** (except get viruses)
  
-  * Be **careful** when you use remote servers **shared** with other users +  * **Be very careful when you use remote interactive servers shared with other users** 
-    * Others should not even notice that you are there! Yes, you can use computing resources cleanly and efficiently...+    * Others should not even notice that you(r processes) are there!\\ Yes, **you** can use computing resources cleanly and efficiently... 
 +    * Note: notebook servers are shared servers!
     * If you have some heavy duty processing to do:     * If you have some heavy duty processing to do:
-      * Use shared servers only for quick tests (with a subset your huge dataset) +      * Use shared interactive servers only for quick tests (ideally with a subset of your huge dataset) 
-        * notebook servers are shared servers! +      * Use dedicated batch/computing clusters for //heavy// processing 
-      * Use dedicated batch/computing clusters for heavy processing +        * //heavy// = 
-        * heavy = will run a long time and/or use a lot of CPU and/or use a lot of RAM and/or will do a lot of disk reading/writing, ... +          * will run for a **long** time 
-        * Resources you use in High Performance Centers will probably be deducted from a (shared) //project//+          * and/or use a **lot** of CPU (one or many CPUs) 
 +          * and/or use a **lot** of RAM 
 +          * and/or do a **lot** of disk reading/writing, 
 +          * ... 
 +        * resources you use in High Performance Computing centers will probably be deducted from a (shared) //project//
  
-  * When possible, avoid duplicating data and run your computations //near// the data +  * **Disk space (and backup space) is also an important resource!** 
-    * Disk space is also a resource! Do not forget to remove temporary data+    * When possible, **avoid duplicating data**, and run your computations //near// the data 
 +      * e.g. use the //[[https://mesocentre.ipsl.fr/jeux-de-donnees/|mirror data]]// stored in ''/bdd/'' on the ''spirit[x]'' servers 
 +    * Use //scratch// disks, when available, for **temporary data** and do not forget to remove temporary data 
 +      * LSCE: ''/home/scratch01/$USER'' 
 +      * spiritx: ''/scratchx/$USER'' 
 +      * TGCC: ''$SCRATCH'' 
 +    * Do not store temporary data, or replicated data, or data that can be easily re-created, on backed-up disks
  
-  * Ask your experienced colleagues when you are not sure about something (sometimes even if you are sure...)+  * **Ask your experienced colleagues** when you are not sure about something (sometimes even if you are sure...) 
 +    * you can get better domain specific answers from your colleagues than from chatbots
     * There are no stupid questions (especially for beginners)     * There are no stupid questions (especially for beginners)
     * There are always different ways to do things, some are much less efficient than others     * There are always different ways to do things, some are much less efficient than others
  
-  * Learn how to monitor resources usage: ''top'', ''df'', ''du'', ...+  * **Learn how to monitor resources usage**: 
 +    * CPU and RAM: ''[[other:newppl:starting#determining_the_load_of_a_linux_server|top]]'' 
 +    * Disk: ''[[other:newppl:starting#quota_command|quota]]'', ''[[other:newppl:starting#du_disk_usage_command|du]]'', ''[[other:newppl:starting#df_command|df]]'' 
 +    * Listing and killing processes: ''[[other:newppl:starting#listing_processes_with_ps|ps]]'', ''[[other:newppl:starting#terminating_processes_will_kill|kill]]'' 
 +    * ... 
 + 
 +  * **Read (some of) the //local// documentation**\\ <wrap hi>[[https://en.wikipedia.org/wiki/RTFM|RTFM]]</wrap> (Read The Fine Manual) 
 +    * [[other:newppl:starting#useful_documentation_for_beginners|Some Linux related documents]] (by JYP) 
 +      * Learn at least how to use the ''[[other:newppl:starting#the_man_command|man]]''(ual) command !! 
 +    * [[https://intranet.lsce.ipsl.fr/informatique/en/|LSCE documentation]] 
 +    * [[https://documentations.ipsl.fr/MESO_User/|IPSL Computing and Data Center documentation]] 
 +    * [[https://www-hpc.cea.fr/tgcc-public/en/html/tgcc-public.html|TGCC documentation]] 
 +    * [[http://www.idris.fr/en/docs/jean-zay/nouvel-utilisateur/|Jean Zay @ IDRIS documentation]] 
 +    * ... 
 + 
 +===== Accessing the servers ===== 
 + 
 +==== Connecting to the servers ==== 
 + 
 +  * We use **''[[other:ssh|ssh]]''** and //ssh clients// to connect securely to the servers we work with 
 +    * Use the [[other:putty_conf|PuTTY]] ssh //client// to easily open a terminal on a remote server from a Windows computer 
 + 
 +  * Some computing resources can also be accessed with a web browser and a [[https://jupyter.org/|JupyterHub server]] 
 +==== Transferring data ==== 
 + 
 +  * Servers can usually only access data on local (in the same computer room) disks. If you need to work locally on data available on a remote server, you first have to download the remote data 
 +    * Notable exceptions : 
 +      * ''spirit**x**'' servers in Palaiseau campus can access ''spirit'' servers disks at Sorbonne Université, in //read-only// mode, and the other way around ([[https://documentations.ipsl.fr/MESO_User/|details]]) 
 +      * some of the data available in ''/bdd'' on the ''spirit[x]'' servers are actually distributed from HPC centers 
 +    * Reminder: when possible, **avoid duplicating data**, and **perform the data analysis near the (original) data** 
 +    * When possible, **//reduce// the data** before download, in order to transfer only the subset you need: e.g. select only the regions and variables you need, pre-compute spatial or temporal means, etc...\\ You can easily reduce NetCDF data with: 
 +      * [[https://code.mpimet.mpg.de/projects/cdo/wiki|cdo]] (Climate Data Operators) 
 +      * [[https://nco.sourceforge.net/|NCO]] (netCDF Operators) 
 +    * It is of course a good idea to document (in a simple text file) where the files come from, how they were pre-processed, ... 
 + 
 +  * We use transfer programs based on the //ssh protocol// to move data securely 
 +    * [[other:ssh#copying_files_with_scp|scp]] (secure copy): **easily** copy files and directories recursively 
 +      * Use the [[other:win10apps#winscp|WinSCP]] //scp client// to easily copy files to/from a Windows computer 
 +    * [[other:ssh#mirroring_directories_with_rsync|rsync]]: mirror directories (more complex) 
 + 
 + 
 +==== Checking data integrity ==== 
 + 
 +If you need to check that a (potentially big) file is identical to its remote version, you can first check that the file sizes are identical, and then **check data integrity** by comparing the //checksum// (with ''sha256sum'' or ''[[https://en.wikipedia.org/wiki/Md5sum|md5sum]]'') of the original and copied files. It is more efficient and secure to compare this unique fingerprint than to transfer a file again (and use ''diff'') 
 + 
 +e.g. with a 14 Gb file on spirit: 
 + 
 +<code>$ cd /bdd/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/piControl/r1i1p1f1/Omon/thetao/gn/latest 
 + 
 +$ ls -l thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 
 +[...] 14293466154 [...] thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 
 + 
 +$ ls -lh thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 
 +[...] 14G [...] thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 
 + 
 +$ md5sum thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 
 +5378634e97136d75e4517cb72e0cda9a  thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 
 + 
 +$ sha256sum thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 
 +90cf48c540216897bf4d9841acd7034f85cc10e233eed74b1aa27fd77a8da1e6  thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc</code> 
 +===== LSCE servers ===== 
 + 
 + 
 +==== The LSCE gateway ==== 
 + 
 +''ssh1.lsce.ipsl.fr'' 
 +    * Use ''ssh ssh1.lsce.ipsl.fr'' when you are **outside the LSCE wired network**, before accessing any other LSCE server: 
 +      * By **//outside LSCE//**, we mean that your computer is: 
 +        * connected to the LSCE WiFi network (instead of the wired network) 
 +        * or physically outside LSCE 
 +    * You can also use ''ssh1'' when you are **copying files to/from outside LSCE** with ''scp'' or ssh-based tools 
 +    * <wrap hi>The ''ssh1'' gateway can **only** be used for copying files or accessing the LSCE interactive servers</wrap>! 
 +      * **You have to** connect to an ''obelix//NN//'' interactive server before doing any real work 
 + 
 + 
 +==== The LSCE interactive servers ==== 
 + 
 +''obelix//NN//'' (''obelix2'' to ''obelix5'') 
 +    * Use ''ssh obelix'' to access the LSCE servers from **inside LSCE**.\\ The //load balancing// system will send you to the ''obelix//NN//'' server that has currently the smallest load. 
 +      * By **//inside LSCE//**, we mean that: 
 +        * your computer is connected to the LSCE **wired** network 
 +        * or you are using the [[https://intranet.lsce.ipsl.fr/informatique/fr/vpn-forticlient.php#vpnlsce|LSCE VPN]] 
 +        * or you have a //terminal// connected to the ''ssh1'' gateway 
 +    * <wrap hi>Never forget that you are **sharing** these interactive servers with other users!</wrap> 
 +      * <wrap hi>Do not use too much CPU and/or memory for a long time</wrap>. Heavy computation should be done on the [[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|LSCE cluster]] 
 +      * Learn how to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the top command]] to determine the current load of a server and the CPU/memory usage of your processes ! 
 +    * You can use one of the following commands if you want to know on which ''obelixNN'' you are 
 +      * <code>$ ssh obelix 
 +Last login: Mon Jun  3 08:49:53 2019 from somewhere 
 +$ echo $HOST 
 +obelix5 
 +$ hostname 
 +obelix5</code> 
 +    * In a python script, you can use the following to get some information 
 +      * <code>>>> import os 
 +>>> os.getlogin(), os.getenv('HOST') 
 +('mylogin', 'obelix5')</code> 
 + 
 + 
 +==== The LSCE cluster ==== 
 + 
 +  * **[[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers 
 +    * //Heavy duty// programs are programs that will prevent other users from working on the same server because these programs use a lot of CPU and/or memory for more than a few minutes. Ask your advisor, if you are not sure 
 + 
 + 
 +==== The LSCE JupyterHub notebook server ==== 
 + 
 +    * Read the [[other:python:starting#using_jupyterhub_servers|Using JupyterHub servers]] section of the //Working with Python// page 
 + 
 + 
 +===== IPSL servers ===== 
 + 
 + 
 + 
 +  * **The [[https://documentations.ipsl.fr/spirit/|IPSL Computing and Data Center]]**, aka //Mésocentre ESPRI// (''Spirit'' / ''SpiritX'' / ''HAL'') 
 +    * Note: the ''Spirit''/''X'' clusters have replaced the ''ciclad''/''climserv'' cluster (in 2022) 
 +    * You will need a specific account (different from your //LSCE// account) to access these servers and use data files that are already available/mirrored there (e.g. **''CMIPn'' data**).\\ **Do not duplicate** at LSCE data  already available on IPSL servers and that you can process there! 
 +    * [[https://documentations.ipsl.fr/spirit/getting_started/account.html|Requesting an IPSL account]] 
 +    * [[https://documentations.ipsl.fr/spirit/|IPSL ESPRI Mesocenter documentation]] 
 +      * [[https://documentations.ipsl.fr/spirit/spirit_clusters/head_nodes.html|spirit clusters]] 
 +    * [[https://documentations.ipsl.fr/spirit/getting_started/support.html|Getting help]] 
 +===== HPC centers =====
  
  
other/available_servers.1786117695.txt.gz · Last modified: by jypeter
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki