Differences

This shows you the differences between two versions of the page.

--- other:available_servers [2026/08/24 11:45] – [Transferring data] Started adding content jypeter
+++ other:available_servers [2026/08/25 16:19] (current) – [IPSL servers] Added some IPSL content jypeter
@@ Line 54: / Line 54: @@
   * We use **''[[other:ssh|ssh]]''** and //ssh clients// to connect securely to the servers we work with
+    * Use the [[other:putty_conf|PuTTY]] ssh //client// to easily open a terminal on a remote server from a Windows computer
   * Some computing resources can also be accessed with a web browser and a [[https://jupyter.org/|JupyterHub server]]
 ==== Transferring data ====
-  * Servers can access data on local disks. If you need to work locally on data available on a remote server, you have to transfer the data first
+  * Servers can usually only access data on local (in the same computer room) disks. If you need to work locally on data available on a remote server, you first have to download the remote data
-    * avoid duplicating data
+    * Notable exceptions :
-    * computation near the data
+      * ''spirit**x**'' servers in Palaiseau campus can access ''spirit'' servers disks at Sorbonne Université, in //read-only// mode, and the other way around ([[https://documentations.ipsl.fr/MESO_User/|details]])
-    * reduce the data
+      * some of the data available in ''/bdd'' on the ''spirit[x]'' servers are actually distributed from HPC centers
-    * IPSL data on the spirit[x] servers
+    * Reminder: when possible, **avoid duplicating data**, and **perform the data analysis near the (original) data**
-    * opendap servers
+    * When possible, **//reduce// the data** before download, in order to transfer only the subset you need: e.g. select only the regions and variables you need, pre-compute spatial or temporal means, etc...\\ You can easily reduce NetCDF data with:
+      * [[https://code.mpimet.mpg.de/projects/cdo/wiki|cdo]] (Climate Data Operators)
+      * [[https://nco.sourceforge.net/|NCO]] (netCDF Operators)
+    * It is of course a good idea to document (in a simple text file) where the files come from, how they were pre-processed, ...
+  * We use transfer programs based on the //ssh protocol// to move data securely
+    * [[other:ssh#copying_files_with_scp|scp]] (secure copy): **easily** copy files and directories recursively
+      * Use the [[other:win10apps#winscp|WinSCP]] //scp client// to easily copy files to/from a Windows computer
+    * [[other:ssh#mirroring_directories_with_rsync|rsync]]: mirror directories (more complex)
+==== Checking data integrity ====
+If you need to check that a (potentially big) file is identical to its remote version, you can first check that the file sizes are identical, and then **check data integrity** by comparing the //checksum// (with ''sha256sum'' or ''[[https://en.wikipedia.org/wiki/Md5sum|md5sum]]'') of the original and copied files. It is more efficient and secure to compare this unique fingerprint than to transfer a file again (and use ''diff'')
+e.g. with a 14 Gb file on spirit:
+<code>$ cd /bdd/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/piControl/r1i1p1f1/Omon/thetao/gn/latest
+$ ls -l thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
+[...] 14293466154 [...] thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
+$ ls -lh thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
+[...] 14G [...] thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
+$ md5sum thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
+5378634e97136d75e4517cb72e0cda9a  thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
+$ sha256sum thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
+cf48c540216897bf4d9841acd7034f85cc10e233eed74b1aa27fd77a8da1e6  thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc</code>
 ===== LSCE servers =====
+==== The LSCE gateway ====
+''ssh1.lsce.ipsl.fr''
+    * Use ''ssh ssh1.lsce.ipsl.fr'' when you are **outside the LSCE wired network**, before accessing any other LSCE server:
+      * By **//outside LSCE//**, we mean that your computer is:
+        * connected to the LSCE WiFi network (instead of the wired network)
+        * or physically outside LSCE
+    * You can also use ''ssh1'' when you are **copying files to/from outside LSCE** with ''scp'' or ssh-based tools
+    * <wrap hi>The ''ssh1'' gateway can **only** be used for copying files or accessing the LSCE interactive servers</wrap>!
+      * **You have to** connect to an ''obelix//NN//'' interactive server before doing any real work
+==== The LSCE interactive servers ====
+''obelix//NN//'' (''obelix2'' to ''obelix5'')
+    * Use ''ssh obelix'' to access the LSCE servers from **inside LSCE**.\\ The //load balancing// system will send you to the ''obelix//NN//'' server that has currently the smallest load.
+      * By **//inside LSCE//**, we mean that:
+        * your computer is connected to the LSCE **wired** network
+        * or you are using the [[https://intranet.lsce.ipsl.fr/informatique/fr/vpn-forticlient.php#vpnlsce|LSCE VPN]]
+        * or you have a //terminal// connected to the ''ssh1'' gateway
+    * <wrap hi>Never forget that you are **sharing** these interactive servers with other users!</wrap>
+      * <wrap hi>Do not use too much CPU and/or memory for a long time</wrap>. Heavy computation should be done on the [[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|LSCE cluster]]
+      * Learn how to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the top command]] to determine the current load of a server and the CPU/memory usage of your processes !
+    * You can use one of the following commands if you want to know on which ''obelixNN'' you are
+      * <code>$ ssh obelix
+Last login: Mon Jun  3 08:49:53 2019 from somewhere
+$ echo $HOST
+obelix5
+$ hostname
+obelix5</code>
+    * In a python script, you can use the following to get some information
+      * <code>>>> import os
+>>> os.getlogin(), os.getenv('HOST')
+('mylogin', 'obelix5')</code>
+==== The LSCE cluster ====
+  * **[[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers
+    * //Heavy duty// programs are programs that will prevent other users from working on the same server because these programs use a lot of CPU and/or memory for more than a few minutes. Ask your advisor, if you are not sure
+==== The LSCE JupyterHub notebook server ====
+    * Read the [[other:python:starting#using_jupyterhub_servers|Using JupyterHub servers]] section of the //Working with Python// page
@@ Line 70: / Line 147: @@
+  * **The [[https://documentations.ipsl.fr/spirit/|IPSL Computing and Data Center]]**, aka //Mésocentre ESPRI// (''Spirit'' / ''SpiritX'' / ''HAL'')
+    * Note: the ''Spirit''/''X'' clusters have replaced the ''ciclad''/''climserv'' cluster (in 2022)
+    * You will need a specific account (different from your //LSCE// account) to access these servers and use data files that are already available/mirrored there (e.g. **''CMIPn'' data**).\\ **Do not duplicate** at LSCE data  already available on IPSL servers and that you can process there!
+    * [[https://documentations.ipsl.fr/spirit/getting_started/account.html|Requesting an IPSL account]]
+    * [[https://documentations.ipsl.fr/spirit/|IPSL ESPRI Mesocenter documentation]]
+      * [[https://documentations.ipsl.fr/spirit/spirit_clusters/head_nodes.html|spirit clusters]]
+    * [[https://documentations.ipsl.fr/spirit/getting_started/support.html|Getting help]]
 ===== HPC centers =====