====== Linux servers used by LSCE users ====== ===== Common sense reminders ===== * You can do what you want on **your local laptop/desktop** (except get viruses) * **Be very careful when you use remote interactive servers shared with other users** * Others should not even notice that you(r processes) are there!\\ Yes, **you** can use computing resources cleanly and efficiently... * Note: notebook servers are shared servers! * If you have some heavy duty processing to do: * Use shared interactive servers only for quick tests (ideally with a subset of your huge dataset) * Use dedicated batch/computing clusters for //heavy// processing * //heavy// = * will run for a **long** time * and/or use a **lot** of CPU (one or many CPUs) * and/or use a **lot** of RAM * and/or do a **lot** of disk reading/writing, * ... * resources you use in High Performance Computing centers will probably be deducted from a (shared) //project// * **Disk space (and backup space) is also an important resource!** * When possible, **avoid duplicating data**, and run your computations //near// the data * e.g. use the //[[https://mesocentre.ipsl.fr/jeux-de-donnees/|mirror data]]// stored in ''/bdd/'' on the ''spirit[x]'' servers * Use //scratch// disks, when available, for **temporary data** and do not forget to remove temporary data * LSCE: ''/home/scratch01/$USER'' * spiritx: ''/scratchx/$USER'' * TGCC: ''$SCRATCH'' * Do not store temporary data, or replicated data, or data that can be easily re-created, on backed-up disks * **Ask your experienced colleagues** when you are not sure about something (sometimes even if you are sure...) * you can get better domain specific answers from your colleagues than from chatbots * There are no stupid questions (especially for beginners) * There are always different ways to do things, some are much less efficient than others * **Learn how to monitor resources usage**: * CPU and RAM: ''[[other:newppl:starting#determining_the_load_of_a_linux_server|top]]'' * Disk: ''[[other:newppl:starting#quota_command|quota]]'', ''[[other:newppl:starting#du_disk_usage_command|du]]'', ''[[other:newppl:starting#df_command|df]]'' * Listing and killing processes: ''[[other:newppl:starting#listing_processes_with_ps|ps]]'', ''[[other:newppl:starting#terminating_processes_will_kill|kill]]'' * ... * **Read (some of) the //local// documentation**\\ [[https://en.wikipedia.org/wiki/RTFM|RTFM]] (Read The Fine Manual) * [[other:newppl:starting#useful_documentation_for_beginners|Some Linux related documents]] (by JYP) * Learn at least how to use the ''[[other:newppl:starting#the_man_command|man]]''(ual) command !! * [[https://intranet.lsce.ipsl.fr/informatique/en/|LSCE documentation]] * [[https://documentations.ipsl.fr/MESO_User/|IPSL Computing and Data Center documentation]] * [[https://www-hpc.cea.fr/tgcc-public/en/html/tgcc-public.html|TGCC documentation]] * [[http://www.idris.fr/en/docs/jean-zay/nouvel-utilisateur/|Jean Zay @ IDRIS documentation]] * ... ===== Accessing the servers ===== ==== Connecting to the servers ==== * We use **''[[other:ssh|ssh]]''** and //ssh clients// to connect securely to the servers we work with * Use the [[other:putty_conf|PuTTY]] ssh //client// to easily open a terminal on a remote server from a Windows computer * Some computing resources can also be accessed with a web browser and a [[https://jupyter.org/|JupyterHub server]] ==== Transferring data ==== * Servers can usually only access data on local (in the same computer room) disks. If you need to work locally on data available on a remote server, you first have to download the remote data * Notable exceptions : * ''spirit**x**'' servers in Palaiseau campus can access ''spirit'' servers disks at Sorbonne Université, in //read-only// mode, and the other way around ([[https://documentations.ipsl.fr/MESO_User/|details]]) * some of the data available in ''/bdd'' on the ''spirit[x]'' servers are actually distributed from HPC centers * Reminder: when possible, **avoid duplicating data**, and **perform the data analysis near the (original) data** * When possible, **//reduce// the data** before download, in order to transfer only the subset you need: e.g. select only the regions and variables you need, pre-compute spatial or temporal means, etc...\\ You can easily reduce NetCDF data with: * [[https://code.mpimet.mpg.de/projects/cdo/wiki|cdo]] (Climate Data Operators) * [[https://nco.sourceforge.net/|NCO]] (netCDF Operators) * It is of course a good idea to document (in a simple text file) where the files come from, how they were pre-processed, ... * We use transfer programs based on the //ssh protocol// to move data securely * [[other:ssh#copying_files_with_scp|scp]] (secure copy): **easily** copy files and directories recursively * Use the [[other:win10apps#winscp|WinSCP]] //scp client// to easily copy files to/from a Windows computer * [[other:ssh#mirroring_directories_with_rsync|rsync]]: mirror directories (more complex) ==== Checking data integrity ==== If you need to check that a (potentially big) file is identical to its remote version, you can first check that the file sizes are identical, and then **check data integrity** by comparing the //checksum// (with ''sha256sum'' or ''[[https://en.wikipedia.org/wiki/Md5sum|md5sum]]'') of the original and copied files. It is more efficient and secure to compare this unique fingerprint than to transfer a file again (and use ''diff'') e.g. with a 14 Gb file on spirit: $ cd /bdd/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/piControl/r1i1p1f1/Omon/thetao/gn/latest $ ls -l thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc [...] 14293466154 [...] thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc $ ls -lh thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc [...] 14G [...] thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc $ md5sum thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 5378634e97136d75e4517cb72e0cda9a thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc $ sha256sum thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc 90cf48c540216897bf4d9841acd7034f85cc10e233eed74b1aa27fd77a8da1e6 thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc ===== LSCE servers ===== ==== The LSCE gateway ==== ''ssh1.lsce.ipsl.fr'' * Use ''ssh ssh1.lsce.ipsl.fr'' when you are **outside the LSCE wired network**, before accessing any other LSCE server: * By **//outside LSCE//**, we mean that your computer is: * connected to the LSCE WiFi network (instead of the wired network) * or physically outside LSCE * You can also use ''ssh1'' when you are **copying files to/from outside LSCE** with ''scp'' or ssh-based tools * The ''ssh1'' gateway can **only** be used for copying files or accessing the LSCE interactive servers! * **You have to** connect to an ''obelix//NN//'' interactive server before doing any real work ==== The LSCE interactive servers ==== ''obelix//NN//'' (''obelix2'' to ''obelix5'') * Use ''ssh obelix'' to access the LSCE servers from **inside LSCE**.\\ The //load balancing// system will send you to the ''obelix//NN//'' server that has currently the smallest load. * By **//inside LSCE//**, we mean that: * your computer is connected to the LSCE **wired** network * or you are using the [[https://intranet.lsce.ipsl.fr/informatique/fr/vpn-forticlient.php#vpnlsce|LSCE VPN]] * or you have a //terminal// connected to the ''ssh1'' gateway * Never forget that you are **sharing** these interactive servers with other users! * Do not use too much CPU and/or memory for a long time. Heavy computation should be done on the [[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|LSCE cluster]] * Learn how to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the top command]] to determine the current load of a server and the CPU/memory usage of your processes ! * You can use one of the following commands if you want to know on which ''obelixNN'' you are * $ ssh obelix Last login: Mon Jun 3 08:49:53 2019 from somewhere $ echo $HOST obelix5 $ hostname obelix5 * In a python script, you can use the following to get some information * >>> import os >>> os.getlogin(), os.getenv('HOST') ('mylogin', 'obelix5') ==== The LSCE cluster ==== * **[[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers * //Heavy duty// programs are programs that will prevent other users from working on the same server because these programs use a lot of CPU and/or memory for more than a few minutes. Ask your advisor, if you are not sure ==== The LSCE JupyterHub notebook server ==== * Read the [[other:python:starting#using_jupyterhub_servers|Using JupyterHub servers]] section of the //Working with Python// page ===== IPSL servers ===== * **The [[https://documentations.ipsl.fr/spirit/|IPSL Computing and Data Center]]**, aka //Mésocentre ESPRI// (''Spirit'' / ''SpiritX'' / ''HAL'') * Note: the ''Spirit''/''X'' clusters have replaced the ''ciclad''/''climserv'' cluster (in 2022) * You will need a specific account (different from your //LSCE// account) to access these servers and use data files that are already available/mirrored there (e.g. **''CMIPn'' data**).\\ **Do not duplicate** at LSCE data already available on IPSL servers and that you can process there! * [[https://documentations.ipsl.fr/spirit/getting_started/account.html|Requesting an IPSL account]] * [[https://documentations.ipsl.fr/spirit/|IPSL ESPRI Mesocenter documentation]] * [[https://documentations.ipsl.fr/spirit/spirit_clusters/head_nodes.html|spirit clusters]] * [[https://documentations.ipsl.fr/spirit/getting_started/support.html|Getting help]] ===== HPC centers ===== /* standard page footer */ \\ \\ \\ ---- [ [[pmip3:|PMIP3 Wiki Home]] ] - [ [[pmip3:wiki_help|Help!]] ] - [ [[wiki:syntax|Wiki syntax]] ]