Differences

This shows you the differences between two versions of the page.

--- other:newppl:starting [2024/03/20 11:27] – [Determining the load of a Linux server] Added details about TIME and mem usage jypeter
+++ other:newppl:starting [2025/08/29 15:26] (current) – [Dealing with disk space] Finished rewriting the section jypeter
@@ Line 17: / Line 17: @@
   * [[https://www.science-accueil.org/en/|Science Accueil]]
     * [[https://www.science-accueil.org/en/request-services/|Service request page]]
+  * [[https://gate.paris-saclay.fr/en/|GATE desk]]
   * {{:other:newppl:uvsq-guide-du-personnel-nov2019.pdf|Guide du Personnel UVSQ}}
@@ Line 34: / Line 35: @@
 <WRAP center round important 60%>
-It's very important that you update your phone/office information when you arrive at LSCE, so that other people can find you using the tools above!
+It's very important that you update **your** phone/office information when you arrive at LSCE, so that other people can find you using the tools above!
 </WRAP>
-=== Finding your phone number ===
+=== Finding YOUR phone number ===
-Use the following sequence of keys on the //Mitel// phone in your office:
+Use the following sequence of keys on the phone in your office:
-  - Select **MENU** (use the downward arrow)
+  - Select **MENU** (then use the downward arrow)
   - Select **PARAMETRES** (use the downward arrow)
   - Select **GENERAL** (use the downward arrow)
@@ Line 47: / Line 48: @@
     * You can be called from outside with ''01 69 08 NN NN''
-=== Updating your entry in the directory ===
+=== Updating YOUR entry in the phone directory ===
-  - Find your name in the [[https://intranet.lsce.ipsl.fr/en/Phocea/Membres/Annuaire/index.php|phone directory]]
+  - Find your name in the [[https://intranet.lsce.ipsl.fr/en/Phocea/Membres/Annuaire/index.php|intranet phone directory]]
   - Click on **Modifier mes coordonnées** (in blue on the right side)
   - Click on **Informations dans l'annuaire** and update the different fields
   - Click on **Envoyez votre demande**
+  - Wait (for someone to validate your request)
 ===== The LSCE mailing lists =====
@@ Line 220: / Line 222: @@
 $ hostname
 obelix5</code>
+    * In a python script, you can use the following to get some information
+      * <code>>>> import os
+>>> os.getlogin(), os.getenv('HOST')
+('mylogin', 'obelix5')</code>
   * **[[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers
     * //Heavy duty// programs are programs that will prevent other users from working on the same server because these programs use a lot of CPU and/or memory for more than a few minutes. Ask your advisor, if you are not sure
+  * The **''JupyterHub'' notebook servers**:
+    * Read the [[other:python:starting#using_jupyterhub_servers|Using JupyterHub servers]] section of the //Working with Python// page
   * **The [[https://documentations.ipsl.fr/spirit/|IPSL Computing and Data Center]]**, aka //Mésocentre ESPRI// (''Spirit'' / ''SpiritX'' / ''HAL'')
     * Note: the ''Spirit''/''X'' clusters have replaced the ''ciclad''/''climserv'' cluster (in 2022)
     * You will need a specific account (different from your //LSCE// account) to access these servers and use data files that are already available/mirrored there (e.g. **''CMIPn'' data**).\\ **Do not duplicate** at LSCE data  already available on IPSL servers and that you can process there!
-    * [[https://mesocentre.ipsl.fr/account-opening/|Requesting an IPSL account]]
+    * [[https://documentations.ipsl.fr/spirit/getting_started/account.html|Requesting an IPSL account]]
     * [[https://documentations.ipsl.fr/spirit/|IPSL ESPRI Mesocenter documentation]]
       * [[https://documentations.ipsl.fr/spirit/spirit_clusters/head_nodes.html|spirit clusters]]
@@ Line 367: / Line 377: @@
     * Use the following command to access the backup: ''cd ~/.snapshot''
-  * **Quota**: YES!\\ There is a limit to what you can store in your home directory. If you reach this limit, you may experience some seemingly random errors (can't connect to the servers, the new files have a 0 size, ...)\\ The example below shows how to determine the //quota// on the LSCE disks: you can never go above the size displayed in the ''limit column'', and you cannot stay more than 7 days above the limit displayed in the ''quota'' column! The ''grace'' column will display either ''None'' (you cannot write on the disk anymore, unless you get below the ''quota'' limit) or ''N days'' (you can use the disk, but you have ''N days'' to go below the ''quota'' limit)\\ <code>
+  * **Quota**: YES!\\ There is a limit to what you can store in your home directory. If you reach this limit, you may experience some seemingly random errors (can't connect to the servers, the new files have a 0 size, ...)
- > quota -s
-Disk quotas for user johndoe (uid 6369):
-     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
-prolix3:/users/   4658M   4883M   4981M           23503       0       0</code>
 ==== The scratch directory ====
@@ Line 403: / Line 407: @@
   * **Quota**: ask your advisor
-==== Determining the space used by directories ====
+==== Dealing with disk space ====
-Your Linux environment may sometimes stop working correctly because you have exceeded the allowed quota on your disks (check your quotas with ''quota -s''). You can use the following command to get the size of all the directories and files in the current directory, sorted by size: ''du -sh * | sort -rh''
+<WRAP center round important 60%>
+Running out of disk space may have strange (and difficult to debug) side effects!
+</WRAP>
-<code> >cd /home/scratch01/johndoe
+**Disk space** on remote servers may sometimes seems infinite, but **is a limited resource**, especially if other users are not as careful as you!
-/home/scratch01/johndoe
- >du -sh * | sort -rh
+You should regularly check how much space you are using (''quota'', ''du''), and how much space is left (''df'')
+=== 'quota' command ===
+The ''quota -s'' command will tell you:
+  * If your disk usage (total size and/or number of files) is limited on some of the disks you use
+    * It is usually the case on the //home// directories (LSCE, spirit, computing centers, ...)
+  * How much disk space you are currently using on the disks with //quota// control
+The examples below shows how to display the current //quota// on the LSCE and spirit disks:
+  * //home// quota at LSCE
+    * <code> > quota -s
+Disk quotas for user johndoe (uid 0007):
+     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
+prolix3:/users/   4658M   4883M   4981M           23503       0       0</code>
+  * //home// quota on spiritx
+    * <code> $ quota -s
+Disk quotas for user janedoe (uid 0008):
+     Filesystem   space   quota   limit   grace   files   quota   limit   grace
+.17.3.169:/home
+K  32768M  33792M             339    400k    420k</code>
+  * The ''blocks'' or ''space'' column display your current usage of the disk where the quota is activated
+  * You can never go above the size displayed in the **//hard//** ''limit'' column
+  * You cannot stay more than 7 days above the **//soft//** limit displayed in the ''quota'' column!
+  * The ''grace'' column will display either:
+    * nothing: everything is fine!!
+    * ''N days'': you can still use this disk, but you have ''N days'' (decreasing from 7 days) to remove files and get back below the //soft// ''quota'' limit
+    * ''None'': you cannot write to this disk anymore, unless you remove files to get below the ''quota'' limit
+=== 'du' (disk usage) command ===
+You can use ''du -sh some_dir'' to determine the space used by ''some_dir'' (and all its sub-directories)
+Examples:
+  * At LSCE ''du -sh /home/users/johndoe'' will display the size of the //home// directory. It should be roughly the same value as the ''blocks'' column displayed by ''quota''
+  * You can use the following command to get the size of all the directories and files in the current directory, sorted by decreasing size:
+    * ''du -sh * | sort -rh''
+      * If you don't understand the //pipe// (''|'' character) above, read //Combining commands// in //[[#linux_unix|Useful Linux commands for working with (potentially big) text files (houchesa4_updated.pdf)]]//
+    * <code> > cd /home/scratch01/johndoe
+ > du -sh * | sort -rh
 M     GLAC1DHiceF26.nc
 M     dashboard
@@ Line 419: / Line 468: @@
 ...</code>
-FIXME Add something about ''df''
+=== 'df' command ===
+''df -h'' will display the total size of a disk, and how much space is left. A disk may be full, even if you have not reached your quota (because it is full of fools)
+Examples:
+  * <code> > df -h /home/users
+Filesystem      Size  Used Avail Use% Mounted on
+prolix3:/users  3.6T  2.1T  1.4T  60% /home/users</code>
+  * <code> >df -h /home/scratch01
+Filesystem                       Size  Used Avail Use% Mounted on
+prolix3.lsce.ipsl.fr:/scratch01   14T   13T  127G 100% /home/scratch01</code>
 ===== Which programs should you use? =====
-Your advisor will let you know which (version of which) programs you should use. Some programs are available by default, and you have to use the ''module'' command to access other programs.
+Your advisor will let you know which (version of which) programs you should use. Some programs are available by default, and you have to use the ''[[other:newppl:starting#using_module_to_access_optional_programs|module]]'' command to access other programs (or access a specific version of a program).
 ==== Which program am I using? ====
@@ Line 445: / Line 507: @@
 ==== Using module to access optional programs ====
-Main ''module'' options:
+=== Documentation ===
+  * [[https://documentations.ipsl.fr/spirit/common/module_command.html|IPSL Meso documentation]]
+  * [[https://modules.readthedocs.io/|Full and official module documentation]]
+=== Most useful module options ===
   * ''module list'': list the currently loaded modules
-  * ''module avail'': list the available modules
+  * ''module avail [mod_name]'': list the available modules (or only the ones with a name starting with ''mod_name'')
+  * '' module avail -l 2>&1 | grep subname'': list the modules that have ''subname'' in their name, and their installation date
+    * <wrap hi>This only works in a //bash// shell!</wrap> In a //tcsh// shell, start by invoking a //bash// shell with ''bash -l''
   * ''module load module_a ... module_n'': load modules (automatically including dependencies)
     * ''module load module_a'': load default version of //module_a//
@@ Line 453: / Line 523: @@
   * ''module purge'': remove all modules
-Detailed example:
+=== Detailed example ===
 <code> > which ncview
 ncview: Command not found.
@@ Line 460: / Line 531: @@
 No Modulefiles Currently Loaded.
- > module avail
+ > module avail ncview
-[...]
+ncview/2.1.7
-batch_env          grib_api/1.14.0    netcdf/4           python/3.6
-[...]
-glost/0.3.1        ncview/2.1.7       python/2.7
  > module load ncview netcdf/4
@@ Line 476: / Line 544: @@
  > which ncdump
-/usr/local/install/netcdf-4.3.2/bin/ncdump</code>
+/usr/local/install/netcdf-4.3.2/bin/ncdump
+ > bash -l
+    $ module avail -l 2>&1 | grep gcc
+binutils/2.43.1-gcc14-acc                            2025/01/27 21:09:36
+gcc/14.2.0-openacc                                   2025/01/24  8:11:14
+gcc/8.3.0                                            2021/04/21  8:27:35
+netcdf/4.9.2-gcc14-acc                               2025/01/27 22:02:57
+netcdf/4.9.2-gcc8                                    2024/06/05 20:49:36
+R/4.0.3-gcc8                                         2022/02/03 12:53:55</code>
 ===== Useful documentation for beginners =====
@@ Line 487: / Line 564: @@
   * Useful Linux commands for working with (potentially big) text files ({{:other:newppl:houchesa4_updated.pdf}})
   * Using [[other:ssh|ssh]] and [[other:ssh#copying_files_with_scp|scp]]
+  * See also the //old// [[http://www.idris.fr/formations/utilisation-unix/unix-u-cours.html|IDRIS (advanced) Linux training]] (French tutorial)
 ==== Text editors ====
@@ Line 497: / Line 576: @@
 There are **lots of text editors available** on Linux computers/servers (''emacs'', ''nano'', ''vi'', ''vim'', ''gvim'', ''gedit'', ...). You can use them:
   * In basic **text terminals**
-    * useful if you are in a hurry, and/or working on a remote server with a bad network connection or have no local [[other:x_conf|X server]] running
+    * **If you only need to read a text file**, use a //pager// like [[http://www.greenwoodsoftware.com/less|less]] (or ''more''): ''less my_text_file.txt''
-    * you should know the basic commands and shortcuts! Some text editors (e.g ''nano'') fortunately display some help at the bottom of the screen
+      * ''SPACE'' goes down one page
+      * ''b'' (//back//) goes back one page
+      * ''/STRING'' looks for ''STRING'' in the text (then ''n''ext, ''p''revious, ''q''uit)
+      * ''q'' exits
+    * **[[#gnu_nano|nano]] is probably the easiest editor in text mode** to use!
+    * Useful if you are in a hurry, and/or working on a remote server with a bad network connection or have no local [[other:x_conf|X server]] running
+    * You should know the basic commands and shortcuts!
   * With a **full Graphical User Interface** (aka //GUI//) allowing you to use the mouse+menus/buttons/etc...
@@ Line 508: / Line 594: @@
 <note important>Read the **dedicated [[other:emacs_doc|Using emacs]] page**, and become a power user!</note>
+=== Gnu nano ===
+<note tip>Gnu [[https://www.nano-editor.org/|nano]] is probably the easiest text editor in text mode to use!</note>
+You can easily view/edit a file by just typing ''nano my_text_file.txt'' in a terminal, and looking at the instructions displayed at the bottom of the screen (e.g. ''^X Exit'' means that you can exit the editor by typing ''CTRL-X'')
 === vi (vim, gvim) ===
@@ Line 523: / Line 613: @@
   * What are binary ({{:other:newppl:binaire_jyp.pdf}}) and NetCDF ({{:other:newppl:netcdf_jyp.pdf}}) files?
+    * A reminder about [[other:python:misc_by_jyp#numerical_values|numerical precision]]
     * [[http://www.unidata.ucar.edu/software/netcdf/|Network Common Data Form (NetCDF)]] official page
       * [[http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#How-many-netCDF-formats-are-there-and-what-are-the-differences-among-them|The different netCDF formats]] (and do not forget to use ''ncdump -k data.nc'' if you need to know the format)
@@ Line 532: / Line 623: @@
   * Displaying the content of a NetCDF file in a text format\\ Check the ''ncdump'' [[https://docs.unidata.ucar.edu/nug/current/netcdf_utilities_guide.html#ncdump_guide|options]]. People usually want to quickly check the metadata, and use other programs to work with the actual data
-    * ''ncdump some_climate_data_file.nc'': dump the full content to text
+    * ''ncdump -h some_climate_data_file.nc'': only display the metadata (//h_eader// of the file)
-    * ''ncdump -h some_climate_data_file.nc'': only display the metadata
+    * ''ncdump some_climate_data_file.nc | less'': dump the **full** content to text, and use ''less'' to display it page by page
+      * Maybe it is time to learn how to {{:other:newppl:houchesa4_updated.pdf|work efficiently with big text files}}
   * You can use [[other:win7apps#panoply|Panoply]] and [[http://meteora.ucsd.edu/~pierce/ncview_home_page.html|ncview]] for quickly visualizing NetCDF files
   * Manipulating NetCDF files:
     * [[https://code.zmaw.de/projects/cdo/wiki/Cdo#Documentation|Climate Data Operators]] (''cdo'')
+      * Use ''[[#using_module_to_access_optional_programs|module]] load cdo'' on the LSCE and IPSL servers
     * [[http://nco.sourceforge.net/#RTFM|netCDF Operator]] (//NCO// operators)
+      * Use ''[[#using_module_to_access_optional_programs|module]] load nco'' on the LSCE and IPSL servers
 === NetCDF related Conventions ===
+  * [[http://cfconventions.org/|NetCDF CF (Climate and Forecast) Metadata Conventions]]
   * [[https://arxiv.org/abs/1911.08638|Gridspec: A standard for the description of grids used in Earth System models]]
     * A nice document for discovering the different type of (ir)regular grids used in climate models!
-  * [[http://cfconventions.org/|NetCDF CF (Climate and Forecast) Metadata Conventions]]
   * [[https://ugrid-conventions.github.io/ugrid-conventions/|UGRID Conventions]] for storing unstructured (or flexible mesh) model data
   * [[http://sgrid.github.io/sgrid/|SGRID Conventions]] for storing staggered data on structured grids that are consistent with the UGRID conventions
@@ Line 554: / Line 650: @@
 Check the [[other:python:jyp_steps#using_netcdf_files_with_python|dedicated section]] of the //Python// page
-==== Programming languages ====
+==== Programming languages, parallelism, ... ====
   * [[other:index#python_notes|JYP's Python notes]]
+  * [[http://www.idris.fr/eng/formations/supports_de_cours-eng.html|IDRIS classes and tutorials]]
 ===== Getting help (from the LSCE system administrators) =====
@@ Line 599: / Line 696: @@
 {{ :other:newppl:xnview_capture.jpg?200 |Click to get a larger version}}
-===== Getting help, when using the IPSL Mesocenter =====
+===== Getting help, when using the IPSL Mesocenter (ESPRI) =====
-Check the [[https://mesocentre.ipsl.fr/contacts//|Contacts]] page
+Check the [[https://documentations.ipsl.fr/spirit/getting_started/support.html|ESPRI User Support]] page
 ===== What next? =====