| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| other:newppl:starting [2024/09/12 17:08] – [Available servers] Added information on the JupyterHub servers jypeter | other:newppl:starting [2025/08/29 15:26] (current) – [Dealing with disk space] Finished rewriting the section jypeter |
|---|
| * [[https://www.science-accueil.org/en/|Science Accueil]] | * [[https://www.science-accueil.org/en/|Science Accueil]] |
| * [[https://www.science-accueil.org/en/request-services/|Service request page]] | * [[https://www.science-accueil.org/en/request-services/|Service request page]] |
| | * [[https://gate.paris-saclay.fr/en/|GATE desk]] |
| * {{:other:newppl:uvsq-guide-du-personnel-nov2019.pdf|Guide du Personnel UVSQ}} | * {{:other:newppl:uvsq-guide-du-personnel-nov2019.pdf|Guide du Personnel UVSQ}} |
| |
| |
| * The **''JupyterHub'' notebook servers**: | * The **''JupyterHub'' notebook servers**: |
| * [[https://intranet.lsce.ipsl.fr/informatique/en/jupyter/index.php|LSCE JupyterHub server information]] | * Read the [[other:python:starting#using_jupyterhub_servers|Using JupyterHub servers]] section of the //Working with Python// page |
| * [[https://documentations.ipsl.fr/spirit/jupyterhub/documentation.html|IPSL JupyterHub server information]] | |
| * You will **have to** be on the LSCE wired network, or use the [[https://intranet.lsce.ipsl.fr/informatique/en/intra-cea.php|LSCE VPN]], if you want to use the LSCE or IPSL JupyterHub servers ! | |
| * You can use a [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#using_a_non-standard_kernel|non standard kernel]] if you know what you are doing | |
| |
| * **The [[https://documentations.ipsl.fr/spirit/|IPSL Computing and Data Center]]**, aka //Mésocentre ESPRI// (''Spirit'' / ''SpiritX'' / ''HAL'') | * **The [[https://documentations.ipsl.fr/spirit/|IPSL Computing and Data Center]]**, aka //Mésocentre ESPRI// (''Spirit'' / ''SpiritX'' / ''HAL'') |
| * Note: the ''Spirit''/''X'' clusters have replaced the ''ciclad''/''climserv'' cluster (in 2022) | * Note: the ''Spirit''/''X'' clusters have replaced the ''ciclad''/''climserv'' cluster (in 2022) |
| * You will need a specific account (different from your //LSCE// account) to access these servers and use data files that are already available/mirrored there (e.g. **''CMIPn'' data**).\\ **Do not duplicate** at LSCE data already available on IPSL servers and that you can process there! | * You will need a specific account (different from your //LSCE// account) to access these servers and use data files that are already available/mirrored there (e.g. **''CMIPn'' data**).\\ **Do not duplicate** at LSCE data already available on IPSL servers and that you can process there! |
| * [[https://mesocentre.ipsl.fr/account-opening/|Requesting an IPSL account]] | * [[https://documentations.ipsl.fr/spirit/getting_started/account.html|Requesting an IPSL account]] |
| * [[https://documentations.ipsl.fr/spirit/|IPSL ESPRI Mesocenter documentation]] | * [[https://documentations.ipsl.fr/spirit/|IPSL ESPRI Mesocenter documentation]] |
| * [[https://documentations.ipsl.fr/spirit/spirit_clusters/head_nodes.html|spirit clusters]] | * [[https://documentations.ipsl.fr/spirit/spirit_clusters/head_nodes.html|spirit clusters]] |
| * Use the following command to access the backup: ''cd ~/.snapshot'' | * Use the following command to access the backup: ''cd ~/.snapshot'' |
| |
| * **Quota**: YES!\\ There is a limit to what you can store in your home directory. If you reach this limit, you may experience some seemingly random errors (can't connect to the servers, the new files have a 0 size, ...)\\ The example below shows how to determine the //quota// on the LSCE disks: you can never go above the size displayed in the ''limit column'', and you cannot stay more than 7 days above the limit displayed in the ''quota'' column! The ''grace'' column will display either ''None'' (you cannot write on the disk anymore, unless you get below the ''quota'' limit) or ''N days'' (you can use the disk, but you have ''N days'' to go below the ''quota'' limit)\\ <code> | * **Quota**: YES!\\ There is a limit to what you can store in your home directory. If you reach this limit, you may experience some seemingly random errors (can't connect to the servers, the new files have a 0 size, ...) |
| > quota -s | |
| | |
| Disk quotas for user johndoe (uid 6369): | |
| Filesystem blocks quota limit grace files quota limit grace | |
| prolix3:/users/ 4658M 4883M 4981M 23503 0 0</code> | |
| ==== The scratch directory ==== | ==== The scratch directory ==== |
| |
| * **Quota**: ask your advisor | * **Quota**: ask your advisor |
| |
| ==== Determining the space used by directories ==== | ==== Dealing with disk space ==== |
| |
| Your Linux environment may sometimes stop working correctly because you have exceeded the allowed quota on your disks (check your quotas with ''quota -s''). You can use the following command to get the size of all the directories and files in the current directory, sorted by size: ''du -sh * | sort -rh'' | <WRAP center round important 60%> |
| | Running out of disk space may have strange (and difficult to debug) side effects! |
| | </WRAP> |
| |
| <code> >cd /home/scratch01/johndoe | **Disk space** on remote servers may sometimes seems infinite, but **is a limited resource**, especially if other users are not as careful as you! |
| /home/scratch01/johndoe | |
| |
| >du -sh * | sort -rh | You should regularly check how much space you are using (''quota'', ''du''), and how much space is left (''df'') |
| | |
| | === 'quota' command === |
| | |
| | The ''quota -s'' command will tell you: |
| | * If your disk usage (total size and/or number of files) is limited on some of the disks you use |
| | * It is usually the case on the //home// directories (LSCE, spirit, computing centers, ...) |
| | * How much disk space you are currently using on the disks with //quota// control |
| | |
| | The examples below shows how to display the current //quota// on the LSCE and spirit disks: |
| | * //home// quota at LSCE |
| | * <code> > quota -s |
| | |
| | Disk quotas for user johndoe (uid 0007): |
| | Filesystem blocks quota limit grace files quota limit grace |
| | prolix3:/users/ 4658M 4883M 4981M 23503 0 0</code> |
| | * //home// quota on spiritx |
| | * <code> $ quota -s |
| | Disk quotas for user janedoe (uid 0008): |
| | Filesystem space quota limit grace files quota limit grace |
| | 172.17.3.169:/home |
| | 26516K 32768M 33792M 339 400k 420k</code> |
| | |
| | * The ''blocks'' or ''space'' column display your current usage of the disk where the quota is activated |
| | * You can never go above the size displayed in the **//hard//** ''limit'' column |
| | * You cannot stay more than 7 days above the **//soft//** limit displayed in the ''quota'' column! |
| | * The ''grace'' column will display either: |
| | * nothing: everything is fine!! |
| | * ''N days'': you can still use this disk, but you have ''N days'' (decreasing from 7 days) to remove files and get back below the //soft// ''quota'' limit |
| | * ''None'': you cannot write to this disk anymore, unless you remove files to get below the ''quota'' limit |
| | |
| | |
| | |
| | === 'du' (disk usage) command === |
| | |
| | You can use ''du -sh some_dir'' to determine the space used by ''some_dir'' (and all its sub-directories) |
| | |
| | Examples: |
| | * At LSCE ''du -sh /home/users/johndoe'' will display the size of the //home// directory. It should be roughly the same value as the ''blocks'' column displayed by ''quota'' |
| | * You can use the following command to get the size of all the directories and files in the current directory, sorted by decreasing size: |
| | * ''du -sh * | sort -rh'' |
| | * If you don't understand the //pipe// (''|'' character) above, read //Combining commands// in //[[#linux_unix|Useful Linux commands for working with (potentially big) text files (houchesa4_updated.pdf)]]// |
| | * <code> > cd /home/scratch01/johndoe |
| | |
| | > du -sh * | sort -rh |
| 58M GLAC1DHiceF26.nc | 58M GLAC1DHiceF26.nc |
| 47M dashboard | 47M dashboard |
| ...</code> | ...</code> |
| |
| FIXME Add something about ''df'' | === 'df' command === |
| | |
| | ''df -h'' will display the total size of a disk, and how much space is left. A disk may be full, even if you have not reached your quota (because it is full of fools) |
| | |
| | Examples: |
| | |
| | * <code> > df -h /home/users |
| | Filesystem Size Used Avail Use% Mounted on |
| | prolix3:/users 3.6T 2.1T 1.4T 60% /home/users</code> |
| | |
| | * <code> >df -h /home/scratch01 |
| | Filesystem Size Used Avail Use% Mounted on |
| | prolix3.lsce.ipsl.fr:/scratch01 14T 13T 127G 100% /home/scratch01</code> |
| ===== Which programs should you use? ===== | ===== Which programs should you use? ===== |
| |
| Your advisor will let you know which (version of which) programs you should use. Some programs are available by default, and you have to use the ''module'' command to access other programs. | Your advisor will let you know which (version of which) programs you should use. Some programs are available by default, and you have to use the ''[[other:newppl:starting#using_module_to_access_optional_programs|module]]'' command to access other programs (or access a specific version of a program). |
| |
| ==== Which program am I using? ==== | ==== Which program am I using? ==== |
| ==== Using module to access optional programs ==== | ==== Using module to access optional programs ==== |
| |
| Most useful ''[[https://modules.readthedocs.io/|module]]'' options: | === Documentation === |
| | |
| | * [[https://documentations.ipsl.fr/spirit/common/module_command.html|IPSL Meso documentation]] |
| | * [[https://modules.readthedocs.io/|Full and official module documentation]] |
| | |
| | === Most useful module options === |
| * ''module list'': list the currently loaded modules | * ''module list'': list the currently loaded modules |
| * ''module avail [mod_name]'': list the available modules (or only the ones with a name starting with ''mod_name'') | * ''module avail [mod_name]'': list the available modules (or only the ones with a name starting with ''mod_name'') |
| | * '' module avail -l 2>&1 | grep subname'': list the modules that have ''subname'' in their name, and their installation date |
| | * <wrap hi>This only works in a //bash// shell!</wrap> In a //tcsh// shell, start by invoking a //bash// shell with ''bash -l'' |
| * ''module load module_a ... module_n'': load modules (automatically including dependencies) | * ''module load module_a ... module_n'': load modules (automatically including dependencies) |
| * ''module load module_a'': load default version of //module_a// | * ''module load module_a'': load default version of //module_a// |
| * ''module purge'': remove all modules | * ''module purge'': remove all modules |
| |
| Detailed example: | === Detailed example === |
| <code> > which ncview | <code> > which ncview |
| ncview: Command not found. | ncview: Command not found. |
| |
| > which ncdump | > which ncdump |
| /usr/local/install/netcdf-4.3.2/bin/ncdump</code> | /usr/local/install/netcdf-4.3.2/bin/ncdump |
| | |
| | > bash -l |
| | $ module avail -l 2>&1 | grep gcc |
| | binutils/2.43.1-gcc14-acc 2025/01/27 21:09:36 |
| | gcc/14.2.0-openacc 2025/01/24 8:11:14 |
| | gcc/8.3.0 2021/04/21 8:27:35 |
| | netcdf/4.9.2-gcc14-acc 2025/01/27 22:02:57 |
| | netcdf/4.9.2-gcc8 2024/06/05 20:49:36 |
| | R/4.0.3-gcc8 2022/02/03 12:53:55</code> |
| |
| ===== Useful documentation for beginners ===== | ===== Useful documentation for beginners ===== |
| * Useful Linux commands for working with (potentially big) text files ({{:other:newppl:houchesa4_updated.pdf}}) | * Useful Linux commands for working with (potentially big) text files ({{:other:newppl:houchesa4_updated.pdf}}) |
| * Using [[other:ssh|ssh]] and [[other:ssh#copying_files_with_scp|scp]] | * Using [[other:ssh|ssh]] and [[other:ssh#copying_files_with_scp|scp]] |
| | |
| | * See also the //old// [[http://www.idris.fr/formations/utilisation-unix/unix-u-cours.html|IDRIS (advanced) Linux training]] (French tutorial) |
| |
| ==== Text editors ==== | ==== Text editors ==== |
| * Use ''[[#using_module_to_access_optional_programs|module]] load cdo'' on the LSCE and IPSL servers | * Use ''[[#using_module_to_access_optional_programs|module]] load cdo'' on the LSCE and IPSL servers |
| * [[http://nco.sourceforge.net/#RTFM|netCDF Operator]] (//NCO// operators) | * [[http://nco.sourceforge.net/#RTFM|netCDF Operator]] (//NCO// operators) |
| * Use ''[[#using_module_to_access_optional_programs|module]] load cdo'' on the LSCE and IPSL servers | * Use ''[[#using_module_to_access_optional_programs|module]] load nco'' on the LSCE and IPSL servers |
| |
| |
| Check the [[other:python:jyp_steps#using_netcdf_files_with_python|dedicated section]] of the //Python// page | Check the [[other:python:jyp_steps#using_netcdf_files_with_python|dedicated section]] of the //Python// page |
| |
| ==== Programming languages ==== | ==== Programming languages, parallelism, ... ==== |
| |
| * [[other:index#python_notes|JYP's Python notes]] | * [[other:index#python_notes|JYP's Python notes]] |
| | * [[http://www.idris.fr/eng/formations/supports_de_cours-eng.html|IDRIS classes and tutorials]] |
| |
| ===== Getting help (from the LSCE system administrators) ===== | ===== Getting help (from the LSCE system administrators) ===== |
