| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| other:newppl:starting [2025/08/29 11:27] – [The home directory] Moved the "quota" section jypeter | other:newppl:starting [2026/09/10 10:48] (current) – [Available servers] Updated intranet links jypeter |
|---|
| ==== Working directly on your desktop ==== | ==== Working directly on your desktop ==== |
| |
| Always remember that **your local LSCE desktop/laptop can [[#accessing_remote_disks|access remote disks on the LSCE servers]]**. | * **your local LSCE desktop/laptop can [[#accessing_remote_disks|access (some) disks on the LSCE servers]]**, if it is connected to the LSCE wired network |
| |
| There are **lots** of things you can do directly on your **local** computer (displaying pdf, images, using a text editor, ...), rather than on the remote servers. For example, if a script running on the servers generates a pdf file, it is **more efficient** to open this pdf file using ''Acrobat''/''evince'' on your local computer, than by using ''evince'' on the server. | * You can do **many things** directly on your **local** computer (displaying pdf, images, using a text editor, ...), rather than on the obelix servers. For example, if a script running on the servers generates a pdf file, it is **more efficient** to open this pdf file using ''Acrobat''/''evince'' on your local computer, than by using ''evince'' on the server. |
| |
| ==== Available servers ==== | ==== Available servers ==== |
| * By **//inside LSCE//**, we mean that: | * By **//inside LSCE//**, we mean that: |
| * your computer is connected to the LSCE **wired** network | * your computer is connected to the LSCE **wired** network |
| * or you are using the [[https://intranet.lsce.ipsl.fr/informatique/fr/vpn-forticlient.php#vpnlsce|LSCE VPN]] | * or you are using the [[https://private.lsce.ipsl.fr/intranet/accueil-intranet-informatique/outils-pour-le-teletravail/vpn-forticlient/|LSCE VPN]] |
| * or you have a //terminal// connected to the ''ssh1'' gateway | * or you have a //terminal// connected to the ''ssh1'' gateway |
| * <wrap hi>Never forget that you are **sharing** these interactive servers with other users!</wrap> | * <wrap hi>Never forget that you are **sharing** these interactive servers with other users!</wrap> |
| * <wrap hi>Do not use too much CPU and/or memory for a long time</wrap>. Heavy computation should be done on the [[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|LSCE cluster]] | * <wrap hi>Do not use too much CPU and/or memory for a long time</wrap>. Heavy computation should be done on the [[https://private.lsce.ipsl.fr/intranet/calcul-stockage-informatique/batch/|LSCE cluster]] |
| * Learn how to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the top command]] to determine the current load of a server and the CPU/memory usage of your processes ! | * Learn how to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the top command]] to determine the current load of a server and the CPU/memory usage of your processes ! |
| * You can use one of the following commands if you want to know on which ''obelixNN'' you are | * You can use one of the following commands if you want to know on which ''obelixNN'' you are |
| ('mylogin', 'obelix5')</code> | ('mylogin', 'obelix5')</code> |
| |
| * **[[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers | * **[[https://private.lsce.ipsl.fr/intranet/calcul-stockage-informatique/batch/|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers |
| * //Heavy duty// programs are programs that will prevent other users from working on the same server because these programs use a lot of CPU and/or memory for more than a few minutes. Ask your advisor, if you are not sure | * //Heavy duty// programs are programs that will prevent other users from working on the same server because these programs use a lot of CPU and/or memory for more than a few minutes. Ask your advisor, if you are not sure |
| |
| | **P** | Sort by CPU usage (''%CPU'' columun) (Default) | | | **P** | Sort by CPU usage (''%CPU'' columun) (Default) | |
| |
| ==== Using the kill command to terminate processes ==== | ==== Listing processes with 'ps' ==== |
| | |
| | |
| | ==== Terminating processes will 'kill' ==== |
| |
| FIXME | FIXME |
| * **Quota**: ask your advisor | * **Quota**: ask your advisor |
| |
| ==== Determining the space used by directories ==== | ==== Dealing with disk space ==== |
| |
| Your Linux environment may sometimes stop working correctly because you have exceeded the allowed quota on your disks (check your quotas with ''quota -s''). You can use the following command to get the size of all the directories and files in the current directory, sorted by size: ''du -sh * | sort -rh'' | <WRAP center round important 60%> |
| | Running out of disk space may have strange (and difficult to debug) side effects! |
| | </WRAP> |
| |
| <code> >cd /home/scratch01/johndoe | **Disk space** on remote servers may sometimes seems infinite, but **is a limited resource**, especially if other users are not as careful as you! |
| /home/scratch01/johndoe | |
| |
| >du -sh * | sort -rh | You should regularly check how much space you are using (''quota'', ''du''), and how much space is left (''df'') |
| | |
| | === 'quota' command === |
| | |
| | The ''quota -s'' command will tell you: |
| | * If your disk usage (total size and/or number of files) is limited on some of the disks you use |
| | * It is usually the case on the //home// directories (LSCE, spirit, computing centers, ...) |
| | * How much disk space you are currently using on the disks with //quota// control |
| | |
| | The examples below shows how to display the current //quota// on the LSCE and spirit disks: |
| | * //home// quota at LSCE |
| | * <code> > quota -s |
| | |
| | Disk quotas for user johndoe (uid 0007): |
| | Filesystem blocks quota limit grace files quota limit grace |
| | prolix3:/users/ 4658M 4883M 4981M 23503 0 0</code> |
| | * //home// quota on spiritx |
| | * <code> $ quota -s |
| | Disk quotas for user janedoe (uid 0008): |
| | Filesystem space quota limit grace files quota limit grace |
| | 172.17.3.169:/home |
| | 26516K 32768M 33792M 339 400k 420k</code> |
| | |
| | * The ''blocks'' or ''space'' column display your current usage of the disk where the quota is activated |
| | * You can never go above the size displayed in the **//hard//** ''limit'' column |
| | * You cannot stay more than 7 days above the **//soft//** limit displayed in the ''quota'' column! |
| | * The ''grace'' column will display either: |
| | * nothing: everything is fine!! |
| | * ''N days'': you can still use this disk, but you have ''N days'' (decreasing from 7 days) to remove files and get back below the //soft// ''quota'' limit |
| | * ''None'': you cannot write to this disk anymore, unless you remove files to get below the ''quota'' limit |
| | |
| | |
| | |
| | === 'du' (disk usage) command === |
| | |
| | You can use ''du -sh some_dir'' to determine the space used by ''some_dir'' (and all its sub-directories) |
| | |
| | Examples: |
| | * At LSCE ''du -sh /home/users/johndoe'' will display the size of the //home// directory. It should be roughly the same value as the ''blocks'' column displayed by ''quota'' |
| | * You can use the following command to get the size of all the directories and files in the current directory, sorted by decreasing size: |
| | * ''du -sh * | sort -rh'' |
| | * If you don't understand the //pipe// (''|'' character) above, read //Combining commands// in //[[#linux_unix|Useful Linux commands for working with (potentially big) text files (houchesa4_updated.pdf)]]// |
| | * <code> > cd /home/scratch01/johndoe |
| | |
| | > du -sh * | sort -rh |
| 58M GLAC1DHiceF26.nc | 58M GLAC1DHiceF26.nc |
| 47M dashboard | 47M dashboard |
| ...</code> | ...</code> |
| |
| FIXME Add something about ''df'' | === 'df' command === |
| | |
| | ''df -h'' will display the total size of a disk, and how much space is left. A disk may be full, even if you have not reached your quota (because it is full of fools) |
| | |
| | Examples: |
| | |
| | * <code> > df -h /home/users |
| | Filesystem Size Used Avail Use% Mounted on |
| | prolix3:/users 3.6T 2.1T 1.4T 60% /home/users</code> |
| | |
| | * <code> >df -h /home/scratch01 |
| | Filesystem Size Used Avail Use% Mounted on |
| | prolix3.lsce.ipsl.fr:/scratch01 14T 13T 127G 100% /home/scratch01</code> |
| ===== Which programs should you use? ===== | ===== Which programs should you use? ===== |
| |
| |
| ==== Linux/UNIX ==== | ==== Linux/UNIX ==== |
| | |
| | === The 'man' command === |
| | |
| | You can use the ''man'' command in any terminal to access the full ''man''ual of a command |
| | |
| | * Examples: |
| | * ''man man'' => ''man - an interface to the on-line reference manuals'' |
| | * ''man ls'' |
| | * <code>NAME |
| | ls - list directory contents |
| | |
| | SYNOPSIS |
| | ls [OPTION]... [FILE]... |
| | |
| | DESCRIPTION |
| | List information about the FILEs (the current directory by default). Sort entries alphabetically if none of -cftuvSUX nor --sort is specified. |
| | |
| | Mandatory arguments to long options are mandatory for short options too. |
| | |
| | -a, --all |
| | do not ignore entries starting with . |
| | [...]</code> |
| | |
| | * Reading a //man// page (''man less'' for all the options) |
| | * ''SPACE'' / ''b'': next / previous page |
| | * ''q'': quit |
| | * ''/pattern'': search ''pattern'' in the //man// page |
| | * ''n'' / ''N'': when searching, look for next / previous occurrence of ''pattern'' |
| | |
| | === Documents === |
| |
| * {{:other:newppl:unix_doc_jyp.pdf|UNIX_doc_JYP.pdf}}: some basic Linux documentation. Make sure you read it! | * {{:other:newppl:unix_doc_jyp.pdf|UNIX_doc_JYP.pdf}}: some basic Linux documentation. Make sure you read it! |
| * Using [[other:ssh|ssh]] and [[other:ssh#copying_files_with_scp|scp]] | * Using [[other:ssh|ssh]] and [[other:ssh#copying_files_with_scp|scp]] |
| |
| * See also the //old// [[http://www.idris.fr/formations/utilisation-unix/unix-u-cours.html|IDRIS (advanced) Linux training]] (French tutorial) | * See also [[http://www.idris.fr/docs/jean-zay/cheatsheet/|IDRIS cheatsheet]] (French tutorial) |
| |
| | |
| | === Example commands === |
| | |
| | Misc (re)discovered Linux commands. Use ''[[#the_man_command|man]]'' for all the details |
| | |
| | * ''column'': columnate lists |
| | * ''find'': search for files in a directory hierarchy |
| | * [[https://meetcyber.net/the-linux-command-i-used-wrong-for-years-e2492de5cfdf|examples]] |
| | * (try to) exclude directories using the [[https://www.theunixschool.com/2012/07/find-command-15-examples-to-exclude.html|-prune]] option. Good luck! |
| | * ''timeout'': run a command with a time limit |
| | * ''tree'': list contents of directories in a tree-like format |
| | * ''tree -d -L 2 -a /home/scratch01/$USER'': list 2 (''-L 2'') levels of directories (''-d''), including hidden ones (''-a'') |
| | * ''watch'': execute a program periodically, showing output fullscreen |
| | |
| | |
| | === Other commands === |
| | |
| | Commands **useless for beginners, and probably most users**, but we needed a place to list them... |
| | |
| | * ''ss'': another utility to [[https://systemweakness.com/the-linux-command-that-quietly-became-my-favorite-debugging-tool-bf51221c3471|investigate sockets]], similar to netstat |
| ==== Text editors ==== | ==== Text editors ==== |
| |
| <note important>A //jupyter notebook// is **not** a text editor! | <note important> |
| | * <wrap em>Pleeeeeease do not use high-impact vscode on the multi-users servers when you just need a text editor and/or a file explorer</wrap> |
| |
| **You need a correct text editor to efficiently work with scripts and programs** | |
| | * A //jupyter notebook// is **not** a text editor... |
| | * ... but it can be useful when you are a beginner, or an intern staying only for one month |
| | |
| | * **You need a correct text editor if you want to <wrap em>work efficiently</wrap> with scripts and programs** |
| |
| {{ :other:newppl:real_programmers.png?direct&600 |}} [[https://www.explainxkcd.com/wiki/index.php/378:_Real_Programmers|xkcd & Real Programmers]]</note> | {{ :other:newppl:real_programmers.png?direct&600 |}} [[https://www.explainxkcd.com/wiki/index.php/378:_Real_Programmers|xkcd & Real Programmers]]</note> |
| |
| There are **lots of text editors available** on Linux computers/servers (''emacs'', ''nano'', ''vi'', ''vim'', ''gvim'', ''gedit'', ...). You can use them: | * A **text editor** allows you to easily and <wrap em>efficiently</wrap> create/edit/view a **text file** |
| | * <wrap em>Efficiently</wrap>? A text editor appropriate for your server should need only one process, use almost no CPU power, and not much RAM |
| | * **Other users of the same server should not be able to notice that you are there** :-) |
| | * If your text file is very big, you probably need to read {{:other:newppl:houchesa4_updated.pdf|Useful Linux commands for working with (potentially big) text files}} |
| | * By the way, a **text file** is a collection of readable [[other:python:misc_by_jyp#strings|strings]] followed by [[https://mojoauth.com/blog/end-of-line-characters|EOL (End Of Line) characters]] :-) |
| | * Other files are called **binary files** (e.g. [[#netcdf_and_related_conventions|NetCDF]]) |
| | * Real <del>nerds</del>scientists should have at least some notions of [[other:python:misc_by_jyp#data_representation|Data representation]] |
| |
| * In basic **text terminals** | * **Most** text editors will use the text file extension (''.txt'', ''.sh'', ''.py'', ''.f90'', ...) to recognize the file content and automatically use colored //<wrap em>syntax highlighting</wrap>// to make the text more readable.\\ **Some** editors will also help you with //<wrap em>code indentation</wrap>// (the ''emacs'' //python mode// is quite efficient) |
| * **If you only need to read a text file**, use a //pager// like [[http://www.greenwoodsoftware.com/less|less]] (or ''more''): ''less my_text_file.txt'' | * Example //syntax highlighting//: |
| * ''SPACE'' goes down one page | * <code bash># This is a comment in an example shell script that would have a '.sh'/'.ksh' extension |
| * ''b'' (//back//) goes back one page | all_files=`ls` |
| * ''/STRING'' looks for ''STRING'' in the text (then ''n''ext, ''p''revious, ''q''uit) | echo $all_files</code> |
| * ''q'' exits | |
| * **[[#gnu_nano|nano]] is probably the easiest editor in text mode** to use! | |
| * Useful if you are in a hurry, and/or working on a remote server with a bad network connection or have no local [[other:x_conf|X server]] running | |
| * You should know the basic commands and shortcuts! | |
| |
| * With a **full Graphical User Interface** (aka //GUI//) allowing you to use the mouse+menus/buttons/etc... | * There are **lots of text editors available** on Linux computers/servers, that you can use directly in **text terminals**, or with a **graphical interface** |
| * you will need to have a //local// [[other:x_conf|X server]] running, if you want to use the text editor GUI on a //remote// server | * In basic **text terminals**: |
| | * Editors => ''[[#gnu_nano|nano]]'', ''[[#vi_vim_gvim|vi]]'' or ''[[#vi_vim_gvim|vim]]'', ''[[other:emacs_doc|emacs]]'' |
| | * Note: if you only need to **read** a text file, use a //pager// like [[http://www.greenwoodsoftware.com/less|less]] (or ''more''): |
| | * e.g. ''less my_text_file.txt'' |
| | * ''SPACE'' goes down one page |
| | * ''b'' (//back//) goes back one page |
| | * ''/STRING'' looks for ''STRING'' in the text (then ''n''ext, ''p''revious, ''q''uit) |
| | * ''q'' exits |
| | * ''[[#gnu_nano|nano]]'' is probably the easiest editor in text mode to use...\\ ...because the commands are displayed at the bottom of the windows |
| | * Editors in text terminals are useful when you are in a hurry, and/or working on a remote server with a bad network connection or have no local [[other:x_conf|X server]] running |
| | * You should know the basic commands and shortcuts of at least one editor you can use in a text terminal |
| | * With a **full Graphical User Interface** (aka //GUI//) allowing you to use the mouse+menus/buttons/etc... |
| | * Editors => ''gedit'', ''geany'', ''[[other:emacs_doc|emacs]]'', ''[[#vi_vim_gvim|gvim]]'', ... |
| | * You will need to have a //local// [[other:x_conf|X server]] running, if you want to open the text editor GUI on a //remote// server |
| |
| === emacs === | === emacs === |
| === vi (vim, gvim) === | === vi (vim, gvim) === |
| |
| * [[https://www.atmos.albany.edu/daes/atmclasses/atm350/vi_cheat_sheet.pdf|vi cheat sheet]] | You can use ''vi''/''vim'' in a basic text terminal. ''gvim'', the graphical version, is easier to use, but you need to have an [[other:x_conf|X server]] running in order to open the graphical window |
| * [[https://web.mit.edu/merolish/Public/vi-ref.pdf|vi reference card]] | |
| | * [[https://www.atmos.albany.edu/daes/atmclasses/atm350/vi_cheat_sheet.pdf|vi / vim cheat sheet]] |
| | * [[https://web.mit.edu/merolish/Public/vi-ref.pdf|vi / vim reference card]] |
| |
| === Notepad++ === | === Notepad++ === |
| |
| [[other:win10apps#notepad|Notepad++]] is a nice and powerful text editor for **Windows**, but it is also very easy to [[other:emacs_doc#windows_installation|install emacs on Windows]] | [[other:win10apps#notepad|Notepad++]] is a nice and powerful text editor for **Windows** |
| | |
| | Note: it is also very easy to [[other:emacs_doc#windows_installation|install emacs on Windows]] :-) |
| ==== NetCDF and related Conventions ==== | ==== NetCDF and related Conventions ==== |
| |
