Differences

This shows you the differences between two versions of the page.

--- other:newppl:starting [2025/07/23 17:18] – [Using module to access optional programs] Added how to use 'grep' jypeter
+++ other:newppl:starting [2026/09/10 10:48] (current) – [Available servers] Updated intranet links jypeter
@@ Line 17: / Line 17: @@
   * [[https://www.science-accueil.org/en/|Science Accueil]]
     * [[https://www.science-accueil.org/en/request-services/|Service request page]]
+  * [[https://gate.paris-saclay.fr/en/|GATE desk]]
   * {{:other:newppl:uvsq-guide-du-personnel-nov2019.pdf|Guide du Personnel UVSQ}}
@@ Line 190: / Line 191: @@
 ==== Working directly on your desktop ====
-Always remember that **your local LSCE desktop/laptop can [[#accessing_remote_disks|access remote disks on the LSCE servers]]**.
+  * **your local LSCE desktop/laptop can [[#accessing_remote_disks|access (some) disks on the LSCE servers]]**, if it is connected to the LSCE wired network
-There are **lots** of things you can do directly on your **local** computer (displaying pdf, images, using a text editor, ...), rather than on the remote servers. For example, if a script running on the servers generates a pdf file, it is **more efficient** to open this pdf file using ''Acrobat''/''evince'' on your local computer, than by using ''evince'' on the server.
+  * You can do **many things** directly on your **local** computer (displaying pdf, images, using a text editor, ...), rather than on the obelix servers. For example, if a script running on the servers generates a pdf file, it is **more efficient** to open this pdf file using ''Acrobat''/''evince'' on your local computer, than by using ''evince'' on the server.
 ==== Available servers ====
@@ Line 209: / Line 210: @@
       * By **//inside LSCE//**, we mean that:
         * your computer is connected to the LSCE **wired** network
-        * or you are using the [[https://intranet.lsce.ipsl.fr/informatique/fr/vpn-forticlient.php#vpnlsce|LSCE VPN]]
+        * or you are using the [[https://private.lsce.ipsl.fr/intranet/accueil-intranet-informatique/outils-pour-le-teletravail/vpn-forticlient/|LSCE VPN]]
         * or you have a //terminal// connected to the ''ssh1'' gateway
     * <wrap hi>Never forget that you are **sharing** these interactive servers with other users!</wrap>
-      * <wrap hi>Do not use too much CPU and/or memory for a long time</wrap>. Heavy computation should be done on the [[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|LSCE cluster]]
+      * <wrap hi>Do not use too much CPU and/or memory for a long time</wrap>. Heavy computation should be done on the [[https://private.lsce.ipsl.fr/intranet/calcul-stockage-informatique/batch/|LSCE cluster]]
       * Learn how to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the top command]] to determine the current load of a server and the CPU/memory usage of your processes !
     * You can use one of the following commands if you want to know on which ''obelixNN'' you are
@@ Line 226: / Line 227: @@
 ('mylogin', 'obelix5')</code>
-  * **[[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers
+  * **[[https://private.lsce.ipsl.fr/intranet/calcul-stockage-informatique/batch/|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers
     * //Heavy duty// programs are programs that will prevent other users from working on the same server because these programs use a lot of CPU and/or memory for more than a few minutes. Ask your advisor, if you are not sure
@@ Line 314: / Line 315: @@
 |  **P**  | Sort by CPU usage  (''%CPU'' columun) (Default) |
-==== Using the kill command to terminate processes ====
+==== Listing processes with 'ps' ====
+==== Terminating processes will 'kill' ====
 FIXME
@@ Line 376: / Line 380: @@
     * Use the following command to access the backup: ''cd ~/.snapshot''
-  * **Quota**: YES!\\ There is a limit to what you can store in your home directory. If you reach this limit, you may experience some seemingly random errors (can't connect to the servers, the new files have a 0 size, ...)\\ The example below shows how to determine the //quota// on the LSCE disks: you can never go above the size displayed in the ''limit column'', and you cannot stay more than 7 days above the limit displayed in the ''quota'' column! The ''grace'' column will display either ''None'' (you cannot write on the disk anymore, unless you get below the ''quota'' limit) or ''N days'' (you can use the disk, but you have ''N days'' to go below the ''quota'' limit)\\ <code>
+  * **Quota**: YES!\\ There is a limit to what you can store in your home directory. If you reach this limit, you may experience some seemingly random errors (can't connect to the servers, the new files have a 0 size, ...)
- > quota -s
-Disk quotas for user johndoe (uid 6369):
-     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
-prolix3:/users/   4658M   4883M   4981M           23503       0       0</code>
 ==== The scratch directory ====
@@ Line 412: / Line 410: @@
   * **Quota**: ask your advisor
-==== Determining the space used by directories ====
+==== Dealing with disk space ====
-Your Linux environment may sometimes stop working correctly because you have exceeded the allowed quota on your disks (check your quotas with ''quota -s''). You can use the following command to get the size of all the directories and files in the current directory, sorted by size: ''du -sh * | sort -rh''
+<WRAP center round important 60%>
+Running out of disk space may have strange (and difficult to debug) side effects!
+</WRAP>
-<code> >cd /home/scratch01/johndoe
+**Disk space** on remote servers may sometimes seems infinite, but **is a limited resource**, especially if other users are not as careful as you!
-/home/scratch01/johndoe
- >du -sh * | sort -rh
+You should regularly check how much space you are using (''quota'', ''du''), and how much space is left (''df'')
+=== 'quota' command ===
+The ''quota -s'' command will tell you:
+  * If your disk usage (total size and/or number of files) is limited on some of the disks you use
+    * It is usually the case on the //home// directories (LSCE, spirit, computing centers, ...)
+  * How much disk space you are currently using on the disks with //quota// control
+The examples below shows how to display the current //quota// on the LSCE and spirit disks:
+  * //home// quota at LSCE
+    * <code> > quota -s
+Disk quotas for user johndoe (uid 0007):
+     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
+prolix3:/users/   4658M   4883M   4981M           23503       0       0</code>
+  * //home// quota on spiritx
+    * <code> $ quota -s
+Disk quotas for user janedoe (uid 0008):
+     Filesystem   space   quota   limit   grace   files   quota   limit   grace
+.17.3.169:/home
+K  32768M  33792M             339    400k    420k</code>
+  * The ''blocks'' or ''space'' column display your current usage of the disk where the quota is activated
+  * You can never go above the size displayed in the **//hard//** ''limit'' column
+  * You cannot stay more than 7 days above the **//soft//** limit displayed in the ''quota'' column!
+  * The ''grace'' column will display either:
+    * nothing: everything is fine!!
+    * ''N days'': you can still use this disk, but you have ''N days'' (decreasing from 7 days) to remove files and get back below the //soft// ''quota'' limit
+    * ''None'': you cannot write to this disk anymore, unless you remove files to get below the ''quota'' limit
+=== 'du' (disk usage) command ===
+You can use ''du -sh some_dir'' to determine the space used by ''some_dir'' (and all its sub-directories)
+Examples:
+  * At LSCE ''du -sh /home/users/johndoe'' will display the size of the //home// directory. It should be roughly the same value as the ''blocks'' column displayed by ''quota''
+  * You can use the following command to get the size of all the directories and files in the current directory, sorted by decreasing size:
+    * ''du -sh * | sort -rh''
+      * If you don't understand the //pipe// (''|'' character) above, read //Combining commands// in //[[#linux_unix|Useful Linux commands for working with (potentially big) text files (houchesa4_updated.pdf)]]//
+    * <code> > cd /home/scratch01/johndoe
+ > du -sh * | sort -rh
 M     GLAC1DHiceF26.nc
 M     dashboard
@@ Line 428: / Line 471: @@
 ...</code>
-FIXME Add something about ''df''
+=== 'df' command ===
+''df -h'' will display the total size of a disk, and how much space is left. A disk may be full, even if you have not reached your quota (because it is full of fools)
+Examples:
+  * <code> > df -h /home/users
+Filesystem      Size  Used Avail Use% Mounted on
+prolix3:/users  3.6T  2.1T  1.4T  60% /home/users</code>
+  * <code> >df -h /home/scratch01
+Filesystem                       Size  Used Avail Use% Mounted on
+prolix3.lsce.ipsl.fr:/scratch01   14T   13T  127G 100% /home/scratch01</code>
 ===== Which programs should you use? =====
@@ Line 505: / Line 561: @@
 ==== Linux/UNIX ====
+=== The 'man' command ===
+You can use the ''man'' command in any terminal to access the full ''man''ual of a command
+  * Examples:
+    * ''man man'' => ''man - an interface to the on-line reference manuals''
+    * ''man ls''
+      * <code>NAME
+       ls - list directory contents
+SYNOPSIS
+       ls [OPTION]... [FILE]...
+DESCRIPTION
+       List information about the FILEs (the current directory by default).  Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
+       Mandatory arguments to long options are mandatory for short options too.
+       -a, --all
+              do not ignore entries starting with .
+[...]</code>
+  * Reading a //man// page (''man less'' for all the options)
+    * ''SPACE''  /  ''b'': next / previous page
+    * ''q'': quit
+    * ''/pattern'': search ''pattern'' in the //man// page
+      * ''n''  /  ''N'': when searching, look for next / previous occurrence of ''pattern''
+=== Documents ===
   * {{:other:newppl:unix_doc_jyp.pdf|UNIX_doc_JYP.pdf}}: some basic Linux documentation. Make sure you read it!
@@ Line 512: / Line 598: @@
   * Using [[other:ssh|ssh]] and [[other:ssh#copying_files_with_scp|scp]]
-  * See also the //old// [[http://www.idris.fr/formations/utilisation-unix/unix-u-cours.html|IDRIS (advanced) Linux training]] (French tutorial)
+  * See also [[http://www.idris.fr/docs/jean-zay/cheatsheet/|IDRIS cheatsheet]]  (French tutorial)
+=== Example commands ===
+Misc (re)discovered Linux commands. Use ''[[#the_man_command|man]]'' for all the details
+  * ''column'': columnate lists
+  * ''find'': search for files in a directory hierarchy
+    * [[https://meetcyber.net/the-linux-command-i-used-wrong-for-years-e2492de5cfdf|examples]]
+    * (try to) exclude directories using the [[https://www.theunixschool.com/2012/07/find-command-15-examples-to-exclude.html|-prune]] option. Good luck!
+  * ''timeout'': run a command with a time limit
+  * ''tree'': list contents of directories in a tree-like format
+    * ''tree -d -L 2 -a /home/scratch01/$USER'': list 2 (''-L 2'') levels of directories (''-d''), including hidden ones (''-a'')
+  * ''watch'': execute a program periodically, showing output fullscreen
+=== Other commands ===
+Commands **useless for beginners, and probably most users**, but we needed a place to list them...
+  * ''ss'': another utility to [[https://systemweakness.com/the-linux-command-that-quietly-became-my-favorite-debugging-tool-bf51221c3471|investigate sockets]], similar to netstat
 ==== Text editors ====
-<note important>A //jupyter notebook// is **not** a text editor!
+<note important>
+  * <wrap em>Pleeeeeease do not use high-impact vscode on the multi-users servers when you just need a text editor and/or a file explorer</wrap>
+  * A //jupyter notebook// is **not** a text editor...
+    * ... but it can be useful when you are a beginner, or an intern staying only for one month
-**You need a correct text editor to efficiently work with scripts and programs**
+  * **You need a correct text editor if you want to <wrap em>work efficiently</wrap> with scripts and programs**
 {{ :other:newppl:real_programmers.png?direct&600 |}} [[https://www.explainxkcd.com/wiki/index.php/378:_Real_Programmers|xkcd & Real Programmers]]</note>
-There are **lots of text editors available** on Linux computers/servers (''emacs'', ''nano'', ''vi'', ''vim'', ''gvim'', ''gedit'', ...). You can use them:
+  * A **text editor** allows you to easily and <wrap em>efficiently</wrap> create/edit/view a **text file**
+    * <wrap em>Efficiently</wrap>? A text editor appropriate for your server should need only one process, use almost no CPU power, and not much RAM
+      * **Other users of the same server should not be able to notice that you are there** :-)
+      * If your text file is very big, you probably need to read {{:other:newppl:houchesa4_updated.pdf|Useful Linux commands for working with (potentially big) text files}}
+    * By the way, a **text file** is a collection of readable [[other:python:misc_by_jyp#strings|strings]] followed by [[https://mojoauth.com/blog/end-of-line-characters|EOL (End Of Line) characters]] :-)
+    * Other files are called **binary files** (e.g. [[#netcdf_and_related_conventions|NetCDF]])
+      * Real <del>nerds</del>scientists should have at least some notions of [[other:python:misc_by_jyp#data_representation|Data representation]]
-  * In basic **text terminals**
+  * **Most** text editors will use the text file extension (''.txt'', ''.sh'', ''.py'', ''.f90'', ...) to recognize the file content and automatically use colored //<wrap em>syntax highlighting</wrap>// to make the text more readable.\\ **Some** editors will also help you with //<wrap em>code indentation</wrap>// (the ''emacs'' //python mode// is quite efficient)
-    * **If you only need to read a text file**, use a //pager// like [[http://www.greenwoodsoftware.com/less|less]] (or ''more''): ''less my_text_file.txt''
+    * Example //syntax highlighting//:
-      * ''SPACE'' goes down one page
+      * <code bash># This is a comment in an example shell script that would have a '.sh'/'.ksh' extension
-      * ''b'' (//back//) goes back one page
+all_files=`ls`
-      * ''/STRING'' looks for ''STRING'' in the text (then ''n''ext, ''p''revious, ''q''uit)
+echo $all_files</code>
-      * ''q'' exits
-    * **[[#gnu_nano|nano]] is probably the easiest editor in text mode** to use!
-    * Useful if you are in a hurry, and/or working on a remote server with a bad network connection or have no local [[other:x_conf|X server]] running
-    * You should know the basic commands and shortcuts!
-  * With a **full Graphical User Interface** (aka //GUI//) allowing you to use the mouse+menus/buttons/etc...
+  * There are **lots of text editors available** on Linux computers/servers, that you can use directly in **text terminals**, or with a **graphical interface**
-    * you will need to have a //local// [[other:x_conf|X server]] running, if you want to use the text editor GUI on a //remote// server
+    * In basic **text terminals**:
+      * Editors => ''[[#gnu_nano|nano]]'', ''[[#vi_vim_gvim|vi]]'' or ''[[#vi_vim_gvim|vim]]'', ''[[other:emacs_doc|emacs]]''
+      * Note: if you only need to **read** a text file, use a //pager// like [[http://www.greenwoodsoftware.com/less|less]] (or ''more''):
+        * e.g. ''less my_text_file.txt''
+          * ''SPACE'' goes down one page
+          * ''b'' (//back//) goes back one page
+          * ''/STRING'' looks for ''STRING'' in the text (then ''n''ext, ''p''revious, ''q''uit)
+          * ''q'' exits
+      * ''[[#gnu_nano|nano]]'' is probably the easiest editor in text mode to use...\\ ...because the commands are displayed at the bottom of the windows
+      * Editors in text terminals are useful when you are in a hurry, and/or working on a remote server with a bad network connection or have no local [[other:x_conf|X server]] running
+      * You should know the basic commands and shortcuts of at least one editor you can use in a text terminal
+    * With a **full Graphical User Interface** (aka //GUI//) allowing you to use the mouse+menus/buttons/etc...
+      * Editors => ''gedit'', ''geany'', ''[[other:emacs_doc|emacs]]'', ''[[#vi_vim_gvim|gvim]]'', ...
+      * You will need to have a //local// [[other:x_conf|X server]] running, if you want to open the text editor GUI on a //remote// server
 === emacs ===
@@ Line 549: / Line 675: @@
 === vi (vim, gvim) ===
-  * [[https://www.atmos.albany.edu/daes/atmclasses/atm350/vi_cheat_sheet.pdf|vi cheat sheet]]
+You can use ''vi''/''vim'' in a basic text terminal. ''gvim'', the graphical version, is easier to use, but you need to have an [[other:x_conf|X server]] running in order to open the graphical window
-  * [[https://web.mit.edu/merolish/Public/vi-ref.pdf|vi reference card]]
+  * [[https://www.atmos.albany.edu/daes/atmclasses/atm350/vi_cheat_sheet.pdf|vi / vim cheat sheet]]
+  * [[https://web.mit.edu/merolish/Public/vi-ref.pdf|vi / vim reference card]]
 === Notepad++ ===
-[[other:win10apps#notepad|Notepad++]] is a nice and powerful text editor for **Windows**, but it is also very easy to [[other:emacs_doc#windows_installation|install emacs on Windows]]
+[[other:win10apps#notepad|Notepad++]] is a nice and powerful text editor for **Windows**
+Note: it is also very easy to [[other:emacs_doc#windows_installation|install emacs on Windows]] :-)
 ==== NetCDF and related Conventions ====
@@ Line 578: / Line 708: @@
       * Use ''[[#using_module_to_access_optional_programs|module]] load cdo'' on the LSCE and IPSL servers
     * [[http://nco.sourceforge.net/#RTFM|netCDF Operator]] (//NCO// operators)
-      * Use ''[[#using_module_to_access_optional_programs|module]] load cdo'' on the LSCE and IPSL servers
+      * Use ''[[#using_module_to_access_optional_programs|module]] load nco'' on the LSCE and IPSL servers