This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:newppl:starting [2023/01/12 15:23] jypeter [NetCDF and file formats] |
other:newppl:starting [2023/09/11 08:14] jypeter [Miscellaneous information] |
||
---|---|---|---|
Line 13: | Line 13: | ||
* [[other:newppl:riealgo|Adding money online to your restaurant account]] | * [[other:newppl:riealgo|Adding money online to your restaurant account]] | ||
* [[https://intranet.lsce.ipsl.fr/Phocea/Page/index.php?id=97|Survival kit and other information for CDD and interns]] (LSCE intranet) | * [[https://intranet.lsce.ipsl.fr/Phocea/Page/index.php?id=97|Survival kit and other information for CDD and interns]] (LSCE intranet) | ||
- | * [[http://internationaloffice.ceasaclay.com/?lang=en|CEA Saclay International Office]] //Our multilingual staff is readily available to assist you through immigration procedures and to help you settle in France// | + | * [[https://www.universite-paris-saclay.fr/en/campus-life/international-welcome-desk|International welcome desk]] //Université Paris-Saclay supports international students, PhD students and researchers as soon as they arrive in France to ensure their stay take place under the best conditions// |
- | * [[https://www.universite-paris-saclay.fr/en/campus-life/international-welcome-desk|International welcome desk]] //Université Paris-Saclay provides you with a variety of services and documentation to welcome international students and researchers under the best conditions// | + | |
* [[https://logement.campus-paris-saclay.fr/| Trouver un logement dans une résidence universitaire]] (in French... //Finding a room in a student dorm//) | * [[https://logement.campus-paris-saclay.fr/| Trouver un logement dans une résidence universitaire]] (in French... //Finding a room in a student dorm//) | ||
* [[https://www.science-accueil.org/en/|Science Accueil]] | * [[https://www.science-accueil.org/en/|Science Accueil]] | ||
Line 190: | Line 189: | ||
==== Available servers ==== | ==== Available servers ==== | ||
- | * **The LSCE interactive servers**: ''obelix//NN//''\\ Use ''ssh obelix'' to access these servers, and the //load balancing// system will send you to the server that has currently the smallest load. | + | * **The LSCE //gateway//**: ''ssh1.lsce.ipsl.fr'' |
- | * <wrap hi>Never forget that you are sharing these servers with other users!</wrap> | + | * Use ''ssh ssh1.lsce.ipsl.fr'' when you are **outside the LSCE wired network**, before accessing any other LSCE server: |
- | * <wrap hi>Do not use too much CPU and/or memory for a long time</wrap>. Heavy computation should be done on the [[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|LSCE cluster]] | + | * By **//outside LSCE//**, we mean that your computer is: |
+ | * connected to the LSCE WiFi network (instead of the wired network) | ||
+ | * or physically outside LSCE | ||
+ | * You can also use ''ssh1'' when you are **copying files to/from outside LSCE** with ''scp'' or ssh-based tools | ||
+ | * <wrap hi>The ''ssh1'' gateway can **only** be used for copying files or accessing the LSCE interactive servers</wrap>! | ||
+ | * **You have to** connect to an ''obelix//NN//'' interactive server before doing any real work | ||
+ | |||
+ | * **The LSCE interactive servers**: ''obelix//NN//'' (''obelix2'' to ''obelix5'') | ||
+ | * Use ''ssh obelix'' to access the LSCE servers from **inside LSCE**.\\ The //load balancing// system will send you to the ''obelix//NN//'' server that has currently the smallest load. | ||
+ | * By **//inside LSCE//**, we mean that: | ||
+ | * your computer is connected to the LSCE **wired** network | ||
+ | * or you are using the [[https://intranet.lsce.ipsl.fr/informatique/fr/vpn-forticlient.php#vpnlsce|LSCE VPN]] | ||
+ | * or you have a //terminal// connected to the ''ssh1'' gateway | ||
+ | * <wrap hi>Never forget that you are **sharing** these interactive servers with other users!</wrap> | ||
+ | * <wrap hi>Do not use too much CPU and/or memory for a long time</wrap>. Heavy computation should be done on the [[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|LSCE cluster]] | ||
* Learn how to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the top command]] to determine the current load of a server and the CPU/memory usage of your processes ! | * Learn how to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the top command]] to determine the current load of a server and the CPU/memory usage of your processes ! | ||
- | * <code>$ ssh obelix | + | * You can use one of the following commands if you want to know on which ''obelixNN'' you are |
+ | * <code>$ ssh obelix | ||
Last login: Mon Jun 3 08:49:53 2019 from somewhere | Last login: Mon Jun 3 08:49:53 2019 from somewhere | ||
- | # You can use one of the following commands if you want to know on which obelix you are | ||
$ echo $HOST | $ echo $HOST | ||
obelix5 | obelix5 | ||
Line 202: | Line 215: | ||
obelix5</code> | obelix5</code> | ||
- | * **[[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for really heavy duty programs, rather than killing the multi-users interactive servers | + | * **[[https://intranet.lsce.ipsl.fr/informatique/en/calcul/batch.php|The LSCE cluster]]** (aka the //batch system//)\\ Use this cluster for **//heavy duty// programs**, rather than killing the multi-users interactive servers |
+ | * //Heavy duty// programs are programs that will prevent other users from working on the same server because these programs use a lot of CPU and/or memory for more than a few minutes. Ask your advisor, if you are not sure | ||
- | * **The [[http://mesocentre.ipsl.fr/|IPSL/ESPRI Mesocenter servers/clusters]]** (''ciclad'' / ''climserv'' / etc...)\\ You will need an account (different from your //LSCE// account) to access these servers and use data files that are already available/mirrored there (e.g. **''CMIPn'' data**). **Do not duplicate** at LSCE data that is already available on IPSL servers and that you can process there! | + | * **The [[https://documentations.ipsl.fr/spirit/|IPSL Computing and Data Center]]**, aka //Mésocentre ESPRI// (''Spirit'' / ''SpiritX'' / ''HAL'') |
- | * [[http://mesocentre.ipsl.fr/account-opening/|Requesting a new account]] | + | * Note: the ''Spirit''/''X'' clusters have replaced the ''ciclad''/''climserv'' cluster (in 2022) |
- | * [[https://documentations.ipsl.fr/MESO_User/|IPSL ESPRI Mesocenter documentation]] (//English//) | + | * You will need a specific account (different from your //LSCE// account) to access these servers and use data files that are already available/mirrored there (e.g. **''CMIPn'' data**).\\ **Do not duplicate** at LSCE data already available on IPSL servers and that you can process there! |
- | * [[https://mesocentre.ipsl.fr/quick-start/|Utilisation du Mésocentre IPSL]] (//French//, the //English// documentation link above has more information) | + | * [[https://mesocentre.ipsl.fr/account-opening/|Requesting an IPSL account]] |
+ | * [[https://documentations.ipsl.fr/spirit/|IPSL ESPRI Mesocenter documentation]] | ||
+ | * [[https://documentations.ipsl.fr/spirit/spirit_clusters/head_nodes.html|spirit clusters]] | ||
+ | * [[https://documentations.ipsl.fr/spirit/getting_started/support.html|Getting help]] | ||
* **The ''asterix//NN//'' servers**\\ Some wise LSCE elders may mention these servers, but they don't exist any more! | * **The ''asterix//NN//'' servers**\\ Some wise LSCE elders may mention these servers, but they don't exist any more! | ||
Line 306: | Line 323: | ||
* tcsh shell users: ''.cshrc'' and ''.login'' | * tcsh shell users: ''.cshrc'' and ''.login'' | ||
* bash shell users: ''.profile'' and ''.bashrc'' | * bash shell users: ''.profile'' and ''.bashrc'' | ||
- | * emacs text editor: ''.emacs'' | + | * [[other:emacs_doc|emacs text editor]]: ''.emacs'' |
* You can go to your home directory by typing ''cd'' (no arguments),\\ and to the home directory of somebody else with ''cd ~login_name'' | * You can go to your home directory by typing ''cd'' (no arguments),\\ and to the home directory of somebody else with ''cd ~login_name'' | ||
* At LSCE, you can't read the content of somebody else's //home// directory by default | * At LSCE, you can't read the content of somebody else's //home// directory by default | ||
Line 471: | Line 488: | ||
[[other:win10apps#notepad|Notepad++]] is a nice and powerful text editor for **Windows**, but it is also very easy to [[other:emacs_doc#windows_installation|install emacs on Windows]] | [[other:win10apps#notepad|Notepad++]] is a nice and powerful text editor for **Windows**, but it is also very easy to [[other:emacs_doc#windows_installation|install emacs on Windows]] | ||
- | ==== NetCDF and file formats ==== | + | ==== NetCDF and related Conventions ==== |
+ | |||
+ | === Super-short introduction to NetCDF === | ||
* What are binary ({{:other:newppl:binaire_jyp.pdf}}) and NetCDF ({{:other:newppl:netcdf_jyp.pdf}}) files? | * What are binary ({{:other:newppl:binaire_jyp.pdf}}) and NetCDF ({{:other:newppl:netcdf_jyp.pdf}}) files? | ||
Line 478: | Line 497: | ||
* The NetCDF file format is **self-documented**, and the metadata of climate data files often follows the [[http://cfconventions.org/|CF (Climate and Forecast) Metadata Conventions]] | * The NetCDF file format is **self-documented**, and the metadata of climate data files often follows the [[http://cfconventions.org/|CF (Climate and Forecast) Metadata Conventions]] | ||
* The files in the [[https://esgf-node.ipsl.upmc.fr/projects/esgf-ipsl/|CMIPn (n ≥ 5) database]] are written with the [[https://cmor.llnl.gov/|CMOR 3 library]] in order to make sure that they follow exactly the requested standards | * The files in the [[https://esgf-node.ipsl.upmc.fr/projects/esgf-ipsl/|CMIPn (n ≥ 5) database]] are written with the [[https://cmor.llnl.gov/|CMOR 3 library]] in order to make sure that they follow exactly the requested standards | ||
- | * You can use [[other:win7apps#panoply|Panoply]] and [[http://meteora.ucsd.edu/~pierce/ncview_home_page.html|ncview]] for visualizing quickly the NetCDF files | + | |
- | * [[https://code.zmaw.de/projects/cdo/wiki/Cdo#Documentation|Climate Data Operators]] (''cdo'') for manipulating NetCDF files (and some other formats) | + | |
- | * [[http://nco.sourceforge.net/#RTFM|netCDF Operator]] (//NCO//) for manipulating NetCDF files | + | === Some ways of working with NetCDF without programming === |
- | * [[http://cfconventions.org/|NetCDF CF (Climate and Forecast) Metadata Conventions]]: this will help you understand the information you get when you do a ''ncdump -h some_climate_data_file.nc'' | + | |
- | * Using Python to work with NetCDF data: | + | * Displaying the content of a NetCDF file in a text format\\ Check the ''ncdump'' [[https://docs.unidata.ucar.edu/nug/current/netcdf_utilities_guide.html#ncdump_guide|options]]. People usually want to quickly check the metadata, and use other programs to work with the actual data |
- | * Using the [[http://uv-cdat.llnl.gov/documentation/cdms/cdms.html|cdms2]] module if you work with the [[other:python:starting#uv-cdat|UV-CDAT distribution]] | + | * ''ncdump some_climate_data_file.nc'': dump the full content to text |
- | * Using the [[http://unidata.github.io/netcdf4-python/|netCDF4]] module | + | * ''ncdump -h some_climate_data_file.nc'': only display the metadata |
+ | * You can use [[other:win7apps#panoply|Panoply]] and [[http://meteora.ucsd.edu/~pierce/ncview_home_page.html|ncview]] for quickly visualizing NetCDF files | ||
+ | * Manipulating NetCDF files: | ||
+ | * [[https://code.zmaw.de/projects/cdo/wiki/Cdo#Documentation|Climate Data Operators]] (''cdo'') | ||
+ | * [[http://nco.sourceforge.net/#RTFM|netCDF Operator]] (//NCO// operators) | ||
+ | |||
+ | |||
+ | === NetCDF related Conventions === | ||
+ | |||
+ | * [[https://arxiv.org/abs/1911.08638|Gridspec: A standard for the description of grids used in Earth System models]] | ||
+ | * A nice document for discovering the different type of (ir)regular grids used in climate models! | ||
+ | |||
+ | * [[http://cfconventions.org/|NetCDF CF (Climate and Forecast) Metadata Conventions]] | ||
+ | * [[https://ugrid-conventions.github.io/ugrid-conventions/|UGRID Conventions]] for storing unstructured (or flexible mesh) model data | ||
+ | * [[http://sgrid.github.io/sgrid/|SGRID Conventions]] for storing staggered data on structured grids that are consistent with the UGRID conventions | ||
+ | |||
+ | === Using NetCDF with Python === | ||
+ | |||
+ | |||
+ | Check the [[other:python:jyp_steps#using_netcdf_files_with_python|dedicated section]] of the //Python// page | ||
==== Programming languages ==== | ==== Programming languages ==== |