User Tools

Site Tools


other:python:starting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
other:python:starting [2019/04/24 15:28]
jypeter [LSCE distribution] Improved
other:python:starting [2023/05/17 08:10] (current)
jypeter [Using a non-standard kernel] Added mkdir to create the kernels directory
Line 1: Line 1:
 ====== Working with Python ====== ====== Working with Python ======
  
-<note tip>This page will tell you how to select a python distribution and start (and exit!) the python interpreter+<note tip>This page will tell you how to select a //python distribution// and start (and exit!) the python interpreter
  
-You can then read the [[jyp_steps|JYP'​s recommended steps for learning python]] for really working with python+After reading this page, you can read the [[jyp_steps|JYP'​s recommended steps for learning python]] for really working with python
 </​note>​ </​note>​
 +
 +Note: the former and **outdated** version (before the massive usage of //conda//) is still [[other:​python:​starting_170810|available for reference]]
  
 ===== Where to start ===== ===== Where to start =====
Line 10: Line 12:
 In order to start working with Python, you need to have a //Python distribution//​ installed on your local computer or on the remote Linux server(s) you work on. A //​distribution//​ provides a //Python interpreter//,​ and Python extensions (aka Python //modules// or //​packages//​). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type ''​which python''​ on Linux to determine where the python executable is located) In order to start working with Python, you need to have a //Python distribution//​ installed on your local computer or on the remote Linux server(s) you work on. A //​distribution//​ provides a //Python interpreter//,​ and Python extensions (aka Python //modules// or //​packages//​). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type ''​which python''​ on Linux to determine where the python executable is located)
  
-If you are using a Linux computer or a Mac, you should already have a **default python** installed. The following example shows where the python interpreter is installed on the //obelix// LSCE servers (if it's in ''/​usr/​bin'',​ it's the default python) and which version it is (example below: version //2.7.5// compiled in April 2019)+If you are using a Linux computer or a Mac, you should already have a **default python** installed. The following example shows where the python interpreter is installed on the //obelix// LSCE servers (if it's in ''/​usr/​bin'',​ it's the //default// python) and which version it is (example below: version //2.7.5// compiled in April 2019)
  
 <​code>#​ Which is the current python (e.g. where is it located)? <​code>#​ Which is the current python (e.g. where is it located)?
Line 28: Line 30:
 </​code>​ </​code>​
  
-===== Python distributions available for LSCE users =====+===== Ultra quick-start on the interactive ​LSCE servers ​=====
  
-This section will help you choose a distribution in the big Python ecosystem (many distributions,​ python version 2.7.* or 3.*, ...). You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/​duplicate the data!) and try to identify who is maintaining it, if you need help or additional packages.+==== Initialization ====
  
-Only install ​a distribution ​yourself if you need it on your local computer, ​or if you need to install some modules that can't be installed ​by the contacts listed belowA python distribution ​can require several Gb of disk spaceso do not install it in a backed up //home// directory!+  - Start terminal on an //obelix// server ([[https://​wiki.lsce.ipsl.fr/​pmip3/​doku.php/​other:​newppl:​starting#​which_linux_servers_should_you_use|more info]]) 
 +  - Type ''​python''​ to access the //default Python 2// interpreter available on the servers... 
 +  - ...or **follow the steps below to use the more complete //​CDAT// ​distribution ​maintained by JYP**.\\ Remember that **you can exit the interpreter by typing CTRL-D** (or ''​quit()''​ or ''​exit()''​) 
 +    - Determine ​if you are using a [[other:​newppl:​starting#​which_shell_are_you_using|bash or tcsh shell]] (all the new LSCE accounts use the //bash// shell), then type the following command ​to //​initialize conda// (//conda// helps you manage your python environment) 
 +      * In a //bash// shell: ​''​source ~jypeter/​.conda3_jyp.sh''​ 
 +      * In a //tcsh// shell: ''​source ~jypeter/​.conda3_jyp.csh''​ 
 +    - Choose which environment you want by typing ​the activation command: 
 +      * Python 3.x: ''​conda activate cdatm_py3''​ 
 +      * You can use ''​conda env list''​ to find out which environments are available, but the one you most likely want is ''​cdatm_py3''​ 
 +    - If you need an (older) Python 2.7 environment,​ replace the steps above with: 
 +      * ''​source ~jypeter/​.conda3_19-06_jyp.sh''​ 
 +      * ''​conda activate cdatm_py2''​ 
 +    - Type ''​python''​ to start the interpreter 
 +    - If you need a list of the installed packages (and their version)use 
 +      * ''​conda list''​ 
 +      * A partial list of what is installed is available ​in the [[other:​uvcdat:​cdat_conda:​cdat_8_2_1#​extra_packages_list|Extra packages list]] 
 +    - Note: if you will have to use python regularly, you should add the ''​source''​ line above to your ''​.cshrc'' ​/''​.login''​ file (//tcsh// users) or ''​.bashrc''/''​.profile''​ (//bash// users) 
 +      * **Do not** add the ''​conda activate cdatm_pyN''​ line to the shell config files, because this will create potential side-effects!\\ Only type the activation command in the terminal(s) where you will need to use this specific version of Python
  
-You can use either //Python 2// or //Python 3//. Most packages are now available in both versions, but you should make sure that the most important package/s you need is/are available in the selected version. Check the [[https://​wiki.lsce.ipsl.fr/​pmip3/​doku.php/​other:​python:​jyp_steps#​python_27_vs_python_3|differences between both versions]] and try to write scripts that will work in both versions! 8-)+=== Example ===
  
-==== LSCE distribution ====+<​code>​ >ssh obelix 
 +Last login: Wed Feb  1 09:56:29 2023 from somewhere
  
-Contact: the LSCE system administrators ([[help-lsce@lsce.ipsl.fr]])+bash-4.2$ which python 
 +/​usr/​bin/​python
  
-<​code>​ +bash-4.2$ source ~jypeter/.conda3_jyp.sh
- > module avail +
-[...] +
-4ARTIC/​3.6 ​        ​grib_api/​1.14 ​     netcdf/​3 ​          ​python/​2.7.5 +
-batch_env ​         grib_api/​1.14.0 ​   netcdf/          python/3.+
-[...] +
-glost/​0.3.1 ​       ncview/2.1.7       ​python/2.7 +
-[...]+
  
- > module load python/2.7+bash-4.2$ which python 
 +/usr/​bin/​python
  
- > which python +bash-4.2$ conda activate cdatm_py3
-/​usr/​local/​install/​python-2.7/​bin/​python+
  
- > python +(cdatm_py3bash-4.2$ which python 
-Python 2.7.15 |Anaconda, Inc.| (default, Oct 10 2018, 21:32:13) +/home/​share/​unix_files/​cdat/​miniconda3_21-02/​envs/​cdatm_py3/​bin/​python
-[GCC 7.3.0] on linux2 +
-Type "​help",​ "​copyright",​ "​credits"​ or "​license"​ for more information. +
->>>​ +
-</code>+
  
-==== TGCC distribution ==== +(cdatm_py3bash-4.2$ python 
- +Python 3.8.8 | packaged by conda-forge | (default, ​Feb 20 202116:22:27
-Contact: the TGCC hotline ​([[hotline.tgcc@cea.fr]]) +[GCC 9.3.0] on linux
- +
-<​code>​ +
-$ module avail +
-[...] +
-cmake/​2.8.9(default) ​     hwloc/​1.5 ​                ​netcdf-utils/4.3.3.1_hdf5 python/2.7.8              tix/8.4.3 +
-cmake/​3.2.2 ​              ​hwloc/​1.7.1(default) ​     octave/​3.6.3 ​             python/​3.3.2 ​             tk/​8.5(default) +
-[...] +
- +
-module load python/2.7.8 +
-load module python/2.7.8 (Python) +
- +
-$ which python +
-/​usr/​local/​ccc_python/​2.7.8_201409/​bin/​python +
- +
-$ python +
-Python 2.7.8 (default, ​Aug 27 201417:50:16+
-[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2+
 Type "​help",​ "​copyright",​ "​credits"​ or "​license"​ for more information. Type "​help",​ "​copyright",​ "​credits"​ or "​license"​ for more information.
->>>​+>>> ​print('​hello!'​) 
 +hello! 
 +>>> ​  # Type CTRL-D to exit the python interpreter
 </​code>​ </​code>​
  
-==== UV-CDAT ​====+==== Useful keyboard shortcuts ​====
  
-Contact: Jean-Yves Peterschmitt (//JYP//) @ LSCE+^  Key  ^  Effect ​ ^ 
 +|  **CTRL-D**  |  Exit the interpreter ​ | 
 +|  **↑ and ↓**  |  Go to previous/next line(s) ​ | 
 +|  **CTRL-A** ​ |  Go to the beginning of the line  | 
 +|  **CTRL-E** ​ |  Go to the end of the line  | 
 +|  **CTRL-K** ​ |  Erase from the cursor to the end of the line  | 
 +|  **CTRL-U** ​ |  Erase from the beginning of the line to the cursor ​ | 
 +|  **TAB** x 2  |  Do some TAB-completion (context dependent)\\ e.g. ''​a = np.cum**TABTAB**'' ​ | 
 +|  **CTRL-C** ​ |  Interrupt a running script ​ | 
 +|  **CTRL-Z** ​ |  Suspend the interpreter and go back to the shell\\ **Do not forget** to go back to the interpreter with ''​fg''​\\ or to kill it (with ''​jobs''​ and ''​kill %NN''​ |
  
-=== What is UV-CDAT? ​===+==== Executing a python script ====
  
-[[http://uv-cdat.llnl.gov/|UV-CDAT]] (//Ultrascale Visualization - Climate Data Analysis Tools//) is a python distribution developed specifically for the climate scientists, and is recommended by JYP.+Note: Python is an [[https://en.wikipedia.org/wiki/​Interpreted_language|interpreted language]] and we speak of //Python scripts// rather than //Python programs//.
  
-UV-CDAT is **available for Mac and Linux**.+  ​''​python'':​ start the interpreter and start working interactively 
 +  ​''​python script.py'':​ execute //​script.py// ​and **exit** 
 +  * ''​python -i script'':​ execute //script.py// and **stay in the interpreter** (''​-i''​ = //​**I**nteractive//​)
  
-You can find out which **standard packages** are installed in the //Packages built by CDAT// column below. UV-CDAT also provides some **specific packages** that you will not find in other python distributions:​ **cdms2**, **cdutil**, **cdtime**, **genutil**,​ **vcs**, **cmor**, ...+Type ''​man python''​ if you want to see what other command line options are available
  
-**Extra packages** are listed: +==== Stand-alone script ====
-  * in the apropriate //​Installation notes// pages for CDAT>=''​2.8.0''​ +
-  * on the [[other:​uvcdat:​extra|Extra packages installed in UV-CDAT by JYP]] page for CDAT%%<=%%''​2.10''​+
  
-=== UV-CDAT versions maintained by JYP ===+You don't have to explicitly call the python interpreter,​ if the interpreter is specified in a ''#​!''​ [[https://​en.wikipedia.org/​wiki/​Shebang_%28Unix%29|shebang comment]] on **the first line of the script** with, and the script has its //execution bit// set (''​chmod +x my_script.py''​). ​
  
-The following versions are maintained by JYP, on the Linux servers where LSCE users have accounts+If you have a ''​basic_script.py''​ file with the following content in a directory... 
 +<​code>​ 
 +#​!/​usr/​bin/​env python
  
-^  CDAT version ​ ^  JYP\\ version ​ ^  python\\ version ​ ^  Availability? ​ ^  Packages built by CDAT  ^  Installation notes  ^ +import sys
-|  **1.1.0** ​ |  08  |  2.7.3  |  LSCE, curie  |  n/a  |  [[other:​uvcdat:​installed#​section110|1.1.0 notes]] ​ | +
-|  **1.5.1** ​ |  10  |  2.7.4  |  LSCE, ciclad ​ |  {{:​other:​python:​cdat-1.5.1_jyp-10_build_info.txt}} ​ |  [[other:​uvcdat:​installed#​section151|1.5.1 notes]] ​ | +
-|  **2.1.0** ​ |  13  |  2.7.8  |  LSCE, ciclad ​ |  {{:​other:​python:​cdat-2.1.0_jyp-13_build_info.txt}} ​ |  [[other:​uvcdat:​installed#​section210|2.1.0 notes]] ​ | +
-|  **2.4.0** ​ |  //Not installed// ​ ||||| +
-|  **2.8.0** ​ |  14  |  2.7.12 ​ |  LSCE  |  {{:​other:​uvcdat:​uvcdat-2.8.0_install_dry_with-forge_160221.txt}} ​ |  [[:​other:​uvcdat:​cdat_conda:​cdat_2_8_0|2.8.0 notes]] ​ |+
  
-Notes: +script_name ​sys.argv[0]
-  * For the conda based CDAT versions (>=''​2.8.0''​),​ you can get the list of currently available packages by typing: ''​conda list''​ +
-  * For older CDAT versions (<​=''​2.1.0''​),​ the ''​cdat-VVV_jyp-NNN_build_info.txt''​ files have been generated with\\ ''​cd /​my_build_directory_path;​ cat build_info.txt | sort > cdat-VVV_JYP-NNN_build_info.txt''​+
  
-=== The basics of UV-CDAT initialization ===+print('​Hello world, I am the ' + script_name + ' script'​)
  
-The way you initialize UV-CDAT depends on: +The end</code>
-  * which **server** you want to use it on +
-  * which **shell** you are using in your terminals: //tcsh// or other shells +
-    * Read [[other:​newppl:​starting#​which_shell_are_you_using|Which shell are you using?]] if you are not sure +
-    * Warning: the shell in a (batch) script may be different from your interactive shell! +
-  * which **version** of UV-CDAT you want to use... +
-    * Note: You should use the most recent version of UV-CDAT available on a given server, unless you have a good reason to use an older version (e.g. if you need to use the //old-style vcs// of version 1.5.1). +
-Note: if you get an error when importing //cdms2// or //vcs/it means that either you have forgotten to initialize UV-CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed on your system, or another python distribution!+
  
-<​code>​$ python -c '​import cdms2vcs' +...and if you are in the same directory and have initialized the Python distribution you wantyou can then run the script the following way:
-Traceback (most recent call last): +
-  File "<​string>",​ line 1, in <​module>​ +
-ImportErrorNo module named cdms2+
  
-# Am I using the correct python distribution?​ +<​code> ​> chmod +x basic_script.py
-$ which python +
-/​usr/​bin/​python</code>+
  
-=== conda-based versions of UV-CDAT ===+ > ls -l basic_script.py 
 +-rwxr-xr-x [...] basic_script.py
  
-<WRAP center round important 60%+ ./​basic_script.py 
-\\ The steps detailed in this section are for [[other:​uvcdat:​cdat_conda:​|UV-CDAT 2.8.0 and later]] +Hello world, I am the ./​basic_script.py script 
-</WRAP>+</code>
  
-Note: if you need to maintain a **local** version on **your** machine, you can read [[other:​uvcdat:​conda_notes|Installing and maintaining UV-CDAT with conda]] and the [[other:​uvcdat:​cdat_conda:​index#​versions|versions specific notes]].+==== What next? ====
  
-== Configuring .bashrc ==+  - Read the rest of this page to get a better understanding of what you have learned in this section, and use Python more efficiently 
 +  - Learn Python by reading what you need on the [[other:​python:​jyp_steps|JYP'​s recommended steps for learning python]] page
  
-You should **add** the following line at the end of your ''​~/​.bashrc''​ configuration file (create the ''​~/​.bashrc''​ file if it does not exist yet). This will basically make sure that your shell can find the ''​conda''​ executable it needs for initializing CDAT, and define a ''​wp''​ alias that you can use to find where the ''​python''​ executable is located 
  
-^  Server ​ ^  Line to add to ''​.bashrc'' ​ ^ +===== Using the LSCE jupyterhub server =====
-|  **LSCE**  | ''​source ~jypeter/​.conda_jyp.sh''​\\ \\ Note:{{ :​other:​python:​conda_jyp.txt |conda_jyp}} content | +
-|  **ciclad** ​ | ''​source ~jypmce/​.conda_jyp.sh''​ |+
  
 +The [[https://​intranet.lsce.ipsl.fr/​informatique/​en/​jupyter/​index.php|LSCE jupyterhub notebook server]] is available for users with a LSCE Linux account: ​
 +  * from a computer on the LSCE wired network: https://​jupyter.lsce.ipsl.fr/​
 +  * from other computers: start the [[https://​intranet.lsce.ipsl.fr/​informatique/​fr/​vpn-forticlient.php#​vpnlsce|LSCE VPN]] before opening https://​jupyter.lsce.ipsl.fr/​
  
-Note: when you add the specified line to your ''​bashrc''​ file, you will then always get the python supplied by conda, instead of the system'​s python, when you use ''​bash''​. If you don't like this permanent behavior, just type ''​source ~jyp_login_on_this_server/​.conda_jyp.sh''​ in the bash shell where you want to use conda+CDAT, rather than adding it to the ''​.bashrc''​ file...+==== Using a non-standard kernel ====
  
-== Initializing UV-CDAT ==+Note: a //kernel// is a specific python environment/​distribution
  
-<WRAP center round tip 60%> +The LSCE jupyterhub server can be used to open //​Notebooks//​ using pre-configured Python or R //kernels//. It is also possible to define additional per-user kernels by adding [[https://​jupyter-client.readthedocs.io/​en/​stable/​kernels.html#​kernel-specs|kernel configuration directories]] in the user's home directory
-When you initialize UV-CDAT, it will be available only in the **current** window/shellIn the other windows, you still get whatever your default python ​is. This should minimize potential side-effects +
-</WRAP>+
  
-  - If [[other:newppl:starting#which_shell_are_you_using|your shell]] is not //bash//start a bash shell by typing ''​bash''​ +The steps below show how to add a specific ​[[other:python:starting#cdat|CDAT environment]] kernelbut can be easily adapted for using other kernels
-    * You can later go back to your previous shell by typing ''​exit''​ or ''​CTRL-D''​ +
-  - Activate the //conda environment//​ that uses the version of UV-CDAT you want to use +
-    * ''​source activate conda_env_name''​ +
-    * Use the table below to determine which environments are available on which server +
-  - Use CDAT !+
  
-^  UV-CDAT version ​ ^  Server ​ ^  Environment name\\ ​and activation line  ^  Packages\\ ​(''​conda list''​ ^ +  ​Initialize your (favorite) Python environment in a terminal, ​and (as usualcheck which Python executable you are using\\ <​code>​ >which python 
-|  [[:​other:​uvcdat:​cdat_conda:​cdat_2_8_0|2.8.0]] ​ |  **LSCE**\\ **ciclad** ​ |**cdatm14** (''​source activate cdatm14''​) ​ |  {{ :​other:​uvcdat:​cdat_conda:​conda_list_lsce_cdatm14_170310.txt |cdatm14 installed packages}} ​ | +/usr/bin/python
-|  [[:​other:​uvcdat:​cdat_conda:​cdat_2_10|2.10]] ​ |  **LSCE** ​ |**cdatm15** (''​source activate cdatm15''​) ​ |  {{ :​other:​uvcdat:​cdat_conda:​uvcdat-2.10_list_170721.txt |cdatm14 installed packages}} ​ | +
-|  **cdatm** will always point to the latest stable version ​ |||| +
-|  ​//latest stable version//  |  **LSCE** ​ |**cdatm** (''​source activate cdatm''​) ​ |  ''​conda list'' ​ |+
  
-=== Older versions of UV-CDAT ===+ >​source ~jypeter/​.conda3_21-02_jyp.csh 
 + >​conda activate cdatm_py3
  
-<WRAP center round important 60%+(cdatm_py3) ​ >which python 
-The steps detailed in this section are for **UV-CDAT ''​2.1.0''​ and earlier** +/​home/​share/​unix_files/​cdat/​miniconda3_21-02/​envs/​cdatm_py3/​bin/​python</code
-</WRAP>+  - Determine where the kernel configuration files of the **active** python are located\\ <​code>​ >jupyter kernelspec list 
 +Available kernels: 
 +  python3 ​    /​home/​share/​unix_files/​cdat/​miniconda3_21-02/​envs/​cdatm19_nompi_py3/​share/​jupyter/​kernels/​python3</​code>​ 
 +  - Copy the existing kernel directory that you have found to your home directory. You can use the existing kernel name, or use a new kernel name (using only ASCII letters, numbers, ​ ''​-''​ hyphen, ​''​.'' ​period ​and ''​_''​ underscore)\\ ​<code> >mkdir -p ~/.local/​share/​jupyter/​kernels 
 + >cp -pr /​home/​share/​unix_files/​cdat/​miniconda3_21-02/​envs/​cdatm19_nompi_py3/​share/​jupyter/​kernels/​python3 ~/​.local/​share/​jupyter/​kernels 
 + >​cp -pr /​home/​share/​unix_files/​cdat/​miniconda3_21-02/​envs/​cdatm19_nompi_py3/​share/​jupyter/​kernels/​python3 ~/​.local/​share/​jupyter/​kernels/​my_favorite_kernel
  
-== Initializing UV-CDAT for tcsh users ==+ >​ls ~/​.local/​share/​jupyter/​kernels/​ 
 +my_favorite_kernel/ ​   python3/ 
 +</​code>​ 
 +  ​Check that the ''​kernelspec list''​ command now finds the new kernel(s)\\ <​code>​ >jupyter kernelspec list 
 +Available kernels: 
 +  my_favorite_kernel ​   your_home_dir/​.local/​share/​jupyter/​kernels/​my_favorite_kernel 
 +  python3 ​              ​your_home_dir/​.local/​share/​jupyter/​kernels/​python3</​code>​ 
 +  - You can optionally change the value of the ''​display_name''​ parameter in the ''​kernel.json''​ configuration file if you want to customize the name that will appear in the jupyterhub kernel menus\\ <​code>​ >ls your_home_dir/​.local/​share/​jupyter/​kernels/​my_favorite_kernel 
 +kernel.json ​ logo-32x32.png ​ logo-64x64.png
  
-If you use **tcsh** as [[other:​newppl:​starting#​which_shell_are_you_using|your login shell]], you can define a ''​cdatm''​ alias on all the IPSL servers on which UV-CDAT is installed (see details below), and move easily from one version of UV-CDAT to another: + >cat your_home_dir/.local/share/jupyter/kernels/my_favorite_kernel/kernel.json 
-  * type ''​cdatm''​ to initialize the default version of UV-CDAT +{ 
-  * type ''​cdatm %%--%%config ​//CONFIG_NAME//''​ to initialize a specific version + "​argv":​ [ 
-    * You can find the allowed values of //CONFIG_NAME//​ (e.g. ''​LSCE_10'',​ ''​LSCE_13'',​ ...) in one of the tables below +  "​/home/​share/​unix_files/​cdat/​miniconda3_21-02/​envs/​cdatm19_nompi_py3/bin/python", 
- +  "​-m",​ 
-<​code>​ > which python +  "​ipykernel_launcher",​ 
-/usr/​bin/​python +  "​-f", 
- +  "​{connection_file}"​ 
- > cdatm + ]
-** CDAT 'uv-1.5.1' initialized! + "​display_name"​"My favorite Python kernel",​ 
-** If you have problems with this installation+ "​language":​ "python"
-** it is maintained byJean-Yves Peterschmitt - LSCE +
- +
- > which python +
-/​home/​share/​unix_files/​cdat/​versions/​cdat_install_uv-1.5.1_x86_64_gcc4_10/​bin/​python+
 </​code>​ </​code>​
 +  - refresh the https://​jupyter.lsce.ipsl.fr/​ page and the new kernel(s) should appear in the kernels'​ related menus
 +===== Python distributions available for LSCE users =====
  
-You need to add the following line to the ''​.cshrc''​ file in your home directoryin order to define ​the ''​cdatm''​ alias. You can also define ​the simple ''​wp''​ aliasin order to always be able to determine quickly which python ​is initialized in the current terminal.+This section will help you choose a distribution in the big Python ecosystem (many distributions,​ python version 2.7.* or 3.*...) on some of the servers used by LSCE users. You should use a distribution that is already available near your data (e.g. do the computation on a server near your datado not move/​duplicate the data!) and try to identify who is maintaining it, if you need help or additional packages.
  
-|  **LSCE** ​ | ''alias cdatm '​source ~jypeter/​CDAT/​Install/​cdat_multi/​cdat_multi.login ~jypeter/CDAT/Install/cdat_multi'''​ | +Only install a distribution yourself if you need it on your local computer (desktop or laptop), or if you need to install some modules that can't be installed by the contacts listed belowA python distribution will require several Gb of disk space, so do not install it in your backed up //home// directory!
-|  **ciclad** ​ | ''​alias cdatm '​source ~jypmce/CDAT/​Install/​cdat_multi/​cdat_multi.login ~jypmce/​CDAT/​Install/​cdat_multi'''​| +
-|  **curie** ​ | ''​alias cdatm '​source ~p25jype/​CDAT/​Install/​cdat_multi/​cdat_multi.login ~p25jype/​CDAT/​Install/​cdat_multi'''​ | +
-|  **All** servers ​ | ''​alias wp 'which python''​ |+
  
-== Initializing UV-CDAT ​in any kind of shellor in a batch script ==+You can use either //Python 2// or //Python 3//. Most packages are now available ​in both versionsbut you should make sure that the most important package/s you need is/are available in the selected Python version. You can check the [[https://​wiki.lsce.ipsl.fr/​pmip3/​doku.php/​other:​python:​jyp_steps#​python_27_vs_python_3|differences between both versions]] and try to write scripts that will work in both versions!
  
-You first need to know where the ''​python''​ program provided by UV-CDAT is installed, something like ''/​path/​to/​UV-CDAT/​bin/​python''​+==== LSCE distribution ====
  
-Then, all you need to do in order to initialize UV-CDAT is to //source// the ''​setup_runtime''​ file appropriate for [[other:​newppl:​starting#​which_shell_are_you_using|your shell]]+  * Contact: the LSCE system administrators ([[help-lsce@lsce.ipsl.fr]]) 
-  * tcsh: type ''​source ​/path/to/UV-CDAT/bin/​setup_runtime**.csh**''​ +  * Where: ​//obelix// interactive servers and //obelix// cluster at LSCE 
-  * bash, sh/ksh: type ''​source /​path/​to/​UV-CDAT/​bin/​setup_runtime**.sh**''​+  * Initialization type: [[other:​newppl:​starting#​using_module_to_access_optional_programs|module based]] //conda// 
 +  * What's installed: type ''​conda list'' ​after initializing the LSCE distribution
  
-The table below lists, for each server and version, the path where you can find the //setup_runtime// files+<​code>​ 
 +$ module -t avail 
 +[...] 
 +python/2.7 
 +python/2.7.5 
 +python/2-ramces 
 +python/
 +python/​3.9 
 +python/​4artic 
 +python/​esmpy 
 +python/​intel 
 +[...]
  
-^  Server ​ ^  CDAT\\ version ​ ^  JYP\\ version ​ ^  Path  ^ +module load python/3.9
-|  LSCE  |  1.1.0  |  LSCE_08 ​ |  ''/​home/​share/​unix_files/​cdat/​versions/​cdat_install_uv-1.1.0_x86_64_gcc4_08/​bin'' ​ | +
-|  LSCE  |  1.5.1  |  LSCE_10 ​ |  ''/​home/​share/​unix_files/​cdat/​versions/​cdat_install_uv-1.5.1_x86_64_gcc4_10/​bin'' ​ | +
-|  LSCE  |  2.1.0  |  LSCE_13 ​ |  ''/​home/​share/​unix_files/​cdat/​versions/​cdat_install_uv-2.1.0_x86_64_gcc4_13/​bin'' ​ | +
-|  ciclad ​ |  1.5.1  |  ciclad_10-ng ​ |  ''/​data/​jypmce/​cdat/​versions/​cdat_install_uv-1.5.1_x86_64_gcc4_10-ng/​bin'' ​ | +
-|  ciclad ​ |  2.1.0  |  ciclad_13 ​ |  ''/​data/​jypmce/​cdat/​versions/​cdat_install_uv-2.1.0_x86_64_gcc4_13/​bin'' ​ | +
-|  curie  |  1.1.0  |  curie_08 ​ |  ''/​ccc/​work/​cont003/​dsm/​p25jype/​cdat/​versions/​cdat_install_uv-1.1.0_x86_64_gcc4_curie_08/​bin''​\\ **On curie**, you need to use **setup_cdat**.sh or **setup_cdat**.csh to initialize CDAT,\\ and also to load modules providing access to old librairies\\ ''​module load gnu/4.6.3''​ and ''​module load qt/4.8.6'' ​ |+
  
-Example: if you want to use UV-CDAT 2.1.0 at LSCE in a bash shell, you need to do the following (you only need the //source// lineThe other lines are optional, just for checking that the initialization was OK)+$ which python 
 +/usr/local/install/python-3.9/​bin/​python
  
-<​code>​bash-4.1source /​home/​share/​unix_files/​cdat/​versions/​cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/​setup_runtime.sh+python 
 +Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
 +[GCC 7.5.0] :: Anaconda, Inc. on linux 
 +Type "​help",​ "​copyright",​ "​credits"​ or "​license"​ for more information. 
 +>>>​ 
 +</code>
  
-bash-4.1$ which python +==== CDAT ====
-/​home/​share/​unix_files/​cdat/​versions/​cdat_install_uv-2.1.0_x86_64_gcc4_13/​bin/​python+
  
-bash-4.1$ python ​-c '​import cdms2, vcs' +[[https://​cdat.llnl.gov/​|CDAT]] (//​Community Data Analysis Tools//) is a powerful and complete front-end to a rich set of visual-data exploration and analysis capabilities well suited for **climate** data analysis problems. CDAT is **recommended by JYP**.
-bash-4.1$</​code>​+
  
-You probably don't want to have to determine what you should type each time you want to use UV-CDAT in a window, so you should define a simple shortcut alias that you can type each time you need to initialize UV-CDAT. If, for example, you want to use UV-CDAT 2.1.0 at LSCE: +CDAT is **available for MacLinux and Windows 10 (Windows 10 + [[other:​win10wsl|Windows Subsystem ​for Linuxand Ubuntu]])**.
-  ​tcsh: add the following alias definitions to ''​~/​.cshrc''​\\ ''​alias cdat13 '​source /​home/​share/​unix_files/​cdat/​versions/​cdat_install_uv-2.1.0_x86_64_gcc4_13/​bin/​setup_runtime.csh'''​\\ ​ ''​alias wp 'which python'''​ +
-  ​bash: add the following alias definition to ''​~/​.bashrc''​''​~/​.profile''​ or ''​~/​.bash_profile'' ​(whichever works for youor read ''​man bash''​...)\\ ''​alias cdat13='​source /​home/​share/​unix_files/​cdat/​versions/​cdat_install_uv-2.1.0_x86_64_gcc4_13/​bin/​setup_runtime.sh'''​\\ ''​alias wp='​which python'''​+
  
-If you are going to use a UV-CDAT python script on a cluster or supercomputer,​ using some kind of batch system using a bash or ksh script, use the following lines before you run the python script +  * Contact: Jean-Yves Peterschmitt (//JYP//) @ LSCE 
-<​code>​source ​/home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.sh+  * Where: ​//obelix// interactive servers and cluster at LSCE, //ciclad// interactive servers and cluster at IPSL, irene @ TGCC 
 +  * Initialization type: conda based 
 +  * What's installed: type ''​conda list''​ after initializing a specific CDAT distribution
  
-# Check which python we will be using +=== CDAT versions maintained by JYP ===
-mypython=`which python` +
-echo +
-echo -e "This script will use the following python: $mypython\n\n"</​code>​+
  
-==== Canopy ====+The following versions are maintained by JYP, on the Linux servers where LSCE users have accounts
  
-[[https://www.enthought.com/​products/​canopy/​|Enthought Canopy]] is //a Scientific and Analytic Python Deployment with Integrated Analysis Environment//​ provided by [[https://www.enthought.com/|Enthought]]. It used to be called EPD (//​Enthought Python Distribution//​)+^  CDAT version ​ ^  JYP\\ version ​ ^  python\\ version ​ ^  Availability? ​ ^  Available packages ^  Installation notes  ^ 
 +|  **8.2.1** ​ |  19  |  3.8.8  |  LSCE  |  {{  ​:other:​uvcdat:​cdat_conda:​cdat_8.2.1_nompi_21-03-05.txt |Default CDAT 8.2.1 packages}}\\ [[other:​uvcdat:​cdat_conda:​cdat_8_2_1#​extra_packages_list|Extra packages]]  ​|  ​[[other:uvcdat:​cdat_conda:​cdat_8_2_1|8.2.1 notes]] ​ | 
 +|  **8.1** ​ |  18  |  2.7.15\\ 3.6.7  |  LSCE, ciclad ​ |  {{:​other:​uvcdat:​cdat_conda:​cdat-8.1_py2_list_190307.txt|Default CDAT 8.1 packages}}\\ [[other:​uvcdat:​cdat_conda:​cdat_8_1#​extra_packages_list|Extra packages]]  ​| ​ [[other:​uvcdat:​cdat_conda:​cdat_8_1|8.1 notes]] ​ |
  
-Canopy is **available for Windows, Mac and Linux**. You can download the free //Canopy Express// that will already provide many extensions. If you are entitled to use //Canopy Academic//, login from inside Canopy Express and download the extra modules you need+== Initialization ==
  
 +Read the [[https://​wiki.lsce.ipsl.fr/​pmip3/​doku.php/​other:​python:​starting#​ultra_quick-start_on_the_interactive_lsce_servers|Ultra quick-start on the LSCE servers]] above
  
-==== Anaconda ====+Note: on the ciclad cluster, use ''​source ~jypmce/​.conda3_jyp.sh''​ to initialize conda
  
-[[https://​www.continuum.io/​why-anaconda|Anaconda]] is a distribution similar to [[#canopy]] provided by [[https://​www.continuum.io/​|Continuum Analytics]].+=== CDAT at TGCC ===
  
-Anaconda is **available for WindowsMac and Linux**+Note: TGCC also supports its own [[#​tgcc_distribution|TGCC distribution]]that may be more up-to-date, if you don't need specific CDAT modules
  
-Note: Anaconda provides and uses ''​conda''​ for its installationSince UV-CDAT ​is now also installed and maintained ​with ''​conda''​, you can read the [[other:uvcdat:​conda_notes|Installing and maintaining UV-CDAT with conda]] page for more information,​ even if you are not going to use UV-CDAT+CDAT **8.0** is installed ​at TGCC and can be initialized ​with
 +  * Python 2: ''​module load cdat''​ 
 +  * Python 3''​module load flavor/​cdat/​python3 cdat''​
  
-===== Launching Python =====+=== A common CDAT-related error ===
  
-Once you have initialized the [[#​some_python_distributions|python distribution]] you want to use, follow the instructions below to //start the interpreter from a shell// (on a Windows computer, you will have to start python from the //Start// menu or by clicking a shortcut ​on your desktop).+Note: if you get an error when importing ​//cdms2// or //vcs// it means that either you have forgotten to initialize CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed ​on your system, or another (non-CDATpython distribution!
  
-Remember that on Linux/Mac you can check which python ​you are using by typing ​''​which python''​!+<​code>​$ ​python ​-c 'import cdms2, vcs' 
 +Traceback (most recent call last): 
 +  File "<​string>",​ line 1, in <​module>​ 
 +ImportError:​ No module named cdms2
  
-==== Useful keyboard shortcuts ====+# Am I using the correct python distribution?​ 
 +$ which python 
 +/​usr/​bin/​python</​code>​
  
-^  Key  ^  Effect ​ ^ +==== TGCC distribution ====
-|  **CTRL-D** ​ |  Exit the interpreter ​ | +
-|  **↑ and ↓**  |  Go to previous/​next line(s) ​ | +
-|  **CTRL-A** ​ |  Go to the beginning of the line  | +
-|  **CTRL-E** ​ |  Go to the end of the line  | +
-|  **CTRL-K** ​ |  Erase from the cursor to the end of the line  | +
-|  **CTRL-U** ​ |  Erase from the beginning of the line to the cursor ​ | +
-|  **TAB** x 2  |  Do some TAB-completion (context dependent)\\ e.g. ''​a ​np.cum**TABTAB**'' ​ | +
-|  **CTRL-C** ​ |  Interrupt a running script ​ | +
-|  **CTRL-Z** ​ |  Suspend the interpreter and go back to the shell\\ **Do not forget** to go back to the interpreter with ''​fg''​\\ or to kill it (with ''​jobs''​ and ''​kill %NN''​) ​ |+
  
-==== Stand-alone script ====+  * Contact: the TGCC hotline ([[hotline.tgcc@cea.fr]]) 
 +  * Where: CEA TGCC 
 +  * Initialization type: [[other:​newppl:​starting#​using_module_to_access_optional_programs|module based]] 
 +  * What's installed: it depends...
  
-A python script is just like any other shell script. You don't have to explicitly call the python interpreter,​ if the interpreter is specified in a ''#​!'' ​[[https://en.wikipedia.org/wiki/​Shebang_%28Unix%29|shebang comment]] on the first line of the script with, and the script has its execution bit set (''​chmod +x my_script.py''​). +<​code>#​ Get the default version of the Python based modules 
 +$ module avail -t -d | egrep '(python|cdat)' 
 +flavor/cdat/standard(default) 
 +flavor/​nest/​python2(default) 
 +flavor/​pytorch/​python2(default) 
 +flavor/​tensorflow/​gpu_python2(default) 
 +intelpython2/​2019.0(default) 
 +intelpython3/​2019.0(default) 
 +python/2.7.14(default) 
 +python3/3.6.4(default) 
 +cdat/8.0(default)
  
-<​code>​ +$ module load python 
-jypeter@asterix1 - ...jypeter - 54 >cat basic_script.py +[...] 
-#​!/​usr/​bin/​env ​python+load module ​python/2.7.14 (Python)
  
-import sys+$ which python 
 +/​ccc/​products/​python-2.7.14/​intel--17.0.4.196__openmpi--2.0.2/​default/​bin/​python
  
-script_name = sys.argv[0] +$ python 
- +Python 2.7.14 (default, Jan 11 201816:43:59
-print('Hello worldI am the ' + script_name + ' script'​+[GCC 4.8.5] on linux2 
- +Type "​help",​ "​copyright",​ "​credits"​ or "​license"​ for more information
-# The end +>>>
- +
-jypeter@asterix1 - ...jypeter - 55 >chmod +x basic_script.py +
- +
-ypeter@asterix1 - ...jypeter - 56 ># Type here what is required to initialize the distribution you want +
- +
-jypeter@asterix1 - ...jypeter - 57 >./​basic_script.py +
-Hello world, I am the ./​basic_script.py script+
 </​code>​ </​code>​
  
-==== Standard interpreter ==== 
  
-  * ''​python'':​ start the interpreter +===== ipython =====
-  * ''​python script.py'':​ execute //​script.py//​ and **exit** +
-  * ''​python -i script'':​ execute //​script.py//​ and **stay in the interpreter** (''​-i'' ​//​**I**nteractive//​) +
- +
-Type ''​man python''​ if you want to see what other command line options are available+
  
 ==== ipython interpreter ==== ==== ipython interpreter ====
  
-The //ipython// interpreter provides more options and commands than the standard python interpreter,​ but takes more time to load. If you are going to develop by starting and exiting the interpreter many times, it's faster to use the standard interpreter+The //ipython// interpreter provides more options and commands than the standard python interpreter,​ but takes more time to start. If you are going to develop by starting and exiting the interpreter many times, it's faster to use the standard interpreter
  
 Starting ipython: ''​ipython''​ Starting ipython: ''​ipython''​
Line 321: Line 319:
   * [[https://​damontallen.github.io/​IPython-quick-ref-sheets/​|Ipython-quick-ref-sheets]]   * [[https://​damontallen.github.io/​IPython-quick-ref-sheets/​|Ipython-quick-ref-sheets]]
  
 +
 +==== LSCE jupyter notebook server ====
 +
 +The different Python distributions available on the LSCE servers (interactive servers and cluster) can also be used on the [[https://​intranet.lsce.ipsl.fr/​informatique/​en/​jupyter/​index.php|LSCE jupyter notebook server]]:
 +  * If your computer is on the LSCE wired network
 +  * Or if you are using the [[https://​intranet.lsce.ipsl.fr/​informatique/​fr/​vpn-forticlient.php|LSCE VPN]]
 ==== ipython notebook ==== ==== ipython notebook ====
  
-The ipython notebook is a way to interact with python (and other supported interpreted languages) inside a web browser. You can mix //cells// with python commands, cells with the output of the python commands (possibly graphics generated by the commands), and text (using some wiki-like rich text format). This interactive web page, aka notebook, can be saved in a ''​my_notebook.ipynb''​ file and re-used later (e.g. the notebook provided in the [[other:​python:​jyp_steps#​part_1|introduction to Python, part 1]]).+<WRAP center round tip 60%> 
 +If you want to use ipython notebooks at LSCE, you should **use the [[other:​python:​starting#​lsce_jupyter_notebook_server|LSCE notebook server]]** rather than trying to use (and probably overloading) the interactive servers! 
 +</​WRAP>​ 
 + 
 + 
 +<note warning>​FIXME Add a link to the new notebook page</​note>​ 
 + 
 +[[https://​www.dataquest.io/​blog/​jupyter-notebook-tutorial/​|Jupyter Notebook for Beginners: A Tutorial]] 
 + 
 +The **ipython notebook is a way to interact with python (and other supported interpreted languages) inside a web browser**. You can mix //cells// with python commands, cells with the output of the python commands (possibly graphics generated by the commands), and text (using some wiki-like rich text format). This interactive web page, aka notebook, can be saved in a ''​my_notebook.ipynb''​ file and re-used later (e.g. the notebook provided in the [[other:​python:​jyp_steps#​part_1|introduction to Python, part 1]]).
  
 Starting the notebook server: ''​ipython notebook''​ Starting the notebook server: ''​ipython notebook''​
  
-<note warning>​**Warning!** When you start the notebook server, you will start a python process and a web browser, and a new python process will be started each time you open a new notebook.+<note warning> 
 +  * **Warning!** When you start the notebook server, you will start a python process and a web browser, and a new python process will be started each time you open a new notebook.
  
-Please make sure that you shutdown cleanly each notebook and the server when you are finished, in order not to clutter the local computer or remote server with lots of python processes (especially ​server shared by many users!).+  * Please make sure that you **shutdown cleanly each notebook and the server when you are finished**, in order not to clutter the local computer or remote server with lots of python processes (especially ​an interactive ​server shared by many users!).
  
-You may want to [[other:​newppl:​starting#​determining_the_load_of_a_linux_server|use the '​top'​ command]] in order to monitor what is happening</​note>​+  * You may want to [[other:​newppl:​starting#​determining_the_load_of_a_linux_server|use the '​top'​ command]] in order to monitor what is happening ​on the interactive server where you are using your notebook server 
 +</​note>​
  
 /* standard page footer */ /* standard page footer */
other/python/starting.1556119713.txt.gz · Last modified: 2019/04/24 15:28 by jypeter