PMIP3 wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Working with Python ======

<note tip>This page will tell you how to select a //python distribution// and start (and exit!) the python interpreter

After reading this page, you can read the [[jyp_steps|JYP's recommended steps for learning python]] for really working with python
</note>

Note: the former and **outdated** version (before the massive usage of //conda//) is still [[other:python:starting_170810|available for reference]]

===== Where to start =====

In order to start working with Python, you need to have a //Python distribution// installed on your local computer or on the remote Linux server(s) you work on. A //distribution// provides a //Python interpreter//, and Python extensions (aka Python //modules// or //packages//). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type ''which python'' on Linux to determine where the python executable is located)

If you are using a Linux computer or a Mac, you should already have a **default python** installed. The following example shows where the python interpreter is installed on the //obelix// LSCE servers (if it's in ''/usr/bin'', it's the //default// python) and which version it is (example below: version //2.7.5// compiled in April 2019)

<code># Which is the current python (e.g. where is it located)?
 > which python
/usr/bin/python

# Where is it coming from ('rpm' works on a RedHat-like Linux machine)?
 > rpm -qf /usr/bin/python
python-2.7.5-77.el7_6.x86_64

# You also get some information when you start the interpreter
 > python
Python 2.7.5 (default, Apr  9 2019, 14:30:50)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
</code>

===== Ultra quick-start on the interactive LSCE servers =====

==== Initialization ====

  - Start a terminal on an //obelix// server ([[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:newppl:starting#which_linux_servers_should_you_use|more info]])
  - Type ''python'' to access the //default Python 2// interpreter available on the servers...
  - ...or **follow the steps below to use the more complete //CDAT// distribution maintained by JYP**.\\ Remember that **you can exit the interpreter by typing CTRL-D** (or ''quit()'' or ''exit()'')
    - Determine if you are using a [[other:newppl:starting#which_shell_are_you_using|bash or tcsh shell]] (all the new LSCE accounts use the //bash// shell), then type the following command to //initialize conda// (//conda// helps you manage your python environment)
      * In a //bash// shell: ''source ~jypeter/.conda3_jyp.sh''
      * In a //tcsh// shell: ''source ~jypeter/.conda3_jyp.csh''
    - Choose which environment you want by typing the activation command:
      * Python 3.x: ''conda activate cdatm_py3''
      * You can use ''conda env list'' to find out which environments are available, but the one you most likely want is ''cdatm_py3''
    - If you need an (older) Python 2.7 environment, replace the steps above with:
      * ''source ~jypeter/.conda3_19-06_jyp.sh''
      * ''conda activate cdatm_py2''
    - Type ''python'' to start the interpreter
    - If you need a list of the installed packages (and their version), use
      * ''conda list''
      * A partial list of what is installed is available in the [[other:uvcdat:cdat_conda:cdat_8_2_1#extra_packages_list|Extra packages list]]
    - Note: if you will have to use python regularly, you should add the ''source'' line above to your ''.cshrc'' /''.login'' file (//tcsh// users) or ''.bashrc''/''.profile'' (//bash// users)
      * **Do not** add the ''conda activate cdatm_pyN'' line to the shell config files, because this will create potential side-effects!\\ Only type the activation command in the terminal(s) where you will need to use this specific version of Python

=== Example ===

<code> >ssh obelix
Last login: Wed Feb  1 09:56:29 2023 from somewhere

bash-4.2$ which python
/usr/bin/python

bash-4.2$ source ~jypeter/.conda3_jyp.sh

bash-4.2$ which python
/usr/bin/python

bash-4.2$ conda activate cdatm_py3

(cdatm_py3) bash-4.2$ which python
/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python

(cdatm_py3) bash-4.2$ python
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print('hello!')
hello!
>>>   # Type CTRL-D to exit the python interpreter
</code>

==== Useful keyboard shortcuts ====

^  Key  ^  Effect  ^
|  **CTRL-D**  |  Exit the interpreter  |
|  **↑ and ↓**  |  Go to previous/next line(s)  |
|  **CTRL-A**  |  Go to the beginning of the line  |
|  **CTRL-E**  |  Go to the end of the line  |
|  **CTRL-K**  |  Erase from the cursor to the end of the line  |
|  **CTRL-U**  |  Erase from the beginning of the line to the cursor  |
|  **TAB** x 2  |  Do some TAB-completion (context dependent)\\ e.g. ''a = np.cum**TABTAB**''  |
|  **CTRL-C**  |  Interrupt a running script  |
|  **CTRL-Z**  |  Suspend the interpreter and go back to the shell\\ **Do not forget** to go back to the interpreter with ''fg''\\ or to kill it (with ''jobs'' and ''kill %NN'')  |

==== Executing a python script ====

Note: Python is an [[https://en.wikipedia.org/wiki/Interpreted_language|interpreted language]] and we speak of //Python scripts// rather than //Python programs//.

  * ''python'': start the interpreter and start working interactively
  * ''python script.py'': execute //script.py// and **exit**
  * ''python -i script'': execute //script.py// and **stay in the interpreter** (''-i'' = //**I**nteractive//)

Type ''man python'' if you want to see what other command line options are available

==== Stand-alone script ====

You don't have to explicitly call the python interpreter, if the interpreter is specified in a ''#!'' [[https://en.wikipedia.org/wiki/Shebang_%28Unix%29|shebang comment]] on **the first line of the script** with, and the script has its //execution bit// set (''chmod +x my_script.py''). 

If you have a ''basic_script.py'' file with the following content in a directory...
<code>
#!/usr/bin/env python

import sys

script_name = sys.argv[0]

print('Hello world, I am the ' + script_name + ' script')

# The end</code>

...and if you are in the same directory and have initialized the Python distribution you want, you can then run the script the following way:

<code> > chmod +x basic_script.py

 > ls -l basic_script.py
-rwxr-xr-x [...] basic_script.py

 > ./basic_script.py
Hello world, I am the ./basic_script.py script
</code>

==== What next? ====

  - Read the rest of this page to get a better understanding of what you have learned in this section, and use Python more efficiently
  - Learn Python by reading what you need on the [[other:python:jyp_steps|JYP's recommended steps for learning python]] page


===== Using JupyterHub servers =====

==== JupyterHub servers available for LSCE users ====

  * [[https://intranet.lsce.ipsl.fr/informatique/en/jupyter/index.php|LSCE JupyterHub server information]]
  * [[https://documentations.ipsl.fr/spirit/jupyterhub/documentation.html|IPSL JupyterHub server information]]
  * You will **have to** be on the LSCE wired network, or use the [[https://intranet.lsce.ipsl.fr/informatique/en/intra-cea.php|LSCE VPN]], if you want to use the LSCE or IPSL JupyterHub servers !
    * Otherwise, check the IPSL server documentation, if you want to try **accessing the IPSL server using an ssh tunnel**...

==== Notebooks and jupyter servers documentation ====

FIXME


==== Using a non-standard kernel ====

Note: a //kernel// is a specific python environment/distribution

The LSCE jupyterhub server can be used to open //Notebooks// using pre-configured Python or R //kernels//. It is also possible to define additional per-user kernels by adding [[https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs|kernel configuration directories]] in the user's home directory

The steps below show how to add a specific [[other:python:starting#cdat|CDAT environment]] kernel, but can be easily adapted for using other kernels

  - Initialize your (favorite) Python environment in a terminal, and (as usual) check which Python executable you are using\\ <code> >which python
/usr/bin/python

 >source ~jypeter/.conda3_21-02_jyp.csh
 >conda activate cdatm_py3

(cdatm_py3)  >which python
/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python</code>
  - Determine where the kernel configuration files of the **active** python are located\\ <code> >jupyter kernelspec list
Available kernels:
  python3     /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/share/jupyter/kernels/python3</code>
  - Copy the existing kernel directory that you have found to your home directory (more precisely to the ''~/.local/share/jupyter/kernels'' directory). You can use the existing kernel name (probably ''python3''), or use a new kernel name (using only ASCII letters, numbers,  ''-'' hyphen, ''.'' period and ''_'' underscore)\\ <code> >mkdir -p ~/.local/share/jupyter/kernels
 >cp -pr /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/share/jupyter/kernels/python3 ~/.local/share/jupyter/kernels
 >cp -pr /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/share/jupyter/kernels/python3 ~/.local/share/jupyter/kernels/my_favorite_kernel

 >ls ~/.local/share/jupyter/kernels/
my_favorite_kernel/    python3/
</code>
  - Check that the ''kernelspec list'' command now finds the new kernel(s)\\ <code> >jupyter kernelspec list
Available kernels:
  my_favorite_kernel    your_home_dir/.local/share/jupyter/kernels/my_favorite_kernel
  python3               your_home_dir/.local/share/jupyter/kernels/python3</code>
  - You can optionally change the value of the ''display_name'' parameter in the ''kernel.json'' configuration file if you want to customize the name that will appear in the jupyterhub kernel menus\\ <code> >ls your_home_dir/.local/share/jupyter/kernels/my_favorite_kernel
kernel.json  logo-32x32.png  logo-64x64.png

 >cat ~/.local/share/jupyter/kernels/my_favorite_kernel/kernel.json
{
 "argv": [
  "/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "My favorite Python kernel",
 "language": "python"
}
</code>
  - Refresh the ''https://jupyter.lsce.ipsl.fr/'' (if you are using the LSCE notebook server) page and the new kernel(s) should appear in the kernels' related menus
===== Python distributions available for LSCE users =====

This section will help you choose a distribution in the big Python ecosystem (many distributions, python version 2.7.* or 3.*, ...) on some of the servers used by LSCE users. You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/duplicate the data!) and try to identify who is maintaining it, if you need help or additional packages.

Only install a distribution yourself if you need it on your local computer (desktop or laptop), or if you need to install some modules that can't be installed by the contacts listed below. A python distribution will require several Gb of disk space, so do not install it in your backed up //home// directory!

You can use either //Python 2// or //Python 3//. Most packages are now available in both versions, but you should make sure that the most important package/s you need is/are available in the selected Python version. You can check the [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:jyp_steps#python_27_vs_python_3|differences between both versions]] and try to write scripts that will work in both versions!

==== LSCE distribution ====

  * Contact: the [[other:newppl:starting#getting_help_from_the_lsce_system_administrators|LSCE system administrators]]
  * Where: //obelix// interactive servers and //obelix// cluster at LSCE
  * Initialization type: [[other:newppl:starting#using_module_to_access_optional_programs|module based]] + //conda//
  * What's installed: type ''conda list'' after initializing on of the LSCE distributions

<code>
$ module avail python
python/2.7      python/2-ramces python/3.9      python/intel
python/2.7.5    python/3        python/4artic

$ module load python/3.9

$ which python
/usr/local/install/python-3.9/bin/python

$ python
Python 3.9.15 (main, Nov 24 2022, 14:31:59)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
</code>

==== CDAT ====

[[https://cdat.llnl.gov/|CDAT]] (//Community Data Analysis Tools//) is a powerful and complete front-end to a rich set of visual-data exploration and analysis capabilities well suited for **climate** data analysis problems. CDAT is **recommended by JYP**.

CDAT is **available for Mac, Linux and Windows 10 (Windows 10 + [[other:win10wsl|Windows Subsystem for Linux, and Ubuntu]])**.

  * Contact: Jean-Yves Peterschmitt (//JYP//) @ LSCE
  * Where: //obelix// interactive servers and cluster at LSCE, //ciclad// interactive servers and cluster at IPSL, irene @ TGCC
  * Initialization type: conda based
  * What's installed: type ''conda list'' after initializing a specific CDAT distribution

=== CDAT versions maintained by JYP ===

The following versions are maintained by JYP, on the Linux servers where LSCE users have accounts

^  CDAT version  ^  JYP\\ version  ^  python\\ version  ^  Availability?  ^  Available packages ^  Installation notes  ^
|  **8.2.1**  |  19  |  3.8.8  |  LSCE  |  {{  :other:uvcdat:cdat_conda:cdat_8.2.1_nompi_21-03-05.txt |Default CDAT 8.2.1 packages}}\\ [[other:uvcdat:cdat_conda:cdat_8_2_1#extra_packages_list|Extra packages]]  |  [[other:uvcdat:cdat_conda:cdat_8_2_1|8.2.1 notes]]  |
|  **8.1**  |  18  |  2.7.15\\ 3.6.7  |  LSCE, ciclad  |  {{:other:uvcdat:cdat_conda:cdat-8.1_py2_list_190307.txt|Default CDAT 8.1 packages}}\\ [[other:uvcdat:cdat_conda:cdat_8_1#extra_packages_list|Extra packages]]  |  [[other:uvcdat:cdat_conda:cdat_8_1|8.1 notes]]  |

== Initialization ==

Read the [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#ultra_quick-start_on_the_interactive_lsce_servers|Ultra quick-start on the LSCE servers]] above

Note: on the ciclad cluster, use ''source ~jypmce/.conda3_jyp.sh'' to initialize conda

=== CDAT at TGCC ===

Note: TGCC also supports its own [[#tgcc_distribution|TGCC distribution]], that may be more up-to-date, if you don't need specific CDAT modules

CDAT **8.0** is installed at TGCC and can be initialized with:
  * Python 2: ''module load cdat''
  * Python 3: ''module load flavor/cdat/python3 cdat''

=== A common CDAT-related error ===

Note: if you get an error when importing //cdms2// or //vcs// it means that either you have forgotten to initialize CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed on your system, or another (non-CDAT) python distribution!

<code>$ python -c 'import cdms2, vcs'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named cdms2

# Am I using the correct python distribution?
$ which python
/usr/bin/python</code>

==== TGCC distribution ====

  * Contact: the TGCC hotline ([[hotline.tgcc@cea.fr]])
  * Where: CEA TGCC
  * Initialization type: [[other:newppl:starting#using_module_to_access_optional_programs|module based]]
  * What's installed: it depends...

<code># Get the default version of the Python based modules
$ module avail -t -d | egrep '(python|cdat)'
flavor/cdat/standard(default)
flavor/nest/python2(default)
flavor/pytorch/python2(default)
flavor/tensorflow/gpu_python2(default)
intelpython2/2019.0(default)
intelpython3/2019.0(default)
python/2.7.14(default)
python3/3.6.4(default)
cdat/8.0(default)

$ module load python
[...]
load module python/2.7.14 (Python)

$ which python
/ccc/products/python-2.7.14/intel--17.0.4.196__openmpi--2.0.2/default/bin/python

$ python
Python 2.7.14 (default, Jan 11 2018, 16:43:59)
[GCC 4.8.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
</code>


===== ipython =====

==== ipython interpreter ====

The //ipython// interpreter provides more options and commands than the standard python interpreter, but takes more time to start. If you are going to develop by starting and exiting the interpreter many times, it's faster to use the standard interpreter

Starting ipython: ''ipython''

  * [[http://ipython.readthedocs.org/en/stable/interactive/tutorial.html|Using IPython for interactive work]]
  * [[https://damontallen.github.io/IPython-quick-ref-sheets/|Ipython-quick-ref-sheets]]


==== LSCE jupyter notebook server ====

The different Python distributions available on the LSCE servers (interactive servers and cluster) can also be used on the [[https://intranet.lsce.ipsl.fr/informatique/en/jupyter/index.php|LSCE jupyter notebook server]]:
  * If your computer is on the LSCE wired network
  * Or if you are using the [[https://intranet.lsce.ipsl.fr/informatique/fr/vpn-forticlient.php|LSCE VPN]]
==== ipython notebook ====

<WRAP center round tip 60%>
If you want to use ipython notebooks at LSCE, you should **use the [[other:python:starting#lsce_jupyter_notebook_server|LSCE notebook server]]** rather than trying to use (and probably overloading) the interactive servers!
</WRAP>


<note warning>FIXME Add a link to the new notebook page</note>

[[https://www.dataquest.io/blog/jupyter-notebook-tutorial/|Jupyter Notebook for Beginners: A Tutorial]]

The **ipython notebook is a way to interact with python (and other supported interpreted languages) inside a web browser**. You can mix //cells// with python commands, cells with the output of the python commands (possibly graphics generated by the commands), and text (using some wiki-like rich text format). This interactive web page, aka notebook, can be saved in a ''my_notebook.ipynb'' file and re-used later (e.g. the notebook provided in the [[other:python:jyp_steps#part_1|introduction to Python, part 1]]).

Starting the notebook server: ''ipython notebook''

<note warning>
  * **Warning!** When you start the notebook server, you will start a python process and a web browser, and a new python process will be started each time you open a new notebook.

  * Please make sure that you **shutdown cleanly each notebook and the server when you are finished**, in order not to clutter the local computer or remote server with lots of python processes (especially an interactive server shared by many users!).

  * You may want to [[other:newppl:starting#determining_the_load_of_a_linux_server|use the 'top' command]] in order to monitor what is happening on the interactive server where you are using your notebook server
</note>

/* standard page footer */

\\ \\ \\ 
----
[ [[pmip3:|PMIP3 Wiki Home]] ] -
[ [[pmip3:wiki_help|Help!]] ] -
[ [[wiki:syntax|Wiki syntax]] ]