Working with Python

This page will tell you how to select a python distribution and start (and exit!) the python interpreterAfter reading this page, you can read the JYP's recommended steps for learning python for really working with python

Note: the former and outdated version (before the massive usage of conda) is still available for reference

Where to start

In order to start working with Python, you need to have a Python distribution installed on your local computer or on the remote Linux server(s) you work on. A distribution provides a Python interpreter, and Python extensions (aka Python modules or packages). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type which python on Linux to determine where the python executable is located)

If you are using a Linux computer or a Mac, you should already have a default python installed. The following example shows where the python interpreter is installed on the obelix LSCE servers (if it's in /usr/bin, it's the default python) and which version it is (example below: version 2.7.5 compiled in April 2019)

# Which is the current python (e.g. where is it located)?
> which python
/usr/bin/python
# Where is it coming from ('rpm' works on a RedHat-like Linux machine)?
> rpm -qf /usr/bin/python
python-2.7.5-77.el7_6.x86_64
# You also get some information when you start the interpreter
> python
Python 2.7.5 (default, Apr 9 2019, 14:30:50)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Ultra quick-start on the interactive LSCE servers

Initialization

Start a terminal on an obelix server (more info)

Type python to access the default Python 2 interpreter available on the servers...

...or follow the steps below to use the more complete CDAT distribution maintained by JYP.
Remember that you can exit the interpreter by typing CTRL-D (or quit() or exit())

Determine if you are using a bash or tcsh shell (all the new LSCE accounts use the bash shell), then type the following command to initialize conda (conda helps you manage your python environment)

In a bash shell: source ~jypeter/.conda3_jyp.sh

In a tcsh shell: source ~jypeter/.conda3_jyp.csh

Choose which flavor of Python you want by typing the activation command:

Python 2.7.x: conda activate cdatm_py2

Python 3.x: conda activate cdatm_py3

Type python to start the interpreter

Note: if you will have to use python regularly, you should add the source line above to your .cshrc /.login file (tcsh users) or .bashrc/.profile (bash users)

Do not add the conda activate cdatm_pyN line to the shell config files, because this will create potential side-effects!
Only type the activation command in the terminal(s) where you will need to use this specific version of Python

Example

>ssh obelix
Last login: Mon Jun 3 08:49:53 2019 from somewhere
bash-4.2$ which python
/usr/bin/python
bash-4.2$ source ~jypeter/.conda3_jyp.sh
bash-4.2$ which python
/usr/bin/python
bash-4.2$ conda activate cdatm_py2
(cdatm_py2) bash-4.2$ which python
/home/share/unix_files/cdat/miniconda3/envs/cdatm_py2/bin/python
(cdatm_py2) bash-4.2$ python
Python 2.7.15 | packaged by conda-forge | (default, Feb 28 2019, 04:00:11)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print('hello!')
hello!
>>> # Type CTRL-D to exit the python interpreter

Useful keyboard shortcuts

Key	Effect
CTRL-D	Exit the interpreter
↑ and ↓	Go to previous/next line(s)
CTRL-A	Go to the beginning of the line
CTRL-E	Go to the end of the line
CTRL-K	Erase from the cursor to the end of the line
CTRL-U	Erase from the beginning of the line to the cursor
TAB x 2	Do some TAB-completion (context dependent) e.g. a = np.cumTABTAB
CTRL-C	Interrupt a running script
CTRL-Z	Suspend the interpreter and go back to the shell Do not forget to go back to the interpreter with fg or to kill it (with jobs and kill %NN)

Executing a python script

Note: Python is an interpreted language and we speak of Python scripts rather than Python programs.

python: start the interpreter and start working interactively

python script.py: execute script.py and exit

python -i script: execute script.py and stay in the interpreter (-i = Interactive)

Type man python if you want to see what other command line options are available

Stand-alone script

You don't have to explicitly call the python interpreter, if the interpreter is specified in a #! shebang comment on the first line of the script with, and the script has its execution bit set (chmod +x my_script.py).

If you have a basic_script.py file with the following content in a directory...

#!/usr/bin/env python
import sys
script_name = sys.argv[0]
print('Hello world, I am the ' + script_name + ' script')
# The end

...and if you are in the same directory and have initialized the Python distribution you want, you can then run the script the following way:

> chmod +x basic_script.py
> ls -l basic_script.py
-rwxr-xr-x [...] basic_script.py
> ./basic_script.py
Hello world, I am the ./basic_script.py script

What next?

Read the rest of this page to get a better understanding of what you have learned in this section, and use Python more efficiently

Learn Python by reading what you need on the JYP's recommended steps for learning python page

Using the LSCE jupyterhub server

The LSCE jupyterhub notebook server is available for users with a LSCE Linux account:

from a computer on the LSCE wired network: https://jupyter.lsce.ipsl.fr/

from other computers: start the LSCE VPN before opening https://jupyter.lsce.ipsl.fr/

Using a non-standard kernel

Note: a kernel is a specific python environment/distribution

The LSCE jupyterhub server can be used to open Notebooks using pre-configured Python or R kernels. It is also possible to define additional per-user kernels by adding kernel configuration directories in the user's home directory

The steps below show how to add a specific CDAT environment kernel, but can be easily adapted for using other kernels

Initialize your (favorite) Python environment in a terminal, and (as usual) check which Python executable you are using

>which python
/usr/bin/python
>source ~jypeter/.conda3_21-02_jyp.csh
>conda activate cdatm_py3
(cdatm_py3) >which python
/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python

Determine where the kernel configuration files of the active python are located

>jupyter kernelspec list
Available kernels:
python3 /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/share/jupyter/kernels/python3

Copy the existing kernel directory that you have found to your home directory. You can use the existing kernel name, or use a new kernel name (using only ASCII letters, numbers, - hyphen, . period and _ underscore)

>cp -pr /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/share/jupyter/kernels/python3 ~/.local/share/jupyter/kernels
>cp -pr /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/share/jupyter/kernels/python3 ~/.local/share/jupyter/kernels/my_favorite_kernel
>ls ~/.local/share/jupyter/kernels/
my_favorite_kernel/ python3/

Check that the kernelspec list command now finds the new kernel(s)

>jupyter kernelspec list
Available kernels:
my_favorite_kernel your_home_dir/.local/share/jupyter/kernels/my_favorite_kernel
python3 your_home_dir/.local/share/jupyter/kernels/python3

You can optionally change the value of the display_name parameter in the kernel.json configuration file if you want to customize the name that will appear in the jupyterhub kernel menus

>ls your_home_dir/.local/share/jupyter/kernels/my_favorite_kernel
kernel.json logo-32x32.png logo-64x64.png
>cat your_home_dir/.local/share/jupyter/kernels/my_favorite_kernel/kernel.json
{
"argv": [
"/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "My favorite Python kernel",
"language": "python"

refresh the https://jupyter.lsce.ipsl.fr/ page and the new kernel(s) should appear in the kernels' related menus

Python distributions available for LSCE users

This section will help you choose a distribution in the big Python ecosystem (many distributions, python version 2.7.* or 3.*, ...) on some of the servers used by LSCE users. You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/duplicate the data!) and try to identify who is maintaining it, if you need help or additional packages.

Only install a distribution yourself if you need it on your local computer (desktop or laptop), or if you need to install some modules that can't be installed by the contacts listed below. A python distribution will require several Gb of disk space, so do not install it in your backed up home directory!

You can use either Python 2 or Python 3. Most packages are now available in both versions, but you should make sure that the most important package/s you need is/are available in the selected Python version. You can check the differences between both versions and try to write scripts that will work in both versions!

LSCE distribution

Contact: the LSCE system administrators (help-lsce@lsce.ipsl.fr)

Where: obelix interactive servers and obelix cluster at LSCE

Initialization type: module based + conda

What's installed: type conda list after initializing the LSCE distribution

$ module -t avail
[...]
python/2.7
python/2.7.5
python/2-ramces
python/3
python/3.9
python/4artic
python/esmpy
python/intel
[...]
$ module load python/3.9
$ which python
/usr/local/install/python-3.9/bin/python
$ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

CDAT

CDAT (Community Data Analysis Tools) is a powerful and complete front-end to a rich set of visual-data exploration and analysis capabilities well suited for climate data analysis problems. CDAT is recommended by JYP.

CDAT is available for Mac, Linux and Windows 10 (Windows 10 + Windows Subsystem for Linux, and Ubuntu).

Contact: Jean-Yves Peterschmitt (JYP) @ LSCE

Where: obelix interactive servers and cluster at LSCE, ciclad interactive servers and cluster at IPSL, irene @ TGCC

Initialization type: conda based

What's installed: type conda list after initializing a specific CDAT distribution

CDAT versions maintained by JYP

The following versions are maintained by JYP, on the Linux servers where LSCE users have accounts

CDAT version	JYP version	python version	Availability?	Available packages	Installation notes
8.2.1	19	3.8.8	LSCE	Default CDAT 8.2.1 packages Extra packages	8.2.1 notes
8.1	18	2.7.15 3.6.7	LSCE, ciclad	Default CDAT 8.1 packages Extra packages	8.1 notes

Initialization

Read the Ultra quick-start on the LSCE servers above

Note: on the ciclad cluster, use source ~jypmce/.conda3_jyp.sh to initialize conda

CDAT at TGCC

Note: TGCC also supports its own TGCC distribution, that may be more up-to-date, if you don't need specific CDAT modules

CDAT 8.0 is installed at TGCC and can be initialized with:

Python 2: module load cdat

Python 3: module load flavor/cdat/python3 cdat

A common CDAT-related error

Note: if you get an error when importing cdms2 or vcs it means that either you have forgotten to initialize CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed on your system, or another (non-CDAT) python distribution!

$ python -c 'import cdms2, vcs'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named cdms2
# Am I using the correct python distribution?
$ which python
/usr/bin/python

TGCC distribution

Contact: the TGCC hotline (hotline.tgcc@cea.fr)

Where: CEA TGCC

Initialization type: module based

What's installed: it depends...

# Get the default version of the Python based modules
$ module avail -t -d | egrep '(python|cdat)'
flavor/cdat/standard(default)
flavor/nest/python2(default)
flavor/pytorch/python2(default)
flavor/tensorflow/gpu_python2(default)
intelpython2/2019.0(default)
intelpython3/2019.0(default)
python/2.7.14(default)
python3/3.6.4(default)
cdat/8.0(default)
$ module load python
[...]
load module python/2.7.14 (Python)
$ which python
/ccc/products/python-2.7.14/intel--17.0.4.196__openmpi--2.0.2/default/bin/python
$ python
Python 2.7.14 (default, Jan 11 2018, 16:43:59)
[GCC 4.8.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Anaconda

This is the recommended (by JYP) Python distribution/environment. It is available for Windows, Mac and Linux

The Anaconda distribution is provided by the Anaconda company (that was previously called Continuum Analytics). It is similar to EDM, but probably more widely used than EDM

Note: Anaconda provides and uses conda for its installation. Since CDAT is also installed and maintained with conda, you can read the Installing and maintaining UV-CDAT with conda page for more information, even if you are not going to use CDAT

Enthought Deployment Manager (EDM)

This was previously called Enthought Python Distribution (EPD) and then Enthought Canopy

Enthought Deployment Manager (EDM) is Enthought’s mechanism to deliver scientific software applications and development environments

There are native installers for Windows (.msi), Mac OS X (.pkg), RHEL/Fedora (.rpm), and Debian/Ubuntu (.deb)

ipython

ipython interpreter

The ipython interpreter provides more options and commands than the standard python interpreter, but takes more time to start. If you are going to develop by starting and exiting the interpreter many times, it's faster to use the standard interpreter

Starting ipython: ipython

Using IPython for interactive work

Ipython-quick-ref-sheets

ipython notebook

Add a link to the new notebook page

Jupyter Notebook for Beginners: A Tutorial

The ipython notebook is a way to interact with python (and other supported interpreted languages) inside a web browser. You can mix cells with python commands, cells with the output of the python commands (possibly graphics generated by the commands), and text (using some wiki-like rich text format). This interactive web page, aka notebook, can be saved in a my_notebook.ipynb file and re-used later (e.g. the notebook provided in the introduction to Python, part 1).

Starting the notebook server: ipython notebook

Warning! When you start the notebook server, you will start a python process and a web browser, and a new python process will be started each time you open a new notebook.Please make sure that you shutdown cleanly each notebook and the server when you are finished, in order not to clutter the local computer or remote server with lots of python processes (especially a server shared by many users!). You may want to use the 'top' command in order to monitor what is happening

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]