User Tools

Site Tools


other:python:starting_170810

Working with Python

This page will tell you how to select a python distribution and start (and exit!) the python interpreter

You can then read the JYP's recommended steps for learning python for really working with python

Where to start

In order to start working with Python, you need to have a Python distribution installed on your local computer or on the remote Linux server(s) you work on. A distribution provides a Python interpreter, and Python extensions (aka Python modules or packages). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type which python on Linux to determine where the python executable is located)

If you are using a Linux computer or a Mac, you should already have a default python installed. The following example shows where the python interpreter is installed on the asterix LSCE server (if it's in /usr/bin, it's the default python) and which version it is (example below: version 2.6.6 compiled in May 2015)

# Which is the current python (e.g. where is it located)?
 > which python
/usr/bin/python

# Where is it coming from ('rpm' works on a RedHat-like Linux machine)?
 > rpm -qf /usr/bin/python
python-2.6.6-64.el6.x86_64

# You also get some information when you start the interpreter
 > python
Python 2.6.6 (r266:84292, May 22 2015, 08:34:51)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Python distributions available for LSCE users

This section will help you choose a distribution in the big Python ecosystem (many distributions, python version 2.7.* or 3.*, …). You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/duplicate the data!) and try to identify who is maintaining it, if you need help or extra packages.

Only install a distribution yourself if you need it on your local computer, or if you need to install some modules that can't be installed by the contacts listed below. A python distribution can require several Gb of disk space, so do not install it in a backed up home directory!

We suggest that you use Python 2 rather than Python 3. Or you can check the differences between both versions and try to write scripts that will work in both versions! 8-)

LSCE distribution

Contact: the LSCE system administrators (help-lsce@lsce.ipsl.fr)

 > module avail
[...]
castem/12        ferret/6.9.5     hdf5/1.8.9       netcdf/4p        python/2.7.5     sun-java/7.0.45
[...]

 > module load python/2.7.5

 > which python
/usr/local/install/python-2.7.5/bin/python

 > python
Python 2.7.5 (default, Sep 18 2013, 15:47:43)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

TGCC distribution

Contact: the TGCC hotline (hotline.tgcc@cea.fr)

$ module avail
[...]
cmake/2.8.9(default)      hwloc/1.5                 netcdf-utils/4.3.3.1_hdf5 python/2.7.8              tix/8.4.3
cmake/3.2.2               hwloc/1.7.1(default)      octave/3.6.3              python/3.3.2              tk/8.5(default)
[...]

$ module load python/2.7.8
load module python/2.7.8 (Python)

$ which python
/usr/local/ccc_python/2.7.8_201409/bin/python

$ python
Python 2.7.8 (default, Aug 27 2014, 17:50:16)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

UV-CDAT

Contact: Jean-Yves Peterschmitt (JYP) @ LSCE

What is UV-CDAT?

UV-CDAT (Ultrascale Visualization - Climate Data Analysis Tools) is a python distribution developed specifically for the climate scientists, and is recommended by JYP.

UV-CDAT is available for Mac and Linux.

You can find out which standard packages are installed in the Packages built by CDAT column below. UV-CDAT also provides some specific packages that you will not find in other python distributions: cdms2, cdutil, cdtime, genutil, vcs, cmor, …

Extra packages are listed:

UV-CDAT versions maintained by JYP

The following versions are maintained by JYP, on the Linux servers where LSCE users have accounts

CDAT version JYP
version
python
version
Availability? Packages built by CDAT Installation notes
1.1.0 08 2.7.3 LSCE, curie n/a 1.1.0 notes
1.5.1 10 2.7.4 LSCE, ciclad cdat-1.5.1_jyp-10_build_info.txt 1.5.1 notes
2.1.0 13 2.7.8 LSCE, ciclad cdat-2.1.0_jyp-13_build_info.txt 2.1.0 notes
2.4.0 Not installed
2.8.0 14 2.7.12 LSCE uvcdat-2.8.0_install_dry_with-forge_160221.txt 2.8.0 notes

Notes:

  • For the conda based CDAT versions (>=2.8.0), you can get the list of currently available packages by typing: conda list
  • For older CDAT versions (⇐2.1.0), the cdat-VVV_jyp-NNN_build_info.txt files have been generated with
    cd /my_build_directory_path; cat build_info.txt | sort > cdat-VVV_JYP-NNN_build_info.txt

The basics of UV-CDAT initialization

The way you initialize UV-CDAT depends on:

  • which server you want to use it on
  • which shell you are using in your terminals: tcsh or other shells
    • Read Which shell are you using? if you are not sure
    • Warning: the shell in a (batch) script may be different from your interactive shell!
  • which version of UV-CDAT you want to use…
    • Note: You should use the most recent version of UV-CDAT available on a given server, unless you have a good reason to use an older version (e.g. if you need to use the old-style vcs of version 1.5.1).

Note: if you get an error when importing cdms2 or vcs it means that either you have forgotten to initialize UV-CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed on your system, or another python distribution!

$ python -c 'import cdms2, vcs'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named cdms2

# Am I using the correct python distribution?
$ which python
/usr/bin/python

conda-based versions of UV-CDAT


The steps detailed in this section are for UV-CDAT 2.8.0 and later

Note: if you need to maintain a local version on your machine, you can read Installing and maintaining UV-CDAT with conda and the versions specific notes.

Configuring .bashrc

You should add the following line at the end of your ~/.bashrc configuration file (create the ~/.bashrc file if it does not exist yet). This will basically make sure that your shell can find the conda executable it needs for initializing CDAT, and define a wp alias that you can use to find where the python executable is located

Server Line to add to .bashrc
LSCE source ~jypeter/.conda_jyp.sh

Note:conda_jyp content
ciclad source ~jypmce/.conda_jyp.sh

Note: when you add the specified line to your bashrc file, you will then always get the python supplied by conda, instead of the system's python, when you use bash. If you don't like this permanent behavior, just type source ~jyp_login_on_this_server/.conda_jyp.sh in the bash shell where you want to use conda+CDAT, rather than adding it to the .bashrc file…

Initializing UV-CDAT

When you initialize UV-CDAT, it will be available only in the current window/shell. In the other windows, you still get whatever your default python is. This should minimize potential side-effects

  1. If your shell is not bash, start a bash shell by typing bash
    • You can later go back to your previous shell by typing exit or CTRL-D
  2. Activate the conda environment that uses the version of UV-CDAT you want to use
    • source activate conda_env_name
    • Use the table below to determine which environments are available on which server
  3. Use CDAT !
UV-CDAT version Server Environment name
and activation line
Packages
(conda list)
2.8.0 LSCE
ciclad
cdatm14 (source activate cdatm14) cdatm14 installed packages
2.10 LSCE cdatm15 (source activate cdatm15) cdatm14 installed packages
cdatm will always point to the latest stable version
latest stable version LSCE cdatm (source activate cdatm) conda list

Older versions of UV-CDAT

The steps detailed in this section are for UV-CDAT 2.1.0 and earlier

Initializing UV-CDAT for tcsh users

If you use tcsh as your login shell, you can define a cdatm alias on all the IPSL servers on which UV-CDAT is installed (see details below), and move easily from one version of UV-CDAT to another:

  • type cdatm to initialize the default version of UV-CDAT
  • type cdatm --config CONFIG_NAME to initialize a specific version
    • You can find the allowed values of CONFIG_NAME (e.g. LSCE_10, LSCE_13, …) in one of the tables below
 > which python
/usr/bin/python

 > cdatm
** CDAT 'uv-1.5.1' initialized!
** If you have problems with this installation,
** it is maintained by: Jean-Yves Peterschmitt - LSCE

 > which python
/home/share/unix_files/cdat/versions/cdat_install_uv-1.5.1_x86_64_gcc4_10/bin/python

You need to add the following line to the .cshrc file in your home directory, in order to define the cdatm alias. You can also define the simple wp alias, in order to always be able to determine quickly which python is initialized in the current terminal.

LSCE alias cdatm 'source ~jypeter/CDAT/Install/cdat_multi/cdat_multi.login ~jypeter/CDAT/Install/cdat_multi'
ciclad alias cdatm 'source ~jypmce/CDAT/Install/cdat_multi/cdat_multi.login ~jypmce/CDAT/Install/cdat_multi'
curie alias cdatm 'source ~p25jype/CDAT/Install/cdat_multi/cdat_multi.login ~p25jype/CDAT/Install/cdat_multi'
All servers alias wp 'which python
Initializing UV-CDAT in any kind of shell, or in a batch script

You first need to know where the python program provided by UV-CDAT is installed, something like /path/to/UV-CDAT/bin/python

Then, all you need to do in order to initialize UV-CDAT is to source the setup_runtime file appropriate for your shell:

  • tcsh: type source /path/to/UV-CDAT/bin/setup_runtime.csh
  • bash, sh/ksh: type source /path/to/UV-CDAT/bin/setup_runtime.sh

The table below lists, for each server and version, the path where you can find the setup_runtime files

Server CDAT
version
JYP
version
Path
LSCE 1.1.0 LSCE_08 /home/share/unix_files/cdat/versions/cdat_install_uv-1.1.0_x86_64_gcc4_08/bin
LSCE 1.5.1 LSCE_10 /home/share/unix_files/cdat/versions/cdat_install_uv-1.5.1_x86_64_gcc4_10/bin
LSCE 2.1.0 LSCE_13 /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin
ciclad 1.5.1 ciclad_10-ng /data/jypmce/cdat/versions/cdat_install_uv-1.5.1_x86_64_gcc4_10-ng/bin
ciclad 2.1.0 ciclad_13 /data/jypmce/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin
curie 1.1.0 curie_08 /ccc/work/cont003/dsm/p25jype/cdat/versions/cdat_install_uv-1.1.0_x86_64_gcc4_curie_08/bin
On curie, you need to use setup_cdat.sh or setup_cdat.csh to initialize CDAT,
and also to load modules providing access to old librairies
module load gnu/4.6.3 and module load qt/4.8.6

Example: if you want to use UV-CDAT 2.1.0 at LSCE in a bash shell, you need to do the following (you only need the source line. The other lines are optional, just for checking that the initialization was OK)

bash-4.1$ source /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.sh

bash-4.1$ which python
/home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/python

bash-4.1$ python -c 'import cdms2, vcs'
bash-4.1$

You probably don't want to have to determine what you should type each time you want to use UV-CDAT in a window, so you should define a simple shortcut alias that you can type each time you need to initialize UV-CDAT. If, for example, you want to use UV-CDAT 2.1.0 at LSCE:

  • tcsh: add the following alias definitions to ~/.cshrc
    alias cdat13 'source /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.csh'
    alias wp 'which python'
  • bash: add the following alias definition to ~/.bashrc, ~/.profile or ~/.bash_profile (whichever works for you, or read man bash…)
    alias cdat13='source /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.sh'
    alias wp='which python'

If you are going to use a UV-CDAT python script on a cluster or supercomputer, using some kind of batch system using a bash or ksh script, use the following lines before you run the python script

source /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.sh

# Check which python we will be using
mypython=`which python`
echo
echo -e "This script will use the following python: $mypython\n\n"

Canopy

Enthought Canopy is a Scientific and Analytic Python Deployment with Integrated Analysis Environment provided by Enthought. It used to be called EPD (Enthought Python Distribution)

Canopy is available for Windows, Mac and Linux. You can download the free Canopy Express that will already provide many extensions. If you are entitled to use Canopy Academic, login from inside Canopy Express and download the extra modules you need

Anaconda

Anaconda is a distribution similar to canopy provided by Continuum Analytics.

Anaconda is available for Windows, Mac and Linux

Note: Anaconda provides and uses conda for its installation. Since UV-CDAT is now also installed and maintained with conda, you can read the Installing and maintaining UV-CDAT with conda page for more information, even if you are not going to use UV-CDAT

Launching Python

Once you have initialized the python distribution you want to use, follow the instructions below to start the interpreter from a shell (on a Windows computer, you will have to start python from the Start menu or by clicking a shortcut on your desktop).

Remember that on Linux/Mac you can check which python you are using by typing which python!

Useful keyboard shortcuts

Key Effect
CTRL-D Exit the interpreter
↑ and ↓ Go to previous/next line(s)
CTRL-A Go to the beginning of the line
CTRL-E Go to the end of the line
CTRL-K Erase from the cursor to the end of the line
CTRL-U Erase from the beginning of the line to the cursor
TAB x 2 Do some TAB-completion (context dependent)
e.g. a = np.cumTABTAB
CTRL-C Interrupt a running script
CTRL-Z Suspend the interpreter and go back to the shell
Do not forget to go back to the interpreter with fg
or to kill it (with jobs and kill %NN)

Stand-alone script

A python script is just like any other shell script. You don't have to explicitly call the python interpreter, if the interpreter is specified in a #! shebang comment on the first line of the script with, and the script has its execution bit set (chmod +x my_script.py).

jypeter@asterix1 - ...jypeter - 54 >cat basic_script.py
#!/usr/bin/env python

import sys

script_name = sys.argv[0]

print('Hello world, I am the ' + script_name + ' script')

# The end

jypeter@asterix1 - ...jypeter - 55 >chmod +x basic_script.py

ypeter@asterix1 - ...jypeter - 56 ># Type here what is required to initialize the distribution you want

jypeter@asterix1 - ...jypeter - 57 >./basic_script.py
Hello world, I am the ./basic_script.py script

Standard interpreter

  • python: start the interpreter
  • python script.py: execute script.py and exit
  • python -i script: execute script.py and stay in the interpreter (-i = Interactive)

Type man python if you want to see what other command line options are available

ipython interpreter

The ipython interpreter provides more options and commands than the standard python interpreter, but takes more time to load. If you are going to develop by starting and exiting the interpreter many times, it's faster to use the standard interpreter

Starting ipython: ipython

ipython notebook

The ipython notebook is a way to interact with python (and other supported interpreted languages) inside a web browser. You can mix cells with python commands, cells with the output of the python commands (possibly graphics generated by the commands), and text (using some wiki-like rich text format). This interactive web page, aka notebook, can be saved in a my_notebook.ipynb file and re-used later (e.g. the notebook provided in the introduction to Python, part 1).

Starting the notebook server: ipython notebook

Warning! When you start the notebook server, you will start a python process and a web browser, and a new python process will be started each time you open a new notebook.

Please make sure that you shutdown cleanly each notebook and the server when you are finished, in order not to clutter the local computer or remote server with lots of python processes (especially a server shared by many users!).

You may want to use the 'top' command in order to monitor what is happening





[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]

other/python/starting_170810.txt · Last modified: 2019/04/25 07:16 by jypeter