This is an old revision of the document!
After reading this page, you can read the JYP's recommended steps for learning python for really working with python
Note: the former and outdated version (before the massive usage of conda) is still available for reference
In order to start working with Python, you need to have a Python distribution installed on your local computer or on the remote Linux server(s) you work on. A distribution provides a Python interpreter, and Python extensions (aka Python modules or packages). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type which python
on Linux to determine where the python executable is located)
If you are using a Linux computer or a Mac, you should already have a default python installed. The following example shows where the python interpreter is installed on the obelix LSCE servers (if it's in /usr/bin
, it's the default python) and which version it is (example below: version 2.7.5 compiled in April 2019)
# Which is the current python (e.g. where is it located)? > which python /usr/bin/python # Where is it coming from ('rpm' works on a RedHat-like Linux machine)? > rpm -qf /usr/bin/python python-2.7.5-77.el7_6.x86_64 # You also get some information when you start the interpreter > python Python 2.7.5 (default, Apr 9 2019, 14:30:50) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>
python
to access the default Python 2 interpreter available on the servers…quit()
or exit()
)source ~jypeter/.conda3_jyp.sh
source ~jypeter/.conda3_jyp.csh
conda activate cdatm_py2
conda activate cdatm_py3
python
to start the interpretersource
line above to your .cshrc
/.login
file (tcsh users) or .bashrc
/.profile
(bash users)conda activate cdatm_pyN
line to the shell config files, because this will create potential side-effects!>ssh obelix Last login: Mon Jun 3 08:49:53 2019 from somewhere bash-4.2$ which python /usr/bin/python bash-4.2$ source ~jypeter/.conda3_jyp.sh bash-4.2$ which python /usr/bin/python bash-4.2$ conda activate cdatm_py2 (cdatm_py2) bash-4.2$ which python /home/share/unix_files/cdat/miniconda3/envs/cdatm_py2/bin/python (cdatm_py2) bash-4.2$ python Python 2.7.15 | packaged by conda-forge | (default, Feb 28 2019, 04:00:11) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> print('hello!') hello! >>> # Type CTRL-D to exit the python interpreter
Key | Effect |
---|---|
CTRL-D | Exit the interpreter |
↑ and ↓ | Go to previous/next line(s) |
CTRL-A | Go to the beginning of the line |
CTRL-E | Go to the end of the line |
CTRL-K | Erase from the cursor to the end of the line |
CTRL-U | Erase from the beginning of the line to the cursor |
TAB x 2 | Do some TAB-completion (context dependent) e.g. a = np.cumTABTAB |
CTRL-C | Interrupt a running script |
CTRL-Z | Suspend the interpreter and go back to the shell Do not forget to go back to the interpreter with fg or to kill it (with jobs and kill %NN ) |
Note: Python is an interpreted language and we speak of Python scripts rather than Python programs.
python
: start the interpreter and start working interactivelypython script.py
: execute script.py and exitpython -i script
: execute script.py and stay in the interpreter (-i
= Interactive)
Type man python
if you want to see what other command line options are available
You don't have to explicitly call the python interpreter, if the interpreter is specified in a #!
shebang comment on the first line of the script with, and the script has its execution bit set (chmod +x my_script.py
).
If you have a basic_script.py
file with the following content in a directory…
#!/usr/bin/env python import sys script_name = sys.argv[0] print('Hello world, I am the ' + script_name + ' script') # The end
…and if you are in the same directory and have initialized the Python distribution you want, you can then run the script the following way:
> chmod +x basic_script.py > ls -l basic_script.py -rwxr-xr-x [...] basic_script.py > ./basic_script.py Hello world, I am the ./basic_script.py script
This section will help you choose a distribution in the big Python ecosystem (many distributions, python version 2.7.* or 3.*, …) on some of the servers used by LSCE users. You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/duplicate the data!) and try to identify who is maintaining it, if you need help or additional packages.
Only install a distribution yourself if you need it on your local computer (desktop or laptop), or if you need to install some modules that can't be installed by the contacts listed below. A python distribution will require several Gb of disk space, so do not install it in your backed up home directory!
You can use either Python 2 or Python 3. Most packages are now available in both versions, but you should make sure that the most important package/s you need is/are available in the selected Python version. You can check the differences between both versions and try to write scripts that will work in both versions!
conda list
after initializing the LSCE distribution> module avail [...] 4ARTIC/3.6 grib_api/1.14 netcdf/3 python/2.7.5 batch_env grib_api/1.14.0 netcdf/4 python/3.6 [...] glost/0.3.1 ncview/2.1.7 python/2.7 [...] > module load python/2.7 > which python /usr/local/install/python-2.7/bin/python > python Python 2.7.15 |Anaconda, Inc.| (default, Oct 10 2018, 21:32:13) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>
CDAT (Community Data Analysis Tools) is a powerful and complete front-end to a rich set of visual-data exploration and analysis capabilities well suited for climate data analysis problems, and is recommended by JYP.
CDAT is available for Mac, Linux and Windows 10 (Windows 10 + Windows Subsystem for Linux, and Ubuntu).
conda list
after initializing a specific CDAT distributionThe following versions are maintained by JYP, on the Linux servers where LSCE users have accounts
CDAT version | JYP version | python version | Availability? | Available packages | Installation notes |
---|---|---|---|---|---|
8.1 | 18 | 2.7.15 3.6.7 | LSCE, ciclad | Default CDAT packages Extra packages | 8.1 notes |
Read the Ultra quick-start on the LSCE servers above
Note: on the ciclad cluster, use source ~jypmce/.conda3_jyp.sh
to initialize conda
Note: TGCC also supports its own TGCC distribution, that may be more up-to-date, if you don't need specific CDAT modules
CDAT 8.0 is installed at TGCC and can be initialized with:
module load cdat
module load flavor/cdat/python3 cdat
Note: if you get an error when importing cdms2 or vcs it means that either you have forgotten to initialize CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed on your system, or another (non-CDAT) python distribution!
$ python -c 'import cdms2, vcs' Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named cdms2 # Am I using the correct python distribution? $ which python /usr/bin/python
# Get the default version of the Python based modules $ module avail -t -d | egrep '(python|cdat)' flavor/cdat/standard(default) flavor/nest/python2(default) flavor/pytorch/python2(default) flavor/tensorflow/gpu_python2(default) intelpython2/2019.0(default) intelpython3/2019.0(default) python/2.7.14(default) python3/3.6.4(default) cdat/8.0(default) $ module load python [...] load module python/2.7.14 (Python) $ which python /ccc/products/python-2.7.14/intel--17.0.4.196__openmpi--2.0.2/default/bin/python $ python Python 2.7.14 (default, Jan 11 2018, 16:43:59) [GCC 4.8.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>
Enthought Deployment Manager (EDM) is Enthought’s mechanism to deliver scientific software applications and development environments
There are native installers for Windows (.msi), Mac OS X (.pkg), RHEL/Fedora (.rpm), and Debian/Ubuntu (.deb)
Anaconda is a distribution similar to canopy provided by Continuum Analytics.
Anaconda is available for Windows, Mac and Linux
Note: Anaconda provides and uses conda
for its installation. Since CDAT is now also installed and maintained with conda
, you can read the Installing and maintaining UV-CDAT with conda page for more information, even if you are not going to use CDAT
The ipython interpreter provides more options and commands than the standard python interpreter, but takes more time to start. If you are going to develop by starting and exiting the interpreter many times, it's faster to use the standard interpreter
Starting ipython: ipython
Jupyter Notebook for Beginners: A Tutorial
The ipython notebook is a way to interact with python (and other supported interpreted languages) inside a web browser. You can mix cells with python commands, cells with the output of the python commands (possibly graphics generated by the commands), and text (using some wiki-like rich text format). This interactive web page, aka notebook, can be saved in a my_notebook.ipynb
file and re-used later (e.g. the notebook provided in the introduction to Python, part 1).
Starting the notebook server: ipython notebook
Please make sure that you shutdown cleanly each notebook and the server when you are finished, in order not to clutter the local computer or remote server with lots of python processes (especially a server shared by many users!).
You may want to use the 'top' command in order to monitor what is happening
[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]