Differences

This shows you the differences between two versions of the page.

--- other:python:starting [2017/03/10 17:03]
jypeter
+++ other:python:starting [2019/04/25 16:07]
jypeter Lots of changes
@@ Line 5: / Line 5: @@
 You can then read the [[jyp_steps|JYP's recommended steps for learning python]] for really working with python
 </note>
+Note: the former and **outdated** version (before the massive usage of //conda//) is still [[other:python:starting_170810|available for reference]]
 ===== Where to start =====
@@ Line 10: / Line 12: @@
 In order to start working with Python, you need to have a //Python distribution// installed on your local computer or on the remote Linux server(s) you work on. A //distribution// provides a //Python interpreter//, and Python extensions (aka Python //modules// or //packages//). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type ''which python'' on Linux to determine where the python executable is located)
-If you are using a Linux computer or a Mac, you should already have a **default python** installed. The following example shows where the python interpreter is installed on the //asterix// LSCE server (if it's in ''/usr/bin'', it's the default python) and which version it is (example below: version //2.6.6// compiled in May 2015)
+If you are using a Linux computer or a Mac, you should already have a **default python** installed. The following example shows where the python interpreter is installed on the //obelix// LSCE servers (if it's in ''/usr/bin'', it's the default python) and which version it is (example below: version //2.7.5// compiled in April 2019)
 <code># Which is the current python (e.g. where is it located)?
@@ Line 18: / Line 20: @@
 # Where is it coming from ('rpm' works on a RedHat-like Linux machine)?
  > rpm -qf /usr/bin/python
-python-2.6.6-64.el6.x86_64
+python-2.7.5-77.el7_6.x86_64
 # You also get some information when you start the interpreter
  > python
-Python 2.6.6 (r266:84292, May 22 2015, 08:34:51)
+Python 2.7.5 (default, Apr  9 2019, 14:30:50)
-[GCC 4.4.7 20120313 (Red Hat 4.4.7-15)] on linux2
+[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>>
@@ Line 30: / Line 32: @@
 ===== Python distributions available for LSCE users =====
-This section will help you choose a distribution in the big Python ecosystem (many distributions, python version 2.7.* or 3.*, ...). You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/duplicate the data!) and try to identify who is maintaining it, if you need help or extra packages.
+This section will help you choose a distribution in the big Python ecosystem (many distributions, python version 2.7.* or 3.*, ...) on some of the servers used by LSCE users. You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/duplicate the data!) and try to identify who is maintaining it, if you need help or additional packages.
-Only install a distribution yourself if you need it on your local computer, or if you need to install some modules that can't be installed by the contacts listed below. A python distribution can require several Gb of disk space, so do not install it in a backed up //home// directory!
+Only install a distribution yourself if you need it on your local computer (desktop or laptop), or if you need to install some modules that can't be installed by the contacts listed below. A python distribution will require several Gb of disk space, so do not install it in a backed up //home// directory!
-We suggest that you **use Python 2 rather than Python 3**. Or you can check the [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:jyp_steps#python_27_vs_python_3|differences between both versions]] and try to write scripts that will work in both versions! 8-)
+You can use either //Python 2// or //Python 3//. Most packages are now available in both versions, but you should make sure that the most important package/s you need is/are available in the selected Python version. You can check the [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:jyp_steps#python_27_vs_python_3|differences between both versions]] and try to write scripts that will work in both versions!
 ==== LSCE distribution ====
-Contact: the LSCE system administrators ([[help-lsce@lsce.ipsl.fr]])
+  * Contact: the LSCE system administrators ([[help-lsce@lsce.ipsl.fr]])
+  * Where: //obelix// interactive servers and cluster at LSCE
+  * Initialization type: [[other:newppl:starting#using_module_to_access_optional_programs|module based]] + //conda//
+  * What's installed: type ''conda list'' after initializing the LSCE distribution
 <code>
  > module avail
 [...]
-castem/12        ferret/6.9.5     hdf5/1.8.9       netcdf/4p        python/2.7.5     sun-java/7.0.45
+ARTIC/3.6         grib_api/1.14      netcdf/3           python/2.7.5
+batch_env          grib_api/1.14.0    netcdf/4           python/3.6
+[...]
+glost/0.3.1        ncview/2.1.7       python/2.7
 [...]
- > module load python/2.7.5
+ > module load python/2.7
  > which python
-/usr/local/install/python-2.7.5/bin/python
+/usr/local/install/python-2.7/bin/python
  > python
-Python 2.7.5 (default, Sep 18 2013, 15:47:43)
+Python 2.7.15 |Anaconda, Inc.| (default, Oct 10 2018, 21:32:13)
-[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
+[GCC 7.3.0] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>>
@@ Line 60: / Line 68: @@
 ==== TGCC distribution ====
-Contact: the TGCC hotline ([[hotline.tgcc@cea.fr]])
+  * Contact: the TGCC hotline ([[hotline.tgcc@cea.fr]])
+  * Where: CEA TGCC
+  * Initialization type: [[other:newppl:starting#using_module_to_access_optional_programs|module based]]
+  * What's installed: it depends...
-<code>
+<code># Get the default version of the Python based modules
-$ module avail
+$ module avail -t -d | egrep '(python|cdat)'
+flavor/cdat/standard(default)
+flavor/nest/python2(default)
+flavor/pytorch/python2(default)
+flavor/tensorflow/gpu_python2(default)
+intelpython2/2019.0(default)
+intelpython3/2019.0(default)
+python/2.7.14(default)
+python3/3.6.4(default)
+cdat/8.0(default)
+$ module load python
 [...]
-cmake/2.8.9(default)      hwloc/1.5                 netcdf-utils/4.3.3.1_hdf5 python/2.7.8              tix/8.4.3
+load module python/2.7.14 (Python)
-cmake/3.2.2               hwloc/1.7.1(default)      octave/3.6.3              python/3.3.2              tk/8.5(default)
-[...]
-$ module load python/2.7.8
-load module python/2.7.8 (Python)
 $ which python
-/usr/local/ccc_python/2.7.8_201409/bin/python
+/ccc/products/python-2.7.14/intel--17.0.4.196__openmpi--2.0.2/default/bin/python
 $ python
-Python 2.7.8 (default, Aug 27 2014, 17:50:16)
+Python 2.7.14 (default, Jan 11 2018, 16:43:59)
-[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
+[GCC 4.8.5] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>>
 </code>
-==== UV-CDAT ====
+==== CDAT ====
-Contact: Jean-Yves Peterschmitt (//JYP//) @ LSCE
+[[https://uvcdat.llnl.gov/|CDAT]] (//Community Data Analysis Tools//) is a powerful and complete front-end to a rich set of visual-data exploration and analysis capabilities well suited for **climate** data analysis problems, and is recommended by JYP.
-=== What is UV-CDAT? ===
+CDAT is **available for Mac, Linux and Windows 10 (Windows 10 + [[other:win10wsl|Windows Subsystem for Linux, and Ubuntu]])**.
-[[http://uv-cdat.llnl.gov/|UV-CDAT]] (//Ultrascale Visualization - Climate Data Analysis Tools//) is a python distribution developed specifically for the climate scientists, and is recommended by JYP.
+  * Contact: Jean-Yves Peterschmitt (//JYP//) @ LSCE
+  * Where: //obelix// interactive servers and cluster at LSCE, //ciclad// interactive servers and cluster at IPSL, irene @ TGCC
+  * Initialization type: conda based or [[other:newppl:starting#using_module_to_access_optional_programs|module based]]
+  * What's installed: type ''conda list'' after initializing a specific CDAT distribution
-UV-CDAT is **available for Mac and Linux**.
+=== CDAT versions maintained by JYP ===
-You can find out which **standard packages** are installed in the //Packages built by CDAT// column below. UV-CDAT also provides some **specific packages** that you will not find in other python distributions: **cdms2**, **cdutil**, **cdtime**, **genutil**, **vcs**, **cmor**, ...
+The following versions are maintained by JYP, on the Linux servers where LSCE users have accounts
-**Extra packages** are listed:
+^  CDAT version  ^  JYP\\ version  ^  python\\ version  ^  Availability?  ^  Available packages ^  Installation notes  ^
-  * in the apropriate //Installation notes// pages for CDAT>=''2.8.0''
+|  **8.1**  |  18  |  2.7.15\\ 3.6.7  |  LSCE, ciclad  |  {{ :other:uvcdat:cdat_conda:cdat-8.1_py2_list_190307.txt |Default CDAT packages}}\\ [[other:uvcdat:cdat_conda:cdat_8_1#extra_packages_list|Extra packages]]  |  [[other:uvcdat:cdat_conda:cdat_8_1|8.1 notes]]  |
-  * on the [[other:uvcdat:extra|Extra packages installed in UV-CDAT by JYP]] page for CDAT%%<=%%''2.10''
-=== UV-CDAT versions maintained by JYP ===
-The following versions are maintained by JYP, on the Linux servers where LSCE users have accounts
-^  CDAT version  ^  JYP\\ version  ^  python\\ version  ^  Availability?  ^  Packages built by CDAT  ^  Installation notes  ^
+=== CDAT at TGCC ===
-|  **1.1.0**  |  08  |  2.7.3  |  LSCE, curie  |  n/a  |  [[other:uvcdat:installed#section110|1.1.0 notes]]  |
-|  **1.5.1**  |  10  |  2.7.4  |  LSCE, ciclad  |  {{:other:python:cdat-1.5.1_jyp-10_build_info.txt}}  |  [[other:uvcdat:installed#section151|1.5.1 notes]]  |
-|  **2.1.0**  |  13  |  2.7.8  |  LSCE, ciclad  |  {{:other:python:cdat-2.1.0_jyp-13_build_info.txt}}  |  [[other:uvcdat:installed#section210|2.1.0 notes]]  |
-|  **2.4.0**  |  //Not installed//  |||||
-|  **2.8.0**  |  14  |  2.7.12  |  LSCE  |  {{:other:uvcdat:uvcdat-2.8.0_install_dry_with-forge_160221.txt}}  |  [[:other:uvcdat:cdat_conda:cdat_2_8_0|2.8.0 notes]]  |
-Notes:
+CDAT **8.0** is installed at TGCC and can be initialized with:
-  * For the conda based CDAT versions (>=''2.8.0''), you can get the list of currently available packages by typing: ''conda list''
+  * Python 2: ''module load cdat''
-  * For older CDAT versions (<=''2.1.0''), the ''cdat-VVV_jyp-NNN_build_info.txt'' files have been generated with\\ ''cd /my_build_directory_path; cat build_info.txt | sort > cdat-VVV_JYP-NNN_build_info.txt''
+  * Python 3: ''module load flavor/cdat/python3 cdat''
-=== The basics of UV-CDAT initialization ===
+=== The basics of CDAT initialization ===
-The way you initialize UV-CDAT depends on:
+The way you initialize CDAT depends on:
   * which **server** you want to use it on
   * which **shell** you are using in your terminals: //tcsh// or other shells
     * Read [[other:newppl:starting#which_shell_are_you_using|Which shell are you using?]] if you are not sure
     * Warning: the shell in a (batch) script may be different from your interactive shell!
-  * which **version** of UV-CDAT you want to use...
+  * which **version** of CDAT you want to use...
-    * Note: You should use the most recent version of UV-CDAT available on a given server, unless you have a good reason to use an older version (e.g. if you need to use the //old-style vcs// of version 1.5.1).
-Note: if you get an error when importing //cdms2// or //vcs// it means that either you have forgotten to initialize UV-CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed on your system, or another python distribution!
+Note: if you get an error when importing //cdms2// or //vcs// it means that either you have forgotten to initialize CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed on your system, or another (non-CDAT) python distribution!
 <code>$ python -c 'import cdms2, vcs'
@@ Line 133: / Line 143: @@
 /usr/bin/python</code>
-=== conda-based versions of UV-CDAT ===
+=== conda-based versions of CDAT ===
 <WRAP center round important 60%>
@@ Line 139: / Line 149: @@
 </WRAP>
-Note: if you need to maintain a **local** version on **your** machine, you can read [[other:uvcdat:conda_notes|Installing and maintaining UV-CDAT with conda]] and the [[other:uvcdat:cdat_conda:index#versions|versions specific notes]].
+Note: if you need to maintain a **local** version on **your** machine, you can read [[other:uvcdat:conda_notes|Installing and maintaining CDAT with conda]] and the [[other:uvcdat:cdat_conda:index#versions|versions specific notes]].
 == Configuring .bashrc ==
@@ Line 152: / Line 162: @@
 Note: when you add the specified line to your ''bashrc'' file, you will then always get the python supplied by conda, instead of the system's python, when you use ''bash''. If you don't like this permanent behavior, just type ''source ~jyp_login_on_this_server/.conda_jyp.sh'' in the bash shell where you want to use conda+CDAT, rather than adding it to the ''.bashrc'' file...
-== Initializing UV-CDAT ==
+== Initializing CDAT ==
 <WRAP center round tip 60%>
-When you initialize UV-CDAT, it will be available only in the **current** window/shell. In the other windows, you still get whatever your default python is. This should minimize potential side-effects
+When you initialize CDAT, it will be available only in the **current** window/shell. In the other windows, you still get whatever your default python is. This should minimize potential side-effects
 </WRAP>
   - If [[other:newppl:starting#which_shell_are_you_using|your shell]] is not //bash//, start a bash shell by typing ''bash''
     * You can later go back to your previous shell by typing ''exit'' or ''CTRL-D''
   - Activate the //conda environment// that uses the version of UV-CDAT you want to use
@@ Line 167: / Line 178: @@
 ^  UV-CDAT version  ^  Server  ^  Environment name\\ and activation line  ^  Packages\\ (''conda list'')  ^
 |  [[:other:uvcdat:cdat_conda:cdat_2_8_0|2.8.0]]  |  **LSCE**\\ **ciclad**  |**cdatm14** (''source activate cdatm14'')  |  {{ :other:uvcdat:cdat_conda:conda_list_lsce_cdatm14_170310.txt |cdatm14 installed packages}}  |
+|  [[:other:uvcdat:cdat_conda:cdat_2_10|2.10]]  |  **LSCE**  |**cdatm15** (''source activate cdatm15'')  |  {{ :other:uvcdat:cdat_conda:uvcdat-2.10_list_170721.txt |cdatm14 installed packages}}  |
+|  **cdatm** will always point to the latest stable version  ||||
+|  //latest stable version//  |  **LSCE**  |**cdatm** (''source activate cdatm'')  |  ''conda list''  |
-=== Older versions of UV-CDAT ===
-<WRAP center round important 60%>
-The steps detailed in this section are for **UV-CDAT ''2.1.0'' and earlier**
-</WRAP>
-== Initializing UV-CDAT for tcsh users ==
-If you use **tcsh** as [[other:newppl:starting#which_shell_are_you_using|your login shell]], you can define a ''cdatm'' alias on all the IPSL servers on which UV-CDAT is installed (see details below), and move easily from one version of UV-CDAT to another:
-  * type ''cdatm'' to initialize the default version of UV-CDAT
-  * type ''cdatm %%--%%config //CONFIG_NAME//'' to initialize a specific version
-    * You can find the allowed values of //CONFIG_NAME// (e.g. ''LSCE_10'', ''LSCE_13'', ...) in one of the tables below
-<code> > which python
-/usr/bin/python
- > cdatm
-** CDAT 'uv-1.5.1' initialized!
-** If you have problems with this installation,
-** it is maintained by: Jean-Yves Peterschmitt - LSCE
- > which python
-/home/share/unix_files/cdat/versions/cdat_install_uv-1.5.1_x86_64_gcc4_10/bin/python
-</code>
-You need to add the following line to the ''.cshrc'' file in your home directory, in order to define the ''cdatm'' alias. You can also define the simple ''wp'' alias, in order to always be able to determine quickly which python is initialized in the current terminal.
-|  **LSCE**  | ''alias cdatm 'source ~jypeter/CDAT/Install/cdat_multi/cdat_multi.login ~jypeter/CDAT/Install/cdat_multi''' |
-|  **ciclad**  | ''alias cdatm 'source ~jypmce/CDAT/Install/cdat_multi/cdat_multi.login ~jypmce/CDAT/Install/cdat_multi'''|
-|  **curie**  | ''alias cdatm 'source ~p25jype/CDAT/Install/cdat_multi/cdat_multi.login ~p25jype/CDAT/Install/cdat_multi''' |
-|  **All** servers  | ''alias wp 'which python'' |
-== Initializing UV-CDAT in any kind of shell, or in a batch script ==
-You first need to know where the ''python'' program provided by UV-CDAT is installed, something like ''/path/to/UV-CDAT/bin/python''
-Then, all you need to do in order to initialize UV-CDAT is to //source// the ''setup_runtime'' file appropriate for [[other:newppl:starting#which_shell_are_you_using|your shell]]:
-  * tcsh: type ''source /path/to/UV-CDAT/bin/setup_runtime**.csh**''
-  * bash, sh/ksh: type ''source /path/to/UV-CDAT/bin/setup_runtime**.sh**''
-The table below lists, for each server and version, the path where you can find the //setup_runtime// files
-^  Server  ^  CDAT\\ version  ^  JYP\\ version  ^  Path  ^
-|  LSCE  |  1.1.0  |  LSCE_08  |  ''/home/share/unix_files/cdat/versions/cdat_install_uv-1.1.0_x86_64_gcc4_08/bin''  |
-|  LSCE  |  1.5.1  |  LSCE_10  |  ''/home/share/unix_files/cdat/versions/cdat_install_uv-1.5.1_x86_64_gcc4_10/bin''  |
-|  LSCE  |  2.1.0  |  LSCE_13  |  ''/home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin''  |
-|  ciclad  |  1.5.1  |  ciclad_10-ng  |  ''/data/jypmce/cdat/versions/cdat_install_uv-1.5.1_x86_64_gcc4_10-ng/bin''  |
-|  ciclad  |  2.1.0  |  ciclad_13  |  ''/data/jypmce/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin''  |
-|  curie  |  1.1.0  |  curie_08  |  ''/ccc/work/cont003/dsm/p25jype/cdat/versions/cdat_install_uv-1.1.0_x86_64_gcc4_curie_08/bin''\\ **On curie**, you need to use **setup_cdat**.sh or **setup_cdat**.csh to initialize CDAT,\\ and also to load modules providing access to old librairies\\ ''module load gnu/4.6.3'' and ''module load qt/4.8.6''  |
-Example: if you want to use UV-CDAT 2.1.0 at LSCE in a bash shell, you need to do the following (you only need the //source// line. The other lines are optional, just for checking that the initialization was OK)
-<code>bash-4.1$ source /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.sh
-bash-4.1$ which python
-/home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/python
-bash-4.1$ python -c 'import cdms2, vcs'
-bash-4.1$</code>
-You probably don't want to have to determine what you should type each time you want to use UV-CDAT in a window, so you should define a simple shortcut alias that you can type each time you need to initialize UV-CDAT. If, for example, you want to use UV-CDAT 2.1.0 at LSCE:
-  * tcsh: add the following alias definitions to ''~/.cshrc''\\ ''alias cdat13 'source /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.csh'''\\  ''alias wp 'which python'''
-  * bash: add the following alias definition to ''~/.bashrc'', ''~/.profile'' or ''~/.bash_profile'' (whichever works for you, or read ''man bash''...)\\ ''alias cdat13='source /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.sh'''\\ ''alias wp='which python'''
-If you are going to use a UV-CDAT python script on a cluster or supercomputer, using some kind of batch system using a bash or ksh script, use the following lines before you run the python script
-<code>source /home/share/unix_files/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_13/bin/setup_runtime.sh
-# Check which python we will be using
-mypython=`which python`
-echo
-echo -e "This script will use the following python: $mypython\n\n"</code>
 ==== Canopy ====

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools