Differences

This shows you the differences between two versions of the page.

--- other:python:starting [2016/06/09 09:56]
jypeter [UV-CDAT] Added specific instructions for initializing the old CDAT on curie
+++ other:python:starting [2019/04/24 15:28]
jypeter [LSCE distribution] Improved
@@ Line 6: / Line 6: @@
 </note>
-In order to begin using Python, you need to have a //Python distribution// installed. A //distribution// provides a Python interpreter, and Python extensions (aka Python //modules//). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type ''which python'' on Linux)
+===== Where to start =====
-If you are using a Linux computer or a Mac, you should already have a **default python** installed. The following example shows where the python interpreter is installed on the asterix1 server (if it's in ''/usr/bin'', it's the default python) and which version it is (version //2.6.6// compiled in May 2015)
+In order to start working with Python, you need to have a //Python distribution// installed on your local computer or on the remote Linux server(s) you work on. A //distribution// provides a //Python interpreter//, and Python extensions (aka Python //modules// or //packages//). You may have several distributions installed on your computer and you need to know how to initialize them, and which one you are using at a given time (type ''which python'' on Linux to determine where the python executable is located)
-<code> > which python
+If you are using a Linux computer or a Mac, you should already have a **default python** installed. The following example shows where the python interpreter is installed on the //obelix// LSCE servers (if it's in ''/usr/bin'', it's the default python) and which version it is (example below: version //2.7.5// compiled in April 2019)
+<code># Which is the current python (e.g. where is it located)?
+ > which python
 /usr/bin/python
+# Where is it coming from ('rpm' works on a RedHat-like Linux machine)?
  > rpm -qf /usr/bin/python
-python-2.6.6-64.el6.x86_64
+python-2.7.5-77.el7_6.x86_64
+# You also get some information when you start the interpreter
  > python
-Python 2.6.6 (r266:84292, May 22 2015, 08:34:51)
+Python 2.7.5 (default, Apr  9 2019, 14:30:50)
-[GCC 4.4.7 20120313 (Red Hat 4.4.7-15)] on linux2
+[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>>
 </code>
-===== Some python distributions =====
+===== Python distributions available for LSCE users =====
-This section will help you choose a distribution in the big Python ecosystem (many distributions, python version 2.7.* or 3.*, ...). You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/duplicate the data!).
+This section will help you choose a distribution in the big Python ecosystem (many distributions, python version 2.7.* or 3.*, ...). You should use a distribution that is already available near your data (e.g. do the computation on a server near your data, do not move/duplicate the data!) and try to identify who is maintaining it, if you need help or additional packages.
-Only install a distribution yourself if you need it on your local computer, or if you need to install some modules that can't be installed by the contacts listed below. A python distribution can be quite big, so do not install it in a backed up home directory!
+Only install a distribution yourself if you need it on your local computer, or if you need to install some modules that can't be installed by the contacts listed below. A python distribution can require several Gb of disk space, so do not install it in a backed up //home// directory!
-We suggest that you **use Python 2 rather than Python 3**. Or you can check the [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:jyp_steps#python_27_vs_python_3|differences between both versions]] and try to write scripts that will work in both versions! 8-)
+You can use either //Python 2// or //Python 3//. Most packages are now available in both versions, but you should make sure that the most important package/s you need is/are available in the selected version. Check the [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:jyp_steps#python_27_vs_python_3|differences between both versions]] and try to write scripts that will work in both versions! 8-)
 ==== LSCE distribution ====
-Contact: the LSCE system administrators
+Contact: the LSCE system administrators ([[help-lsce@lsce.ipsl.fr]])
 <code>
  > module avail
 [...]
-castem/12        ferret/6.9.5     hdf5/1.8.9       netcdf/4p        python/2.7.5     sun-java/7.0.45
+ARTIC/3.6         grib_api/1.14      netcdf/3           python/2.7.5
+batch_env          grib_api/1.14.0    netcdf/4           python/3.6
+[...]
+glost/0.3.1        ncview/2.1.7       python/2.7
 [...]
- > module load python/2.7.5
+ > module load python/2.7
  > which python
-/usr/local/install/python-2.7.5/bin/python
+/usr/local/install/python-2.7/bin/python
  > python
-Python 2.7.5 (default, Sep 18 2013, 15:47:43)
+Python 2.7.15 |Anaconda, Inc.| (default, Oct 10 2018, 21:32:13)
-[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
+[GCC 7.3.0] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>>
@@ Line 55: / Line 63: @@
 ==== TGCC distribution ====
-Contact: the TGCC hotline
+Contact: the TGCC hotline ([[hotline.tgcc@cea.fr]])
 <code>
@@ Line 79: / Line 87: @@
 ==== UV-CDAT ====
-Contact: Jean-Yves @ LSCE
+Contact: Jean-Yves Peterschmitt (//JYP//) @ LSCE
 === What is UV-CDAT? ===
-[[http://uv-cdat.llnl.gov/|UV-CDAT]] (//Ultrascale Visualization - Climate Data Analysis Tools//) is a python distribution developed specifically for the climate scientists.
+[[http://uv-cdat.llnl.gov/|UV-CDAT]] (//Ultrascale Visualization - Climate Data Analysis Tools//) is a python distribution developed specifically for the climate scientists, and is recommended by JYP.
-You can find out which standard packages are installed by clicking on the ''cdat-VVV_JYP-NNN_build_info.txt'' files below and by visiting the [[other:uvcdat:extra|Extra packages installed in UV-CDAT by JYP]] page.
+UV-CDAT is **available for Mac and Linux**.
-UV-CDAT also provides some specific packages that you will not find in other python distributions: **cdms2**, **cdutil**, **cdtime**, **genutil**, **vcs**, **cmor2**, ...
+You can find out which **standard packages** are installed in the //Packages built by CDAT// column below. UV-CDAT also provides some **specific packages** that you will not find in other python distributions: **cdms2**, **cdutil**, **cdtime**, **genutil**, **vcs**, **cmor**, ...
+**Extra packages** are listed:
+  * in the apropriate //Installation notes// pages for CDAT>=''2.8.0''
+  * on the [[other:uvcdat:extra|Extra packages installed in UV-CDAT by JYP]] page for CDAT%%<=%%''2.10''
 === UV-CDAT versions maintained by JYP ===
-The following versions are maintained by JYP, on the servers where LSCE users have accounts
+The following versions are maintained by JYP, on the Linux servers where LSCE users have accounts
 ^  CDAT version  ^  JYP\\ version  ^  python\\ version  ^  Availability?  ^  Packages built by CDAT  ^  Installation notes  ^
@@ Line 97: / Line 109: @@
 |  **1.5.1**  |  10  |  2.7.4  |  LSCE, ciclad  |  {{:other:python:cdat-1.5.1_jyp-10_build_info.txt}}  |  [[other:uvcdat:installed#section151|1.5.1 notes]]  |
 |  **2.1.0**  |  13  |  2.7.8  |  LSCE, ciclad  |  {{:other:python:cdat-2.1.0_jyp-13_build_info.txt}}  |  [[other:uvcdat:installed#section210|2.1.0 notes]]  |
-|  **2.4.0**  |  14  |    |    |  //Not installed yet//  |    |
+|  **2.4.0**  |  //Not installed//  |||||
+|  **2.8.0**  |  14  |  2.7.12  |  LSCE  |  {{:other:uvcdat:uvcdat-2.8.0_install_dry_with-forge_160221.txt}}  |  [[:other:uvcdat:cdat_conda:cdat_2_8_0|2.8.0 notes]]  |
 Notes:
-  * The ''cdat-VVV_jyp-NNN_build_info.txt'' files have been generated with\\ ''cd /my_build_directory_path; cat build_info.txt | sort > cdat-VVV_JYP-NNN_build_info.txt''
+  * For the conda based CDAT versions (>=''2.8.0''), you can get the list of currently available packages by typing: ''conda list''
+  * For older CDAT versions (<=''2.1.0''), the ''cdat-VVV_jyp-NNN_build_info.txt'' files have been generated with\\ ''cd /my_build_directory_path; cat build_info.txt | sort > cdat-VVV_JYP-NNN_build_info.txt''
 === The basics of UV-CDAT initialization ===
 The way you initialize UV-CDAT depends on:
-  * which **server** you want to use it
+  * which **server** you want to use it on
   * which **shell** you are using in your terminals: //tcsh// or other shells
-    * Note: the shell in a (batch) script may be different from your interactive shell!
+    * Read [[other:newppl:starting#which_shell_are_you_using|Which shell are you using?]] if you are not sure
+    * Warning: the shell in a (batch) script may be different from your interactive shell!
   * which **version** of UV-CDAT you want to use...
     * Note: You should use the most recent version of UV-CDAT available on a given server, unless you have a good reason to use an older version (e.g. if you need to use the //old-style vcs// of version 1.5.1).
+Note: if you get an error when importing //cdms2// or //vcs// it means that either you have forgotten to initialize UV-CDAT, or that something went wrong during the initialization. In both cases, you are either still using the default python installed on your system, or another python distribution!
-Note: if you are not sure which shell is running in your terminals, you can find out the following way
-<code>
-# Somewhere with tcsh
- > echo $SHELL
- /bin/tcsh
-# Somewhere else with bash
- $ echo $SHELL
- /bin/bash
-</code>
-Note: if you get an error when importing //cdms2// or //vcs// it means that either you have forgotten to initialize UV-CDAT, or that something went wrong during the initialization. In both cases, you are probably still using the default python installed on your system!
 <code>$ python -c 'import cdms2, vcs'
@@ Line 130: / Line 132: @@
 ImportError: No module named cdms2
+# Am I using the correct python distribution?
 $ which python
 /usr/bin/python</code>
-=== Initializing UV-CDAT for tcsh users ===
+=== conda-based versions of UV-CDAT ===
+<WRAP center round important 60%>
+\\ The steps detailed in this section are for [[other:uvcdat:cdat_conda:|UV-CDAT 2.8.0 and later]]
+</WRAP>
+Note: if you need to maintain a **local** version on **your** machine, you can read [[other:uvcdat:conda_notes|Installing and maintaining UV-CDAT with conda]] and the [[other:uvcdat:cdat_conda:index#versions|versions specific notes]].
+== Configuring .bashrc ==
+You should **add** the following line at the end of your ''~/.bashrc'' configuration file (create the ''~/.bashrc'' file if it does not exist yet). This will basically make sure that your shell can find the ''conda'' executable it needs for initializing CDAT, and define a ''wp'' alias that you can use to find where the ''python'' executable is located
+^  Server  ^  Line to add to ''.bashrc''  ^
+|  **LSCE**  | ''source ~jypeter/.conda_jyp.sh''\\ \\ Note:{{ :other:python:conda_jyp.txt |conda_jyp}} content |
+|  **ciclad**  | ''source ~jypmce/.conda_jyp.sh'' |
+Note: when you add the specified line to your ''bashrc'' file, you will then always get the python supplied by conda, instead of the system's python, when you use ''bash''. If you don't like this permanent behavior, just type ''source ~jyp_login_on_this_server/.conda_jyp.sh'' in the bash shell where you want to use conda+CDAT, rather than adding it to the ''.bashrc'' file...
+== Initializing UV-CDAT ==
+<WRAP center round tip 60%>
+When you initialize UV-CDAT, it will be available only in the **current** window/shell. In the other windows, you still get whatever your default python is. This should minimize potential side-effects
+</WRAP>
+  - If [[other:newppl:starting#which_shell_are_you_using|your shell]] is not //bash//, start a bash shell by typing ''bash''
+    * You can later go back to your previous shell by typing ''exit'' or ''CTRL-D''
+  - Activate the //conda environment// that uses the version of UV-CDAT you want to use
+    * ''source activate conda_env_name''
+    * Use the table below to determine which environments are available on which server
+  - Use CDAT !
+^  UV-CDAT version  ^  Server  ^  Environment name\\ and activation line  ^  Packages\\ (''conda list'')  ^
+|  [[:other:uvcdat:cdat_conda:cdat_2_8_0|2.8.0]]  |  **LSCE**\\ **ciclad**  |**cdatm14** (''source activate cdatm14'')  |  {{ :other:uvcdat:cdat_conda:conda_list_lsce_cdatm14_170310.txt |cdatm14 installed packages}}  |
+|  [[:other:uvcdat:cdat_conda:cdat_2_10|2.10]]  |  **LSCE**  |**cdatm15** (''source activate cdatm15'')  |  {{ :other:uvcdat:cdat_conda:uvcdat-2.10_list_170721.txt |cdatm14 installed packages}}  |
+|  **cdatm** will always point to the latest stable version  ||||
+|  //latest stable version//  |  **LSCE**  |**cdatm** (''source activate cdatm'')  |  ''conda list''  |
+=== Older versions of UV-CDAT ===
+<WRAP center round important 60%>
+The steps detailed in this section are for **UV-CDAT ''2.1.0'' and earlier**
+</WRAP>
+== Initializing UV-CDAT for tcsh users ==
-If you use **tcsh** as your login shell, you can define a ''cdatm'' alias on all the IPSL servers on which UV-CDAT is installed (see details below), and move easily from one version of UV-CDAT to another:
+If you use **tcsh** as [[other:newppl:starting#which_shell_are_you_using|your login shell]], you can define a ''cdatm'' alias on all the IPSL servers on which UV-CDAT is installed (see details below), and move easily from one version of UV-CDAT to another:
   * type ''cdatm'' to initialize the default version of UV-CDAT
   * type ''cdatm %%--%%config //CONFIG_NAME//'' to initialize a specific version
@@ Line 159: / Line 206: @@
 |  **All** servers  | ''alias wp 'which python'' |
-=== Initializing UV-CDAT in any kind of shell, or in a batch script ===
+== Initializing UV-CDAT in any kind of shell, or in a batch script ==
 You first need to know where the ''python'' program provided by UV-CDAT is installed, something like ''/path/to/UV-CDAT/bin/python''
-Then, all you need to do in order to initialize UV-CDAT is to //source// the ''setup_runtime'' file appropriate for your shell:
+Then, all you need to do in order to initialize UV-CDAT is to //source// the ''setup_runtime'' file appropriate for [[other:newppl:starting#which_shell_are_you_using|your shell]]:
-  * tcsh: type ''source /path/to/UV-CDAT/bin/setup_runtime.csh''
+  * tcsh: type ''source /path/to/UV-CDAT/bin/setup_runtime**.csh**''
-  * bash, sh/ksh: type ''source /path/to/UV-CDAT/bin/setup_runtime.sh''
+  * bash, sh/ksh: type ''source /path/to/UV-CDAT/bin/setup_runtime**.sh**''
 The table below lists, for each server and version, the path where you can find the //setup_runtime// files
@@ Line 201: / Line 248: @@
 ==== Canopy ====
-[[https://www.enthought.com/products/canopy/|Enthought Canopy]] is //a Scientific and Analytic Python Deployment with Integrated Analysis Environment// provided by [[https://www.enthought.com/|Enthought]]. It is available for Windows, Mac and Linux. You can download the free //Canopy Express// that will already provide many extensions. If you are entitled to use //Canopy Academic//, login from inside Canopy Express and download the extra modules you need
+[[https://www.enthought.com/products/canopy/|Enthought Canopy]] is //a Scientific and Analytic Python Deployment with Integrated Analysis Environment// provided by [[https://www.enthought.com/|Enthought]]. It used to be called EPD (//Enthought Python Distribution//)
+Canopy is **available for Windows, Mac and Linux**. You can download the free //Canopy Express// that will already provide many extensions. If you are entitled to use //Canopy Academic//, login from inside Canopy Express and download the extra modules you need
 ==== Anaconda ====
-[[https://www.continuum.io/why-anaconda|Anaconda]] is a distribution similar to [[#canopy]] provided by [[https://www.continuum.io/|Continuum Analytics]]. It is available for Windows, Mac and Linux
+[[https://www.continuum.io/why-anaconda|Anaconda]] is a distribution similar to [[#canopy]] provided by [[https://www.continuum.io/|Continuum Analytics]].
+Anaconda is **available for Windows, Mac and Linux**
+Note: Anaconda provides and uses ''conda'' for its installation. Since UV-CDAT is now also installed and maintained with ''conda'', you can read the [[other:uvcdat:conda_notes|Installing and maintaining UV-CDAT with conda]] page for more information, even if you are not going to use UV-CDAT
 ===== Launching Python =====
-Once you have initialized the [[#some_python_distributions|python distribution]] you want to use, follow the instructions below to //start the interpreter from a shell// (on a Windows computer, you will have to start python from the //Start// menu).
+Once you have initialized the [[#some_python_distributions|python distribution]] you want to use, follow the instructions below to //start the interpreter from a shell// (on a Windows computer, you will have to start python from the //Start// menu or by clicking a shortcut on your desktop).
-Remember that you can check which python you are using by typing ''which python''!
+Remember that on Linux/Mac you can check which python you are using by typing ''which python''!
 ==== Useful keyboard shortcuts ====
@@ Line 230: / Line 282: @@
 ==== Stand-alone script ====
-A python script is just like any other shell script. You don't have to explicitly call the python interpreter, if the interpreter is specified in a ''#!'' [[https://en.wikipedia.org/wiki/Shebang_%28Unix%29|shebang comment]] on the first line of the script with, and the script has it's execution bit set.
+A python script is just like any other shell script. You don't have to explicitly call the python interpreter, if the interpreter is specified in a ''#!'' [[https://en.wikipedia.org/wiki/Shebang_%28Unix%29|shebang comment]] on the first line of the script with, and the script has its execution bit set (''chmod +x my_script.py'').
 <code>
@@ Line 256: / Line 308: @@
   * ''python'': start the interpreter
   * ''python script.py'': execute //script.py// and **exit**
-  * ''python -i script'': execute //script.py// and **stay in the interpreter**
+  * ''python -i script'': execute //script.py// and **stay in the interpreter** (''-i'' = //**I**nteractive//)
 Type ''man python'' if you want to see what other command line options are available

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools