Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:cdat_8_1 [2019/03/07 16:59] – jypeter
+++ other:uvcdat:cdat_conda:cdat_8_1 [2019/05/17 10:39] – Added ferretmagic jypeter
@@ Line 4: / Line 4: @@
 <WRAP center round tip 60%>
-Follow the instructions about the [[other:python:starting#conda-based_versions_of_uv-cdat|conda-based versions of UV-CDAT initialization]] for actually using an installed version of ''8.1''
+If you have questions about Python, read:
+  * [[other:python:starting|Working with Python]]
+  * [[other:python:jyp_steps|JYP's recommended steps for learning python]]!
 </WRAP>
+===== Using JYP version =====
+If you mostly want to use the CDAT 8.1 installed by JYP, just use the following steps and skip the next sections:
+  * Initialize conda:
+^  Server  ^  tcsh  ^  bash  ^
+|  LSCE  | ''source ~jypeter/.conda3_jyp.csh'' | ''source ~jypeter/.conda3_jyp.sh'' |
+|  ciclad  | ''source ~jypmce/.conda3_jyp.csh'' | ''source ~jypmce/.conda3_jyp.sh'' |
+  * Activate the default environment: ''conda activate cdatm_py2''
+    * Check the [[#environments_summary|Available environments]] for more options
 ===== What's New? =====
@@ Line 16: / Line 29: @@
 We assume that Miniconda3 is already installed. Otherwise, follow the the [[other:uvcdat:cdat_conda:cdat_8_0_py2#installing_miniconda3|Installing Miniconda]] steps (and the //Post-Miniconda3 installation// section) we followed when installing CDAT 8.0
-==== Installing CDAT 8.0 ====
+==== Updating conda ====
+The conda package itself can be updated (if need be) with\\ ''conda update -n base -c defaults conda''
+==== Installing CDAT 8.1 ====
 We first check that we have indeed access to a conda installation, and we assume that we have a write-access to the conda hierarchy, and a few Gb of disk space available
@@ Line 97: / Line 113: @@
 ===== Customizing UV-CDAT for LSCE =====
+==== Testing vcs ====
+Start python and paste the following lines
+<code>import numpy as np, vcs
+id100 = np.identity(100)
+x = vcs.init()
+x.plot(id100)
+</code>
+If you get the kind of errors described in [[https://github.com/CDAT/vcs/issues/393|x.plot crashes with OpenGL2 error]], the drivers on your server are probably not compatible with an interactive use of vcs... The workaround is to **install mesalib and work in //headless mode//** (e.g. without a display)...
+Install //mesalib// with ''conda install -c cdat/label/v81 -c conda-forge mesalib vtk-cdat'' and try pasting again the few code lines above in python. If things work as expected, no canvas will be opened, but you'll be able to save the canvas to a file with
+<code>x.png('test_mesalib')
+x.pdf('test_mesalib')</code>
+Note: using //mesalib// for headless mode work makes it possible to work without an X server, and with no DISPLAY variable defined
 ==== Downloading cdms2/vcs test data ====
@@ Line 104: / Line 137: @@
 <code>$ conda activate cdatm18_py2
-(cdatm17_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data'
+(cdatm18_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)'
 [...]
-Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data
+Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
-(cdatm17_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/uvcdat/sample_data
+(cdatm18_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
-M    /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data
+M    /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
 </code>
@@ Line 137: / Line 170: @@
       * Get version number with:\\ <code>$ drq -v
 dreqPy version 01.00.29 [Version 01.00.29]</code>
+  * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret
+    * ''conda activate cdatm18_py2''\\ ''pip install ferretmagic''
 The following packages have no dependency problems and were installed (or updated) //later//
+  * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files
+    * ''conda install -n cdatm18_py2 -c conda-forge cmor''
+    * Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' ''
+    * <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]]
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
     * ''conda install -n cdatm18_py2 -c conda-forge -c vacumm vacumm''
+    * <wrap hi>Warning!</wrap> [[https://github.com/VACUMM/vacumm/issues/15|standard install of vacumm does not work for Python3]]
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
-    * ''conda install -n cdatm17_py2 -c stefraynaud -c conda-forge spanlib''
+    * ''conda install -n cdatm18_py2 -c stefraynaud -c conda-forge spanlib''
     * Test: ''python -c 'from spanlib.analyzer import Analyzer' ''
+  * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses
+    * ''conda install -n cdatm18_py2 -c conda-forge cartopy''
+  * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs
+    * ''conda install -n cdatm18_py2 -c conda-forge joblib''
+  * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework
+    * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]]
 ==== TODO ====
@@ Line 170: / Line 221: @@
 Some packages change more often than others, and can be easily updated the following way:
-  * [[https://cmor.llnl.gov/|CMOR]]
-    * Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor''
-      * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' ''
   * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]
@@ Line 179: / Line 226: @@
       * Get version number with: ''drq -v''
-==== Installing CLIMAF ====
-The [[https://climaf.readthedocs.io/|CLIMAF]] installation is a special case, because we may want to use it with the Python installed for CDAT, but it is not installed using conda
-We follow the steps for the [[https://climaf.readthedocs.io/en/latest/installing.html#installing-climaf-if-you-are-not-at-cnrm-on-ciclad-or-climserv|installation of a non-standard server]]
-  * Go to the appropriate installation directory (we choose the directory where we have also installed miniconda3):
-    * LSCE: ''cd /home/share/unix_files/cdat''
-    * ciclad: ''/data/jypmce/cdat/''
-  * ''git clone https://github.com/senesis/climaf climaf''
-  * Add CLIMAF to the PYTHONPATH environment variable
-    * LSCE & tcsh: ''setenv PYTHONPATH ${PYTHONPATH}:/home/share/unix_files/cdat/climaf''
-    * LSCE & bash: ''export PYTHONPATH=${PYTHONPATH}:/home/share/unix_files/cdat/climaf''
-    * ciclad: same as above, using ''/data/jypmce/cdat/climaf''
-  * Test the installation
-    * make sure the [[https://climaf.readthedocs.io/en/latest/requirements.html#requirements|Requirements]] are available:
-      * LSCE: ''module load netcdf/4 nco cdo ncl ncview''
-      * ciclad: ''module load netcdf4 nco cdo ncl/6.3.0''
-      * optional: start an X server
-      * <code>cd climaf/testing
-./test_install.sh</code>
-  * Once the tests are OK, add the required //module load// and environment variables definitions to the ''.basrc'' and ''.cshrc'' files. <wrap hi>Note: maybe these lines should go to the ''.profile'' and ''.login'' files...(check [[https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html|Bash Startup Files]])</wrap>
-    * LSCE: add to your .cshrc
-      * <code># Prerequisites for using CLIMAF
-module load netcdf/4 nco cdo ncl ncview
-# CLIMAF environment variables
-setenv CLIMAF /home/share/unix_files/cdat/climaf
-setenv PYTHONPATH "${PYTHONPATH}:${CLIMAF}"
-#setenv PYTHONPATH $CLIMAF
-setenv PATH "${PATH}:${CLIMAF}/bin"
-setenv CLIMAF_CACHE "/home/scratch01/${USER}/climaf_cache"
-setenv TMPDIR $CLIMAF_CACHE
-setenv CLIMAF_LOG_DIR $CLIMAF_CACHE
-#conda activate cdatm-nightly_py2</code>
-    * ciclad: add to your .bashrc
-      * <code># Prerequisites for using CLIMAF when not using the
-# default 'module load climaf'
-module load netcdf4 nco cdo ncl/6.3.0
-# CLIMAF environment variables
-export CLIMAF=/data/jypmce/cdat/climaf
-#export PYTHONPATH=$PYTHONPATH:$CLIMAF
-export PYTHONPATH=$CLIMAF
-export PATH=$PATH:$CLIMAF/bin
-export CLIMAF_CACHE=/data/jypmce/climaf_cache
-export TMPDIR=$CLIMAF_CACHE
-export CLIMAF_LOG_DIR=$CLIMAF_CACHE
-#conda activate cdatm-nightly_py2
-</code>
-  * Open a new shell and <wrap em>run the tests again</wrap>. If there is an error that was not there, check what you have added to the shell init files
-  * [[other:uvcdat:cdat_conda:climaf_test|Test CLIMAF on ciclad]]
 ==== Cleaning up things ====
@@ Line 254: / Line 247: @@
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
   * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library
-  * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in
+  * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in Python
+  * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret
   * [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: high resolution data for ''basemap''
   * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests
@@ Line 263: / Line 257: @@
   * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module
   * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
-  * [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python
+  * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses
   * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python
   * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography
@@ Line 270: / Line 264: @@
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
   * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python
+  * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework
+  * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs
 ==== Removed packages ====
@@ Line 277: / Line 273: @@
 ===== Environments summary =====
-After following the steps above, we get the following environments. Use the ''conda info %%--%%envs'' or the ''conda env list'' command to get the up-to-date list of available environments
+After following the steps above, we get the following environments. Use the ''conda env list'' command (same result as ''conda info %%--%%envs'') to get the up-to-date list of available environments.
+Use ''conda list -n env_name'' to get a detailed list of which packages/modules are installed, or check the [[other:uvcdat:conda_notes#installation_history|conda Installation history]] section to get more details
-^  Environment\\ name  ^  Server  ^  ''conda list''  ^
+^  Environment\\ name  ^  Server  ^   Summary  ^
-| cdat-8.1_py2 |  LSCE\\ ciclad  | |
+| cdat-8.1_py2 |  LSCE\\ ciclad  | CDAT 8.1 & Python 2.7 |
-| cdat-8.1_py3 |  LSCE  | |
+| cdat-8.1_py3 |  LSCE  | CDAT 8.1 & Python 3.6 |
-| cdatm18_py2 |  LSCE\\ ciclad  | |
+| cdatm18_py2\\ or //cdatm_py2// |  LSCE\\ ciclad  | CDAT 8.1 & P 2.7 //JYP version// |
-| cdat-nightly_py2 |  LSCE\\ ciclad  | |
+| cdatm18_py3\\ or //cdatm_py3// |  LSCE  | CDAT 8.1 & P 3.7 //JYP version// |
-| cdatm-nightly_py2 |  LSCE\\ ciclad  | |
+| cdat-nightly_py2 |  LSCE\\ ciclad  | CDAT nightly & Python 2.7 |
+| cdatm-nightly_py2 |  LSCE\\ ciclad  | CDAT nightly & P 2.7 //JYP version// |
 /* standard page footer */