Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:cdat_8_1 [2019/03/07 15:59]
jypeter
+++ other:uvcdat:cdat_conda:cdat_8_1 [2021/03/26 09:10] (current)
jypeter [You want to replicate this installation on your own server or desktop/laptop]
@@ Line 4: / Line 4: @@
 <WRAP center round tip 60%>
-Follow the instructions about the [[other:python:starting#conda-based_versions_of_uv-cdat|conda-based versions of UV-CDAT initialization]] for actually using an installed version of ''8.1''
+If you have questions about Python, read:
+  * [[other:python:starting|Working with Python]]
+  * [[other:python:jyp_steps|JYP's recommended steps for learning python]]!
 </WRAP>
-===== What's New? =====
+===== Using JYP version =====
+If you mostly want to use the CDAT 8.1 installed by JYP, just use the following steps and then read the //What should I do now?// section below
+  * Initialize //conda// with:
+^  Server  ^  tcsh  ^  bash  ^
+|  LSCE  | ''source ~jypeter/.conda3_jyp.csh'' | ''source ~jypeter/.conda3_jyp.sh'' |
+|  ciclad  | ''source ~jypmce/.conda3_jyp.csh'' | ''source ~jypmce/.conda3_jyp.sh'' |
+  * Choose one of the [[#environments_summary|installed environments]] and activate it with: ''conda activate env_name''
+    * python 2.7.x: ''conda activate cdatm_py2''
+    * python 3.x: ''conda activate cdatm_py3'' (not available on ciclad, but could be installed)
+  * Type ''which python'' (or the ''wp'' alias) and make sure you get something like\\ ''[...]/miniconda3/envs/env_name/bin/python''
+===== What should I do now? =====
+==== You just want to use the installed version ====
+You are ready if you have typed the ''conda activate'' command specified in the previous section!
+  * You may want to add the ''source'' line to your shell configuration file so that you don't have to remember it and type it each time you open a new terminal
+    * Note: it's better to type the ''activate'' command in a terminal only when you want to use this specific version of python, rather than having the ''activate'' in your shell configuration file. This may have some side effects in some cases
+  * Have a look at the [[other:uvcdat:cdat_conda:cdat_8_1#extra_packages_list|Extra package list]] at the bottom of this page to see what is available in this distribution
+  * If you have not done it yet, read (or at least have a look at the list of sections):
+    * [[other:python:starting|Working with Python]]
+    * [[other:python:jyp_steps|JYP's recommended steps for learning python]]
+==== You want to replicate this installation on your own server or desktop/laptop ====
+Read carefully the following sections of this page! 8-)
+===== What's New in CDAT 8.1? =====
   * [[https://github.com/cdat/cdat/releases/tag/v8.1|8.1 announcement]] and features summary ([[https://github.com/UV-CDAT/uvcdat/releases|all versions]])
   * Full [[https://uvcdat.llnl.gov/changelog.html#8.1|Change log]]
+Note: this particular CDAT installation at LSCE (and on other machines/servers) provides **many extra (non-CDAT) packages**. You can jump directly to the [[other:uvcdat:cdat_conda:cdat_8_1#extra_packages_list|Extra package list]] at the bottom of this page to see what is available
 ===== Installation with Miniconda3 =====
@@ Line 16: / Line 52: @@
 We assume that Miniconda3 is already installed. Otherwise, follow the the [[other:uvcdat:cdat_conda:cdat_8_0_py2#installing_miniconda3|Installing Miniconda]] steps (and the //Post-Miniconda3 installation// section) we followed when installing CDAT 8.0
-==== Installing CDAT 8.0 ====
+==== Updating conda ====
+The conda package itself can be updated (if need be) with\\ ''conda update -n base -c defaults conda''
+==== Installing CDAT 8.1 ====
 We first check that we have indeed access to a conda installation, and we assume that we have a write-access to the conda hierarchy, and a few Gb of disk space available
@@ Line 97: / Line 136: @@
 ===== Customizing UV-CDAT for LSCE =====
+==== Testing vcs ====
+Start python and paste the following lines
+<code>import numpy as np, vcs
+id100 = np.identity(100)
+x = vcs.init()
+x.plot(id100)
+</code>
+If you get the kind of errors described in [[https://github.com/CDAT/vcs/issues/393|x.plot crashes with OpenGL2 error]], the drivers on your server are probably not compatible with an interactive use of vcs... The workaround is to **install mesalib and work in //headless mode//** (e.g. without a display)...
+Install //mesalib// with ''conda install -c cdat/label/v81 -c conda-forge mesalib vtk-cdat'' and try pasting again the few code lines above in python. If things work as expected, no canvas will be opened, but you'll be able to save the canvas to a file with
+<code>x.png('test_mesalib')
+x.pdf('test_mesalib')</code>
+Note: using //mesalib// for headless mode work makes it possible to work without an X server, and with no DISPLAY variable defined
 ==== Downloading cdms2/vcs test data ====
@@ Line 104: / Line 160: @@
 <code>$ conda activate cdatm18_py2
-(cdatm17_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data'
+(cdatm18_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)'
 [...]
-Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data
+Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
-(cdatm17_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/uvcdat/sample_data
+(cdatm18_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
-M    /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data
+M    /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
 </code>
@@ Line 124: / Line 180: @@
 # Install...
-$ conda install -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
+$ conda install -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret ferret_datasets basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
 [...]
@@ Line 137: / Line 193: @@
       * Get version number with:\\ <code>$ drq -v
 dreqPy version 01.00.29 [Version 01.00.29]</code>
+  * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret
+    * ''conda activate cdatm18_py2''\\ ''pip install ferretmagic''
 The following packages have no dependency problems and were installed (or updated) //later//
+  * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files
+    * ''conda install -n cdatm18_py2 -c conda-forge cmor''
+    * Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' ''
+    * <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]]
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
     * ''conda install -n cdatm18_py2 -c conda-forge -c vacumm vacumm''
+    * <wrap hi>Warning!</wrap> [[https://github.com/VACUMM/vacumm/issues/15|standard install of vacumm does not work for Python3]]
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
-    * ''conda install -n cdatm17_py2 -c stefraynaud -c conda-forge spanlib''
+    * ''conda install -n cdatm18_py2 -c stefraynaud -c conda-forge spanlib''
     * Test: ''python -c 'from spanlib.analyzer import Analyzer' ''
+  * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses
+    * ''conda install -n cdatm18_py2 -c conda-forge cartopy''
+  * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs
+    * ''conda install -n cdatm18_py2 -c conda-forge joblib''
+  * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework
+    * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]]
+  * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python
+    * ''conda install -n cdatm18_py2 -c conda-forge palettable''
 ==== TODO ====
@@ Line 170: / Line 247: @@
 Some packages change more often than others, and can be easily updated the following way:
-  * [[https://cmor.llnl.gov/|CMOR]]
-    * Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor''
-      * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' ''
   * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]
@@ Line 179: / Line 252: @@
       * Get version number with: ''drq -v''
-==== Installing CLIMAF ====
-The [[https://climaf.readthedocs.io/|CLIMAF]] installation is a special case, because we may want to use it with the Python installed for CDAT, but it is not installed using conda
-We follow the steps for the [[https://climaf.readthedocs.io/en/latest/installing.html#installing-climaf-if-you-are-not-at-cnrm-on-ciclad-or-climserv|installation of a non-standard server]]
-  * Go to the appropriate installation directory (we choose the directory where we have also installed miniconda3):
-    * LSCE: ''cd /home/share/unix_files/cdat''
-    * ciclad: ''/data/jypmce/cdat/''
-  * ''git clone https://github.com/senesis/climaf climaf''
-  * Add CLIMAF to the PYTHONPATH environment variable
-    * LSCE & tcsh: ''setenv PYTHONPATH ${PYTHONPATH}:/home/share/unix_files/cdat/climaf''
-    * LSCE & bash: ''export PYTHONPATH=${PYTHONPATH}:/home/share/unix_files/cdat/climaf''
-    * ciclad: same as above, using ''/data/jypmce/cdat/climaf''
-  * Test the installation
-    * make sure the [[https://climaf.readthedocs.io/en/latest/requirements.html#requirements|Requirements]] are available:
-      * LSCE: ''module load netcdf/4 nco cdo ncl ncview''
-      * ciclad: ''module load netcdf4 nco cdo ncl/6.3.0''
-      * optional: start an X server
-      * <code>cd climaf/testing
-./test_install.sh</code>
-  * Once the tests are OK, add the required //module load// and environment variables definitions to the ''.basrc'' and ''.cshrc'' files. <wrap hi>Note: maybe these lines should go to the ''.profile'' and ''.login'' files...(check [[https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html|Bash Startup Files]])</wrap>
-    * LSCE: add to your .cshrc
-      * <code># Prerequisites for using CLIMAF
-module load netcdf/4 nco cdo ncl ncview
-# CLIMAF environment variables
-setenv CLIMAF /home/share/unix_files/cdat/climaf
-setenv PYTHONPATH "${PYTHONPATH}:${CLIMAF}"
-#setenv PYTHONPATH $CLIMAF
-setenv PATH "${PATH}:${CLIMAF}/bin"
-setenv CLIMAF_CACHE "/home/scratch01/${USER}/climaf_cache"
-setenv TMPDIR $CLIMAF_CACHE
-setenv CLIMAF_LOG_DIR $CLIMAF_CACHE
-#conda activate cdatm-nightly_py2</code>
-    * ciclad: add to your .bashrc
-      * <code># Prerequisites for using CLIMAF when not using the
-# default 'module load climaf'
-module load netcdf4 nco cdo ncl/6.3.0
-# CLIMAF environment variables
-export CLIMAF=/data/jypmce/cdat/climaf
-#export PYTHONPATH=$PYTHONPATH:$CLIMAF
-export PYTHONPATH=$CLIMAF
-export PATH=$PATH:$CLIMAF/bin
-export CLIMAF_CACHE=/data/jypmce/climaf_cache
-export TMPDIR=$CLIMAF_CACHE
-export CLIMAF_LOG_DIR=$CLIMAF_CACHE
-#conda activate cdatm-nightly_py2
-</code>
-  * Open a new shell and <wrap em>run the tests again</wrap>. If there is an error that was not there, check what you have added to the shell init files
-  * [[other:uvcdat:cdat_conda:climaf_test|Test CLIMAF on ciclad]]
 ==== Cleaning up things ====
@@ Line 254: / Line 273: @@
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
   * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library
-  * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in
+  * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in Python
+  * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret
   * [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: high resolution data for ''basemap''
   * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests
@@ Line 263: / Line 283: @@
   * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module
   * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
-  * [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python
+  * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses
   * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python
   * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography
@@ Line 270: / Line 290: @@
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
   * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python
+  * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework
+  * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs
+  * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python
 ==== Removed packages ====
@@ Line 277: / Line 300: @@
 ===== Environments summary =====
-After following the steps above, we get the following environments. Use the ''conda info %%--%%envs'' or the ''conda env list'' command to get the up-to-date list of available environments
+After following the steps above, we get the following environments. Use the ''conda env list'' command (same result as ''conda info %%--%%envs'') to get the up-to-date list of available environments.
+Use ''conda list -n env_name'' to get a detailed list of which packages/modules are installed, or check the [[other:uvcdat:conda_notes#installation_history|conda Installation history]] section to get more details
-^  Environment\\ name  ^  Server  ^  ''conda list''  ^
+^  Environment\\ name  ^  Server  ^   Summary  ^
-| cdat-8.1_py2 |  LSCE\\ ciclad  | |
+| cdat-8.1_py2 |  LSCE\\ ciclad  | CDAT 8.1 & Python 2.7 |
-| cdat-8.1_py3 |  LSCE  | |
+| cdat-8.1_py3 |  LSCE  | CDAT 8.1 & Python 3.6 |
-| cdatm18_py2 |  LSCE\\ ciclad  | |
+| cdatm18_py2\\ or //cdatm_py2// |  LSCE\\ ciclad  | CDAT 8.1 & P 2.7 //JYP version// |
-| cdat-nightly_py2 |  LSCE\\ ciclad  | |
+| cdatm18_py3\\ or //cdatm_py3// |  LSCE  | CDAT 8.1 & P 3.7 //JYP version// |
-| cdatm-nightly_py2 |  LSCE\\ ciclad  | |
+| cdat-nightly_py2 |  LSCE\\ ciclad  | CDAT nightly & Python 2.7 |
+| cdatm-nightly_py2 |  LSCE\\ ciclad  | CDAT nightly & P 2.7 //JYP version// |
 /* standard page footer */

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools