Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:cdat_2_10 [2017/08/08 16:47] – More changes... jypeter
+++ other:uvcdat:cdat_conda:cdat_2_10 [2018/10/03 11:56] (current) – Added windspharm jypeter
@@ Line 184: / Line 184: @@
 ===== Getting ready for a moving default CDAT =====
-We create a **symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this version and always get the latest stable version
+We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version
 <code>bash-4.2$ cd /home/share/unix_files/cdat/miniconda2b/envs
@@ Line 209: / Line 209: @@
 root                     /home/share/unix_files/cdat/miniconda2b
 </code>
+===== Customizing UV-CDAT for LSCE =====
 ==== Downloading cdms2/vcs test data ====
@@ Line 250: / Line 253: @@
 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
-.9G	/home/share/unix_files/cdat/miniconda2
+.9G	/home/share/unix_files/cdat/miniconda2</code>
-bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
+List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_170808.txt |}}
-.4G    /home/share/unix_files/cdat/miniconda2/envs/cdatm14
-M     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0</code>
-Lists of installed packages: {{ :other:uvcdat:lsce-extra_01_install_160223.txt |}}
+Disk space used after adding the extra packages, in <wrap em>in the  new directory hierarchy</wrap> (''cdat/miniconda2b'', on the obelix servers)
-The following packages have no dependency problems and were installed //later//
+<code>bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b
-  * High resolution data for ''basemap''
+.4G	/home/share/unix_files/cdat/miniconda2b
-    * ''conda install -n cdatm14 -c conda-forge basemap-data-hires''
-  * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP)
+bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/*
-    * ''conda install -n cdatm14 -c conda-forge -c pcmdi pcmdi_metrics''
+	/home/share/unix_files/cdat/miniconda2b/envs/cdatm
-  * [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files
+.8G	/home/share/unix_files/cdat/miniconda2b/envs/cdatm15
-    * ''conda install -n cdatm14 xlsxwriter''
+M	/home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10
-    * Note: <wrap hi>this is a dependency of ''dreqPy''</wrap>
+# Check the disk space again after cleaning
+bash-4.2$ conda clean --all
+bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b
+.4G	/home/share/unix_files/cdat/miniconda2b</code>
+Packages installed with //pip//
   * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API
     * ''pip install dreqPy''
@@ Line 271: / Line 279: @@
       * Get version number with:\\ <code>$ drq -v
 dreqPy version 01.00.06 [Version 01.00.06]</code>
-  * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files
-    * ''conda install -n cdatm14 -c conda-forge -c pcmdi -c uvcdat cmor''
-      * Did not complete the installation yet, because it requested a downgrade of libnetcdf (see [[https://github.com/PCMDI/cmor/issues/138|github issue]])
-      * Installation issue was solved in ''cmor-3.2.3''
   * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module
     * ''pip install dtw''
     * Requested by Sentia Goursaud
+The following packages have no dependency problems and were installed (or updated) //later//
+  * [[http://www.earthsystemmodeling.org/esmf_releases/non_public/ESMF_7_0_0/esmpy_doc/html/index.html|ESMPy]]: ESMF Python Regridding Interface
+    * We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10
+    * <code>bash-4.1$ conda list | grep -i esm
+esmf                      7.0.0                         6    conda-forge
+esmpy                     7.0.0                    py27_1    conda-forge
+bash-4.1$ conda install -n cdatm15 -c nesii/channel/dev-esmf -c conda-forge esmpy=7.1.0.dev32
+[...]
+bash-4.1$ conda list | grep -i esm
+esmf                      7.1.0.dev32                   1    nesii/channel/dev-esmf
+esmpy                     7.1.0.dev32              py27_1    nesii/channel/dev-esmf</code>
+  * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography
+    * ''conda install -n cdatm15 -c conda-forge cmocean''
+  * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology
+    * ''conda install -n cdatm15 -c conda-forge -c scitools iris''
+  * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
+    * ''conda install -c conda-forge gdal''
+    * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]]
+  * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
+    * ''conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib''
+    * Test: ''python -c 'from spanlib.analyzer import Analyzer' ''
+  * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
+    * ''conda install -n cdatm15 -c conda-forge wrf-python''
+  * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'')
+    * ''conda install -n cdatm15 -c conda-forge glances''
+    * [[https://github.com/nicolargo/glances|glances@github]]
+  * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python
+    * ''conda install -n cdatm15 -c conda-forge windspharm''
+    * [[https://github.com/ajdawson/windspharm|windspharm@github]]
 ==== TODO ====
@@ Line 283: / Line 320: @@
 //Add here packages that would be useful but have some problems that prevent their installation//
-  * [[https://earthsystemcog.org/projects/esmpy/|ESMPy]]: ESMF Python Regridding Interface
+  * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
-    * ''conda install -n cdatm14 -c conda-forge esmpy''
+    * As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])!
-    * Installation of esmpy ''7.0.0'' did not report errors, **but** esmpy seemed to have libraries based importation problems and has been removed
+    * <wrap hi>Problem solved</wrap> with **vacumm 3.4.1-1** on August 31st! Installation was OK with\\ ''conda install -n cdatm15 -c conda-forge -c vacumm vacumm''
-      * ''conda -n cdatm14 remove esmpy''
+  * Install PySPLIT for Sentia\\ [[https://github.com/mscross/pysplit|PySPLIT]] A package for generating HYSPLIT air parcel trajectories trajectories, performing moisture uptake analyses, expediting HYSPLIT cluster analysis, and for visualizing trajectories, clusters, and along-trajectory meteorological data\\ Works with [[https://ready.arl.noaa.gov/HYSPLIT.php|HYSPLiT]]
+    - ''conda install -n cdatm15 -c conda-forge fiona geopandas''
+    - ''pip install pysplit''
 ==== Other packages ====
@@ Line 292: / Line 332: @@
 <note warning>There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them...</note>
-  * ''shapely'' request a downgrade of ''geos'' when installed from //conda-forge// (see [[https://github.com/Toblerity/Shapely/issues/472|github issue]]) but it will work when installed from the default repo
+  * NO such packages now!
-    * ''conda install -n cdatm14 shapely''
-    * We get a slightly older version (//1.5.**16**-py27_0// instead of //1.5.**17**-np111py27_2//)
-    * Note: ''cartopy'' (below) requests the same downgrade of ''geos''...
-  * ''cartopy'' requests a downgrade of ''geos'' and ''matplotlib'' (see [[https://github.com/SciTools/cartopy/issues/853#issuecomment-282023158|github issue]])
-    * We install the requested dependencies and then we **force** the installation of ''cartopy''\\ <code>(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat libxslt lxml owslib pyepsg
-[...]
-The following NEW packages will be INSTALLED:
-    libxslt: 1.1.29-3      conda-forge
+==== Updating some packages ====
-    lxml:    3.7.2-py27_1  conda-forge
-    owslib:  0.14.0-py27_0 conda-forge
-    pyepsg:  0.3.1-py27_0  conda-forge
-[...]
-(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge cartopy
+Some packages change more often than others, and can be easily updated the following way:
-[...]
-The following NEW packages will be INSTALLED:
-    cartopy:    0.15.0-np111py27_0 conda-forge
+  * [[https://cmor.llnl.gov/|CMOR]]
+    * Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor''
+      * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' ''
-The following packages will be DOWNGRADED due to dependency conflicts:
+  * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]
+    * Update with: ''pip install %%--%%upgrade dreqPy''
+      * Get version number with: ''drq -v''
-    geos:       3.6.0-0            conda-forge --> 3.5.1-1           conda-forge
+==== Cleaning up things ====
-    matplotlib: 2.0.0-np111py27_2  conda-forge --> 1.5.3-np111py27_7 conda-forge
-Proceed ([y]/n)? n
+Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg [[https://github.com/PCMDI/pcmdi_metrics/issues/496|pcmdi-metrics]] in 2.8.0 and [[https://github.com/UV-CDAT/cdp/issues/21|cdp]] in 2.10)
-(uvcdat-2.8.0) bash-4.1$ conda install --force -n cdatm14 -c conda-forge cartopy
+We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions
-[...]
+<code>(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls
-The following NEW packages will be INSTALLED:
+50333969    4 -rw-------   2 jypeter  lsce          234 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO
+50333971    4 -rw-------   2 jypeter  lsce          291 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/SOURCES.txt
-    cartopy: 0.15.0-np111py27_0 conda-forge
+50333962    4 -rw-------   2 jypeter  lsce            4 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/top_level.txt
+50333961    4 -rw-------   2 jypeter  lsce            1 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/dependency_links.txt
+50333960    4 -rw-------   2 jypeter  lsce            1 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe
+50333987    8 -rw-------   2 jypeter  lsce         5149 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.pyc
+50333986    8 -rw-------   2 jypeter  lsce         4208 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.py
+50333966    4 -rw-------   2 jypeter  lsce          150 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.pyc
+50333968    4 -rw-------   2 jypeter  lsce          201 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.py
+50333988    8 -rw-------   2 jypeter  lsce         5365 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.pyc
+50333982    4 -rw-------   2 jypeter  lsce         1821 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.py
+50333981    4 -rw-------   2 jypeter  lsce         1773 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.py
+50333973    4 -rw-------   2 jypeter  lsce          670 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.py
+50333980    4 -rw-------   2 jypeter  lsce         1642 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.pyc
+50333979    4 -rw-------   2 jypeter  lsce         1474 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.pyc
+50333967    4 -rw-------   2 jypeter  lsce          187 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.py
+50333989    0 -rw-------   2 jypeter  lsce            0 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.py
+50333985    4 -rw-------   2 jypeter  lsce         3843 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.py
+50333984    4 -rw-------   2 jypeter  lsce         2436 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.pyc
+50333970    4 -rw-------   2 jypeter  lsce          288 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.py
+50333974    4 -rw-------   2 jypeter  lsce          692 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.pyc
+50333978    4 -rw-------   2 jypeter  lsce          878 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.pyc
+50333976    4 -rw-------   2 jypeter  lsce          734 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.py
+50333983    4 -rw-------   2 jypeter  lsce         2269 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.pyc
+50333975    4 -rw-------   2 jypeter  lsce          724 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.pyc
+(cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp*
+(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls</code>
-Proceed ([y]/n)? y</code>
-  * trying to install ''rpy2'' will trigger the installation of ''R''!
-    * We force the **installation of only rpy2** in order to use the ''R'' already installed at LSCE
-      * ''conda install %%--%%force -n cdatm14 -c conda-forge -c uvcdat rpy2''
 ===== Extra packages list =====
@@ Line 340: / Line 390: @@
   * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox
   * [[https://pypi.python.org/pypi/gsw/|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater
-  * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library
-  * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in Python
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
+  * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library
+  * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in
   * [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: high resolution data for ''basemap''
   * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests
   * [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files
+    * Note: <wrap hi>this is a dependency of ''dreqPy''</wrap>
   * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API
   * [[https://cmor.llnl.gov/|CMOR]]:  CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files
   * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module
   * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
   * [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python
   * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python
+  * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography
+  * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology
+  * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
+  * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
+  * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
+  * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'')
+  * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python
 ==== Removed packages ====
-  * [[https://pypi.python.org/pypi/wget|wget]]: pure python download utility
+  * NO removed packages!
-    * Used ''conda install -n cdatm14 -c daler wget=3.2'' in order to get version **3.2**, the same as the one on Pypi
-    * **REMOVED** because it did not work correctly in the script where it was needed: it emulated wget with //ftp// and failed in our case after downloading the first file
 ===== Environments summary =====
-After following the steps above, we get the following environments. Use the ''conda info --envs'' command to get the up-to-date list of available environments
+After following the steps above, we get the following environments. Use the ''conda info %%--%%envs'' or the ''conda env list'' command to get the up-to-date list of available environments
 ^  Environment\\ name  ^  Server  ^  ''conda list''  ^
-|  uvcdat-2.8.0  |  LSCE\\ ciclad  |  {{ :other:uvcdat:cdat_conda:conda_list_lsce_uvcdat-2.8.0_170310.txt |package list}}  |
+|  uvcdat-2.10  |  LSCE  |  {{  |package list}}  |
-|  cdatm14  |  LSCE\\ ciclad  |  {{ :other:uvcdat:cdat_conda:conda_list_lsce_cdatm14_170310.txt|package list}}  |
+|  cdatm15  |  LSCE  |  {{ |package list}}  |
 /* standard page footer */