Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:cdat_2_10 [2017/08/09 17:36] – More changes... jypeter
+++ other:uvcdat:cdat_conda:cdat_2_10 [2018/10/03 11:56] (current) – Added windspharm jypeter
@@ Line 209: / Line 209: @@
 root                     /home/share/unix_files/cdat/miniconda2b
 </code>
+===== Customizing UV-CDAT for LSCE =====
 ==== Downloading cdms2/vcs test data ====
@@ Line 252: / Line 255: @@
 .9G	/home/share/unix_files/cdat/miniconda2</code>
-Lists of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_170808.txt |}}
+List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_170808.txt |}}
+Disk space used after adding the extra packages, in <wrap em>in the  new directory hierarchy</wrap> (''cdat/miniconda2b'', on the obelix servers)
+<code>bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b
+.4G	/home/share/unix_files/cdat/miniconda2b
+bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/*
+	/home/share/unix_files/cdat/miniconda2b/envs/cdatm
+.8G	/home/share/unix_files/cdat/miniconda2b/envs/cdatm15
+M	/home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10
+# Check the disk space again after cleaning
+bash-4.2$ conda clean --all
+bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b
+.4G	/home/share/unix_files/cdat/miniconda2b</code>
 Packages installed with //pip//
@@ Line 264: / Line 283: @@
     * Requested by Sentia Goursaud
-The following packages have no dependency problems and were installed //later//
+The following packages have no dependency problems and were installed (or updated) //later//
   * [[http://www.earthsystemmodeling.org/esmf_releases/non_public/ESMF_7_0_0/esmpy_doc/html/index.html|ESMPy]]: ESMF Python Regridding Interface
     * We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10
@@ Line 277: / Line 296: @@
 esmf                      7.1.0.dev32                   1    nesii/channel/dev-esmf
 esmpy                     7.1.0.dev32              py27_1    nesii/channel/dev-esmf</code>
+  * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography
+    * ''conda install -n cdatm15 -c conda-forge cmocean''
+  * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology
+    * ''conda install -n cdatm15 -c conda-forge -c scitools iris''
+  * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
+    * ''conda install -c conda-forge gdal''
+    * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]]
+  * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
+    * ''conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib''
+    * Test: ''python -c 'from spanlib.analyzer import Analyzer' ''
+  * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
+    * ''conda install -n cdatm15 -c conda-forge wrf-python''
+  * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'')
+    * ''conda install -n cdatm15 -c conda-forge glances''
+    * [[https://github.com/nicolargo/glances|glances@github]]
+  * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python
+    * ''conda install -n cdatm15 -c conda-forge windspharm''
+    * [[https://github.com/ajdawson/windspharm|windspharm@github]]
 ==== TODO ====
@@ Line 284: / Line 322: @@
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
     * As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])!
+    * <wrap hi>Problem solved</wrap> with **vacumm 3.4.1-1** on August 31st! Installation was OK with\\ ''conda install -n cdatm15 -c conda-forge -c vacumm vacumm''
+  * Install PySPLIT for Sentia\\ [[https://github.com/mscross/pysplit|PySPLIT]] A package for generating HYSPLIT air parcel trajectories trajectories, performing moisture uptake analyses, expediting HYSPLIT cluster analysis, and for visualizing trajectories, clusters, and along-trajectory meteorological data\\ Works with [[https://ready.arl.noaa.gov/HYSPLIT.php|HYSPLiT]]
+    - ''conda install -n cdatm15 -c conda-forge fiona geopandas''
+    - ''pip install pysplit''
 ==== Other packages ====
@@ Line 289: / Line 332: @@
 <note warning>There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them...</note>
-  * ''shapely'' request a downgrade of ''geos'' when installed from //conda-forge// (see [[https://github.com/Toblerity/Shapely/issues/472|github issue]]) but it will work when installed from the default repo
+  * NO such packages now!
-    * ''conda install -n cdatm14 shapely''
-    * We get a slightly older version (//1.5.**16**-py27_0// instead of //1.5.**17**-np111py27_2//)
-    * Note: ''cartopy'' (below) requests the same downgrade of ''geos''...
-  * ''cartopy'' requests a downgrade of ''geos'' and ''matplotlib'' (see [[https://github.com/SciTools/cartopy/issues/853#issuecomment-282023158|github issue]])
-    * We install the requested dependencies and then we **force** the installation of ''cartopy''\\ <code>(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat libxslt lxml owslib pyepsg
-[...]
-The following NEW packages will be INSTALLED:
-    libxslt: 1.1.29-3      conda-forge
+==== Updating some packages ====
-    lxml:    3.7.2-py27_1  conda-forge
-    owslib:  0.14.0-py27_0 conda-forge
-    pyepsg:  0.3.1-py27_0  conda-forge
-[...]
-(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge cartopy
+Some packages change more often than others, and can be easily updated the following way:
-[...]
-The following NEW packages will be INSTALLED:
-    cartopy:    0.15.0-np111py27_0 conda-forge
+  * [[https://cmor.llnl.gov/|CMOR]]
+    * Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor''
+      * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' ''
-The following packages will be DOWNGRADED due to dependency conflicts:
+  * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]
+    * Update with: ''pip install %%--%%upgrade dreqPy''
+      * Get version number with: ''drq -v''
-    geos:       3.6.0-0            conda-forge --> 3.5.1-1           conda-forge
+==== Cleaning up things ====
-    matplotlib: 2.0.0-np111py27_2  conda-forge --> 1.5.3-np111py27_7 conda-forge
-Proceed ([y]/n)? n
+Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg [[https://github.com/PCMDI/pcmdi_metrics/issues/496|pcmdi-metrics]] in 2.8.0 and [[https://github.com/UV-CDAT/cdp/issues/21|cdp]] in 2.10)
-(uvcdat-2.8.0) bash-4.1$ conda install --force -n cdatm14 -c conda-forge cartopy
-[...]
-The following NEW packages will be INSTALLED:
-    cartopy: 0.15.0-np111py27_0 conda-forge
+We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions
+<code>(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls
+50333969    4 -rw-------   2 jypeter  lsce          234 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO
+50333971    4 -rw-------   2 jypeter  lsce          291 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/SOURCES.txt
+50333962    4 -rw-------   2 jypeter  lsce            4 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/top_level.txt
+50333961    4 -rw-------   2 jypeter  lsce            1 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/dependency_links.txt
+50333960    4 -rw-------   2 jypeter  lsce            1 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe
+50333987    8 -rw-------   2 jypeter  lsce         5149 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.pyc
+50333986    8 -rw-------   2 jypeter  lsce         4208 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.py
+50333966    4 -rw-------   2 jypeter  lsce          150 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.pyc
+50333968    4 -rw-------   2 jypeter  lsce          201 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.py
+50333988    8 -rw-------   2 jypeter  lsce         5365 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.pyc
+50333982    4 -rw-------   2 jypeter  lsce         1821 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.py
+50333981    4 -rw-------   2 jypeter  lsce         1773 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.py
+50333973    4 -rw-------   2 jypeter  lsce          670 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.py
+50333980    4 -rw-------   2 jypeter  lsce         1642 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.pyc
+50333979    4 -rw-------   2 jypeter  lsce         1474 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.pyc
+50333967    4 -rw-------   2 jypeter  lsce          187 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.py
+50333989    0 -rw-------   2 jypeter  lsce            0 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.py
+50333985    4 -rw-------   2 jypeter  lsce         3843 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.py
+50333984    4 -rw-------   2 jypeter  lsce         2436 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.pyc
+50333970    4 -rw-------   2 jypeter  lsce          288 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.py
+50333974    4 -rw-------   2 jypeter  lsce          692 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.pyc
+50333978    4 -rw-------   2 jypeter  lsce          878 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.pyc
+50333976    4 -rw-------   2 jypeter  lsce          734 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.py
+50333983    4 -rw-------   2 jypeter  lsce         2269 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.pyc
+50333975    4 -rw-------   2 jypeter  lsce          724 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.pyc
+(cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp*
+(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls</code>
-Proceed ([y]/n)? y</code>
-  * trying to install ''rpy2'' will trigger the installation of ''R''!
-    * We force the **installation of only rpy2** in order to use the ''R'' already installed at LSCE
-      * ''conda install %%--%%force -n cdatm14 -c conda-forge -c uvcdat rpy2''
 ===== Extra packages list =====
@@ Line 337: / Line 390: @@
   * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox
   * [[https://pypi.python.org/pypi/gsw/|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater
+  * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
   * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library
   * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in
@@ Line 346: / Line 400: @@
   * [[https://cmor.llnl.gov/|CMOR]]:  CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files
   * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module
   * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
   * [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python
   * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python
+  * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography
+  * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology
+  * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
+  * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
+  * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
+  * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'')
+  * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python
 ==== Removed packages ====
-  * [[https://pypi.python.org/pypi/wget|wget]]: pure python download utility
+  * NO removed packages!
-    * Used ''conda install -n cdatm14 -c daler wget=3.2'' in order to get version **3.2**, the same as the one on Pypi
-    * **REMOVED** because it did not work correctly in the script where it was needed: it emulated wget with //ftp// and failed in our case after downloading the first file
 ===== Environments summary =====
-After following the steps above, we get the following environments. Use the ''conda info --envs'' command to get the up-to-date list of available environments
+After following the steps above, we get the following environments. Use the ''conda info %%--%%envs'' or the ''conda env list'' command to get the up-to-date list of available environments
 ^  Environment\\ name  ^  Server  ^  ''conda list''  ^
-|  uvcdat-2.8.0  |  LSCE\\ ciclad  |  {{ :other:uvcdat:cdat_conda:conda_list_lsce_uvcdat-2.8.0_170310.txt |package list}}  |
+|  uvcdat-2.10  |  LSCE  |  {{  |package list}}  |
-|  cdatm14  |  LSCE\\ ciclad  |  {{ :other:uvcdat:cdat_conda:conda_list_lsce_cdatm14_170310.txt|package list}}  |
+|  cdatm15  |  LSCE  |  {{ |package list}}  |
 /* standard page footer */