User Tools

Site Tools


other:uvcdat:cdat_conda:cdat_2_10

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
other:uvcdat:cdat_conda:cdat_2_10 [2017/08/09 15:36]
jypeter More changes...
other:uvcdat:cdat_conda:cdat_2_10 [2018/10/03 09:56] (current)
jypeter Added windspharm
Line 209: Line 209:
 root                     /​home/​share/​unix_files/​cdat/​miniconda2b root                     /​home/​share/​unix_files/​cdat/​miniconda2b
 </​code>​ </​code>​
 +
 +===== Customizing UV-CDAT for LSCE =====
 +
  
 ==== Downloading cdms2/vcs test data ==== ==== Downloading cdms2/vcs test data ====
Line 252: Line 255:
 9.9G /​home/​share/​unix_files/​cdat/​miniconda2</​code>​ 9.9G /​home/​share/​unix_files/​cdat/​miniconda2</​code>​
  
-Lists of installed packages: {{ :​other:​uvcdat:​cdat_conda:​lsce-extra_01_install_170808.txt |}}+List of installed packages: {{ :​other:​uvcdat:​cdat_conda:​lsce-extra_01_install_170808.txt |}} 
 + 
 +Disk space used after adding the extra packages, in <wrap em>in the  new directory hierarchy</​wrap>​ (''​cdat/​miniconda2b'',​ on the obelix servers) 
 + 
 +<​code>​bash-4.2$ du -sh /​home/​share/​unix_files/​cdat/​miniconda2b  
 +6.4G /​home/​share/​unix_files/​cdat/​miniconda2b 
 + 
 +bash-4.2$ du -sh /​home/​share/​unix_files/​cdat/​miniconda2b/​envs/​* 
 +0 /​home/​share/​unix_files/​cdat/​miniconda2b/​envs/​cdatm 
 +3.8G /​home/​share/​unix_files/​cdat/​miniconda2b/​envs/​cdatm15 
 +630M /​home/​share/​unix_files/​cdat/​miniconda2b/​envs/​uvcdat-2.10 
 + 
 +# Check the disk space again after cleaning 
 +bash-4.2$ conda clean --all 
 + 
 +bash-4.2$ du -sh /​home/​share/​unix_files/​cdat/​miniconda2b 
 +5.4G /​home/​share/​unix_files/​cdat/​miniconda2b</​code>​
  
 Packages installed with //pip// Packages installed with //pip//
Line 264: Line 283:
     * Requested by Sentia Goursaud     * Requested by Sentia Goursaud
  
-The following packages have no dependency problems and were installed //later//+The following packages have no dependency problems and were installed ​(or updated) ​//later//
   * [[http://​www.earthsystemmodeling.org/​esmf_releases/​non_public/​ESMF_7_0_0/​esmpy_doc/​html/​index.html|ESMPy]]:​ ESMF Python Regridding Interface   * [[http://​www.earthsystemmodeling.org/​esmf_releases/​non_public/​ESMF_7_0_0/​esmpy_doc/​html/​index.html|ESMPy]]:​ ESMF Python Regridding Interface
     * We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10     * We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10
Line 277: Line 296:
 esmf                      7.1.0.dev32 ​                  ​1 ​   nesii/​channel/​dev-esmf esmf                      7.1.0.dev32 ​                  ​1 ​   nesii/​channel/​dev-esmf
 esmpy                     ​7.1.0.dev32 ​             py27_1 ​   nesii/​channel/​dev-esmf</​code>​ esmpy                     ​7.1.0.dev32 ​             py27_1 ​   nesii/​channel/​dev-esmf</​code>​
 +  * [[http://​matplotlib.org/​cmocean/​|cmocean]]:​ beautiful colormaps for oceanography
 +    * ''​conda install -n cdatm15 -c conda-forge cmocean''​
 +  * [[http://​scitools.org.uk/​iris/​index.html|iris]]:​ A Python library for Meteorology and Climatology
 +    * ''​conda install -n cdatm15 -c conda-forge -c scitools iris''​
 +  * [[http://​www.gdal.org/​|OSGeo/​GDAL]]:​ Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
 +    * ''​conda install -c conda-forge gdal''​
 +    * [[https://​pcjericks.github.io/​py-gdalogr-cookbook/​|Python GDAL/OGR Cookbook]]
 +  * [[https://​github.com/​stefraynaud/​spanlib|spanlib]]:​ Spectral Analysis Library
 +    * ''​conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib''​
 +    * Test: ''​python -c 'from spanlib.analyzer import Analyzer'​ ''​
 +  * [[http://​wrf-python.readthedocs.io/​en/​latest/​|wrf-python]]:​ A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
 +    * ''​conda install -n cdatm15 -c conda-forge wrf-python''​
 +  * [[https://​glances.readthedocs.io/​en/​stable/​index.html|glances]]:​ a cross-platform monitoring tool (similar to ''​top''​)
 +    * ''​conda install -n cdatm15 -c conda-forge glances''​
 +    * [[https://​github.com/​nicolargo/​glances|glances@github]]
 +  * [[https://​ajdawson.github.io/​windspharm/​latest/​|windspharm]]:​ spherical harmonic wind analysis in Python
 +    * ''​conda install -n cdatm15 -c conda-forge windspharm''​
 +    * [[https://​github.com/​ajdawson/​windspharm|windspharm@github]]
 +
  
 ==== TODO ==== ==== TODO ====
Line 284: Line 322:
   * [[http://​www.ifremer.fr/​vacumm/​|vacumm]]:​ Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations   * [[http://​www.ifremer.fr/​vacumm/​|vacumm]]:​ Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
     * As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://​github.com/​VACUMM/​vacumm/​issues/​9|github issue]])!     * As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://​github.com/​VACUMM/​vacumm/​issues/​9|github issue]])!
 +    * <wrap hi>​Problem solved</​wrap>​ with **vacumm 3.4.1-1** on August 31st! Installation was OK with\\ ''​conda install -n cdatm15 -c conda-forge -c vacumm vacumm''​
 +
 +  * Install PySPLIT for Sentia\\ [[https://​github.com/​mscross/​pysplit|PySPLIT]] A package for generating HYSPLIT air parcel trajectories trajectories,​ performing moisture uptake analyses, expediting HYSPLIT cluster analysis, and for visualizing trajectories,​ clusters, and along-trajectory meteorological data\\ Works with [[https://​ready.arl.noaa.gov/​HYSPLIT.php|HYSPLiT]]
 +    - ''​conda install -n cdatm15 -c conda-forge fiona geopandas''​
 +    - ''​pip install pysplit''​
  
 ==== Other packages ==== ==== Other packages ====
Line 289: Line 332:
 <note warning>​There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them...</​note>​ <note warning>​There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them...</​note>​
  
-  * ''​shapely''​ request a downgrade of ''​geos''​ when installed from //​conda-forge//​ (see [[https://​github.com/​Toblerity/​Shapely/​issues/​472|github issue]]) but it will work when installed from the default repo +  * NO such packages ​now!
-    * ''​conda install -n cdatm14 shapely''​ +
-    * We get a slightly older version (//​1.5.**16**-py27_0//​ instead of //​1.5.**17**-np111py27_2//​) +
-    * Note: ''​cartopy''​ (below) requests the same downgrade of ''​geos''​... +
-  * ''​cartopy''​ requests a downgrade of ''​geos''​ and ''​matplotlib''​ (see [[https://​github.com/​SciTools/​cartopy/​issues/​853#​issuecomment-282023158|github issue]]) +
-    * We install the requested dependencies and then we **force** the installation of ''​cartopy''​\\ <​code>​(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat libxslt lxml owslib pyepsg +
-[...] +
-The following NEW packages ​will be INSTALLED:+
  
-    libxslt: 1.1.29-3 ​     conda-forge +==== Updating some packages ====
-    lxml:    3.7.2-py27_1 ​ conda-forge +
-    owslib: ​ 0.14.0-py27_0 conda-forge +
-    pyepsg: ​ 0.3.1-py27_0 ​ conda-forge +
-[...]+
  
-(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge cartopy +Some packages ​change more often than others, and can be easily updated the following way:
-[...] +
-The following NEW packages ​will be INSTALLED:+
  
-    cartopy   0.15.0-np111py27_0 ​conda-forge+  * [[https://cmor.llnl.gov/​|CMOR]] 
 +    * Update with: ''​conda update ​-n cdatm15 -c conda-forge ​-c pcmdi -c uvcdat cmor''​ 
 +      * Get version number with: ''​python -c 'from cmor import *; print CMOR_VERSION_MAJOR,​ CMOR_VERSION_MINOR,​ CMOR_VERSION_PATCH'​ ''​
  
-The following packages will be DOWNGRADED due to dependency conflicts:+  * [[https://​earthsystemcog.org/​projects/​wip/​CMIP6DataRequest|dreqPy]] 
 +    * Update with: ''​pip install %%--%%upgrade dreqPy''​ 
 +      * Get version number with: ''​drq -v''​
  
-    geos:       ​3.6.0-0 ​           conda-forge --> 3.5.1-1 ​          ​conda-forge +==== Cleaning up things ====
-    matplotlib: 2.0.0-np111py27_2 ​ conda-forge --> 1.5.3-np111py27_7 conda-forge+
  
-Proceed ​([y]/n)? n +Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions ​(eg [[https://​github.com/​PCMDI/​pcmdi_metrics/​issues/​496|pcmdi-metrics]] in 2.8.0 and [[https://​github.com/​UV-CDAT/​cdp/​issues/​21|cdp]] in 2.10)
- +
-(uvcdat-2.8.0) bash-4.1$ conda install --force -n cdatm14 -c conda-forge cartopy +
-[...] +
-The following NEW packages will be INSTALLED:+
  
-    cartopy: 0.15.0-np111py27_0 conda-forge+We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions 
 +<​code>​(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls 
 +50333969 ​   4 -rw------- ​  2 jypeter ​ lsce          234 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp-1.0.3-py2.7.egg-info/​PKG-INFO 
 +50333971 ​   4 -rw------- ​  2 jypeter ​ lsce          291 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp-1.0.3-py2.7.egg-info/​SOURCES.txt 
 +50333962 ​   4 -rw------- ​  2 jypeter ​ lsce            4 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp-1.0.3-py2.7.egg-info/​top_level.txt 
 +50333961 ​   4 -rw------- ​  2 jypeter ​ lsce            1 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp-1.0.3-py2.7.egg-info/​dependency_links.txt 
 +50333960 ​   4 -rw------- ​  2 jypeter ​ lsce            1 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp-1.0.3-py2.7.egg-info/​not-zip-safe 
 +50333987 ​   8 -rw------- ​  2 jypeter ​ lsce         5149 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_viewer.pyc 
 +50333986 ​   8 -rw------- ​  2 jypeter ​ lsce         4208 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_metric.py 
 +50333966 ​   4 -rw------- ​  2 jypeter ​ lsce          150 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​__init__.pyc 
 +50333968 ​   4 -rw------- ​  2 jypeter ​ lsce          201 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_provenance.py 
 +50333988 ​   8 -rw------- ​  2 jypeter ​ lsce         5365 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_metric.pyc 
 +50333982 ​   4 -rw------- ​  2 jypeter ​ lsce         1821 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_parser.py 
 +50333981 ​   4 -rw------- ​  2 jypeter ​ lsce         1773 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_parameter.py 
 +50333973 ​   4 -rw------- ​  2 jypeter ​ lsce          670 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_output.py 
 +50333980 ​   4 -rw------- ​  2 jypeter ​ lsce         1642 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_output.pyc 
 +50333979 ​   4 -rw------- ​  2 jypeter ​ lsce         1474 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​_cache.pyc 
 +50333967 ​   4 -rw------- ​  2 jypeter ​ lsce          187 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_tool.py 
 +50333989 ​   0 -rw------- ​  2 jypeter ​ lsce            0 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​__init__.py 
 +50333985 ​   4 -rw------- ​  2 jypeter ​ lsce         3843 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_viewer.py 
 +50333984 ​   4 -rw------- ​  2 jypeter ​ lsce         2436 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_parameter.pyc 
 +50333970 ​   4 -rw------- ​  2 jypeter ​ lsce          288 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_io.py 
 +50333974 ​   4 -rw------- ​  2 jypeter ​ lsce          692 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_tool.pyc 
 +50333978 ​   4 -rw------- ​  2 jypeter ​ lsce          878 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_io.pyc 
 +50333976 ​   4 -rw------- ​  2 jypeter ​ lsce          734 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​_cache.py 
 +50333983 ​   4 -rw------- ​  2 jypeter ​ lsce         2269 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_parser.pyc 
 +50333975 ​   4 -rw------- ​  2 jypeter ​ lsce          724 Apr  4 20:35 cdatm15/​lib/​python2.7/​site-packages/​cdp/​cdp_provenance.pyc 
 +(cdatm15) bash-4.2$ chmod -R a+r cdatm15/​lib/​python2.7/​site-packages/​cdp* 
 +(cdatm15) bash-4.2$ find cdatm15 \! -perm /​g+r,​o+r ​-ls</​code>​
  
-Proceed ([y]/n)? y</​code>​ 
-  * trying to install ''​rpy2''​ will trigger the installation of ''​R''​! 
-    * We force the **installation of only rpy2** in order to use the ''​R''​ already installed at LSCE 
-      * ''​conda install %%--%%force -n cdatm14 -c conda-forge -c uvcdat rpy2''​ 
  
 ===== Extra packages list ===== ===== Extra packages list =====
Line 337: Line 390:
   * [[http://​pythonhosted.org/​seawater/​|seawater]]:​ Python re-write of the CSIRO seawater toolbox   * [[http://​pythonhosted.org/​seawater/​|seawater]]:​ Python re-write of the CSIRO seawater toolbox
   * [[https://​pypi.python.org/​pypi/​gsw/​|gsw]]:​ Python implementation of the Thermodynamic Equation Of Seawater   * [[https://​pypi.python.org/​pypi/​gsw/​|gsw]]:​ Python implementation of the Thermodynamic Equation Of Seawater
 +  * [[http://​www.ifremer.fr/​vacumm/​|vacumm]]:​ Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
   * [[http://​unidata.github.io/​netcdf4-python/​|netcdf4]]:​ a Python interface to the netCDF C library   * [[http://​unidata.github.io/​netcdf4-python/​|netcdf4]]:​ a Python interface to the netCDF C library
   * [[http://​ferret.pmel.noaa.gov/​Ferret/​documentation/​pyferret|pyferret]]:​ Ferret encapsulated in    * [[http://​ferret.pmel.noaa.gov/​Ferret/​documentation/​pyferret|pyferret]]:​ Ferret encapsulated in 
Line 346: Line 400:
   * [[https://​cmor.llnl.gov/​|CMOR]]: ​ CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files   * [[https://​cmor.llnl.gov/​|CMOR]]: ​ CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files
   * [[https://​github.com/​pierre-rouanet/​dtw|dtw]]:​ DTW (Dynamic Time Warping) python module   * [[https://​github.com/​pierre-rouanet/​dtw|dtw]]:​ DTW (Dynamic Time Warping) python module
- 
   * [[http://​toblerity.org/​shapely/​project.html|shapely]]:​ a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)   * [[http://​toblerity.org/​shapely/​project.html|shapely]]:​ a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
   * [[http://​scitools.org.uk/​cartopy/​|cartopy]]:​ a library providing cartographic tools for python   * [[http://​scitools.org.uk/​cartopy/​|cartopy]]:​ a library providing cartographic tools for python
   * [[https://​rpy2.readthedocs.io/​en/​version_2.8.x/​|rpy2]]:​ providing simple and robust access to R from within Python   * [[https://​rpy2.readthedocs.io/​en/​version_2.8.x/​|rpy2]]:​ providing simple and robust access to R from within Python
 +  * [[http://​matplotlib.org/​cmocean/​|cmocean]]:​ beautiful colormaps for oceanography
 +  * [[http://​scitools.org.uk/​iris/​index.html|iris]]:​ A Python library for Meteorology and Climatology
 +  * [[http://​www.gdal.org/​|OSGeo/​GDAL]]:​ Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
 +  * [[https://​github.com/​stefraynaud/​spanlib|spanlib]]:​ Spectral Analysis Library
 +  * [[http://​wrf-python.readthedocs.io/​en/​latest/​|wrf-python]]:​ A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
 +  * [[https://​glances.readthedocs.io/​en/​stable/​index.html|glances]]:​ a cross-platform monitoring tool (similar to ''​top''​)
 +  * [[https://​ajdawson.github.io/​windspharm/​latest/​|windspharm]]:​ spherical harmonic wind analysis in Python
  
 ==== Removed packages ==== ==== Removed packages ====
  
-  * [[https://​pypi.python.org/​pypi/​wget|wget]]:​ pure python download utility +  * NO removed packages!
-    * Used ''​conda install -n cdatm14 -c daler wget=3.2''​ in order to get version **3.2**, the same as the one on Pypi +
-    * **REMOVED** because it did not work correctly in the script where it was needed: it emulated wget with //ftp// and failed in our case after downloading the first file+
  
 ===== Environments summary ===== ===== Environments summary =====
  
-After following the steps above, we get the following environments. Use the ''​conda info --envs''​ command to get the up-to-date list of available environments+After following the steps above, we get the following environments. Use the ''​conda info %%--%%envs''​ or the ''​conda env list''​ command to get the up-to-date list of available environments
  
 ^  Environment\\ name  ^  Server ​ ^  ''​conda list'' ​ ^ ^  Environment\\ name  ^  Server ​ ^  ''​conda list'' ​ ^
-|  uvcdat-2.8.0  ​| ​ LSCE\\ ciclad ​ ​| ​ {{ :​other:​uvcdat:​cdat_conda:​conda_list_lsce_uvcdat-2.8.0_170310.txt ​|package list}} ​ | +|  uvcdat-2.10  ​| ​ LSCE  |  {{  |package list}} ​ | 
-|  ​cdatm14 ​ ​| ​ LSCE\\ ciclad ​ ​| ​ {{ :​other:​uvcdat:​cdat_conda:​conda_list_lsce_cdatm14_170310.txt|package list}} ​ |+|  ​cdatm15 ​ ​| ​ LSCE  |  {{ |package list}} ​ |
  
 /* standard page footer */ /* standard page footer */
other/uvcdat/cdat_conda/cdat_2_10.1502292990.txt.gz · Last modified: 2017/08/09 15:36 by jypeter