This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
other:uvcdat:cdat_conda:cdat_2_10 [2017/08/09 15:59] jypeter |
other:uvcdat:cdat_conda:cdat_2_10 [2018/10/03 09:56] jypeter Added windspharm |
||
---|---|---|---|
Line 209: | Line 209: | ||
root /home/share/unix_files/cdat/miniconda2b | root /home/share/unix_files/cdat/miniconda2b | ||
</code> | </code> | ||
+ | |||
+ | ===== Customizing UV-CDAT for LSCE ===== | ||
+ | |||
==== Downloading cdms2/vcs test data ==== | ==== Downloading cdms2/vcs test data ==== | ||
Line 252: | Line 255: | ||
9.9G /home/share/unix_files/cdat/miniconda2</code> | 9.9G /home/share/unix_files/cdat/miniconda2</code> | ||
- | Lists of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_170808.txt |}} | + | List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_170808.txt |}} |
- | </code> | + | |
Disk space used after adding the extra packages, in <wrap em>in the new directory hierarchy</wrap> (''cdat/miniconda2b'', on the obelix servers) | Disk space used after adding the extra packages, in <wrap em>in the new directory hierarchy</wrap> (''cdat/miniconda2b'', on the obelix servers) | ||
Line 281: | Line 283: | ||
* Requested by Sentia Goursaud | * Requested by Sentia Goursaud | ||
- | The following packages have no dependency problems and were installed //later// | + | The following packages have no dependency problems and were installed (or updated) //later// |
* [[http://www.earthsystemmodeling.org/esmf_releases/non_public/ESMF_7_0_0/esmpy_doc/html/index.html|ESMPy]]: ESMF Python Regridding Interface | * [[http://www.earthsystemmodeling.org/esmf_releases/non_public/ESMF_7_0_0/esmpy_doc/html/index.html|ESMPy]]: ESMF Python Regridding Interface | ||
* We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10 | * We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10 | ||
Line 294: | Line 296: | ||
esmf 7.1.0.dev32 1 nesii/channel/dev-esmf | esmf 7.1.0.dev32 1 nesii/channel/dev-esmf | ||
esmpy 7.1.0.dev32 py27_1 nesii/channel/dev-esmf</code> | esmpy 7.1.0.dev32 py27_1 nesii/channel/dev-esmf</code> | ||
+ | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography | ||
+ | * ''conda install -n cdatm15 -c conda-forge cmocean'' | ||
+ | * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology | ||
+ | * ''conda install -n cdatm15 -c conda-forge -c scitools iris'' | ||
+ | * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | ||
+ | * ''conda install -c conda-forge gdal'' | ||
+ | * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] | ||
+ | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
+ | * ''conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib'' | ||
+ | * Test: ''python -c 'from spanlib.analyzer import Analyzer' '' | ||
+ | * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | ||
+ | * ''conda install -n cdatm15 -c conda-forge wrf-python'' | ||
+ | * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'') | ||
+ | * ''conda install -n cdatm15 -c conda-forge glances'' | ||
+ | * [[https://github.com/nicolargo/glances|glances@github]] | ||
+ | * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | ||
+ | * ''conda install -n cdatm15 -c conda-forge windspharm'' | ||
+ | * [[https://github.com/ajdawson/windspharm|windspharm@github]] | ||
+ | |||
==== TODO ==== | ==== TODO ==== | ||
Line 301: | Line 322: | ||
* [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | ||
* As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])! | * As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])! | ||
+ | * <wrap hi>Problem solved</wrap> with **vacumm 3.4.1-1** on August 31st! Installation was OK with\\ ''conda install -n cdatm15 -c conda-forge -c vacumm vacumm'' | ||
+ | |||
+ | * Install PySPLIT for Sentia\\ [[https://github.com/mscross/pysplit|PySPLIT]] A package for generating HYSPLIT air parcel trajectories trajectories, performing moisture uptake analyses, expediting HYSPLIT cluster analysis, and for visualizing trajectories, clusters, and along-trajectory meteorological data\\ Works with [[https://ready.arl.noaa.gov/HYSPLIT.php|HYSPLiT]] | ||
+ | - ''conda install -n cdatm15 -c conda-forge fiona geopandas'' | ||
+ | - ''pip install pysplit'' | ||
==== Other packages ==== | ==== Other packages ==== | ||
Line 306: | Line 332: | ||
<note warning>There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them...</note> | <note warning>There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them...</note> | ||
- | * ''shapely'' request a downgrade of ''geos'' when installed from //conda-forge// (see [[https://github.com/Toblerity/Shapely/issues/472|github issue]]) but it will work when installed from the default repo | + | * NO such packages now! |
- | * ''conda install -n cdatm14 shapely'' | + | |
- | * We get a slightly older version (//1.5.**16**-py27_0// instead of //1.5.**17**-np111py27_2//) | + | |
- | * Note: ''cartopy'' (below) requests the same downgrade of ''geos''... | + | |
- | * ''cartopy'' requests a downgrade of ''geos'' and ''matplotlib'' (see [[https://github.com/SciTools/cartopy/issues/853#issuecomment-282023158|github issue]]) | + | |
- | * We install the requested dependencies and then we **force** the installation of ''cartopy''\\ <code>(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat libxslt lxml owslib pyepsg | + | |
- | [...] | + | |
- | The following NEW packages will be INSTALLED: | + | |
- | libxslt: 1.1.29-3 conda-forge | + | ==== Updating some packages ==== |
- | lxml: 3.7.2-py27_1 conda-forge | + | |
- | owslib: 0.14.0-py27_0 conda-forge | + | |
- | pyepsg: 0.3.1-py27_0 conda-forge | + | |
- | [...] | + | |
- | (uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge cartopy | + | Some packages change more often than others, and can be easily updated the following way: |
- | [...] | + | |
- | The following NEW packages will be INSTALLED: | + | |
- | cartopy: 0.15.0-np111py27_0 conda-forge | + | * [[https://cmor.llnl.gov/|CMOR]] |
+ | * Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor'' | ||
+ | * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' '' | ||
- | The following packages will be DOWNGRADED due to dependency conflicts: | + | * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]] |
+ | * Update with: ''pip install %%--%%upgrade dreqPy'' | ||
+ | * Get version number with: ''drq -v'' | ||
- | geos: 3.6.0-0 conda-forge --> 3.5.1-1 conda-forge | + | ==== Cleaning up things ==== |
- | matplotlib: 2.0.0-np111py27_2 conda-forge --> 1.5.3-np111py27_7 conda-forge | + | |
- | Proceed ([y]/n)? n | + | Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg [[https://github.com/PCMDI/pcmdi_metrics/issues/496|pcmdi-metrics]] in 2.8.0 and [[https://github.com/UV-CDAT/cdp/issues/21|cdp]] in 2.10) |
- | + | ||
- | (uvcdat-2.8.0) bash-4.1$ conda install --force -n cdatm14 -c conda-forge cartopy | + | |
- | [...] | + | |
- | The following NEW packages will be INSTALLED: | + | |
- | cartopy: 0.15.0-np111py27_0 conda-forge | + | We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions |
+ | <code>(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls | ||
+ | 50333969 4 -rw------- 2 jypeter lsce 234 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO | ||
+ | 50333971 4 -rw------- 2 jypeter lsce 291 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/SOURCES.txt | ||
+ | 50333962 4 -rw------- 2 jypeter lsce 4 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/top_level.txt | ||
+ | 50333961 4 -rw------- 2 jypeter lsce 1 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/dependency_links.txt | ||
+ | 50333960 4 -rw------- 2 jypeter lsce 1 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe | ||
+ | 50333987 8 -rw------- 2 jypeter lsce 5149 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.pyc | ||
+ | 50333986 8 -rw------- 2 jypeter lsce 4208 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.py | ||
+ | 50333966 4 -rw------- 2 jypeter lsce 150 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.pyc | ||
+ | 50333968 4 -rw------- 2 jypeter lsce 201 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.py | ||
+ | 50333988 8 -rw------- 2 jypeter lsce 5365 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.pyc | ||
+ | 50333982 4 -rw------- 2 jypeter lsce 1821 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.py | ||
+ | 50333981 4 -rw------- 2 jypeter lsce 1773 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.py | ||
+ | 50333973 4 -rw------- 2 jypeter lsce 670 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.py | ||
+ | 50333980 4 -rw------- 2 jypeter lsce 1642 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.pyc | ||
+ | 50333979 4 -rw------- 2 jypeter lsce 1474 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.pyc | ||
+ | 50333967 4 -rw------- 2 jypeter lsce 187 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.py | ||
+ | 50333989 0 -rw------- 2 jypeter lsce 0 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.py | ||
+ | 50333985 4 -rw------- 2 jypeter lsce 3843 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.py | ||
+ | 50333984 4 -rw------- 2 jypeter lsce 2436 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.pyc | ||
+ | 50333970 4 -rw------- 2 jypeter lsce 288 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.py | ||
+ | 50333974 4 -rw------- 2 jypeter lsce 692 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.pyc | ||
+ | 50333978 4 -rw------- 2 jypeter lsce 878 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.pyc | ||
+ | 50333976 4 -rw------- 2 jypeter lsce 734 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.py | ||
+ | 50333983 4 -rw------- 2 jypeter lsce 2269 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.pyc | ||
+ | 50333975 4 -rw------- 2 jypeter lsce 724 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.pyc | ||
+ | (cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp* | ||
+ | (cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls</code> | ||
- | Proceed ([y]/n)? y</code> | ||
- | * trying to install ''rpy2'' will trigger the installation of ''R''! | ||
- | * We force the **installation of only rpy2** in order to use the ''R'' already installed at LSCE | ||
- | * ''conda install %%--%%force -n cdatm14 -c conda-forge -c uvcdat rpy2'' | ||
===== Extra packages list ===== | ===== Extra packages list ===== | ||
Line 354: | Line 390: | ||
* [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox | * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox | ||
* [[https://pypi.python.org/pypi/gsw/|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater | * [[https://pypi.python.org/pypi/gsw/|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater | ||
+ | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | ||
* [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library | * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library | ||
* [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in | * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in | ||
Line 363: | Line 400: | ||
* [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | ||
* [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | ||
- | |||
* [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) | * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) | ||
* [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python | * [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python | ||
* [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python | * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python | ||
+ | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography | ||
+ | * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology | ||
+ | * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | ||
+ | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
+ | * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | ||
+ | * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'') | ||
+ | * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | ||
==== Removed packages ==== | ==== Removed packages ==== | ||
- | * [[https://pypi.python.org/pypi/wget|wget]]: pure python download utility | + | * NO removed packages! |
- | * Used ''conda install -n cdatm14 -c daler wget=3.2'' in order to get version **3.2**, the same as the one on Pypi | + | |
- | * **REMOVED** because it did not work correctly in the script where it was needed: it emulated wget with //ftp// and failed in our case after downloading the first file | + | |
===== Environments summary ===== | ===== Environments summary ===== | ||
- | After following the steps above, we get the following environments. Use the ''conda info --envs'' command to get the up-to-date list of available environments | + | After following the steps above, we get the following environments. Use the ''conda info %%--%%envs'' or the ''conda env list'' command to get the up-to-date list of available environments |
^ Environment\\ name ^ Server ^ ''conda list'' ^ | ^ Environment\\ name ^ Server ^ ''conda list'' ^ | ||
- | | uvcdat-2.8.0 | LSCE\\ ciclad | {{ :other:uvcdat:cdat_conda:conda_list_lsce_uvcdat-2.8.0_170310.txt |package list}} | | + | | uvcdat-2.10 | LSCE | {{ |package list}} | |
- | | cdatm14 | LSCE\\ ciclad | {{ :other:uvcdat:cdat_conda:conda_list_lsce_cdatm14_170310.txt|package list}} | | + | | cdatm15 | LSCE | {{ |package list}} | |
/* standard page footer */ | /* standard page footer */ |