This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:uvcdat:cdat_conda:cdat_8_0_py2 [2019/03/01 16:47] jypeter [Installing CDAT nightly] |
other:uvcdat:cdat_conda:cdat_8_0_py2 [2019/03/04 16:55] jypeter |
||
---|---|---|---|
Line 259: | Line 259: | ||
<code>$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat python=2.7 cdat</code> | <code>$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat python=2.7 cdat</code> | ||
+ | |||
+ | <wrap hi>Not tested!</wrap> Can this be updated with ''conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat --all'' ??? | ||
=== Python 3.6 version === | === Python 3.6 version === | ||
Line 275: | Line 277: | ||
Note: using hard links, cloning a full environment //only// adds an extra 582M of disk space | Note: using hard links, cloning a full environment //only// adds an extra 582M of disk space | ||
- | <code>$ du -sh /home/share/unix_files/cdat/miniconda3/envs/* | + | <code>$ conda create -n cdatm17_py2 --clone cdat-8.0_py2 |
- | 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | + | |
- | $ conda create -n cdatm17_py2 --clone cdat-8.0_py2 | + | |
Source: /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | Source: /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | ||
Destination: /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | Destination: /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | ||
Line 286: | Line 286: | ||
$ du -sh /home/share/unix_files/cdat/miniconda3/envs/* | $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* | ||
- | 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | + | 2.5G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 |
- | 582M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | + | 392M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 |
+ | 839M /home/share/unix_files/cdat/miniconda3/envs/cdat-nightly_py2 | ||
</code> | </code> | ||
+ | |||
+ | ==== cdat nightly case ==== | ||
+ | |||
+ | ''conda create -n cdatm-nightly_py2 %%--%%clone cdat-nightly_py2'' | ||
===== Getting ready for a moving default CDAT ===== | ===== Getting ready for a moving default CDAT ===== | ||
+ | |||
+ | <note warning>This step should probably be listed at the **end**, especially in a multi-user environment! If there is already a ''cdatm'' link, make sure that the new version is stable and working correctly before updating the ''cdatm'' link</note> | ||
We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version | We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version | ||
Line 334: | Line 341: | ||
After cloning, we are ready to install some extra packages that may be requested by LSCE users | After cloning, we are ready to install some extra packages that may be requested by LSCE users | ||
+ | |||
+ | * We first try to install together as many packages as possible that don't require other channels than //conda-forge//, and that don't request a downgrade of what is already installed | ||
+ | * We then install individual extra packages with ''conda install'' or ''pip install'' | ||
<code># Keep a trace of what will be installed | <code># Keep a trace of what will be installed | ||
- | bash-4.1$ conda install --dry-run -n cdatm15 -c conda-forge -c uvcdat -c pcmdi pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy rpy2 > /home/scratch01/jypeter/lsce-extra_01_install_170808.txt | + | $ conda install --dry-run -n cdatm17_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm > /home/scratch01/jypeter/lsce-extra_01_install_190304.txt |
# Install... | # Install... | ||
- | bash-4.1$ conda install -n cdatm15 -c conda-forge -c uvcdat -c pcmdi pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy rpy2 | + | $ conda install -n cdatm17_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm |
[...] | [...] | ||
+ | |||
+ | # Install extra packages in the nightly version (if you need 'nightly') | ||
+ | $ conda install -n cdatm-nightly_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm | ||
# Check the disk space after installation | # Check the disk space after installation | ||
- | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 | + | $ du -sh /home/share/unix_files/cdat/miniconda3 |
- | 11G /home/share/unix_files/cdat/miniconda2 | + | 7.5G /home/share/unix_files/cdat/miniconda3 |
- | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* | + | $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* |
- | 0 /home/share/unix_files/cdat/miniconda2/envs/cdatm | + | 2.5G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 |
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | + | 1.2G /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 |
- | 3.7G /home/share/unix_files/cdat/miniconda2/envs/cdatm15 | + | 938M /home/share/unix_files/cdat/miniconda3/envs/cdatm-nightly_py2 |
- | 574M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | + | 450M /home/share/unix_files/cdat/miniconda3/envs/cdat-nightly_py2</code> |
- | 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 | + | |
- | # Check the disk space again after cleaning | + | List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_190304.txt |}} |
- | bash-4.1$ conda clean --all | + | |
- | + | ||
- | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 | + | |
- | 9.9G /home/share/unix_files/cdat/miniconda2</code> | + | |
- | + | ||
- | List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_170808.txt |}} | + | |
- | + | ||
- | Disk space used after adding the extra packages, in <wrap em>in the new directory hierarchy</wrap> (''cdat/miniconda2b'', on the obelix servers) | + | |
- | + | ||
- | <code>bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b | + | |
- | 6.4G /home/share/unix_files/cdat/miniconda2b | + | |
- | + | ||
- | bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/* | + | |
- | 0 /home/share/unix_files/cdat/miniconda2b/envs/cdatm | + | |
- | 3.8G /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | + | |
- | 630M /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | + | |
- | + | ||
- | # Check the disk space again after cleaning | + | |
- | bash-4.2$ conda clean --all | + | |
- | + | ||
- | bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b | + | |
- | 5.4G /home/share/unix_files/cdat/miniconda2b</code> | + | |
Packages installed with //pip// | Packages installed with //pip// | ||
Line 383: | Line 373: | ||
* Get version number with:\\ <code>$ drq -v | * Get version number with:\\ <code>$ drq -v | ||
dreqPy version 01.00.06 [Version 01.00.06]</code> | dreqPy version 01.00.06 [Version 01.00.06]</code> | ||
- | * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | ||
- | * ''pip install dtw'' | ||
- | * Requested by Sentia Goursaud | ||
The following packages have no dependency problems and were installed (or updated) //later// | The following packages have no dependency problems and were installed (or updated) //later// | ||
- | * [[http://www.earthsystemmodeling.org/esmf_releases/non_public/ESMF_7_0_0/esmpy_doc/html/index.html|ESMPy]]: ESMF Python Regridding Interface | + | * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology |
- | * We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10 | + | * ''conda install -n cdatm15 -c conda-forge iris'' |
- | * <code>bash-4.1$ conda list | grep -i esm | + | |
- | esmf 7.0.0 6 conda-forge | + | |
- | esmpy 7.0.0 py27_1 conda-forge | + | |
- | bash-4.1$ conda install -n cdatm15 -c nesii/channel/dev-esmf -c conda-forge esmpy=7.1.0.dev32 | ||
- | [...] | ||
- | |||
- | bash-4.1$ conda list | grep -i esm | ||
- | esmf 7.1.0.dev32 1 nesii/channel/dev-esmf | ||
- | esmpy 7.1.0.dev32 py27_1 nesii/channel/dev-esmf</code> | ||
- | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography | ||
- | * ''conda install -n cdatm15 -c conda-forge cmocean'' | ||
- | * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology | ||
- | * ''conda install -n cdatm15 -c conda-forge -c scitools iris'' | ||
- | * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | ||
- | * ''conda install -c conda-forge gdal'' | ||
- | * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] | ||
* [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
* ''conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib'' | * ''conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib'' | ||
* Test: ''python -c 'from spanlib.analyzer import Analyzer' '' | * Test: ''python -c 'from spanlib.analyzer import Analyzer' '' | ||
+ | |||
* [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | ||
* ''conda install -n cdatm15 -c conda-forge wrf-python'' | * ''conda install -n cdatm15 -c conda-forge wrf-python'' | ||
- | * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'') | + | |
+ | * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'') | ||
* ''conda install -n cdatm15 -c conda-forge glances'' | * ''conda install -n cdatm15 -c conda-forge glances'' | ||
* [[https://github.com/nicolargo/glances|glances@github]] | * [[https://github.com/nicolargo/glances|glances@github]] | ||
- | * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | ||
- | * ''conda install -n cdatm15 -c conda-forge windspharm'' | ||
- | * [[https://github.com/ajdawson/windspharm|windspharm@github]] | ||
- | |||
==== TODO ==== | ==== TODO ==== | ||
Line 427: | Line 396: | ||
* As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])! | * As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])! | ||
* <wrap hi>Problem solved</wrap> with **vacumm 3.4.1-1** on August 31st! Installation was OK with\\ ''conda install -n cdatm15 -c conda-forge -c vacumm vacumm'' | * <wrap hi>Problem solved</wrap> with **vacumm 3.4.1-1** on August 31st! Installation was OK with\\ ''conda install -n cdatm15 -c conda-forge -c vacumm vacumm'' | ||
- | |||
- | * Install PySPLIT for Sentia\\ [[https://github.com/mscross/pysplit|PySPLIT]] A package for generating HYSPLIT air parcel trajectories trajectories, performing moisture uptake analyses, expediting HYSPLIT cluster analysis, and for visualizing trajectories, clusters, and along-trajectory meteorological data\\ Works with [[https://ready.arl.noaa.gov/HYSPLIT.php|HYSPLiT]] | ||
- | - ''conda install -n cdatm15 -c conda-forge fiona geopandas'' | ||
- | - ''pip install pysplit'' | ||
==== Other packages ==== | ==== Other packages ==== | ||
Line 456: | Line 421: | ||
We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions | We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions | ||
<code>(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls | <code>(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls | ||
- | 50333969 4 -rw------- 2 jypeter lsce 234 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO | + | [...] |
- | 50333971 4 -rw------- 2 jypeter lsce 291 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/SOURCES.txt | + | |
- | 50333962 4 -rw------- 2 jypeter lsce 4 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/top_level.txt | + | |
- | 50333961 4 -rw------- 2 jypeter lsce 1 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/dependency_links.txt | + | |
- | 50333960 4 -rw------- 2 jypeter lsce 1 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe | + | |
- | 50333987 8 -rw------- 2 jypeter lsce 5149 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.pyc | + | |
- | 50333986 8 -rw------- 2 jypeter lsce 4208 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.py | + | |
- | 50333966 4 -rw------- 2 jypeter lsce 150 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.pyc | + | |
- | 50333968 4 -rw------- 2 jypeter lsce 201 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.py | + | |
- | 50333988 8 -rw------- 2 jypeter lsce 5365 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.pyc | + | |
- | 50333982 4 -rw------- 2 jypeter lsce 1821 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.py | + | |
- | 50333981 4 -rw------- 2 jypeter lsce 1773 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.py | + | |
- | 50333973 4 -rw------- 2 jypeter lsce 670 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.py | + | |
- | 50333980 4 -rw------- 2 jypeter lsce 1642 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.pyc | + | |
- | 50333979 4 -rw------- 2 jypeter lsce 1474 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.pyc | + | |
- | 50333967 4 -rw------- 2 jypeter lsce 187 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.py | + | |
- | 50333989 0 -rw------- 2 jypeter lsce 0 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.py | + | |
- | 50333985 4 -rw------- 2 jypeter lsce 3843 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.py | + | |
- | 50333984 4 -rw------- 2 jypeter lsce 2436 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.pyc | + | |
- | 50333970 4 -rw------- 2 jypeter lsce 288 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.py | + | |
- | 50333974 4 -rw------- 2 jypeter lsce 692 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.pyc | + | |
- | 50333978 4 -rw------- 2 jypeter lsce 878 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.pyc | + | |
- | 50333976 4 -rw------- 2 jypeter lsce 734 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.py | + | |
- | 50333983 4 -rw------- 2 jypeter lsce 2269 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.pyc | + | |
- | 50333975 4 -rw------- 2 jypeter lsce 724 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.pyc | + | |
(cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp* | (cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp* | ||
(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls</code> | (cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls</code> | ||
Line 510: | Line 451: | ||
* [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology | * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology | ||
* [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | ||
+ | * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] | ||
* [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
* [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model |