other:uvcdat:cdat_conda:cdat_2_12
This is an old revision of the document!
Table of Contents
UV-CDAT 2.12 installation notes
Follow the instructions about the conda-based versions of UV-CDAT initialization for actually using an installed version of 2.12
What's New?
- 2.12 announcement and features summary (all versions)
- Full Change log
Installation with Miniconda2
Actual installation
bash-4.1$ conda create -n uvcdat-2.12 -c conda-forge -c uvcdat uvcdat=2.12 # Generate the list of installed packages conda list -n uvcdat-2.12 > /home/scratch01/jypeter/uvcdat-2.12_list_170907.txt
List of installed packages: uvcdat-2.12_list_170907.txt
Disk space after 2.12 installation
bash-4.1$ conda env list # conda environments: # cdatm /home/share/unix_files/cdat/miniconda2/envs/cdatm cdatm14 /home/share/unix_files/cdat/miniconda2/envs/cdatm14 cdatm15 /home/share/unix_files/cdat/miniconda2/envs/cdatm15 uvcdat-2.10 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 uvcdat-2.12 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.12 uvcdat-2.8.0 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 root * /home/share/unix_files/cdat/miniconda2 > du -sh /home/share/unix_files/cdat/miniconda2 15G /home/share/unix_files/cdat/miniconda2 > du -sh /home/share/unix_files/cdat/miniconda2/* 4.4M /home/share/unix_files/cdat/miniconda2/bin 4.0K /home/share/unix_files/cdat/miniconda2/conda-bld 272K /home/share/unix_files/cdat/miniconda2/conda-meta 13G /home/share/unix_files/cdat/miniconda2/envs 20K /home/share/unix_files/cdat/miniconda2/etc 3.8M /home/share/unix_files/cdat/miniconda2/include 99M /home/share/unix_files/cdat/miniconda2/lib 4.0K /home/share/unix_files/cdat/miniconda2/LICENSE.txt 2.1G /home/share/unix_files/cdat/miniconda2/pkgs 376K /home/share/unix_files/cdat/miniconda2/share 364K /home/share/unix_files/cdat/miniconda2/ssl > du -sh /home/share/unix_files/cdat/miniconda2/envs/* 0 /home/share/unix_files/cdat/miniconda2/envs/cdatm 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 4.5G /home/share/unix_files/cdat/miniconda2/envs/cdatm15 630M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 2.7G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.12 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0
Cloning uvcdat to add specific packages for LSCE
This section is about the creation of the cdatm16 environment
Notes about actually using the cdatm16 conda-based python
Cloning
bash-4.1$ conda create -n cdatm16 --clone uvcdat-2.12 Source: /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.12 Destination: /home/share/unix_files/cdat/miniconda2/envs/cdatm16 [...] bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* 0 /home/share/unix_files/cdat/miniconda2/envs/cdatm 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 4.5G /home/share/unix_files/cdat/miniconda2/envs/cdatm15 2.7G /home/share/unix_files/cdat/miniconda2/envs/cdatm16 630M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 242M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.12 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0
Checking the permissions
The files of some packages may have been installed with incorrect access rights, and may not be readable by everybody (e.g. pcmdi_metrics/issues/496 or cdp/issues/21). We have to check and correct this by hand:
(cdatm16) bash-4.1$ find uvcdat-2.12 \! -perm /g+r,o+r -ls 24774102 4 -rw------- 3 jypeter lsce 670 Apr 4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp/cdp_output.py 24774113 4 -rw------- 3 jypeter lsce 2436 Apr 4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp/cdp_parameter.pyc 24774111 4 -rw------- 3 jypeter lsce 1821 Apr 4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp/cdp_parser.py [...] 24774098 4 -rw------- 3 jypeter lsce 234 Apr 4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO 24774089 4 -rw------- 3 jypeter lsce 1 Apr 4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe (cdatm16) bash-4.1$ chmod -R a+r cdatm16/lib/python2.7/site-packages/cdp* chmod -R a+r uvcdat-2.12/lib/python2.7/site-packages/cdp*
Getting ready for a moving default CDAT
See the note about the cdatm link/version. At this point of installation, we are not yet ready to have cdatm point to 2.12.
Customizing UV-CDAT for LSCE
Downloading cdms2/vcs test data
You should download the test data and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any…)
bash-4.1$ source activate cdatm16 (cdatm15) bash-4.1$ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data' [...] Finished downloading sample data to /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data (cdatm16) bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/cdatm16/share/uvcdat/sample_data 174M /home/share/unix_files/cdat/miniconda2/envs/cdatm16/share/uvcdat/sample_data
Packages that have no dependency problems
After cloning, we are ready to install some extra packages that may be requested by LSCE users
# Keep a trace of what will be installed bash-4.1$ conda install --dry-run -n cdatm16 -c conda-forge -c uvcdat -c pcmdi -c scitools -c vacumm pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy iris rpy2 cmocean iris gdal vacumm > /home/scratch01/jypeter/lsce-cdatm16-extra_01_install_170907.txt # Install... bash-4.1$ conda install -n cdatm15 -c conda-forge -c uvcdat -c pcmdi pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy rpy2 [...] # Check the disk space after installation bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 11G /home/share/unix_files/cdat/miniconda2 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* 0 /home/share/unix_files/cdat/miniconda2/envs/cdatm 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 3.7G /home/share/unix_files/cdat/miniconda2/envs/cdatm15 574M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 # Check the disk space again after cleaning bash-4.1$ conda clean --all bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 9.9G /home/share/unix_files/cdat/miniconda2
List of installed packages: lsce-extra_01_install_170808.txt
Disk space used after adding the extra packages, in in the new directory hierarchy (cdat/miniconda2b, on the obelix servers)
bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b 6.4G /home/share/unix_files/cdat/miniconda2b bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/* 0 /home/share/unix_files/cdat/miniconda2b/envs/cdatm 3.8G /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 630M /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 # Check the disk space again after cleaning bash-4.2$ conda clean --all bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b 5.4G /home/share/unix_files/cdat/miniconda2b
Packages installed with pip
- dreqPy: CMIP6 Data Request Python API
pip install dreqPy- Update with:
pip install --upgrade dreqPy- Get version number with:
$ drq -v dreqPy version 01.00.06 [Version 01.00.06]
- dtw: DTW (Dynamic Time Warping) python module
pip install dtw- Requested by Sentia Goursaud
The following packages have no dependency problems and were installed (or updated) later
- ESMPy: ESMF Python Regridding Interface
- We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10
bash-4.1$ conda list | grep -i esm esmf 7.0.0 6 conda-forge esmpy 7.0.0 py27_1 conda-forge bash-4.1$ conda install -n cdatm15 -c nesii/channel/dev-esmf -c conda-forge esmpy=7.1.0.dev32 [...] bash-4.1$ conda list | grep -i esm esmf 7.1.0.dev32 1 nesii/channel/dev-esmf esmpy 7.1.0.dev32 py27_1 nesii/channel/dev-esmf
- cmocean: beautiful colormaps for oceanography
conda install -n cdatm15 -c conda-forge cmocean
- iris: A Python library for Meteorology and Climatology
conda install -n cdatm15 -c conda-forge -c scitools iris
- OSGeo/GDAL: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
conda install -c conda-forge gdal
TODO
Add here packages that would be useful but have some problems that prevent their installation
- vacumm: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
- As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see github issue)!
- Problem solved with vacumm 3.4.1-1 on August 31st! Installation was OK with
conda install -n cdatm15 -c conda-forge -c vacumm vacumm
- Install PySPLIT for Sentia
PySPLIT A package for generating HYSPLIT air parcel trajectories trajectories, performing moisture uptake analyses, expediting HYSPLIT cluster analysis, and for visualizing trajectories, clusters, and along-trajectory meteorological data
Works with HYSPLiTconda install -n cdatm15 -c conda-forge fiona geopandaspip install pysplit
Other packages
There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them…
- NO such packages now!
Updating some packages
Some packages change more often than others, and can be easily updated the following way:
-
- Update with:
conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor- Get version number with:
python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH'
-
- Update with:
pip install --upgrade dreqPy- Get version number with:
drq -v
Cleaning up things
Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg pcmdi-metrics in 2.8.0 and cdp in 2.10)
We check if some of the installed files are missing read access for the group or other, and we manually change the permissions
(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls 50333969 4 -rw------- 2 jypeter lsce 234 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO 50333971 4 -rw------- 2 jypeter lsce 291 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/SOURCES.txt 50333962 4 -rw------- 2 jypeter lsce 4 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/top_level.txt 50333961 4 -rw------- 2 jypeter lsce 1 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/dependency_links.txt 50333960 4 -rw------- 2 jypeter lsce 1 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe 50333987 8 -rw------- 2 jypeter lsce 5149 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.pyc 50333986 8 -rw------- 2 jypeter lsce 4208 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.py 50333966 4 -rw------- 2 jypeter lsce 150 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.pyc 50333968 4 -rw------- 2 jypeter lsce 201 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.py 50333988 8 -rw------- 2 jypeter lsce 5365 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.pyc 50333982 4 -rw------- 2 jypeter lsce 1821 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.py 50333981 4 -rw------- 2 jypeter lsce 1773 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.py 50333973 4 -rw------- 2 jypeter lsce 670 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.py 50333980 4 -rw------- 2 jypeter lsce 1642 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.pyc 50333979 4 -rw------- 2 jypeter lsce 1474 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.pyc 50333967 4 -rw------- 2 jypeter lsce 187 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.py 50333989 0 -rw------- 2 jypeter lsce 0 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.py 50333985 4 -rw------- 2 jypeter lsce 3843 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.py 50333984 4 -rw------- 2 jypeter lsce 2436 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.pyc 50333970 4 -rw------- 2 jypeter lsce 288 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.py 50333974 4 -rw------- 2 jypeter lsce 692 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.pyc 50333978 4 -rw------- 2 jypeter lsce 878 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.pyc 50333976 4 -rw------- 2 jypeter lsce 734 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.py 50333983 4 -rw------- 2 jypeter lsce 2269 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.pyc 50333975 4 -rw------- 2 jypeter lsce 724 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.pyc (cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp* (cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls
Extra packages list
- pillow: the friendly PIL (Python Imaging Library) fork
- pandas: Python Data Analysis Library
- statsmodels: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests
- seaborn: statistical data visualization
- scikit-image: image processing in Python
- seawater: Python re-write of the CSIRO seawater toolbox
- gsw: Python implementation of the Thermodynamic Equation Of Seawater
- netcdf4: a Python interface to the netCDF C library
- pyferret: Ferret encapsulated in
- basemap-data-hires: high resolution data for
basemap - PCMDI metrics package (PMP): objectively compare results from climate models with observations using well-established statistical tests
- XlsxWriter: a Python module for creating Excel XLSX files
- Note: this is a dependency of
dreqPy
- dreqPy: CMIP6 Data Request Python API
- CMOR: CMOR (Climate Model Output Rewriter) is used to produce CF-compliant netCDF files
- dtw: DTW (Dynamic Time Warping) python module
- shapely: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
- cartopy: a library providing cartographic tools for python
- rpy2: providing simple and robust access to R from within Python
- cmocean: beautiful colormaps for oceanography
- iris: A Python library for Meteorology and Climatology
- OSGeo/GDAL: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
Removed packages
- NO removed packages!
Environments summary
After following the steps above, we get the following environments. Use the conda info --envs or the conda env list command to get the up-to-date list of available environments
| Environment name | Server | conda list |
|---|---|---|
| uvcdat-2.10 | LSCE | package list |
| cdatm15 | LSCE | package list |
[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]
other/uvcdat/cdat_conda/cdat_2_12.1505119333.txt.gz · Last modified: 2017/09/11 10:42 by jypeter