UV-CDAT 2.10 installation notes

[ Back to all versions ]

Follow the instructions about the conda-based versions of UV-CDAT initialization for actually using an installed version of 2.10

What's New?

2.10 announcement and features summary (all versions)

Full Change log

Installation with Miniconda2

We used the recommended downgraded version of conda that was available after installing UV-CDAT 2.8

DIsk space used before 2.10 installation (after running conda clean –all)

> du -sh /home/share/unix_files/cdat/miniconda2
6.5G	/home/share/unix_files/cdat/miniconda2
> du -sh /home/share/unix_files/cdat/miniconda2/*
4.4M	/home/share/unix_files/cdat/miniconda2/bin
4.0K	/home/share/unix_files/cdat/miniconda2/conda-bld
272K	/home/share/unix_files/cdat/miniconda2/conda-meta
6.4G	/home/share/unix_files/cdat/miniconda2/envs
20K	/home/share/unix_files/cdat/miniconda2/etc
3.8M	/home/share/unix_files/cdat/miniconda2/include
99M	/home/share/unix_files/cdat/miniconda2/lib
4.0K	/home/share/unix_files/cdat/miniconda2/LICENSE.txt
76K	/home/share/unix_files/cdat/miniconda2/pkgs
376K	/home/share/unix_files/cdat/miniconda2/share
364K	/home/share/unix_files/cdat/miniconda2/ssl
>du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
1.7G	/home/share/unix_files/cdat/miniconda2/envs/keras
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

Actual installation

bash-4.1$ conda create -n uvcdat-2.10 -c conda-forge -c uvcdat uvcdat=2.10
# Generate the list of installed packages
conda list -n uvcdat-2.10 > /home/scratch01/jypeter/uvcdat-2.10_list_170721.txt

List of installed packages: uvcdat-2.10_list_170721.txt

Disk space after 2.10 installation

bash-4.1$ conda env list
cdatm14 /home/share/unix_files/cdat/miniconda2/envs/cdatm14
keras /home/share/unix_files/cdat/miniconda2/envs/keras
uvcdat-2.10 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
uvcdat-2.8.0 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0
root * /home/share/unix_files/cdat/miniconda2
> du -sh /home/share/unix_files/cdat/miniconda2
11G	/home/share/unix_files/cdat/miniconda2
> du -sh /home/share/unix_files/cdat/miniconda2/*
4.4M	/home/share/unix_files/cdat/miniconda2/bin
4.0K	/home/share/unix_files/cdat/miniconda2/conda-bld
272K	/home/share/unix_files/cdat/miniconda2/conda-meta
9.2G	/home/share/unix_files/cdat/miniconda2/envs
20K	/home/share/unix_files/cdat/miniconda2/etc
3.8M	/home/share/unix_files/cdat/miniconda2/include
99M	/home/share/unix_files/cdat/miniconda2/lib
4.0K	/home/share/unix_files/cdat/miniconda2/LICENSE.txt
1.4G	/home/share/unix_files/cdat/miniconda2/pkgs
376K	/home/share/unix_files/cdat/miniconda2/share
364K	/home/share/unix_files/cdat/miniconda2/ssl
> du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
1.7G	/home/share/unix_files/cdat/miniconda2/envs/keras
2.8G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

Disk space after removing the keras environment that we don't need any more

jypeter@asterix2 - ...cdat - 67 >du -sh /home/share/unix_files/cdat/miniconda2
9.0G	/home/share/unix_files/cdat/miniconda2
jypeter@asterix2 - ...cdat - 68 >du -sh /home/share/unix_files/cdat/miniconda2/*
4.4M	/home/share/unix_files/cdat/miniconda2/bin
4.0K	/home/share/unix_files/cdat/miniconda2/conda-bld
272K	/home/share/unix_files/cdat/miniconda2/conda-meta
7.5G	/home/share/unix_files/cdat/miniconda2/envs
20K	/home/share/unix_files/cdat/miniconda2/etc
3.8M	/home/share/unix_files/cdat/miniconda2/include
99M	/home/share/unix_files/cdat/miniconda2/lib
4.0K	/home/share/unix_files/cdat/miniconda2/LICENSE.txt
1.4G	/home/share/unix_files/cdat/miniconda2/pkgs
376K	/home/share/unix_files/cdat/miniconda2/share
364K	/home/share/unix_files/cdat/miniconda2/ssl
jypeter@asterix2 - ...cdat - 69 >du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
2.8G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

Disk space used in a brand new directory hierarchy (cdat/miniconda2b, on the obelix servers)

bash-4.2$ conda env list
uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10
root * /home/share/unix_files/cdat/miniconda2b
bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b
4.3G	/home/share/unix_files/cdat/miniconda2b
bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/*
4.4M	/home/share/unix_files/cdat/miniconda2b/bin
4.0K	/home/share/unix_files/cdat/miniconda2b/conda-bld
280K	/home/share/unix_files/cdat/miniconda2b/conda-meta
2.8G	/home/share/unix_files/cdat/miniconda2b/envs
20K	/home/share/unix_files/cdat/miniconda2b/etc
3.8M	/home/share/unix_files/cdat/miniconda2b/include
100M	/home/share/unix_files/cdat/miniconda2b/lib
8.0K	/home/share/unix_files/cdat/miniconda2b/LICENSE.txt
1.4G	/home/share/unix_files/cdat/miniconda2b/pkgs
16K	/home/share/unix_files/cdat/miniconda2b/share
376K	/home/share/unix_files/cdat/miniconda2b/ssl

Cloning uvcdat to add specific packages for LSCE

This section is about the creation of the cdatm15 environment Notes about actually using the cdatm15 conda-based python

Note: using hard links, cloning a full environment only adds an extra 500M of disk space

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
9.0G	/home/share/unix_files/cdat/miniconda2
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
2.8G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0
bash-4.1$ conda create -n cdatm15 --clone uvcdat-2.10
Source: /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
Destination: /home/share/unix_files/cdat/miniconda2/envs/cdatm15
[...]
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
9.5G	/home/share/unix_files/cdat/miniconda2
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm15
573M	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

Disk space used after cloning, in in the new directory hierarchy (cdat/miniconda2b, on the obelix servers)

bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b
4.8G	/home/share/unix_files/cdat/miniconda2b
bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/*
4.4M	/home/share/unix_files/cdat/miniconda2b/bin
4.0K	/home/share/unix_files/cdat/miniconda2b/conda-bld
280K	/home/share/unix_files/cdat/miniconda2b/conda-meta
3.4G	/home/share/unix_files/cdat/miniconda2b/envs
20K	/home/share/unix_files/cdat/miniconda2b/etc
3.8M	/home/share/unix_files/cdat/miniconda2b/include
100M	/home/share/unix_files/cdat/miniconda2b/lib
8.0K	/home/share/unix_files/cdat/miniconda2b/LICENSE.txt
1.4G	/home/share/unix_files/cdat/miniconda2b/pkgs
16K	/home/share/unix_files/cdat/miniconda2b/share
376K	/home/share/unix_files/cdat/miniconda2b/ssl
bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/*
2.8G	/home/share/unix_files/cdat/miniconda2b/envs/cdatm15
574M	/home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10

Getting ready for a moving default CDAT

We create a cdatm symbolic link in the envs directory, that has a stable name but can be moved to point to the latest default CDAT. In that case, most users can just activate this cdatm version and always get the latest stable version

bash-4.2$ cd /home/share/unix_files/cdat/miniconda2b/envs
bash-4.2$ conda env list
cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15
uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10
root * /home/share/unix_files/cdat/miniconda2b
bash-4.2$ ln -s cdatm15 cdatm
bash-4.2$ conda env list
cdatm /home/share/unix_files/cdat/miniconda2b/envs/cdatm
cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15
uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10
root * /home/share/unix_files/cdat/miniconda2b
bash-4.2$ source activate cdatm
(cdatm) bash-4.2$ conda env list
cdatm * /home/share/unix_files/cdat/miniconda2b/envs/cdatm
cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15
uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10
root /home/share/unix_files/cdat/miniconda2b

Downloading cdms2/vcs test data

You should download the test data and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any…)

bash-4.1$ source activate cdatm15
(cdatm15) bash-4.1$ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data'
[...]
Finished downloading sample data to /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data
(cdatm15) bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data
174M	/home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data

Packages that have no dependency problems

After cloning, we are ready to install some extra packages that may be requested by LSCE users

# Keep a trace of what will be installed
bash-4.1$ conda install --dry-run -n cdatm15 -c conda-forge -c uvcdat -c pcmdi pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy rpy2 > /home/scratch01/jypeter/lsce-extra_01_install_170808.txt
# Install...
bash-4.1$ conda install -n cdatm15 -c conda-forge -c uvcdat -c pcmdi pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy rpy2
[...]
# Check the disk space after installation
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
11G	/home/share/unix_files/cdat/miniconda2
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
0	/home/share/unix_files/cdat/miniconda2/envs/cdatm
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
3.7G	/home/share/unix_files/cdat/miniconda2/envs/cdatm15
574M	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0
# Check the disk space again after cleaning
bash-4.1$ conda clean --all
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
9.9G	/home/share/unix_files/cdat/miniconda2

Lists of installed packages: lsce-extra_01_install_170808.txt

Packages installed with pip

dreqPy: CMIP6 Data Request Python API

pip install dreqPy

Update with: pip install --upgrade dreqPy

Get version number with:

$ drq -v
dreqPy version 01.00.06 [Version 01.00.06]

dtw: DTW (Dynamic Time Warping) python module

pip install dtw

Requested by Sentia Goursaud

The following packages have no dependency problems and were installed later

ESMPy: ESMF Python Regridding Interface

We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10

bash-4.1$ conda list | grep -i esm
esmf 7.0.0 6 conda-forge
esmpy 7.0.0 py27_1 conda-forge
bash-4.1$ conda install -n cdatm15 -c nesii/channel/dev-esmf -c conda-forge esmpy=7.1.0.dev32
[...]
bash-4.1$ conda list | grep -i esm
esmf 7.1.0.dev32 1 nesii/channel/dev-esmf
esmpy 7.1.0.dev32 py27_1 nesii/channel/dev-esmf

TODO

Add here packages that would be useful but have some problems that prevent their installation

vacumm: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations

As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see github issue)!

Other packages

There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them…

shapely request a downgrade of geos when installed from conda-forge (see github issue) but it will work when installed from the default repo

conda install -n cdatm14 shapely

We get a slightly older version (1.5.16-py27_0 instead of 1.5.17-np111py27_2)

Note: cartopy (below) requests the same downgrade of geos…

cartopy requests a downgrade of geos and matplotlib (see github issue)

We install the requested dependencies and then we force the installation of cartopy

(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat libxslt lxml owslib pyepsg
[...]
The following NEW packages will be INSTALLED:
libxslt: 1.1.29-3 conda-forge
lxml: 3.7.2-py27_1 conda-forge
owslib: 0.14.0-py27_0 conda-forge
pyepsg: 0.3.1-py27_0 conda-forge
[...]
(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge cartopy
[...]
The following NEW packages will be INSTALLED:
cartopy: 0.15.0-np111py27_0 conda-forge
The following packages will be DOWNGRADED due to dependency conflicts:
geos: 3.6.0-0 conda-forge --> 3.5.1-1 conda-forge
matplotlib: 2.0.0-np111py27_2 conda-forge --> 1.5.3-np111py27_7 conda-forge
Proceed ([y]/n)? n
(uvcdat-2.8.0) bash-4.1$ conda install --force -n cdatm14 -c conda-forge cartopy
[...]
The following NEW packages will be INSTALLED:
cartopy: 0.15.0-np111py27_0 conda-forge
Proceed ([y]/n)? y

trying to install rpy2 will trigger the installation of R!

We force the installation of only rpy2 in order to use the R already installed at LSCE

conda install --force -n cdatm14 -c conda-forge -c uvcdat rpy2

Extra packages list

pillow: the friendly PIL (Python Imaging Library) fork

pandas: Python Data Analysis Library

statsmodels: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests

seaborn: statistical data visualization

scikit-image: image processing in Python

seawater: Python re-write of the CSIRO seawater toolbox

gsw: Python implementation of the Thermodynamic Equation Of Seawater

netcdf4: a Python interface to the netCDF C library

pyferret: Ferret encapsulated in

basemap-data-hires: high resolution data for basemap

PCMDI metrics package (PMP): objectively compare results from climate models with observations using well-established statistical tests

XlsxWriter: a Python module for creating Excel XLSX files

Note: this is a dependency of dreqPy

dreqPy: CMIP6 Data Request Python API

CMOR: CMOR (Climate Model Output Rewriter) is used to produce CF-compliant netCDF files

dtw: DTW (Dynamic Time Warping) python module

shapely: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)

cartopy: a library providing cartographic tools for python

rpy2: providing simple and robust access to R from within Python

Removed packages

wget: pure python download utility

Used conda install -n cdatm14 -c daler wget=3.2 in order to get version 3.2, the same as the one on Pypi

REMOVED because it did not work correctly in the script where it was needed: it emulated wget with ftp and failed in our case after downloading the first file

Environments summary

After following the steps above, we get the following environments. Use the conda info –envs command to get the up-to-date list of available environments

Environment name	Server	conda list
uvcdat-2.8.0	LSCE ciclad	package list
cdatm14	LSCE ciclad	package list

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]