UV-CDAT 2.8.0 installation notes

UV-CDAT 2.8.0 installation notes

Follow the instructions about the conda-based versions of UV-CDAT initialization for actually using an installed version of 2.8.0

What's New?

2.8 announcement and features summary (all versions)
Full Change log

Installation with Anaconda2

Installing with Miniconda2 rather than Anaconda2 is much better because it will save disk space!

# Keep a trace of what will be installed, with/without conda-forge
conda create --dry-run -n uvcdat-2.8.0 -c uvcdat uvcdat=2.8.0 > /home/scratch01/jypeter/uvcdat-2.8.0_install_dry_no-forge_160220.txt
conda create --dry-run -n uvcdat-2.8.0 -c conda-forge -c uvcdat uvcdat=2.8.0 > /home/scratch01/jypeter/uvcdat-2.8.0_install_dry_with-forge_160220.txt

# Do the actual installation (with conda-forge)
conda create -n uvcdat-2.8.0 -c conda-forge -c uvcdat uvcdat=2.8.0

Lists of installed packages: with conda-forge or without conda-forge

Size of /home/share/unix_files/cdat/anaconda2:

before installation: 2.1G
after installation (with conda-forge): 4.6G

Installation with Miniconda2

We assume that Miniconda2 is already installed

Note: after installation, we use only 2.7G (details below), compared to the 4.6G of the Anaconda2 installation

# Keep a trace of what will be installed (with conda-forge)
conda create --dry-run -n uvcdat-2.8.0 -c conda-forge -c uvcdat uvcdat=2.8.0 > /home/scratch01/jypeter/uvcdat-2.8.0_install_dry_with-forge_160220.txt

# Do the actual installation
conda create -n uvcdat-2.8.0 -c conda-forge -c uvcdat uvcdat=2.8.0

Lists of installed packages: List of installed packages

bash-4.1$ conda info --envs
# conda environments:
#
uvcdat-2.8.0             /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0
root                  *  /home/share/unix_files/cdat/miniconda2

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
2.7G    /home/share/unix_files/cdat/miniconda2

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.0G    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/pkgs
2.6G    /home/share/unix_files/cdat/miniconda2/pkgs

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/*
4.4M    /home/share/unix_files/cdat/miniconda2/bin
4.0K    /home/share/unix_files/cdat/miniconda2/conda-bld
276K    /home/share/unix_files/cdat/miniconda2/conda-meta
2.0G    /home/share/unix_files/cdat/miniconda2/envs
20K     /home/share/unix_files/cdat/miniconda2/etc
3.8M    /home/share/unix_files/cdat/miniconda2/include
99M     /home/share/unix_files/cdat/miniconda2/lib
4.0K    /home/share/unix_files/cdat/miniconda2/LICENSE.txt
665M    /home/share/unix_files/cdat/miniconda2/pkgs
16K     /home/share/unix_files/cdat/miniconda2/share
8.0K    /home/share/unix_files/cdat/miniconda2/ssl

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/*
56M     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/bin
1.9M    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/conda-meta
48K     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/etc
420K    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/imports
67M     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/include
1.7G    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/lib
24K     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/man
3.1M    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/mkspecs
16M     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/mod
3.5M    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/plugins
57M     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/share
4.3M    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/ssl
2.7M    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/tests
12K     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/var
107M    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/vistrails

Downgrading conda

Note: on June 1st 2017, we used the following step in order to downgrade conda and be ready to install UV-CDAT 2.10 that required an older version of conda (strictly older than 4.3.13) than the one used for 2.8

bash-4.1$ conda install -n root "conda<4.3.13"
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/share/unix_files/cdat/miniconda2:

The following packages will be DOWNGRADED due to dependency conflicts:

    conda: 4.3.13-py27_0 --> 4.3.11-py27_0

Proceed ([y]/n)? y

conda-4.3.11-p 100% |###############################################################| Time: 0:00:00   4.28 MB/s

bash-4.1$ conda info
Current conda install:

               platform : linux-64
          conda version : 4.3.11
       conda is private : False
      conda-env version : 4.3.11

Cloning uvcdat to add specific packages for LSCE

This section is about the creation of the cdatm14 environment

Notes about actually using the cdatm14 conda-based python

Note: using hard links, cloning a full environment only adds an extra 100M of disk space

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
2.7G    /home/share/unix_files/cdat/miniconda2

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.0G    /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

bash-4.1$ conda create -n cdatm14 --clone uvcdat-2.8.0
[...]

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
2.8G    /home/share/unix_files/cdat/miniconda2

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.0G    /home/share/unix_files/cdat/miniconda2/envs/cdatm14
82M     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

Downloading cdms2/vcs test data

You should download the test data and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any…)

(cdatm14) bash-4.1$ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data'

Finished downloading sample data to /home/share/unix_files/cdat/miniconda2/envs/cdatm14/share/uvcdat/sample_data

(cdatm14) bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/cdatm14/share/uvcdat/sample_data
174M    /home/share/unix_files/cdat/miniconda2/envs/cdatm14/share/uvcdat/sample_data

Packages that have no dependency problems

After cloning, we are ready to install some extra packages that may be requested by LSCE users

# Keep a trace of what will be installed
bash-4.1$ conda install --dry-run -n cdatm14 -c conda-forge -c uvcdat -c vacumm pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret vacumm > /home/scratch01/jypeter/lsce-extra_01_install_160223.txt

# Install...
bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat -c vacumm pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret vacumm

# Check the disk space after installation
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
3.6G    /home/share/unix_files/cdat/miniconda2

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.4G    /home/share/unix_files/cdat/miniconda2/envs/cdatm14
82M     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

# Check the disk space again after cleaning
bash-4.1$ conda clean all

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
2.9G    /home/share/unix_files/cdat/miniconda2

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
2.4G    /home/share/unix_files/cdat/miniconda2/envs/cdatm14
82M     /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

Lists of installed packages: lsce-extra_01_install_160223.txt

The following packages have no dependency problems and were installed later

High resolution data for basemap
- conda install -n cdatm14 -c conda-forge basemap-data-hires
PCMDI metrics package (PMP)
- conda install -n cdatm14 -c conda-forge -c pcmdi pcmdi_metrics
- Note: There was a read permission problem on the files - Solved in 2.10
XlsxWriter: a Python module for creating Excel XLSX files
- conda install -n cdatm14 xlsxwriter
- Note: this is a dependency of dreqPy
dreqPy: CMIP6 Data Request Python API
- pip install dreqPy
- Update with: pip install --upgrade dreqPy
  - Get version number with:
```
$ drq -v
dreqPy version 01.00.06 [Version 01.00.06]
```
CMOR: CMOR (Climate Model Output Rewriter) is used to produce CF-compliant netCDF files
- conda install -n cdatm14 -c conda-forge -c pcmdi -c uvcdat cmor
  - Did not complete the installation yet, because it requested a downgrade of libnetcdf (see github issue)
  - Installation issue was solved in cmor-3.2.3
dtw: DTW (Dynamic Time Warping) python module
- pip install dtw
- Requested by Sentia Goursaud

TODO

Add here packages that would be useful but have some problems that prevent their installation

ESMPy: ESMF Python Regridding Interface - Works in 2.10!
- conda install -n cdatm14 -c conda-forge esmpy
- Installation of esmpy 7.0.0 did not report errors, but esmpy seemed to have libraries based importation problems and has been removed
  - conda -n cdatm14 remove esmpy
GDAL: Geospatial Data Abstraction Library - Works in 2.10!
- Cannot be installed due to too many dependency problems, and because it requires the downgrade of geos and rpy2

Other packages

There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them…

shapely request a downgrade of geos when installed from conda-forge (see github issue) but it will work when installed from the default repo
- conda install -n cdatm14 shapely
- We get a slightly older version (1.5.16-py27_0 instead of 1.5.17-np111py27_2)
- Note: cartopy (below) requests the same downgrade of geos…

cartopy requests a downgrade of geos and matplotlib (see github issue)

We install the requested dependencies and then we force the installation of cartopy

(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat libxslt lxml owslib pyepsg
[...]
The following NEW packages will be INSTALLED:

    libxslt: 1.1.29-3      conda-forge
    lxml:    3.7.2-py27_1  conda-forge
    owslib:  0.14.0-py27_0 conda-forge
    pyepsg:  0.3.1-py27_0  conda-forge
[...]

(uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge cartopy
[...]
The following NEW packages will be INSTALLED:

    cartopy:    0.15.0-np111py27_0 conda-forge

The following packages will be DOWNGRADED due to dependency conflicts:

    geos:       3.6.0-0            conda-forge --> 3.5.1-1           conda-forge
    matplotlib: 2.0.0-np111py27_2  conda-forge --> 1.5.3-np111py27_7 conda-forge

Proceed ([y]/n)? n

(uvcdat-2.8.0) bash-4.1$ conda install --force -n cdatm14 -c conda-forge cartopy
[...]
The following NEW packages will be INSTALLED:

    cartopy: 0.15.0-np111py27_0 conda-forge

Proceed ([y]/n)? y

trying to install rpy2 will trigger the installation of R!
- We force the installation of only rpy2 in order to use the R already installed at LSCE
  - conda install --force -n cdatm14 -c conda-forge -c uvcdat rpy2

Extra packages list

pillow: the friendly PIL (Python Imaging Library) fork
pandas: Python Data Analysis Library
statsmodels: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests
seaborn: statistical data visualization
scikit-image: image processing in Python
seawater: Python re-write of the CSIRO seawater toolbox
gsw: Python implementation of the Thermodynamic Equation Of Seawater
netcdf4: a Python interface to the netCDF C library
pyferret: Ferret encapsulated in Python
vacumm: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
basemap-data-hires: high resolution data for basemap
PCMDI metrics package (PMP): objectively compare results from climate models with observations using well-established statistical tests
XlsxWriter: a Python module for creating Excel XLSX files
dreqPy: CMIP6 Data Request Python API
CMOR: CMOR (Climate Model Output Rewriter) is used to produce CF-compliant netCDF files
dtw: DTW (Dynamic Time Warping) python module

shapely: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
cartopy: a library providing cartographic tools for python
rpy2: providing simple and robust access to R from within Python

Removed packages

wget: pure python download utility
- Used conda install -n cdatm14 -c daler wget=3.2 in order to get version 3.2, the same as the one on Pypi
- REMOVED because it did not work correctly in the script where it was needed: it emulated wget with ftp and failed in our case after downloading the first file

Environments summary

After following the steps above, we get the following environments. Use the conda info –envs command to get the up-to-date list of available environments

Environment name	Server	`conda list`
uvcdat-2.8.0	LSCE ciclad	package list
cdatm14	LSCE ciclad	package list

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]

Table of Contents