UV-CDAT 2.12 installation notes

UV-CDAT 2.12 installation notes

Follow the instructions about the conda-based versions of UV-CDAT initialization for actually using an installed version of 2.12

What's New?

2.12 announcement and features summary (all versions)
Full Change log

Installation with Miniconda2

Actual installation

bash-4.1$ conda create -n uvcdat-2.12 -c conda-forge -c uvcdat uvcdat=2.12

# Generate the list of installed packages
conda list -n uvcdat-2.12 > /home/scratch01/jypeter/uvcdat-2.12_list_170907.txt

List of installed packages: uvcdat-2.12_list_170907.txt

Disk space after 2.12 installation

bash-4.1$ conda env list
# conda environments:
#
cdatm                    /home/share/unix_files/cdat/miniconda2/envs/cdatm
cdatm14                  /home/share/unix_files/cdat/miniconda2/envs/cdatm14
cdatm15                  /home/share/unix_files/cdat/miniconda2/envs/cdatm15
uvcdat-2.10              /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
uvcdat-2.12              /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.12
uvcdat-2.8.0             /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0
root                  *  /home/share/unix_files/cdat/miniconda2

 > du -sh /home/share/unix_files/cdat/miniconda2
15G	/home/share/unix_files/cdat/miniconda2

 > du -sh /home/share/unix_files/cdat/miniconda2/* 
4.4M	/home/share/unix_files/cdat/miniconda2/bin
4.0K	/home/share/unix_files/cdat/miniconda2/conda-bld
272K	/home/share/unix_files/cdat/miniconda2/conda-meta
13G	/home/share/unix_files/cdat/miniconda2/envs
20K	/home/share/unix_files/cdat/miniconda2/etc
3.8M	/home/share/unix_files/cdat/miniconda2/include
99M	/home/share/unix_files/cdat/miniconda2/lib
4.0K	/home/share/unix_files/cdat/miniconda2/LICENSE.txt
2.1G	/home/share/unix_files/cdat/miniconda2/pkgs
376K	/home/share/unix_files/cdat/miniconda2/share
364K	/home/share/unix_files/cdat/miniconda2/ssl

 > du -sh /home/share/unix_files/cdat/miniconda2/envs/*
0	/home/share/unix_files/cdat/miniconda2/envs/cdatm
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
4.5G	/home/share/unix_files/cdat/miniconda2/envs/cdatm15
630M	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
2.7G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.12
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

Cloning uvcdat to add specific packages for LSCE

This section is about the creation of the cdatm16 environment

Notes about actually using the cdatm16 conda-based python

Cloning

bash-4.1$ conda create -n cdatm16 --clone uvcdat-2.12
Source:      /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.12
Destination: /home/share/unix_files/cdat/miniconda2/envs/cdatm16
[...]

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
0	/home/share/unix_files/cdat/miniconda2/envs/cdatm
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
4.5G	/home/share/unix_files/cdat/miniconda2/envs/cdatm15
2.7G	/home/share/unix_files/cdat/miniconda2/envs/cdatm16
630M	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
242M	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.12
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

Checking the permissions

The files of some packages may have been installed with incorrect access rights, and may not be readable by everybody (e.g. pcmdi_metrics/issues/496 or cdp/issues/21). We have to check and correct this by hand:

(cdatm16) bash-4.1$ find uvcdat-2.12 \! -perm /g+r,o+r -ls
24774102    4 -rw-------   3 jypeter  lsce          670 Apr  4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp/cdp_output.py
24774113    4 -rw-------   3 jypeter  lsce         2436 Apr  4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp/cdp_parameter.pyc
24774111    4 -rw-------   3 jypeter  lsce         1821 Apr  4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp/cdp_parser.py
[...]
24774098    4 -rw-------   3 jypeter  lsce          234 Apr  4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO
24774089    4 -rw-------   3 jypeter  lsce            1 Apr  4 20:35 uvcdat-2.12/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe

(cdatm16) bash-4.1$ chmod -R a+r cdatm16/lib/python2.7/site-packages/cdp* chmod -R a+r uvcdat-2.12/lib/python2.7/site-packages/cdp*

Getting ready for a moving default CDAT

See the note about the cdatm link/version. At this point of installation, we are not yet ready to have cdatm point to 2.12.

Customizing UV-CDAT for LSCE

Downloading cdms2/vcs test data

You should download the test data and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any…)

bash-4.1$ source activate cdatm16

(cdatm15) bash-4.1$ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data'
[...]
Finished downloading sample data to /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data

(cdatm16) bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/cdatm16/share/uvcdat/sample_data
174M	/home/share/unix_files/cdat/miniconda2/envs/cdatm16/share/uvcdat/sample_data

Downgrading numpy

bash-4.1$ conda install -n cdatm16 -c conda-forge -c pcmdi -c uvcdat numpy=1.12
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment /home/share/unix_files/cdat/miniconda2/envs/cdatm16:

The following NEW packages will be INSTALLED:

    blas:         1.1-openblas              conda-forge
    openblas:     0.2.19-2                  conda-forge

The following packages will be UPDATED:

    binaryio:     2.12-np113py27h8073811_0  uvcdat      --> 2.12-np112py27h5813a3b_0      uvcdat     
    cdms2:        2.12-np113py27_0          conda-forge --> 2.12-np112py27_0              conda-forge
    cdtime:       2.12-np113py27_2          conda-forge --> 2.12-np112py27_2              conda-forge
    cssgrid:      2.12-np113py27h3898552_0  uvcdat      --> 2.12-np112py27hf1c8260_0      uvcdat     
    dsgrid:       2.12-np113py27h9d524be_0  uvcdat      --> 2.12-np112py27h822527b_0      uvcdat     
    genutil:      2.12-np113py27_0          conda-forge --> 2.12-np112py27_0              conda-forge
    lmoments:     2.12-np113py27h5574120_0  uvcdat      --> 2.12-np112py27h665d648_0      uvcdat     
    matplotlib:   1.5.3-np113py27_8         conda-forge --> 2.0.0-np112py27_2             conda-forge
    natgrid:      2.12-np113py27h481ab8b_0  uvcdat      --> 2.12-np112py27h569c149_0      uvcdat     
    ort:          2.12-np113py27hd89025a_0  uvcdat      --> 2.12-np112py27h964b06a_0      uvcdat     
    regridpack:   2.12-np113py27hf1cacc0_0  uvcdat      --> 2.12-np112py27h6a169cd_0      uvcdat     
    scikit-learn: 0.19.0-np113py27_0                    --> 0.19.0-py27_blas_openblas_201 conda-forge [blas_openblas]
    scipy:        0.19.1-np113py27_0                    --> 0.19.1-py27_blas_openblas_202 conda-forge [blas_openblas]
    shgrid:       2.12-np113py27h2504d4d_0  uvcdat      --> 2.12-np112py27h6f2dc27_0      uvcdat     
    trends:       2.12-np113py27hedb240a_0  uvcdat      --> 2.12-np112py27he3ac0c2_0      uvcdat     
    uvcmetrics:   2.2.1-np113py27h151233a_0 uvcdat      --> 2.2.1-np112py27_0             uvcdat     
    zonalmeans:   2.12-np113py27h96f9f80_0  uvcdat      --> 2.12-np112py27hbce4e2c_0      uvcdat     

The following packages will be SUPERCEDED by a higher-priority channel:

    numpy:        1.13.1-py27_0                         --> 1.12.1-py27_blas_openblas_200 conda-forge [blas_openblas]

Proceed ([y]/n)?

Packages that have no dependency problems

After cloning, we are ready to install some extra packages that may be requested by LSCE users

# Keep a trace of what will be installed
bash-4.1$ conda install --dry-run -n cdatm16 -c conda-forge -c uvcdat -c pcmdi -c scitools -c vacumm pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy iris rpy2 cmocean iris gdal vacumm > /home/scratch01/jypeter/lsce-cdatm16-extra_01_install_170907.txt

# Install...
bash-4.1$ conda install -n cdatm15 -c conda-forge -c uvcdat -c pcmdi pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy rpy2
[...]

# Check the disk space after installation
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
11G	/home/share/unix_files/cdat/miniconda2

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/*
0	/home/share/unix_files/cdat/miniconda2/envs/cdatm
2.8G	/home/share/unix_files/cdat/miniconda2/envs/cdatm14
3.7G	/home/share/unix_files/cdat/miniconda2/envs/cdatm15
574M	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10
2.0G	/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0

# Check the disk space again after cleaning
bash-4.1$ conda clean --all

bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
9.9G	/home/share/unix_files/cdat/miniconda2

List of installed packages: lsce-extra_01_install_170808.txt

Disk space used after adding the extra packages, in in the new directory hierarchy (cdat/miniconda2b, on the obelix servers)

bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b 
6.4G	/home/share/unix_files/cdat/miniconda2b

bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/*
0	/home/share/unix_files/cdat/miniconda2b/envs/cdatm
3.8G	/home/share/unix_files/cdat/miniconda2b/envs/cdatm15
630M	/home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10

# Check the disk space again after cleaning
bash-4.2$ conda clean --all

bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b
5.4G	/home/share/unix_files/cdat/miniconda2b

Packages installed with pip

dreqPy: CMIP6 Data Request Python API
- pip install dreqPy
- Update with: pip install --upgrade dreqPy
  - Get version number with:
```
$ drq -v
dreqPy version 01.00.06 [Version 01.00.06]
```
dtw: DTW (Dynamic Time Warping) python module
- pip install dtw
- Requested by Sentia Goursaud

The following packages have no dependency problems and were installed (or updated) later

ESMPy: ESMF Python Regridding Interface

We used the following to get a more up-to-date version of esmpy than the one bundled with 2.10

bash-4.1$ conda list | grep -i esm
esmf                      7.0.0                         6    conda-forge
esmpy                     7.0.0                    py27_1    conda-forge

bash-4.1$ conda install -n cdatm15 -c nesii/channel/dev-esmf -c conda-forge esmpy=7.1.0.dev32
[...]

bash-4.1$ conda list | grep -i esm
esmf                      7.1.0.dev32                   1    nesii/channel/dev-esmf
esmpy                     7.1.0.dev32              py27_1    nesii/channel/dev-esmf

cmocean: beautiful colormaps for oceanography
- conda install -n cdatm15 -c conda-forge cmocean
iris: A Python library for Meteorology and Climatology
- conda install -n cdatm15 -c conda-forge -c scitools iris
OSGeo/GDAL: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
- conda install -c conda-forge gdal
- Python GDAL/OGR Cookbook

TODO

Add here packages that would be useful but have some problems that prevent their installation

vacumm: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
- As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see github issue)!
- Problem solved with vacumm 3.4.1-1 on August 31st! Installation was OK with
  conda install -n cdatm15 -c conda-forge -c vacumm vacumm

Install PySPLIT for Sentia
PySPLIT A package for generating HYSPLIT air parcel trajectories trajectories, performing moisture uptake analyses, expediting HYSPLIT cluster analysis, and for visualizing trajectories, clusters, and along-trajectory meteorological data
Works with HYSPLiT
1. conda install -n cdatm15 -c conda-forge fiona geopandas
2. pip install pysplit

Other packages

There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them…

NO such packages now!

Updating some packages

Some packages change more often than others, and can be easily updated the following way:

CMOR
- Update with: conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor
  - Get version number with: python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH'

dreqPy
- Update with: pip install --upgrade dreqPy
  - Get version number with: drq -v

Cleaning up things

Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg pcmdi-metrics in 2.8.0 and cdp in 2.10)

We check if some of the installed files are missing read access for the group or other, and we manually change the permissions

(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls
50333969    4 -rw-------   2 jypeter  lsce          234 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO
50333971    4 -rw-------   2 jypeter  lsce          291 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/SOURCES.txt
50333962    4 -rw-------   2 jypeter  lsce            4 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/top_level.txt
50333961    4 -rw-------   2 jypeter  lsce            1 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/dependency_links.txt
50333960    4 -rw-------   2 jypeter  lsce            1 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe
50333987    8 -rw-------   2 jypeter  lsce         5149 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.pyc
50333986    8 -rw-------   2 jypeter  lsce         4208 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.py
50333966    4 -rw-------   2 jypeter  lsce          150 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.pyc
50333968    4 -rw-------   2 jypeter  lsce          201 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.py
50333988    8 -rw-------   2 jypeter  lsce         5365 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.pyc
50333982    4 -rw-------   2 jypeter  lsce         1821 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.py
50333981    4 -rw-------   2 jypeter  lsce         1773 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.py
50333973    4 -rw-------   2 jypeter  lsce          670 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.py
50333980    4 -rw-------   2 jypeter  lsce         1642 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.pyc
50333979    4 -rw-------   2 jypeter  lsce         1474 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.pyc
50333967    4 -rw-------   2 jypeter  lsce          187 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.py
50333989    0 -rw-------   2 jypeter  lsce            0 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.py
50333985    4 -rw-------   2 jypeter  lsce         3843 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.py
50333984    4 -rw-------   2 jypeter  lsce         2436 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.pyc
50333970    4 -rw-------   2 jypeter  lsce          288 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.py
50333974    4 -rw-------   2 jypeter  lsce          692 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.pyc
50333978    4 -rw-------   2 jypeter  lsce          878 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.pyc
50333976    4 -rw-------   2 jypeter  lsce          734 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.py
50333983    4 -rw-------   2 jypeter  lsce         2269 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.pyc
50333975    4 -rw-------   2 jypeter  lsce          724 Apr  4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.pyc
(cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp*
(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls

Extra packages list

pillow: the friendly PIL (Python Imaging Library) fork
pandas: Python Data Analysis Library
statsmodels: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests
seaborn: statistical data visualization
scikit-image: image processing in Python
seawater: Python re-write of the CSIRO seawater toolbox
gsw: Python implementation of the Thermodynamic Equation Of Seawater
netcdf4: a Python interface to the netCDF C library
pyferret: Ferret encapsulated in
basemap-data-hires: high resolution data for basemap
PCMDI metrics package (PMP): objectively compare results from climate models with observations using well-established statistical tests
XlsxWriter: a Python module for creating Excel XLSX files
- Note: this is a dependency of dreqPy
dreqPy: CMIP6 Data Request Python API
CMOR: CMOR (Climate Model Output Rewriter) is used to produce CF-compliant netCDF files
dtw: DTW (Dynamic Time Warping) python module
shapely: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
cartopy: a library providing cartographic tools for python
rpy2: providing simple and robust access to R from within Python
cmocean: beautiful colormaps for oceanography
iris: A Python library for Meteorology and Climatology
OSGeo/GDAL: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats

Removed packages

NO removed packages!

Environments summary

After following the steps above, we get the following environments. Use the conda info --envs or the conda env list command to get the up-to-date list of available environments

Environment name	Server	`conda list`
uvcdat-2.10	LSCE	package list
cdatm15	LSCE	package list

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]

Table of Contents