CDAT 8.1 installation notes

[ Back to all versions ]

Follow the instructions about the conda-based versions of UV-CDAT initialization for actually using an installed version of 8.1

Using JYP version

If you mostly want to use the CDAT 8.1 installed by JYP, just use the following steps and skip the next sections:

Initialize conda:

Server	tcsh	bash
LSCE	source ~jypeter/.conda3_jyp.csh	source ~jypeter/.conda3_jyp.sh
ciclad	source ~jypmce/.conda3_jyp.csh	source ~jypmce/.conda3_jyp.sh

Activate the default environment: conda activate cdatm_py2

Check the Available environments for more options

What's New?

8.1 announcement and features summary (all versions)

Full Change log

Installation with Miniconda3

We assume that Miniconda3 is already installed. Otherwise, follow the the Installing Miniconda steps (and the Post-Miniconda3 installation section) we followed when installing CDAT 8.0

Updating conda

The conda package itself can be updated (if need be) with
conda update -n base -c defaults conda

Installing CDAT 8.1

We first check that we have indeed access to a conda installation, and we assume that we have a write-access to the conda hierarchy, and a few Gb of disk space available

>conda activate
(base) >conda env list | grep base
base * /home/share/unix_files/cdat/miniconda3
(base) >df -h /home/share/
Filesystem Size Used Avail Use% Mounted on
prolix3:/share 917G 171G 700G 20% /home/share
(base) >du -sh /home/share/unix_files/cdat/miniconda3
7.8G /home/share/unix_files/cdat/miniconda3

We can also use conda env list and remove some obsolete versions with conda remove -n obsolete_name –all to get some space

Python 2.7 version

$ conda create -n cdat-8.1_py2 -c cdat/label/v81 -c conda-forge python=2.7 cdat
# Generate the list of installed packages
$ conda list -n cdat-8.1_py2 > /home/scratch01/jypeter/cdat-8.1_py2_list_190307.txt

List of installed packages: cdat-8.1_py2_list_190307.txt

Python 3.6 version

$ conda create -n cdat-8.1_py3 -c cdat/label/v81 -c conda-forge python=3.6 cdat

Installing CDAT nightly

Notes:

This page is about CDAT 8.1, but we have added a short nightly section here as a convenient shortcut. This should probably be moved to a stand-alone nightly page later

The nightly version is not automatically updated, and may well be out-of-date

Official nightly page

Python 2.7 nightly version

$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge python=2.7 cdat

Not tested! Can this be updated with conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge –all ???

Python 3.6 nightly version

Not installed yet!

$ conda create -n cdat-nightly_py3 -c cdat/label/nightly -c conda-forge python=3.6 cdat

Cloning cdat to add specific packages for LSCE

This section is about the creation of the cdatm18 environment Notes about actually using the cdatm18 conda-based python

Note: using hard links, cloning a full environment uses less disk space than making a real copy

$ conda create -n cdatm18_py2 --clone cdat-8.1_py2
$ du -sh cdat-8.1_py2 cdatm18_py2
1.9G cdat-8.1_py2
448M cdatm18_py2

cdat nightly case

conda create -n cdatm-nightly_py2 --clone cdat-nightly_py2

Getting ready for a moving default CDAT

This step should probably be listed at the end, especially in a multi-user environment! If there is already a cdatm link, make sure that the new version is stable and working correctly before updating the cdatm link

We create a cdatm symbolic link in the envs directory, that has a stable name but can be moved to point to the latest default CDAT. In that case, most users can just activate this cdatm version and always get the latest stable version

$ cd /home/share/unix_files/cdat/miniconda3/envs
$ ln -s cdatm18_py2 cdatm_py2

Customizing UV-CDAT for LSCE

Downloading cdms2/vcs test data

You should download the test data (174M of data…) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any…)

$ conda activate cdatm18_py2
(cdatm18_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)'
[...]
Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
(cdatm18_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
174M /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data

Packages that have no dependency problems

After cloning, we are ready to install some extra packages that may be requested by LSCE users

We first try to install together as many packages as possible that don't require other channels than conda-forge, and that don't request a downgrade of what is already installed

We then install individual extra packages with conda install or pip install

# You can use the following to keep a trace of what will be installed
#$ conda install --dry-run -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm > somewhere/extra_packages.txt
# Install...
$ conda install -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
[...]
# Use a similar install line if you want to install the same packages in the python3 version

List of installed packages: lsce-extra_01_install_190304.txt

Packages installed with pip

dreqPy: CMIP6 Data Request Python API

conda activate cdatm18_py2
pip install dreqPy

Update with: pip install --upgrade dreqPy

Get version number with:

$ drq -v
dreqPy version 01.00.29 [Version 01.00.29]

The following packages have no dependency problems and were installed (or updated) later

CMOR: CMOR (Climate Model Output Rewriter) is used to produce CF-compliant netCDF files

conda install -n cdatm18_py2 -c conda-forge cmor

Get version number with: python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )'

Warning! CMOR currently requires Python 2.7

vacumm: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations

conda install -n cdatm18_py2 -c conda-forge -c vacumm vacumm

Warning! standard install of vacumm does not work for Python3

spanlib: Spectral Analysis Library

conda install -n cdatm18_py2 -c stefraynaud -c conda-forge spanlib

Test: python -c 'from spanlib.analyzer import Analyzer'

CliMAF: a Climate Model Assessment Framework

Installation notes

TODO

Add here packages that would be useful and have not been installed yet, or have some problems that prevent their installation

iris: A Python library for Meteorology and Climatology

conda install -n cdatxxx -c conda-forge iris

wrf-python: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model

conda install -n cdatm15 -c conda-forge wrf-python

glances: a cross-platform monitoring tool (similar to top)

conda install -n cdatm15 -c conda-forge glances

glances@github

Other packages

There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them…

NO such packages now!

Updating some packages

Some packages change more often than others, and can be easily updated the following way:

dreqPy

Update with: pip install --upgrade dreqPy

Get version number with: drq -v

Cleaning up things

Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg pcmdi-metrics in 2.8.0 and cdp in 2.10)

We check if some of the installed files are missing read access for the group or other, and we manually change the permissions

>find /home/share/unix_files/cdat/miniconda3/envs \! -perm /g+r,o+r -ls
# Everything OK!

Extra packages list

pillow: the friendly PIL (Python Imaging Library) fork

pandas: Python Data Analysis Library

statsmodels: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests

seaborn: statistical data visualization

scikit-image: image processing in Python

seawater: Python re-write of the CSIRO seawater toolbox

gsw: Python implementation of the Thermodynamic Equation Of Seawater

vacumm: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations

netcdf4: a Python interface to the netCDF C library

pyferret: Ferret encapsulated in

basemap-data-hires: high resolution data for basemap

PCMDI metrics package (PMP): objectively compare results from climate models with observations using well-established statistical tests

XlsxWriter: a Python module for creating Excel XLSX files

Note: this is a dependency of dreqPy

dreqPy: CMIP6 Data Request Python API

CMOR: CMOR (Climate Model Output Rewriter) is used to produce CF-compliant netCDF files

dtw: DTW (Dynamic Time Warping) python module

shapely: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)

cartopy: a library providing cartographic tools for python

rpy2: providing simple and robust access to R from within Python

cmocean: beautiful colormaps for oceanography

OSGeo/GDAL: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats

Python GDAL/OGR Cookbook

spanlib: Spectral Analysis Library

windspharm: spherical harmonic wind analysis in Python

CliMAF: a Climate Model Assessment Framework

Removed packages

NO removed packages!

Environments summary

After following the steps above, we get the following environments. Use the conda env list command (same result as conda info --envs) to get the up-to-date list of available environments.

Use conda list -n env_name to get a detailed list of which packages/modules are installed, or check the conda Installation history section to get more details

Environment name	Server	Summary
cdat-8.1_py2	LSCE ciclad	CDAT 8.1 & Python 2.7
cdat-8.1_py3	LSCE	CDAT 8.1 & Python 3.6
cdatm18_py2	LSCE ciclad	CDAT 8.1 & P 2.7 JYP version
cdatm18_py3	LSCE	CDAT 8.1 & P 3.7 JYP version
cdat-nightly_py2	LSCE ciclad	CDAT nightly & Python 2.7
cdatm-nightly_py2	LSCE ciclad	CDAT nightly & P 2.7 JYP version

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]