Installing and maintaining UV-CDAT with conda

Check the UV-CDAT conda-based versions installed by JYP page to see what is maintained by JYP
Recent versions (2.8.0 and later?) of UV-CDAT are installed and maintained with the conda package, dependency and environment manager. This page is about conda/Miniconda/Anaconda installation and related notes

What should I install?

The difference between conda, Miniconda and Anaconda is detailed in the General questions part of the FAQ

Installers with a 2 extension will use Python2.7. Those with 3 will use Python3.x

any version of conda can install any version of Python (and UV-CDAT), and you can have several versions of Python (and UV-CDAT) installed at the same time

some parts of CDAT still need Python 2.7, so we will use the 2 version of conda (but it would also work with 3)

You can do almost anything with conda, and you are mostly limited by disk space.

If you know that you are going to use UV-CDAT, you might as well use the conda from Miniconda to avoid uselessly duplicating packages downloaded from the official Anaconda repository (used by Anaconda) with the same packages downloaded from the conda-forge repository (used by UV-CDAT).

Miniconda

Miniconda is basically just a self contained minimal Python distribution, with only the packages required to run conda

Installing Miniconda

Follow the Quick install steps to install Miniconda

Get the Python 2.7 Linux 64-bit (bash installer)

cd /home/scratch01/jypeter

wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh (28 Mo, 16 Feb 2017)

Execute the installer

bash Miniconda2-latest-Linux-x86_64.sh

Accept the license

Specify an explicit installation path outside of the home directory, with enough disk space, preferably on a disk that is not backed up: /home/share/unix_files/cdat/miniconda2

Resulting miniconda2 directory size is 118M

> du -sh miniconda2/
118M miniconda2/
> cd miniconda2/
> du -sh *
4.4M bin
268K conda-meta
4.0K envs
20K etc
3.8M include
99M lib
4.0K LICENSE.txt
23M pkgs
16K share
8.0K ssl

Accept the automatic update of the PATH variable in .bashrc (note: the installer automatically creates a backup of .bashrc)

# added by Miniconda2 4.3.11 installer
export PATH=“/home/share/unix_files/cdat/miniconda2/bin:$PATH”

Open a new terminal (if you use bash) or type bash and check if the python from the conda distribution has been correctly installed

Update and clean the new installation

conda updpate --all

conda clean --all

bash-4.1$ which python
/home/share/unix_files/cdat/miniconda2/bin/python
bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
114M /home/share/unix_files/cdat/miniconda2
bash-4.1$ conda list
# packages in environment at /home/share/unix_files/cdat/miniconda2:
#
cffi 1.9.1 py27_0
conda 4.3.13 py27_0
conda-env 2.6.0 0
cryptography 1.7.1 py27_0
[...]

Follow the 30-minute test drive steps

Removing Miniconda

Remove the installation directory: /home/share/unix_files/cdat/miniconda2

Remove the PATH update line from ~/.bash_profile

Remove the configuration files and directories from the home directory: ~/.condarc ~/.conda ~/.continuum

Anaconda

Installing Anaconda

Follow the Full installation steps to install Miniconda

Installation package Anaconda2-4.3.0-Linux-x86_64.sh is 462M (17 Feb 2017)

Size after initial installation : 2G

jypeter@asterix3 - ...anaconda2 - 93 >du -sh /home/share/unix_files/cdat/anaconda2
2.0G /home/share/unix_files/cdat/anaconda2
jypeter@asterix3 - ...anaconda2 - 94 >du -sh /home/share/unix_files/cdat/anaconda2/*
53M /home/share/unix_files/cdat/anaconda2/bin
3.3M /home/share/unix_files/cdat/anaconda2/conda-meta
564K /home/share/unix_files/cdat/anaconda2/doc
4.0K /home/share/unix_files/cdat/anaconda2/envs
80K /home/share/unix_files/cdat/anaconda2/etc
53M /home/share/unix_files/cdat/anaconda2/include
1.7G /home/share/unix_files/cdat/anaconda2/lib
244K /home/share/unix_files/cdat/anaconda2/libexec
8.0K /home/share/unix_files/cdat/anaconda2/LICENSE.txt
3.1M /home/share/unix_files/cdat/anaconda2/mkspecs
320K /home/share/unix_files/cdat/anaconda2/phrasebooks
186M /home/share/unix_files/cdat/anaconda2/pkgs
340K /home/share/unix_files/cdat/anaconda2/plugins
156K /home/share/unix_files/cdat/anaconda2/qml
4.0K /home/share/unix_files/cdat/anaconda2/sbin
1.3M /home/share/unix_files/cdat/anaconda2/share
8.0K /home/share/unix_files/cdat/anaconda2/ssl
12K /home/share/unix_files/cdat/anaconda2/translations
24K /home/share/unix_files/cdat/anaconda2/var

Size after conda update conda and conda update --all: 2.1G

List of installed packages : unix_files_cdat_anaconda2_170217.txt

Removing Anaconda

The steps are the same as for Removing Miniconda

Notes

Disk usage

conda will use hard links and share packages common to several installed environments to minimize disk usage, but there is no harm in being careful and checking things!

You should avoid installing conda stuff in your home directory (~/.conda)

You should regularly monitor the size of the directories listed in conda info

~/.conda: the hidden conda directory in your home directory

/somewhere/anaconda2 or /somewhere/miniconda2: the main installation directories

/somewhere/anaconda2/envs: the directory where all the different environments will be installed

/somewhere/anaconda2/pkgs: the directory where all the packages used are installed

conda uses hard links to avoid duplicating the files, so you have to be careful when you monitor the size of something else than the main installation directory (the size of hard linked files can be taken into account several times)

>du -sh /home/share/unix_files/cdat/anaconda2
4.6G /home/share/unix_files/cdat/anaconda2
>du -sh /home/share/unix_files/cdat/anaconda2/envs/*
2.0G /home/share/unix_files/cdat/anaconda2/envs/uvcdat-2.8.0
>du -sh /home/share/unix_files/cdat/anaconda2/pkgs
3.8G /home/share/unix_files/cdat/anaconda2/pkgs
>du -sh /home/share/unix_files/cdat/anaconda2/lib
1.7G /home/share/unix_files/cdat/anaconda2/lib

See the note about conda clean --all in the Useful conda commands section

Useful conda commands

There is also a summary available in the Conda cheat sheet

Getting help

conda --help, conda -h
same as conda command --help

Available environments

conda info --envs
same as conda env list

Creating an environment

conda create --name new_environment package1 package2 packageN
conda create -n new_environment package1 package2 packageN

Deleting an environment

conda remove --name crap_environment --all
conda clean --all

Activating or going back to root

source activate existing_environment

source activate will activate the root environment

source deactivate

Installed packages

conda list

This will also list the packages installed with pip install package1 package2 packageN!!

Searching packages

conda search --full-name package_exact_name
conda search package_partial_name

Installing and removing packages
Add -n target_environment to act on another environment than the active one

conda install package1 package2 packageN

Add --channel channel_url_or_alias to install from a specific channel

If there is no URL, conda will use the value of channel_alias that defaults to https://conda.anaconda.org/

UV-CDAT relies on packages maintained in conda-forge (why conda-forge?) and the installation commands should include the following channel option:
--channel conda-forge

conda remove package1 package2 packageN

Updating conda (update the conda package, not the full Miniconda)

conda update conda

conda does not automatically clean the content of the pkgs directory! Use the following to Remove index cache, lock files, tarballs, unused cache packages, and source cache

conda clean --all

Misc

Available 'official' packages

Other packages

Packages in conda-forge

conda Full documentation

conda FAQ

Sharing (replicating) an environment

Centralized installation

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]