User Tools

Site Tools


other:uvcdat:conda_notes

Installing and maintaining UV-CDAT with conda

Check the UV-CDAT conda-based versions installed by JYP page to see what is maintained by JYP

Versions 2.4.1 and later of UV-CDAT are installed and maintained with the conda package, dependency and environment manager. This page is about conda/Miniconda/Anaconda installation and related notes

What should I install?

  • The difference between conda, Miniconda and Anaconda is detailed in the General questions part of the FAQ
  • Installers with a 2 extension will use Python2.7. Those with 3 will use Python3.x
    • any version of conda can install any version of Python (and UV-CDAT), and you can have several versions of Python (and UV-CDAT) installed at the same time
    • some parts of CDAT still need Python 2.7, so we will use the 2 version of conda (but it would also work with 3)
  • You can do almost anything with conda, and you are mostly limited by disk space.
    • If you know that you are going to use UV-CDAT, you might as well use the conda from Miniconda to avoid uselessly duplicating packages downloaded from the official Anaconda repository (used by Anaconda) with the same packages downloaded from the conda-forge repository (used by UV-CDAT).

Miniconda

Note: we have shown in UV-CDAT 2.8.0 installation notes that using Miniconda for installing UV-CDAT requires indeed less space than using Anaconda

Miniconda is basically just a self contained minimal Python distribution, with only the packages required to run conda

Installing Miniconda

  • Follow the Quick install steps to install Miniconda
    • Execute the installer
      • bash Miniconda2-latest-Linux-x86_64.sh
        • Accept the license
        • Specify an explicit installation path outside of the home directory, with enough disk space (more than 3G if you are going to install UV-CDAT and some extra packages), preferably on a disk that is not backed up: /home/share/unix_files/cdat/miniconda2
          • Resulting miniconda2 directory size is 118M
             > du -sh miniconda2/
            118M    miniconda2/
             > cd miniconda2/
             > du -sh *
            4.4M    bin
            268K    conda-meta
            4.0K    envs
            20K     etc
            3.8M    include
            99M     lib
            4.0K    LICENSE.txt
            23M     pkgs
            16K     share
            8.0K    ssl
        • Accept the automatic update of the PATH variable in .bashrc (note: the installer automatically creates a backup of .bashrc)
          • # added by Miniconda2 4.3.21 installer
            export PATH=“/home/share/unix_files/cdat/miniconda2/bin:$PATH”
    • Open a new terminal (if you use bash) or type bash and check if the python from the conda distribution has been correctly installed
      • Update and clean the new installation
        • conda update --all
        • conda clean --all
      • bash-4.1$ which python
        /home/share/unix_files/cdat/miniconda2/bin/python
        bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2
        114M    /home/share/unix_files/cdat/miniconda2
        bash-4.1$ conda list
        # packages in environment at /home/share/unix_files/cdat/miniconda2:
        #
        cffi                      1.9.1                    py27_0
        conda                     4.3.13                   py27_0
        conda-env                 2.6.0                         0
        cryptography              1.7.1                    py27_0
        [...]
    • Follow the 30-minute test drive steps

Removing Miniconda

  • Remove the installation directory: /home/share/unix_files/cdat/miniconda2
  • Remove the PATH update line from ~/.bash_profile
  • Remove the configuration files and directories from the home directory: ~/.condarc ~/.conda ~/.continuum

Anaconda

Installing Anaconda

  • Follow the Full installation steps (and what is detailed above in Installing Miniconda) to install Anaconda
    • Installation package Anaconda2-4.3.0-Linux-x86_64.sh is 462M (17 Feb 2017)
    • Size after initial installation : 2G
      jypeter@asterix3 - ...anaconda2 - 93 >du -sh /home/share/unix_files/cdat/anaconda2
      2.0G    /home/share/unix_files/cdat/anaconda2
      jypeter@asterix3 - ...anaconda2 - 94 >du -sh /home/share/unix_files/cdat/anaconda2/*
      53M     /home/share/unix_files/cdat/anaconda2/bin
      3.3M    /home/share/unix_files/cdat/anaconda2/conda-meta
      564K    /home/share/unix_files/cdat/anaconda2/doc
      4.0K    /home/share/unix_files/cdat/anaconda2/envs
      80K     /home/share/unix_files/cdat/anaconda2/etc
      53M     /home/share/unix_files/cdat/anaconda2/include
      1.7G    /home/share/unix_files/cdat/anaconda2/lib
      244K    /home/share/unix_files/cdat/anaconda2/libexec
      8.0K    /home/share/unix_files/cdat/anaconda2/LICENSE.txt
      3.1M    /home/share/unix_files/cdat/anaconda2/mkspecs
      320K    /home/share/unix_files/cdat/anaconda2/phrasebooks
      186M    /home/share/unix_files/cdat/anaconda2/pkgs
      340K    /home/share/unix_files/cdat/anaconda2/plugins
      156K    /home/share/unix_files/cdat/anaconda2/qml
      4.0K    /home/share/unix_files/cdat/anaconda2/sbin
      1.3M    /home/share/unix_files/cdat/anaconda2/share
      8.0K    /home/share/unix_files/cdat/anaconda2/ssl
      12K     /home/share/unix_files/cdat/anaconda2/translations
      24K     /home/share/unix_files/cdat/anaconda2/var
    • Size after conda update conda and conda update –all: 2.1G

Removing Anaconda

The steps are the same as for Removing Miniconda

Notes

Disk usage

conda will use hard links and share packages common to several installed environments to minimize disk usage, but there is no harm in being careful and checking things!

  • Because of the likely large disk space usage, you should avoid installing conda stuff in your home directory (~/.conda)
  • You should regularly monitor the size of the directories listed in conda info
    • ~/.conda: the hidden conda directory in your home directory
    • /somewhere/anaconda2 or /somewhere/miniconda2: the main installation directories
      • /somewhere/anaconda2/envs: the directory where all the different environments will be installed
      • /somewhere/anaconda2/pkgs: the directory where all the packages used are installed
  • conda uses hard links to avoid duplicating the files, so you have to be careful when you monitor the size of something else than the main installation directory (the size of hard linked files can be taken into account several times)
     >du -sh /home/share/unix_files/cdat/anaconda2
    4.6G    /home/share/unix_files/cdat/anaconda2
     >du -sh /home/share/unix_files/cdat/anaconda2/envs/*
    2.0G    /home/share/unix_files/cdat/anaconda2/envs/uvcdat-2.8.0
     >du -sh /home/share/unix_files/cdat/anaconda2/pkgs
    3.8G    /home/share/unix_files/cdat/anaconda2/pkgs
     >du -sh /home/share/unix_files/cdat/anaconda2/lib
    1.7G    /home/share/unix_files/cdat/anaconda2/lib
  • See the note about conda clean –all in the Useful conda commands section

Useful conda commands

There is also a summary available in the Conda cheat sheet

  • Getting help
    • conda --help, conda -h
      same as conda command --help
  • Available environments
    • conda info –envs
      same as conda env list
  • Creating an environment
    • conda create --name new_environment package1 package2 packageN
      conda create -n new_environment package1 package2 packageN
  • Deleting an environment
    • conda remove --name crap_environment --all
      conda clean –all
    • The following seems to do the same: conda env remove -n crap_environment
  • Activating or going back to root/base
    • conda activate existing_environment (used to be source activate existing_environment in older versions of conda)
      • conda activate will activate the root/base environment
    • You can use which python to determine where is the current Python you are using, or which -a python (-a will only work in bash) to display all the Python executables in the current search path
    • conda deactivate (used to be source deactivate)
  • Installed packages
    • conda list
      • This will also list the packages installed with pip install package1 package2 packageN!!
  • Searching packages
    • conda search --full-name package_exact_name
      conda search package_partial_name
    • See also the --channel option below
  • Installing and removing packages
    Add -n target_environment to act on another environment than the active one
    • conda install package1 package2 packageN
      • Add --channel channel_url_or_alias to install from a specific channel
        • If there is no URL, conda will use the value of channel_alias that defaults to https://conda.anaconda.org/
        • UV-CDAT relies on packages maintained in conda-forge (why conda-forge?) and the installation commands should include the following channel option:
          --channel conda-forge
    • conda remove package1 package2 packageN
  • Updating conda (update the conda package, not the full Miniconda)
    • conda update -n base -c defaults conda
  • Downgrading conda (required for installing UV-CDAT 2.10)
    • bash-4.1$ conda install -n root "conda<4.3.13"
      Fetching package metadata .........
      Solving package specifications: .
      
      Package plan for installation in environment /home/share/unix_files/cdat/miniconda2:
      
      The following packages will be DOWNGRADED due to dependency conflicts:
      
          conda: 4.3.13-py27_0 --> 4.3.11-py27_0
      
      Proceed ([y]/n)? y
      
      conda-4.3.11-p 100% |###############################################################| Time: 0:00:00   4.28 MB/s
      
      bash-4.1$ conda info
      Current conda install:
      
                     platform : linux-64
                conda version : 4.3.11
             conda is private : False
            conda-env version : 4.3.11
  • conda does not automatically clean the content of the pkgs directory! Use the following to Remove index cache, lock files, tarballs, unused cache packages, and source cache
    • conda clean --all

Installation history

You can get the current list of installed packages with conda list, but it may also be useful to know the installation order:

  • conda list --revisions revisions will display exactly what was installed and when (including the updates, and automatic dependencies installation)
    • You can theoretically roll back to a previous revision with: conda install --revision RR
  • There is more information in the conda-meta/history installation log files. e.g., if you need to know the list of installation commands used:
     $ egrep -e 'cmd:' -e '==>' /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/conda-meta/history
    ==> 2019-03-07 16:19:55 <==
    # cmd: /home/share/unix_files/cdat/miniconda3/bin/conda create -n cdatm18_py2 --clone cdat-8.1_py2
    ==> 2019-03-07 16:35:11 <==
    # cmd: /home/share/unix_files/cdat/miniconda3/bin/conda install -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
    ==> 2019-03-07 16:57:50 <==
    # cmd: /home/share/unix_files/cdat/miniconda3/bin/conda install -n cdatm18_py2 -c conda-forge -c vacumm vacumm
    ==> 2019-03-12 14:35:31 <==
    # cmd: /home/share/unix_files/cdat/miniconda3/bin/conda install -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
    ==> 2019-03-12 15:49:54 <==
    # cmd: /home/share/unix_files/cdat/miniconda3/bin/conda install -n cdatm18_py2 -c stefraynaud -c conda-forge spanlib
    ==> 2019-03-12 17:30:26 <==
    # cmd: /home/share/unix_files/cdat/miniconda3/bin/conda install -n cdatm18_py2 -c conda-forge cmor
    • You can also, use a script and the conda module available in the base environment: example

Misc

other/uvcdat/conda_notes.txt · Last modified: 2019/11/05 14:45 by jypeter