User Tools

Site Tools


other:uvcdat:cdat_conda:miniconda3_install

JYP steps for installing Miniconda3

What? Why?

  • Miniconda3 is a minimal python environment/distribution that can be used for creating more complete work environments. It will basically make the conda command available on your computer, and you can then use conda to install and update more python packages and their dependencies.
  • A python environment is basically where you install python. A python distribution is the collection of packages you have chosen to install together in the environment.
  • You don't need to be (and you should not be) root when you install Miniconda3. You just need enough disk space where you have write access
  • The way you do things will depend on how you are going to use python. There may be some slight differences if you are installing the environment just for you, or if you need a more stable environment because you are installing for multiple users.
  • You could also start with the Anaconda installer that will install a much more complete python environment, ready for use, but we choose not to do that because Anaconda requires more disk space at the beginning, and all its packages come from the default channel (or repository) provided by the conda repository. This is not very useful because we will be mostly using (the same) packages provided by the conda-forge channel.

Note: some extra details are available on the much older page Installing and maintaining UV-CDAT with conda. You can check later the Useful conda commands, but the official conda documentation and the Cheat sheet are probably more up-to-date.

Installing miniconda3 on a Linux-like computer

By Linux-like, we mean:

  • A native Linux computer
  • A windows 10 computer with WSL+Ubuntu installed
  • A mac where you can use Linux in a terminal
  • Execute the installer
    • bash Miniconda3-latest-Linux-x86_64.sh
      • Accept the license
      • Note: at the end of the installation, answer no to the following question, so that the installer does not change your existing shell configuration files!
        Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]NO
      • Specify an explicit installation path outside of your home directory, with enough disk space (more than 3 Gb if you are going to install CDAT and some extra packages), preferably on a disk that is not backed up:
        • Installations by JYP:
          • Linux at LSCE: /home/share/unix_files/cdat/miniconda3<possibly_some_version_here>/
          • Linux on ciclad: /data/jypmce/cdat/miniconda3<possibly_some_version_here>/
        • WSL: installing outside of your /home/your_login/ directory does not work. You need to accept the installation in the default location: /home/<your_login>/miniconda3
      • The resulting miniconda3 directory size is 342M
         > du -sh miniconda3
        342M    miniconda3
        
         > cd miniconda3
        
         > du -sh *
        20M     bin
        0       compiler_compat
        4.0K    condabin
        684K    conda-meta
        0       envs
        16K     etc
        5.5M    include
        4.0K    info
        198M    lib
        12K     LICENSE.txt
        114M    pkgs
        604K    share
        4.0K    shell
        0       ssl
        0       x86_64-conda_cos6-linux-gnu
  • Initialize the newly installed conda environment (this will initialize the environment only in the current terminal):
    • bash shell: source <installation_path>/miniconda3/etc/profile.d/conda.sh
    • tcsh shell: source <installation_path>/miniconda3/etc/profile.d/conda.csh
  • Check if you can use the conda command, and use it to initialize the base environment
    • $ which conda
      <installation_path>/miniconda3/condabin/conda
      $ which python
      /usr/bin/python
      $ conda activate
      (base) $ which python
      <installation_path>/miniconda3/bin/python
  • Update the new installation
    • $ conda update --all
      [...]
    • During the update, the miniconda3 directory size goes from 432 Mb to 581 Mb. This directory will keep on growing, which is the reason why you should put it on a (preferably non backed up) disk where you have enough space
  • Make sure we have the latest conda package (just in case we did not get it with the update)
    conda update -n base conda
  • Remove the installer later, when you have tester your installation:
    rm Miniconda3-latest-Linux-x86_64.sh

Initializing conda in new terminals

When you open a terminal, your shell needs to know where to find the conda command used to initialize an environment, or switch between environments

General case

You were asked the following question when installing miniconda3: Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]

  • If you answered yes, the installer probably added some very complicated lines to your shell configuration files, but you probably have conda directly available when you open a new terminal
  • if you answered no (as suggested), use a text editor to add an extra line to the appropriate configuration file
    • bash user: add this line to ~/.bashrc
      source <installation_path>/miniconda3/etc/profile.d/conda.sh
    • tcsh user: add this line to ~/.cshrc
      source <installation_path>/miniconda3/etc/profile.d/conda.csh

We choose not to add a conda activate env_name line to the shell configuration files, in order to avoid side effects. When we open a new terminal, we get the default python available on the system. When we need a specific python environment, we just open a new window and then explicitly type: conda activate env_name

Multi-user installation

In the case of python environments maintained by a single user, but used by several users, we could do the same as in the General case, but it can be useful to have the users source an intermediate initialization file, that will then source the initialization file used in the general case. This makes it easier to maintain and change the environments, without asking users to make changes.

  • ask bash users to add to ~/.bashrc something like
    source ~main_installer_login/.conda3_jyp.sh

    with a .conda3_jyp.sh file looking like conda3_jyp.sh.txt
  • ask tcsh users to add to ~/.cshrc something like
    source ~main_installer_login/.conda3_jyp.csh

    with a .conda3_jyp.csh file looking like conda3_jyp.csh.txt

Fine-tuning conda

conda will probably work fine with the default settings for creating new environments with just one package (e.g. just PyFerret).

We intend to create complex environments (combining lots of packages, with complex dependencies) and it is highly recommended to use the specific settings described in this section

Getting conda configuration information

  • conda config documentation
  • Basic information: conda info
  • Full configuration: conda config –show

Changing the .condarc file

Most of the packages we will install will be provided by the conda-forge channel (-c conda-forge option). The installation steps will also add dependencies that will theoretically come from conda-forge, but could also come from the defaults channel.

The following will make sure that we only get packages from conda-forge unless the requested packages really don't exist on conda-forge (see Managing channels and Using multiple channels => How to fix it)

$ conda config --add channels conda-forge
$ conda config --set channel_priority strict
$ cat ~/.condarc
channels:
  - conda-forge
  - defaults
channel_priority: strict

Using mamba instead of conda

Even when using a proper .condarc in order to get packages coming only from conda-forge, conda install may fail to solve the dependencies after meditating during an enormous time (e.g. Solving environment: failed with initial frozen solve. Retrying with flexible solve.), when we try to deal with complex environments (i.e. we already have lots of installed packages and we want to add even more packages).

Using CDAT as a base environment, and adding a lot of extra packages is a complex environment.

The only solution seems to replace conda with mamba, a reimplementation of the conda package manager in C++ (Making conda fast again):

  • Install mamba in the base (i.e. miniconda) environment
    conda install mamba -n base -c conda-forge
  • Use mamba instead of conda when installing packages.
    Theoretically, you don't need to specify install -c conda forge any more, because conda-forge is now the default source of new packages
    mamba install lots_of_packages





[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]

other/uvcdat/cdat_conda/miniconda3_install.txt · Last modified: 2021/06/28 13:17 by jypeter