JYP steps for installing Miniconda3

Why should you use Miniconda3 ?

Miniconda3 is a minimal/bootstrap Python distribution that can be used for creating more complex Python distributions.
It will basically make a recent conda command available on your computer, and you can then use this conda executable to select an existing distribution, or create new and independent Python distributions.

A conda environment is basically (the short name of the directory) where you install the Python that you will use (independently of Miniconda3 itself). You have to activate this environment in order to use it
A python distribution is the collection of packages (and their dependencies) you have chosen to install together in a given environment.

Example: initializing an environment named cdatm_py3 (assuming it exists)

$ source /path_to_miniconda/etc/profile.d/conda.sh
$ which conda
/path_to_miniconda/condabin/conda
$ conda env list
[...]
cdatm_py3 /path_to_miniconda/envs/cdatm_py3
some_other_env /path_to_miniconda/envs/some_other_env
$ conda activate cdatm_py3
(cdatm_py3) $ which python
/path_to_miniconda/envs/cdatm_py3/bin/python

The special environment where Miniconda3 itself is installed is called base. Note that the python binary of the base environment is not located in the same directory hierarchy of the other environments (i.e. there is no envs/ subdirectory

$ conda activate base
(base) $ which python
/path_to_miniconda/bin/python

You don't need to be (and you should not be) root when you install Miniconda3. You just need enough disk space on a disk where you have write access

WARNING: by default, Miniconda3 will install itself and subsequent Python environments in a sub-directory of your home directory (~/miniconda3 on Linux). Do not use the default installation location, if you want to avoid disk space related problems (disk full, quota exceeded, ...)

WARNING: older versions of Miniconda3 used a hidden sub-directory of your home directory (~/.conda on Linux) for the installation

Choose carefully where you will install Miniconda3, because the size of the installation directory will start at a few Gb and will keep on growing

You could also start with the full Anaconda installer that will install a much more complete python environment, ready for use.
We choose not to use the full Anaconda installer because it requires more disk space than Miniconda3 at the beginning, and all its packages come from the default channel (or repository) provided by the conda repository. This is not very useful (and could even cause complex dependency problems) because we will be mostly using (the same) packages provided by the conda-forge channel, in order to avoid complex package dependency problems.

Note: some extra details are available on the much older page Installing and maintaining UV-CDAT with conda. You can check later the Useful conda commands, but the official conda documentation and the Cheat sheet are probably more up-to-date.

Installing miniconda3 on a Linux-like computer

By Linux-like, we mean: A native Linux computer (or server) A windows 10 computer with WSL+Ubuntu installed A Mac where you can use Linux in a terminal

Downloading the Miniconda3 installer

We use a download link that always point to the latest version of the Linux installer

For the Linux computers we commonly use, we need the Miniconda3 Linux 64-bit link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

Note: Mac users should use the latest macOS installer appropriate for their computer

The installer is based on Python 3.12 as of March 2024

Find some temporary space on Linux (or Windows, if you are using WSL)

e.g. Linux at LSCE:
$ cd /home/scratch01/$USER

e.g. Linux at spiritx:

$ mkdir /homedata/$USER/Scratch
$ cd /homedata/$USER/Scratch

e.g. Linux on a personal PC/Mac desktop/laptop: use a scratch/temporary directory with enough space

Use wget to download the latest installer (138 Mb as of 22 Mar 2024):

$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ ls -lh Miniconda3-latest-Linux-x86_64.sh
[...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh

Using the Miniconda3 installer

We assume below that we are in the directory where we have downloaded the installer

Determine the directory where you will install miniconda3 and all the Python environments based on this version of miniconda3, but do not create the directory (the installation script will do it)
We will later refer to this location as /path_to_miniconda3/

Remember that you will need a few Gb of available space

At LSCE, do not install in a subdirectory of your home directory!

On other servers and computers, you should also avoid installing in your home directory, if you want to be able to easily make backups or your home, without backing up thousands of files related to your Python installation...

We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations

e.g. Linux at LSCE: /home/share/unix_files/cdat/miniconda3_2024-03

e.g. Linux at spiritx: /homedata/$USER/miniconda3_2024-03

Linux (including WSL) on a personal PC/Mac desktop/laptop: find a directory with enough space (including space for adding more packages and their future updates)

Execute the installer with $ bash Miniconda3-latest-Linux-x86_64.sh

Review the license (type <SPACE> several times...) and accept it

When asked for the installation directory, specify the directory location chosen in the previous step, and not the default directory

e.g. specify
/path_to_miniconda3/
instead of the default
$HOME/miniconda3

Answer no to the question Do you wish to update your shell profile to automatically initialize conda?.
Otherwise the installation will make changes to your shell configuration files

If you forget to answer no, you can apparently:

remove the changes to your shell configuration files by typing later: conda init --reverse $SHELL

or at least disable the automatic activation of the environment (that might have side effect) by typing: conda config --set auto_activate_base false

The resulting miniconda3 directory size is 647 Mb (as of March 2024)

$ du -sh miniconda3_2024-03
647M miniconda3_2024-03
$ du -sh miniconda3_2024-03/*
47M miniconda3_2024-03/bin
16K miniconda3_2024-03/cmake
8.0K miniconda3_2024-03/compiler_compat
32M miniconda3_2024-03/_conda
8.0K miniconda3_2024-03/condabin
976K miniconda3_2024-03/conda-meta
4.0K miniconda3_2024-03/envs
28K miniconda3_2024-03/etc
18M miniconda3_2024-03/include
317M miniconda3_2024-03/lib
92K miniconda3_2024-03/LICENSE.txt
1.1M miniconda3_2024-03/man
232M miniconda3_2024-03/pkgs
396K miniconda3_2024-03/sbin
1.5M miniconda3_2024-03/share
12K miniconda3_2024-03/shell
8.0K miniconda3_2024-03/ssl
8.0K miniconda3_2024-03/x86_64-conda_cos7-linux-gnu
8.0K miniconda3_2024-03/x86_64-conda-linux-gnu

Initialize the newly installed conda environment (this will initialize the environment only in the current terminal):

bash shell: source /path_to_miniconda3/etc/profile.d/conda.sh

e.g. (spiritx): source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh

tcsh shell: source /path_to_miniconda3/etc/profile.d/conda.csh

e.g. (LSCE): source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh

Check if you can use the conda command, and use it to initialize the base environment

e.g. on spiritx1:

$ which conda
$ which python
/usr/bin/python
$ source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
$ which conda
/homedata/jypmce/miniconda3_2024-03/condabin/conda
$ which python
/usr/bin/python
$ conda activate base
(base) $ which conda
/homedata/jypmce/miniconda3_2024-03/bin/conda
(base) $ which python
/homedata/jypmce/miniconda3_2024-03/bin/python
(base) $ conda deactivate
$ which conda
/homedata/jypmce/miniconda3_2024-03/condabin/conda
$ which python
/usr/bin/python

Remove the Miniconda3 installer: $ rm Miniconda3-latest-Linux-x86_64.sh

Fine-tuning conda to use (only) conda-forge

conda will probably work fine with the default settings if you create simple new environments with just one package and its dependencies.

Luckily, conda also works much better and faster since the end of 2023 (conda versions starting at 23.10), now that it is using the mamba solver instead of the default solver.

If you have followed all the steps of this section in order to use only conda-forge packages, and there are still some problems with a recent version of conda, try to install and use mamba, as a drop-in replacement of conda

Historical note: before conda used libmamba, we had to use mamba (instead of conda) for dealing with our complex Python environments:

Making conda fast again

Solving environment: failed with initial frozen solve. Retrying with flexible solve.

We are power users, and we intend to create complex Python environments combining lots of packages available from conda-forge, with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge.

=> The solution is to configure conda to always use conda-forge, and completely update Miniconda3 itself with packages coming only from conda-forge

Changing the .condarc file

The following will make sure that we only get packages from conda-forge (same thing as using the -c conda-forge option) unless the requested packages really don't exist on conda-forge. More details in Managing channels

$ cat ~/.condarc
cat: /home/jypmce/.condarc: No such file or directory
$ conda config --get channels
$ conda config --prepend channels conda-forge
$ conda config --set channel_priority strict
$ conda config --get channels
--add channels 'defaults' # lowest priority
--add channels 'conda-forge' # highest priority
$ cat ~/.condarc
channels:
- conda-forge
- defaults
channel_priority: strict

Getting conda configuration information

conda config documentation

Basic information: conda info

Full configuration: conda config --show

Updating (Mini)conda

We have to completely update Miniconda3 at least once after installing Miniconda3 and making conda-forge the highest priority channel

Check the current version of conda

$ conda list -n base | grep conda
# packages in environment at /path_to_miniconda:
conda 24.1.2 py312h06a4308_0
conda-libmamba-solver 23.12.0 pyhd3eb1b0_1
[... and other packages with 'conda' in their name]

Completely update the Miniconda3 installation

$ conda update -n base --all
Channels:
- conda-forge
- defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /homedata/jypmce/miniconda3_2024-03
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-24.3.0 | py312h7900ff3_0 1.1 MB conda-forge
conda-libmamba-solver-24.1.0| pyhd8ed1ab_0 40 KB conda-forge
python-3.12.2 |hab00c5b_0_cpython 30.8 MB conda-forge
[...]
The following NEW packages will be INSTALLED:
[...]
The following packages will be UPDATED:
[...]
conda pkgs/main::conda-24.1.2-py312h06a4308~ --> conda-forge::conda-24.3.0-py312h7900ff3_0
[...]
The following packages will be SUPERSEDED by a higher-priority channel:
[...]
Proceed ([y]/n)? y
[...]
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Check the updated version of conda. Note that (almost) all the packages should now specify that they are provided by conda-forge

$ conda list -n base
conda 24.3.0 py312h7900ff3_0 conda-forge
conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge
python 3.12.2 hab00c5b_0_cpython conda-forge

You can use the following to determine if some packages are not provided by conda-forge

$ conda list -n base | grep -v conda-forge
# Name Version Build Channel
libedit 3.1.20230828 h5eee18b_0
libffi 3.4.4 h6a678d5_0
xz 5.4.5 h5eee18b_0

It should not be a major problem if a few packages are not provided by conda-forge.
It only means that some packages provided by the anaconda channel were more recent than the same packages provided by the conda-forge channel when the packages were updated

libedit example: version 3.1.20191231 @ conda-forge and version 3.1.20230828 @ anaconda

You can later update again the full Miniconda3, or just the conda command.
This should not make any changes to the Python environment(s) have have installed with conda

$ conda update -n base --all
[...]
# All requested packages already installed.
(base) $ conda update -n base conda
[...]
# All requested packages already installed.

Note: during this installation, the miniconda3 directory size grew from 763 Mb to 1.6 Gb, and we have not installed any custom Python environment yet! This is the reason why you should install Miniconda3 on a (preferably non backed up) disk where you have enough space

$ du -sh /homedata/jypmce/miniconda3_2024-03
736M /homedata/jypmce/miniconda3_2024-03
[... update]
$ du -sh /homedata/jypmce/miniconda3_2024-03
1.6G /homedata/jypmce/miniconda3_2024-03

You can clean the initial Miniconda3 installation to free up some disk space. The documentation specifies WARNING: This will break environments with packages installed using symlinks back to the package cache. and you should probably avoid using clean once you have installed new Python environments (unless you are desperate for disk space and know how to do a complete re-installation if something breaks...)

(base) $ conda clean --all
Will remove 154 (180.9 MB) tarball(s).
Proceed ([y]/n)? y
Will remove 1 index cache(s).
Proceed ([y]/n)? y
Will remove 74 (378.0 MB) package(s).
Proceed ([y]/n)? y
There are no tempfile(s) to remove.
There are no logfile(s) to remove.
(base) $ du -sh /homedata/jypmce/miniconda3_2024-03
457M /homedata/jypmce/miniconda3_2024-03

Initializing conda in terminals

When you open a terminal, your shell needs to know where to find the conda command used to initialize an environment, or switch between existing environments. This can be configured in the shell configuration files

Single-user installation

You were asked the following question when installing Miniconda3: Do you wish to update your shell profile to automatically initialize conda? [yes|no]

If you answered yes, the installer probably added some very complicated lines to your shell configuration files, and conda is probably directly available when you open a new terminal.

If you answered no (as suggested), use a text editor to add extra lines to the appropriate shell configuration file

bash shell user : add these lines to ~/.bashrc

source /path_to_miniconda3/etc/profile.d/conda.sh
alias pynit='conda activate base'

e.g. (spiritx):

source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
alias pynit='conda activate base'

tcsh shell user: add these lines to ~/.cshrc

source /path_to_miniconda3/etc/profile.d/conda.csh''
alias pynit 'conda activate base'

e.g. (LSCE):

source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh
alias pynit 'conda activate base'

When you have a custom Python environment installed (e.g. my_power_env, you can update the pynit alias to initialize this environment instead of the base environment

You can also add other aliases, or use another name than pynit!

We choose not to add a conda activate env_name line to the shell configuration files, in order to avoid future side effects. When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type: conda activate env_name

Multi-user installation

In the case of python environments maintained by a single user, but used by several users, we could do the same as in the General case, but it can be useful to have the users source an intermediate initialization file, that will then source the initialization file used in the general case. This makes it easier to maintain and change the environments, without asking users to make changes.

ask bash users to add to ~/.bashrc something like
source ~main_installer_login/.conda3_jyp.sh
with a .conda3_jyp.sh file looking like conda3_jyp.sh.txt

ask tcsh users to add to ~/.cshrc something like
source ~main_installer_login/.conda3_jyp.csh
with a .conda3_jyp.csh file looking like conda3_jyp.csh.txt

Using mamba instead of conda

Even when using a proper .condarc in order to get packages coming only from conda-forge, conda install may fail to solve the dependencies after meditating during an enormous time (e.g. Solving environment: failed with initial frozen solve. Retrying with flexible solve.), when we try to deal with complex environments (i.e. we already have lots of installed packages and we want to add even more packages).

Using CDAT as a base environment, and adding a lot of extra packages is a complex environment.

The only solution seems to replace conda with mamba, a reimplementation of the conda package manager in C++ (Making conda fast again):

Install mamba in the base (i.e. miniconda) environment
conda install mamba -n base -c conda-forge

Use mamba instead of conda when installing packages.
Theoretically, you don't need to specify install -c conda forge any more, because conda-forge is now the default source of new packages
mamba install lots_of_packages

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]