JYP steps for installing Miniconda3

Before installing Python on a shared Linux server

Desperately seeking a specific package

Check if the Python package you are desperately seeking is not already installed, before installing your own Python environment on shared interactive Linux servers, or on batch computing clusters !!

Ask other users who know the local environment

Read the servers/clusters Python-related documentation: Python at LSCE by JYP, LSCE, IPSL Mesocenter, …

Use module avail [mod_name] to find out the Python versions available, and then module load and conda list to get more information on the available Python packages

e.g. on spiritx (modules on spirit)):

$ module avail -t python
python/meso-3.8
python/meso-3.9
python/meso-3.10
python/meso-3.11
jypmce@spiritx1:~$ module avail -l 2>&1 | grep meso
anaconda-meso/2022.10 2023/04/21 18:02:35
anaconda-meso/2023.09-0 2024/01/25 15:07:50
pangeo-meso/2023.04.15 2024/10/18 15:11:30
pangeo-meso/2024.01.22 2024/10/29 15:16:33
pangeo-meso/2025.01.24 2025/02/28 15:45:21
python/meso-3.8 2024/10/18 15:12:15
python/meso-3.9 2024/10/18 15:12:01
python/meso-3.10 2024/10/18 15:11:30
python/meso-3.11 2024/10/29 15:16:33
jypmce@spiritx1:~$ module load pangeo-meso/2025.01.24
(pangeo-meso-2025.01.24) jypmce@spiritx1:~$ which python
/net/nfs/tools/u20/Python/miniforge_24.11.3-0/envs/pangeo-meso-2025.01.24/bin/python
(pangeo-meso-2025.01.24) jypmce@spiritx1:~$ conda list | grep cartopy
cartopy 0.24.0 py312hf9745cd_0 conda-forge

Lots of Python packages do very similar things. Maybe you can use another similar package that is already installed…

If the packages you need are not available, it may be faster and easier to ask JYP or the local system administrators if they can install the packages in an existing environment, rather than doing everything yourself

Using an unlisted Python distribution on a jupyter server

In the case when:

there is a jupyter notebook server connected to your Linux cluster (e.g. https://jupyter.lsce.ipsl.fr/)

and an existing Python environment on this cluster providing all the packages you need

but this Python environment is not directly available on the notebook server (in the listed kernels), you should follow the using a non-standard kernel instructions, rather than starting your own local notebook server (on this multi-user computer)

Why should you use Miniconda3 ?

: try using Miniforge3 instead of Miniconda3 ?

Miniconda3 is a minimal/bootstrap Python distribution that can be used for creating more complex Python distributions.
It will basically make a recent conda command/executable available on your computer, and you can then use this conda executable to select an existing distribution, or create new and independent Python distributions.

A conda environment is basically (the short name of the directory) where you install the Python that you will use (independently of Miniconda3 itself). You have to activate this environment in order to use it
A Python distribution is the collection of packages (and their dependencies) you have chosen to install together in a given environment.

Example: initializing an environment named cdatm_py3 (assuming it exists) in a bash shell

$ source /path_to_miniconda/etc/profile.d/conda.sh
$ which conda
/path_to_miniconda/condabin/conda
$ conda env list
[...]
cdatm_py3 /path_to_miniconda/envs/cdatm_py3
some_other_env /path_to_miniconda/envs/some_other_env
$ conda activate cdatm_py3
(cdatm_py3) $ which python
/path_to_miniconda/envs/cdatm_py3/bin/python

The special environment where Miniconda3 itself is installed is called base (it also used to be called root). Note that the python binary of the special base environment is not located in the same directory hierarchy as the python of the other environments (i.e. there is no envs/ subdirectory

$ conda activate base
(base) $ which python
/path_to_miniconda/bin/python

You don't need to be (and you should not be !!) root when you install Miniconda3. You just need enough disk space on a disk where you have write access

WARNING: by default, Miniconda3 will install itself and subsequent Python environments in a sub-directory of your home directory (~/miniconda3 on Linux). Do not use the default installation location, if you want to avoid disk space related problems (disk full, quota exceeded, …) during the inittial installation or later

WARNING: older versions of Miniconda3 used a hidden sub-directory of the home directory of the person installing Miniconda3 (~/.conda on Linux)

Choose carefully where you will install Miniconda3, because the size of the installation directory will start at a few Gb and will keep on growing

You could also start with the full Anaconda (instead of Miniconda) installer, that will install a much more complete python environment, ready for use.
We choose not to use the full Anaconda installer because:

It requires more disk space than Miniconda3 during the initial installation,

All the Anaconda installer packages come from the default channel (or repository) provided by the conda repository.
This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the conda-forge channel, in order to avoid complex package dependency problems

Downloading Miniconda3, and BASIC installation

Miniconda3 on a Windows computer

Some of the steps below are adapted from the Windows section of "Quick command line install" to install a pure Windows version of Miniconda3

Downloading the Miniconda3 installer on Windows

We use a download link that always point to the latest version of the Windows installer

https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe

The installer is based on Python 3.12.2 and Conda 24.3.0 as of April 2024

Find some temporary disk space on Windows

e.g. C:\Scratch\your_login or whatever you use as a local Scratch folder

Open a terminal, where you can type the required installation commands

Windows Powershell terminal: Start ⇒ W ⇒ Windows PowerShell ⇒ Windows PowerShell

PowerShell in a Windows Terminal

Use curl to download the latest installer (78 Mb as of April 2024):

PS C: cd C:\Scratch\your_login
PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe
PS C:\Scratch\your_login> dir miniconda.exe
[...]
-a---- 19/04/2024 11:18 81274240 miniconda.exe

BASIC Miniconda3 installation on Windows

Determine the folder where you will install Miniconda3 and ALL your future Python environments based on this version of Miniconda3
We will later refer to this installation location as /path_to_miniconda3/ in the Linux sections below

Remember that you will need a few Gb of available space

You should avoid installing Miniconda3 in your Windows home folder (C:\Users\your_login or <some_disk_different_from_c>:\Users\<your_login>), if you want to be able to easily make backups or your home folder, without backing up thousands of files related to your Python environments…

We add some date or version information at the end of the folder name, in order to differentiate it from other (older of future) installations

Example installation folder: C:\Utils\miniconda3_2024-03 (if we start installing in March 2024)

Launch the Miniconda3 installer with

PS C:\Scratch\your_login> .\miniconda.exe

In the installer window:

(Review and) accept the License Agreement

Choose to install for Just Me (the recommended default choice)

Specify the Destination Folder selected above, e.g. C:\Utils\miniconda3_2024-03

Use the pre-selected Advanced Installation Options, i.e

Create start menu shortcutes

Register Miniconda3 as my default Python 3.12 ⇐ Useful?

There is no need to bookmark the web pages suggested by the installer at the end, since this web page gives all the required information

When the installer has finished running, you should have:

A new C:\Utils\miniconda3_2024-03 folder

This folder size will be ~650 Mb and will keep on growing (up to several Gb) as you add more Python environments and packages

⇒ Be sure to have enough space on this disk!

Two new shortcuts in the Windows Start⇒M⇒Miniconda3 menu:

Anaconda Prompt

Anaconda Powershell Prompt

We will now use this Anaconda Powershell shortcut each time we need to use Miniconda3 or Python

It is highly recommended to install Windows Terminal and make it the default terminal application, so that the conda prompts will open in a modern (and highly configurable) terminal

Remove the installer

PS C:\Scratch\your_login> del .\miniconda.exe

Open an Anaconda PowerShell Prompt, and type a few commands to make sure that conda is working

Start⇒M⇒Miniconda3 ⇒ Anaconda Powershell Prompt

(base) PS C:\Users\your_login> conda env list
base * C:\Utils\miniconda3_2024-03
(base) PS C:\Users\your_login> conda list
# packages in environment at C:\Utils\miniconda3_2024-03:
# Name Version Build Channel
[...]
conda 24.3.0 py312haa95532_0
[...]
python 3.12.2 h1d929f7_0
[...]
(base) PS C:\Users\your_login> conda deactivate
PS C:\Users\your_login> conda env list
base C:\Utils\miniconda3_2024-03

Next: be sure to follow the steps in the Fine-tuning conda to use (only) conda-forge, and all the other Miniconda3 related instructions

conda commands will work the same way on all Operating Systems (Windows and Linux-like)

The .condarc is located in your Windows home folder
⇒ C:\Users\your_login\.condarc (or $HOME\.condarc)

Miniconda3 on a Linux-like computer

By Linux-like, we mean: A Linux desktop/laptop The installation will be used by one person A Linux server The installation will be used by one or more persons A windows computer with WSL+Ubuntu installed You should use the Windows installer, if all the packages you need are available for Windows The installation will be used by one person A Mac where you can use Linux in a terminal The installation will be used by one person

Downloading the Miniconda3 installer on Linux

Important: do not install your own Python environment(s) on shared computers before you have made sure that the packages you need are not already installed, or cannot be easily installed in an existing environment (by the person who maintains this environment)!

Read again the Before installing Python on a shared computer section

We use a download link that always point to the latest version of the Linux installer

For the Linux computers we commonly use, we need the Miniconda3 Linux 64-bit link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

Note: Mac users should use the latest macOS installer appropriate for their computer

The installer is based on Python 3.12.2 and Conda 24.3.0 as of April 2024

Find some temporary space on Linux (or Windows, if you are using WSL)

e.g. Linux on the LSCE servers:
$ cd /home/scratch01/$USER

e.g. Linux on the IPSL spiritx cluster:

$ mkdir /homedata/$USER/Scratch
$ cd /homedata/$USER/Scratch

e.g. Linux on a personal PC/Mac desktop/laptop: use a local scratch/temporary directory with enough space

Use wget to download the latest installer (138 Mb as of 22 Mar 2024):

$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ ls -lh Miniconda3-latest-Linux-x86_64.sh
[...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh

Using the Miniconda3 installer on Linux

We assume below that we are in the directory where we have downloaded the installer

Determine the directory where you will install miniconda3 and all the Python environments based on this version of miniconda3, but do not create the directory (the installation script will do it)
We will later refer to this location as /path_to_miniconda3/

Remember that you will need a few Gb of available space

If you want to share this installation with other users, the directory has to be readable by these users

Always avoid installing in your home directory, if you want to be able to easily make backups or your home, without backing up thousands of files related to your Python installation…

We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations

e.g. JYP shared Linux installation at LSCE: /home/share/unix_files/cdat/miniconda3_2024-03

e.g. Linux at spiritx: /homedata/$USER/miniconda3_2024-03

Execute the installer with $ bash Miniconda3-latest-Linux-x86_64.sh

Review the license (type <SPACE> several times…) and accept it

When asked for the installation directory, specify the directory location chosen in the previous step, and not the default directory

e.g. specify
/path_to_miniconda3/
instead of the default
$HOME/miniconda3

Answer no to the question Do you wish to update your shell profile to automatically initialize conda?.
Otherwise the installation will make changes to your shell configuration files

If you forget to answer no, you can apparently:

later remove the changes to your shell configuration files by typing: conda init --reverse $SHELL

or at least disable the automatic activation of the environment (that might have side effect) by typing: conda config --set auto_activate_base false

The resulting initial miniconda3 directory size is 647 Mb (as of March 2024).
Remember that this directory size will keep on growing!

$ du -sh miniconda3_2024-03
647M miniconda3_2024-03
$ du -sh miniconda3_2024-03/*
47M miniconda3_2024-03/bin
16K miniconda3_2024-03/cmake
8.0K miniconda3_2024-03/compiler_compat
32M miniconda3_2024-03/_conda
8.0K miniconda3_2024-03/condabin
976K miniconda3_2024-03/conda-meta
4.0K miniconda3_2024-03/envs
28K miniconda3_2024-03/etc
18M miniconda3_2024-03/include
317M miniconda3_2024-03/lib
92K miniconda3_2024-03/LICENSE.txt
1.1M miniconda3_2024-03/man
232M miniconda3_2024-03/pkgs
396K miniconda3_2024-03/sbin
1.5M miniconda3_2024-03/share
12K miniconda3_2024-03/shell
8.0K miniconda3_2024-03/ssl
8.0K miniconda3_2024-03/x86_64-conda_cos7-linux-gnu
8.0K miniconda3_2024-03/x86_64-conda-linux-gnu

Initialize the newly installed conda environment. This will initialize the environment only in the current terminal. When you have checked that things work properly, you can later make the initialization easier by following the steps in Initializing conda on a Linux-like computer

You have to execute a configuration script, and then activate the environment

Run the init script: source /some_path/conda_init_script

Activate the base environment (these two commands have the same result):

conda activate

or conda activate base

bash shell: source /path_to_miniconda3/etc/profile.d/conda.sh

e.g. (spiritx): source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh

tcsh shell: source /path_to_miniconda3/etc/profile.d/conda.csh

e.g. (LSCE): source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh

After initializing the environment, check if you can access the conda command, and use it to initialize the base environment

e.g. on spiritx1:

$ which conda
/homedata/jypmce/miniconda3_2024-03/condabin/conda
$ which python
/usr/bin/python
$ conda activate base
(base) $ which conda
/homedata/jypmce/miniconda3_2024-03/bin/conda
(base) $ which python
/homedata/jypmce/miniconda3_2024-03/bin/python
(base) $ conda deactivate
$ which conda
/homedata/jypmce/miniconda3_2024-03/condabin/conda
$ which python
/usr/bin/python

Remove the Miniconda3 installer: $ rm Miniconda3-latest-Linux-x86_64.sh

Fine-tuning conda to use (only) conda-forge

Be sure to also follow these steps on a Windows installation !

Why is it vital to use only conda-forge ?

conda will probably work fine with the default settings if you create simple new environments with just one package and its dependencies.

conda works much better and faster since the end of 2023 (conda versions starting at 23.10), now that it is using the mamba solver instead of the default solver.

Use conda --version version to determine which version you are using

$ conda --version
conda 24.5.0

We are power users, and we intend to create complex Python environments combining lots (and lots) of packages available from conda-forge, with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge.

⇒ The sections below configure conda to always use conda-forge, and completely update Miniconda3 itself with packages coming only from conda-forge

Very unlikely situation: if you use a recent version of conda (>= 23.10), and you have followed all the steps below, in order to use only conda-forge packages, and there are still some package installation problems, try to install and use mamba, as a drop-in replacement of conda

Historical note: before conda used libmamba, we had to use mamba (instead of conda) for dealing with our complex Python environments:

Making conda fast again

Solving environment: failed with initial frozen solve. Retrying with flexible solve.

Changing the .condarc file

The following steps will make sure that we only get packages from conda-forge (same thing as using the -c conda-forge option) by default, unless the requested packages really do not exist on conda-forge. More details in Managing channels

$ cat ~/.condarc # On Windows use: cat $HOME\.condarc
cat: /home/jypmce/.condarc: No such file or directory
$ conda config --get channels
$ conda config --prepend channels conda-forge
$ conda config --set channel_priority strict
$ conda config --get channels
--add channels 'defaults' # lowest priority
--add channels 'conda-forge' # highest priority
$ cat ~/.condarc # On Windows use: cat $HOME\.condarc
channels:
- conda-forge
- defaults
channel_priority: strict

Updating Miniconda3 at least once!

Before you go further, do not forget to update conda at least once in order to take all the changes above into account!

Adding mamba (optional)

Most people can (and should!) probably skip the mamba installation Only install mamba if you are sure that you will need some features that are not already provided by conda (e.g. mamba repoquery)

mamba is a fast, robust, and cross-platform package manager […] fully compatible with conda packages, and supports most of conda’s commands

Installing mamba

mamba is used to manage environments, and has to be installed in the base environment (i.e. in the same environment as conda)!

If :

You are sure that you need ''mamba'', even if the developers advise not to mix conda and mamba

You have configured conda to use conda-forge by default

Then (and only then!) you can install mamba:
⇒ conda install -n base mamba

Using mamba

The useful mamba commands (we need) are listed in the Useful conda commands section

We mostly need mamba repoquery

Mamba User Guide

Keeping (Mini)conda up-to-date

The update command

Just type conda update -n base --all, (review) and accept the changes

Things should be OK if no package downgrades are requested

The base (Miniconda3 itself) and the other Python environments installed with the conda executable provided by the base environment are all independent, and can be updated independently

It is up to you to decide if and when you want to update environments…

…but you have to update the base environment at least once in order to replace all of its packages by their equivalent conda-forge packages

You can later update again the full Miniconda3, or just the conda command.
This should not make any changes to the other Python environment(s) installed with Miniconda3

$ conda update -n base --all
[...]
# All requested packages already installed.
(base) $ conda update -n base conda
[...]
# All requested packages already installed.

Note: during our initial installation, the miniconda3 directory size grew from 763 Mb to 1.6 Gb, and we have not installed any custom Python environment yet! This is the reason why you should install Miniconda3 on a (preferably non backed up) disk where you have enough free space

$ du -sh /homedata/jypmce/miniconda3_2024-03
736M /homedata/jypmce/miniconda3_2024-03
[... update]
$ du -sh /homedata/jypmce/miniconda3_2024-03
1.6G /homedata/jypmce/miniconda3_2024-03

The first time we update Miniconda3...

We have to completely update Miniconda3 at least once after installing Miniconda3 and making conda-forge the highest priority channel

Check the current version of conda

$ conda --version
conda 24.1.2

Completely update the Miniconda3 installation

$ conda update -n base --all
Channels:
- conda-forge
- defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /homedata/jypmce/miniconda3_2024-03
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-24.3.0 | py312h7900ff3_0 1.1 MB conda-forge
conda-libmamba-solver-24.1.0| pyhd8ed1ab_0 40 KB conda-forge
python-3.12.2 |hab00c5b_0_cpython 30.8 MB conda-forge
[...]
The following NEW packages will be INSTALLED:
[...]
The following packages will be UPDATED:
[...]
conda pkgs/main::conda-24.1.2-py312h06a4308~ --> conda-forge::conda-24.3.0-py312h7900ff3_0
[...]
The following packages will be SUPERSEDED by a higher-priority channel:
[...]
Proceed ([y]/n)? y
[...]
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Check the updated version of conda and python. Note that (almost) all the packages should now specify that they are provided by conda-forge

$ conda list -n base
[...]
conda 24.3.0 py312h7900ff3_0 conda-forge
conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge
[...]
python 3.12.2 hab00c5b_0_cpython conda-forge
[...]

You can use the following (on Linux) to determine if some packages were not available in conda-forge

$ conda list -n base | grep -v conda-forge
# Name Version Build Channel
libedit 3.1.20230828 h5eee18b_0
libffi 3.4.4 h6a678d5_0
xz 5.4.5 h5eee18b_0

It should not be a major problem if a few packages are not provided by conda-forge.
It only means that some packages provided by the anaconda channel were more recent than the same packages provided by the conda-forge channel when the packages were updated

libedit example: version 3.1.20191231 @ conda-forge and version 3.1.20230828 @ anaconda

Cleaning up Miniconda3...

It seems safe to reclaim some disk space by cleaning up the initial (base) installation, when no other (shared) environments have been installed yet Use cleaning with caution after installing other environments (besides base), especially environments shared by several users. But it should not break your installation

conda clean --all

Documentation

There is no -n option to specify an environment name, because clean will Remove unused packages and caches shared by all the environments

The clean command will mostly deal with the content of the /path_to_miniconda/pkgs/ directory

Example:

(base) $ conda clean --all
Will remove 154 (180.9 MB) tarball(s).
Proceed ([y]/n)? y
Will remove 1 index cache(s).
Proceed ([y]/n)? y
Will remove 74 (378.0 MB) package(s).
Proceed ([y]/n)? y
There are no tempfile(s) to remove.
There are no logfile(s) to remove.
(base) $ du -sh /homedata/jypmce/miniconda3_2024-03
457M /homedata/jypmce/miniconda3_2024-03

It should be safe to use conda clean --all even in a multi-user installation with several environments, if you don't use the -f option

The -f/--force-pkgs-dirs documentation specifies WARNING: This will break environments with packages installed using symlinks back to the package cache.

…but Linux installations probably use hard links rather than soft/symbolic links

…and you will probably never need this option anyway

Remember that your Python installation will keep on growing and growing and growing, and you will need to do some cleaning at some point

Initializing conda in terminals

Initializing conda on a Windows computer

Open a PowerShell terminal with: Start⇒ M ⇒ Miniconda3 ⇒ Anaconda Powershell Prompt

The base environment will be activated by default

If you need another environment, just type: conda activate name_of_the_environment_you_want

TODO

Find out how to use menuinst in order to directly initialize an existing environment (different from base) from the Start menu and from a Windows Terminal

Adding Windows Start menu items

menuinst documentation

Add Windows Terminal profile option

Initializing conda on a Linux-like computer

When you open a terminal, your shell needs to know where to find the conda command used to initialize an environment, or switch between existing environments. This can be configured in your shell configuration files (.bashrc, .cshrc, …)

conda in a single-user Linux-like installation

You were asked the following question when installing Miniconda3: Do you wish to update your shell profile to automatically initialize conda? [yes|no]

If you answered yes, the installer probably added some very complicated lines to your shell configuration files, and conda and the newly installed python are probably directly available when you open a new terminal.
Warning! The automatic activation might have subtle side effects. You can:

Remove the changes to your shell configuration files by typing: conda init --reverse $SHELL

Or at least disable the automatic activation of the environment by typing: conda config --set auto_activate_base false

If you answered no (as suggested), use a text editor to add extra lines to the appropriate shell configuration file
The changes will apply to all the new terminals you open

bash shell user : add these lines to ~/.bashrc

source /path_to_miniconda3/etc/profile.d/conda.sh
alias pynit='conda activate base'

e.g. (spiritx):

source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
alias pynit='conda activate base'

tcsh shell user: add these lines to ~/.cshrc

source /path_to_miniconda3/etc/profile.d/conda.csh''
alias pynit 'conda activate base'

e.g. (LSCE):

source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh
alias pynit 'conda activate base'

After changing your configuration file, just type pynit in any (new) terminal you open, when you need to use Python

When you have a custom Python environment installed (e.g. my_power_env, you can update the pynit alias to initialize this environment instead of the base environment (e.g. conda activate my_power_env)

You can also define other aliases, or use another name than pynit!

conda in a multi-user Linux installation

We need to be extra-careful in the case of python environments maintained by a single user, but used by several users, in order not to break anything

We could use the same configuration file as in the single-user case, but it is more efficient to have the users source an intermediate initialization file. This intermediate init file will:

source the conda initialization file used in the single-user case

Define some aliases that will be useful to all the people using these environments:

pynit ⇒ initialize the base environment (or another default environment chosen by the central maintainer)

wp ⇒ which python

wconda ⇒ which python

more aliases, if required

If changes have to be made later, it is very easy to update this central initialization file accordingly, without asking the other users to make changes

Do not forget to keep a copy of the previous version of the init file!

Use a symbolic link (ln -s) to point from the unchanging central initialization file name to the latest initialization file.
conda@LSCE example in /home/share/unix_files/cdat/conda_init_files/:

$ ls -l
-rw-r--r-- Jun 14 16:37 conda3_jyp_2024-03.csh
-rw-r--r-- Jun 14 17:48 conda3_jyp_2024-03.sh
lrwxrwxrwx Jun 17 15:56 conda3_jyp_latest.csh -> conda3_jyp_2024-03.csh
lrwxrwxrwx Jun 17 15:57 conda3_jyp_latest.sh -> conda3_jyp_2024-03.sh

bash users should add to their ~/.bashrc file something like
source /some_path_accessible_by_all_users/shared_conda_init_file.sh
with a shared_conda_init_file.sh file looking like conda3_jyp_2024-03.sh.txt
Examples:

LSCE:

Add to ~/.bashrc: source /home/share/unix_files/cdat/conda_init_files/conda3_jyp_latest.sh

Note: conda3_jyp_latest.sh is a link to the latest version of the init file,
currently /home/share/unix_files/cdat/conda_init_files/conda3_jyp_2024-03.sh

spiritx and spirit:

We use the fact that file systems on spiritx are accessible in read-only mode on spirit

The python environments are installed on local disks of spiritx, but they can also be used on spirit, even if the remote disks access will add some latency

Add to ~/.bashrc: source /homedata/jypmce/conda_init_files/conda3_jyp_spiritx_latest.sh

Note: conda3_jyp_spiritx_latest.sh is a link to the latest version of the init file,
currently /homedata/jypmce/conda_init_files/conda3_jyp_spiritx_2024-03.sh

tcsh users should add to their ~/.cshrc file something like
source /some_path_accessible_by_all_users/shared_conda_init_file.csh
with a shared_conda_init_file.csh file looking like conda3_jyp_2024-03.csh.txt
Examples:

LSCE:

Add to ~/.cshrc: source /home/share/unix_files/cdat/conda_init_files/conda3_jyp_latest.csh

Note: conda3_jyp_latest.csh is a link to the latest version of the init file,
currently /home/share/unix_files/cdat/conda_init_files/conda3_jyp_2024-03.csh

Using (mini)conda to create a new environment

Once conda is installed (in the base environment provided by Miniconda3), configured and updated, you can use it to create a new environment with the packages that you (or other users, in a shared environment) need

Use the commands described in Useful conda commands

Detailed example: the xcdat environment

: update/rewrite the cdat_2024_03 page

Creating Windows menu shortcuts to start environments

⇒ Move this section somewhere else?

Use conda install and what is described in Easily creating shortcuts with the powershell_shortcut_miniconda and console_shortcut_miniconda packages, and can this be replicated with menuinst ?

Can theoretically be done with menuinst

menuinst @ GH

Find a way to add Windows Terminal profiles

See menuinst+Add Windows Terminal profile option

conda resources

Web sites

Official Documentation site

Cheat sheet (2 pages, for people in a hurry)

Commands' reference

Conda package specification

This helps understand (and use) the content of the /path_to_miniconda/pkgs directory

conda community site:

conda Blog: conda new features, etc…

conda @ GitHub

Package repositories

conda-forge ⇐ recommended source of packages

default

Useful conda commands

This is just a subset of some of the commands, and a subset of their options! For a complete reference, use the official conda website

Get help

conda -h

conda command --help: help for a specific command

Version and configuration information

Version information: conda --version

Detailed information: conda info

Full configuration: conda config --show

conda config documentation

Available environments

conda env list

The currently active environment has a “*” character on its line

(base) $ conda env list
# conda environments:
#
base * /homedata/jypmce/miniconda3_2024-03

Activate an existing environment (available in conda env list) or deactivate, in order to go back to the default Python

conda activate existing_environment

conda activate (without specifying an environment name) will activate the base Miniconda3 environment

Go back to the default Python available on the system

conda deactivate

You can type which python to determine where is the current python executable you are using is
You should always know which Python you are using!

On Linux:

(base) $ which python
/homedata/jypmce/miniconda3_2024-03/bin/python
(base) $ conda deactivate
$ which python
/usr/bin/python

On Windows:

(base) PS C:\> where.exe conda
C:\Utils\miniconda3_2024-03\Library\bin\conda.bat
C:\Utils\miniconda3_2024-03\Scripts\conda.exe
C:\Utils\miniconda3_2024-03\condabin\conda.bat
(base) PS C:\> where.exe python
C:\Utils\miniconda3_2024-03\python.exe
C:\Users\jypeter\AppData\Local\Microsoft\WindowsApps\python.exe
(base) PS C:\Users\jypeter> conda deactivate
PS C:\> where.exe conda
C:\Utils\miniconda3_2024-03\condabin\conda.bat
PS C:\> where.exe python
C:\Users\jypeter\AppData\Local\Microsoft\WindowsApps\python.exe

Note: before conda 4.6, you had to type source activate existing_environment and source deactivate

Get the list of installed packages, their version, and where they come from (e.g. conda-forge, pip)

All the packages: conda list [-n existing_environment]

Some packages: conda list [-n existing_environment] full_or_partial_name

(base) $ conda list libmamba
# Name Version Build Channel
conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge
libmamba 1.5.8 had39da4_0 conda-forge
libmambapy 1.5.8 py312hd9e9ff6_0 conda-forge
(base) $ conda list lib
_libgcc_mutex 0.1 conda_forge conda-forge
conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge
[...]
urllib3 2.2.2 pyhd8ed1ab_1 conda-forge
zlib 1.3.1 h4ab18f5_1 conda-forge

Create an environment and install a few packages and their dependencies

conda create -n new_environment_name package1 package2 packageN

Delete an environment

conda remove -n crap_environment --all

Find out if a package is available on the conda-forge channel (or other channels)

Note: you should always install a package with conda instead of pip or mamba, if it is available on a conda channel

conda search package_exact_name
conda search '*package_partial_name*'

Note: search with wildcards fails in a tcsh shell, as of April 2024 (CSH install/search fail with '*' variable in an argument)

Search results can be filtered by using the conda MatchSpec format

Get very detailed information about a package (versions, size, dependencies) with:
conda search package_exact_name --info

: find an option to get only the latest version available for the current platform

Did not find a way to do that efficiently (June 20th 2024), even with mamba

Install and remove packages
Add -n target_environment to make changes in another environment than the active one

conda install package1 package2 packageN

conda should already be configured to use conda-forge by default if you have read this page carefully

Add -c channel_url_or_alias to install from a specific channel

Specific versions can be specified by using the conda MatchSpec format

conda remove package1 package2 packageN

Matching pip commands (only if the package is not available in conda!)

pip install package

pip uninstall package

Update a package in the current, or a specific environment

conda update -n existing_environment installed_package

Update only the conda package (not the full Miniconda3 base environment) in the base environment

conda update -n base conda

Changes and new conda features are detailed in the conda Blog

Matching pip command (only if the package is not available in conda!)

pip install --upgrade package

Completely update an environment (i.e. update all the packages)

conda update -n existing_environment --all

'base' environment update example

Determine the relationship between installed packages

Packages relationships can be listed with mamba repoquery

Note: mamba repoquery currently only works in the currently active environment

It can be useful to determine:

the packages that are required in order to use a given package ⇒ mamba repoquery depends package

Without mamba repoquery depends, you would have to follow the steps in Listing package dependencies

the packages that need a given package ⇒ mamba repoquery whoneeds package

Example: the tqdm package

conda needs tqdm to display package download and installation progress, and the base dependencies of tqdm are colorama, and of course python

Detailed use:

>mamba repoquery whoneeds --pretty tqdm
tqdm
└─ conda[24.5.0]
>mamba repoquery depends --pretty tqdm
tqdm[4.66.4]
├─ colorama[0.4.6]
└─ python[3.12.3]

The packages that tqdm needs also need packages, and so on…

>mamba repoquery depends --tree tqdm
tqdm[4.66.4]
├─ colorama[0.4.6]
│ └─ python[3.12.3]
│ ├─ bzip2[1.0.8]
│ │ └─ libgcc-ng[14.1.0]
│ │ ├─ _libgcc_mutex[0.1]
│ │ └─ _openmp_mutex[4.5]
│ │ ├─ _libgcc_mutex already visited
│ │ └─ libgomp[14.1.0]
│ │ └─ _libgcc_mutex already visited
│ ├─ libgcc-ng already visited
│ ├─ ld_impl_linux-64[2.40]
│ ├─ libexpat[2.6.2]
│ │ └─ libgcc-ng already visited
[...]

Checking the health of an environment with conda doctor

Some details in Detecting conda environment inconsistencies easily

conda doctor does not do much yet, and not necessarily in a useful way, but this will hopefully improve over time!

'conda doctor' output should be improved

What exactly are altered files?

conda doctor -n base

Example:

>conda doctor -n base
Environment Health Report for: /home/share/unix_files/cdat/miniconda3_2024-03
Altered Files:
python-3.12.3-hab00c5b_0_cpython: 1
Environment listed in environments.txt file: ✅
Missing Files:
✅ There are no packages with missing files.

Cleaning…

conda does not automatically clean the content of the /path_to_miniconda/pkgs directory and the size of /path_to_miniconda/ will keep on growing, as you add new environments, or update existing ones

All the details are in the Cleaning up Miniconda3... section

conda, pip and virtualenv

Read the Conda vs. pip vs. virtualenv commands section

Read the Using pip in a conda environment

Issues may arise when using pip and conda together…

Installation history

You can get the current list of installed packages with conda list -n existing_environment, but it may also be useful to know in which order the packages were installed:

conda list -n existing_environment --revisions: display exactly what was installed and when (including the updates, and automatic dependencies installation)

You can theoretically roll back to a previous RR revision number with: conda install -n existing_environment --revision RR

There is more information in the /path_to_miniconda/conda-meta/history installation log files.

base history file: /path_to_miniconda/conda-meta/history

existing_environment history file: /path_to_miniconda/envs/existing_environment/conda-meta/history

e.g. installation commands for the base environment:

$ egrep -e 'cmd:' -e '==>' /homedata/jypmce/miniconda3_2024-03/conda-meta/history
==> 2024-02-23 18:39:23 <==
# cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW
==> 2024-03-27 16:10:01 <==
# cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all
==> 2024-03-28 11:38:08 <==
[...]
==> 2024-04-24 17:47:32 <==
# cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all

You can also, use a script and the conda module available in the base environment: example

The conda MatchSpec format

The conda search command uses the MatchSpec format

conda install and conda search probably use the same syntax

The MatchSpec format is poorly documented…

Fortunately, you will usually only specify the package name, and want to install the latest version of a package

Using == (exact version) is different from = (“fuzzy” value), which can be confusing

$ conda search zlib==1.2
No match found for: zlib==1.2. Search: *zlib*==1.2
PackagesNotFoundError: The following packages are not available from current channels:
- zlib==1.2
$ conda search zlib=1.2
# Name Version Build Channel
zlib 1.2.8 0 conda-forge
zlib 1.2.8 1 conda-forge
[...]
zlib 1.2.13 h5eee18b_1 pkgs/main
zlib 1.2.13 hd590300_5 conda-forge
$ conda search zlib==1.2.12
zlib 1.2.12 h166bdaf_0 conda-forge
[...]
zlib 1.2.12 h7f8727e_2 pkgs/main

Documentation:

conda Package match specifications

Documentation in the MatchSpec class

Uninstalling Miniconda3

Windows computer

Miniconda3 can be removed like a regular Windows application

Go to the Apps & features panel

Start ⇒ Settings ⇒ Apps ⇒ Apps & features

Select Miniconda3 py3xxxxx

Click on Uninstall and follow the instructions displayed by the uninstaller window

This will remove:

The (very big) directory that you specified when you installed Miniconda3

The conda shortcuts from the Start Menu

This will not remove your local configuration file C:\Users\your_login\.condarc

Linux-like computer

Improving conda and related tools

There is always room for improvement, and this is is some kind of TODO section

conda and mamba ideas

Add a conda command to find the package owning a file

The link also shows how to do it with standard Linux tools (find, grep), based on the Conda package specification page

conda-which seems to work (see test / example) !

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]