This is an old revision of the document!
Check if the Python package you are desperately seeking is not already installed, before installing your own Python on shared interactive Linux servers, or on batch computing clusters !!
module avail [mod_name]
to find out the Python versions available, and then module load
and conda list
to get more information on the available Python packagesspiritx
(modules on spirit)): $ module avail python python/meso-3.8 python/meso-3.9 python/meso-3.10 python/meso-3.11 $ module avail anaconda anaconda-meso/2022.10 anaconda3-py/2020.11 anaconda-meso/2023.09-0 anaconda3-py/2021.11 $ module avail pangeo pangeo-meso/2023.04.15 pangeo-meso/2024.01.22 $ module load pangeo-meso/2024.01.22 (pangeo-meso-2024.01.22) $ which python /net/nfs/tools/u20/Python/miniconda3_py311_23.11.0-2/envs/pangeo-meso-2024.01.22/bin/python $ conda list | grep cartopy cartopy 0.22.0 py311h320fe9a_1 conda-forge
If there is a jupyter notebook server connected to your system, and a Python environment providing all the packages you need, but that is not directly available on the notebook server, follow the using a non-standard kernel instructions, rather than starting your own local server
conda
command/executable available on your computer, and you can then use this conda
executable to select an existing distribution, or create new and independent Python distributions.activate
this environment in order to use itcdatm_py3
(assuming it exists) in a bash shell $ source /path_to_miniconda/etc/profile.d/conda.sh $ which conda /path_to_miniconda/condabin/conda $ conda env list [...] cdatm_py3 /path_to_miniconda/envs/cdatm_py3 some_other_env /path_to_miniconda/envs/some_other_env $ conda activate cdatm_py3 (cdatm_py3) $ which python /path_to_miniconda/envs/cdatm_py3/bin/python
base
(it also used to be called root
). Note that the python
binary of the special base
environment is not located in the same directory hierarchy as the python
of the other environments (i.e. there is no envs/
subdirectory$ conda activate base (base) $ which python /path_to_miniconda/bin/python
~/miniconda3
on Linux). Do not use the default installation location, if you want to avoid disk space related problems (disk full, quota exceeded, …) during the inittial installation or later~/.conda
on Linux)default
channel (or repository) provided by the conda repository.Some of the steps below are adapted from the Windows section of "Quick command line install" to install a pure Windows version of Miniconda3
C:\Scratch\your_login
or whatever you use as a local Scratch folderStart
⇒ W
⇒ Windows PowerShell
⇒ Windows PowerShell
curl
to download the latest installer (78 Mb as of April 2024): PS C: cd C:\Scratch\your_login PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe PS C:\Scratch\your_login> dir miniconda.exe [...] -a---- 19/04/2024 11:18 81274240 miniconda.exe
/path_to_miniconda3/
in the Linux sections belowC:\Users\your_login
or <some_disk_different_from_c>:\Users\<your_login>), if you want to be able to easily make backups or your home, without backing up thousands of files related to your Python environments…C:\Utils\miniconda3_2024-03
(if we start installing in March 2024)PS C:\Scratch\your_login> .\miniconda.exe
C:\Utils\miniconda3_2024-03
C:\Utils\miniconda3_2024-03
folderStart
⇒M
⇒Miniconda3
menu:Anaconda Prompt
Anaconda Powershell Prompt
PS C:\Scratch\your_login> del .\miniconda.exe
conda
is workingStart
⇒M
⇒Miniconda3
⇒ Anaconda Powershell Prompt
(base) PS C:\Users\your_login> conda env list base * C:\Utils\miniconda3_2024-03 (base) PS C:\Users\your_login> conda list # packages in environment at C:\Utils\miniconda3_2024-03: # Name Version Build Channel [...] conda 24.3.0 py312haa95532_0 [...] python 3.12.2 h1d929f7_0 [...] (base) PS C:\Users\your_login> conda deactivate PS C:\Users\your_login> conda env list base C:\Utils\miniconda3_2024-03
conda
commands will work the same way on all Operating Systems (Windows and Linux-like).condarc
is located in your Windows home folderC:\Users\your_login\.condarc
(or $HOME\.condarc
)By Linux-like, we mean:
$ cd /home/scratch01/$USER
$ mkdir /homedata/$USER/Scratch $ cd /homedata/$USER/Scratch
wget
to download the latest installer (138 Mb as of 22 Mar 2024):$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh $ ls -lh Miniconda3-latest-Linux-x86_64.sh [...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh
We assume below that we are in the directory where we have downloaded the installer
/path_to_miniconda3/
/home/share/unix_files/cdat/miniconda3_2024-03
/homedata/$USER/miniconda3_2024-03
$ bash Miniconda3-latest-Linux-x86_64.sh
<SPACE>
several times…) and accept it/path_to_miniconda3/
$HOME/miniconda3
no
to the question Do you wish to update your shell profile to automatically initialize conda?
.no
, you can apparently:conda init --reverse $SHELL
conda config --set auto_activate_base false
miniconda3
directory size is 647 Mb (as of March 2024).$ du -sh miniconda3_2024-03 647M miniconda3_2024-03 $ du -sh miniconda3_2024-03/* 47M miniconda3_2024-03/bin 16K miniconda3_2024-03/cmake 8.0K miniconda3_2024-03/compiler_compat 32M miniconda3_2024-03/_conda 8.0K miniconda3_2024-03/condabin 976K miniconda3_2024-03/conda-meta 4.0K miniconda3_2024-03/envs 28K miniconda3_2024-03/etc 18M miniconda3_2024-03/include 317M miniconda3_2024-03/lib 92K miniconda3_2024-03/LICENSE.txt 1.1M miniconda3_2024-03/man 232M miniconda3_2024-03/pkgs 396K miniconda3_2024-03/sbin 1.5M miniconda3_2024-03/share 12K miniconda3_2024-03/shell 8.0K miniconda3_2024-03/ssl 8.0K miniconda3_2024-03/x86_64-conda_cos7-linux-gnu 8.0K miniconda3_2024-03/x86_64-conda-linux-gnu
source /some_path/conda_init_script
conda activate
conda activate base
source /path_to_miniconda3/etc/profile.d/conda.sh
source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
source /path_to_miniconda3/etc/profile.d/conda.csh
source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh
conda
command, and use it to initialize the base environment$ which conda /homedata/jypmce/miniconda3_2024-03/condabin/conda $ which python /usr/bin/python $ conda activate base (base) $ which conda /homedata/jypmce/miniconda3_2024-03/bin/conda (base) $ which python /homedata/jypmce/miniconda3_2024-03/bin/python (base) $ conda deactivate $ which conda /homedata/jypmce/miniconda3_2024-03/condabin/conda $ which python /usr/bin/python
$ rm Miniconda3-latest-Linux-x86_64.sh
conda
will probably work fine with the default settings if you create simple new environments with just one package and its dependencies.conda
works much better and faster since the end of 2023 (conda
versions starting at 23.10
), now that it is using the mamba solver instead of the default solver.conda --version
version to determine which version you are using$ conda --version conda 24.5.0
conda-forge
, with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge.conda
to always use conda-forge, and completely update Miniconda3 itself with packages coming only from conda-forgeconda
(>= 23.10
), and you have followed all the steps below, in order to use only conda-forge packages, and there are still some package installation problems, try to install and use mamba, as a drop-in replacement of conda
conda
used libmamba
, we had to use mamba
(instead of conda
) for dealing with our complex Python environments:
The following steps will make sure that we only get packages from conda-forge (same thing as using the -c conda-forge
option) by default, unless the requested packages really do not exist on conda-forge. More details in Managing channels
$ cat ~/.condarc # On Windows use: cat $HOME\.condarc cat: /home/jypmce/.condarc: No such file or directory $ conda config --get channels $ conda config --prepend channels conda-forge $ conda config --set channel_priority strict $ conda config --get channels --add channels 'defaults' # lowest priority --add channels 'conda-forge' # highest priority $ cat ~/.condarc # On Windows use: cat $HOME\.condarc channels: - conda-forge - defaults channel_priority: strict
Do not forget to update conda at least once in order to take all the changes above into account!
mamba
installationmamba
if you are sure that you will need some features that are not already provided by conda
(e.g. conda repoquery
)mamba is a fast, robust, and cross-platform package manager […] fully compatible with conda packages, and supports most of conda’s commands
mamba
is used to manage environments, and has to be installed in the base environment (i.e. in the same environment as conda
)!conda install -n base mamba
mamba repoquery
conda update -n base --all
, (review) and accept the changesbase
(Miniconda3 itself) and the other Python environments installed with the conda
executable provided by the base
environment are all independent, and can be updated independentlybase
environment at least once in order to replace all of its packages by their equivalent conda-forge
packages
conda
command.$ conda update -n base --all [...] # All requested packages already installed. (base) $ conda update -n base conda [...] # All requested packages already installed.
miniconda3
directory size grew from 763 Mb to 1.6 Gb, and we have not installed any custom Python environment yet! This is the reason why you should install Miniconda3 on a (preferably non backed up) disk where you have enough free space$ du -sh /homedata/jypmce/miniconda3_2024-03 736M /homedata/jypmce/miniconda3_2024-03 [... update] $ du -sh /homedata/jypmce/miniconda3_2024-03 1.6G /homedata/jypmce/miniconda3_2024-03
We have to completely update Miniconda3 at least once after installing Miniconda3 and making conda-forge the highest priority channel
conda
$ conda --version conda 24.1.2
$ conda update -n base --all Channels: - conda-forge - defaults Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: /homedata/jypmce/miniconda3_2024-03 The following packages will be downloaded: package | build ---------------------------|----------------- conda-24.3.0 | py312h7900ff3_0 1.1 MB conda-forge conda-libmamba-solver-24.1.0| pyhd8ed1ab_0 40 KB conda-forge python-3.12.2 |hab00c5b_0_cpython 30.8 MB conda-forge [...] The following NEW packages will be INSTALLED: [...] The following packages will be UPDATED: [...] conda pkgs/main::conda-24.1.2-py312h06a4308~ --> conda-forge::conda-24.3.0-py312h7900ff3_0 [...] The following packages will be SUPERSEDED by a higher-priority channel: [...] Proceed ([y]/n)? y [...] Preparing transaction: done Verifying transaction: done Executing transaction: done
conda
and python
. Note that (almost) all the packages should now specify that they are provided by conda-forge
$ conda list -n base [...] conda 24.3.0 py312h7900ff3_0 conda-forge conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge [...] python 3.12.2 hab00c5b_0_cpython conda-forge [...]
conda-forge
$ conda list -n base | grep -v conda-forge # Name Version Build Channel libedit 3.1.20230828 h5eee18b_0 libffi 3.4.4 h6a678d5_0 xz 5.4.5 h5eee18b_0
conda-forge
.anaconda
channel were more recent than the same packages provided by the conda-forge
channel when the packages were updated
It seems safe to reclaim some disk space by cleaning up the initial (base) installation.
Use cleaning with caution after installing other environments…
You can clean the initial Miniconda3 installation to free up some disk space.
The documentation specifies WARNING: This will break environments with packages installed using symlinks back to the package cache.
You should probably avoid using clean
once you have installed new Python environments (unless you are desperate for disk space and know how to do a complete re-installation if something breaks…)
(base) $ conda clean --all Will remove 154 (180.9 MB) tarball(s). Proceed ([y]/n)? y Will remove 1 index cache(s). Proceed ([y]/n)? y Will remove 74 (378.0 MB) package(s). Proceed ([y]/n)? y There are no tempfile(s) to remove. There are no logfile(s) to remove. (base) $ du -sh /homedata/jypmce/miniconda3_2024-03 457M /homedata/jypmce/miniconda3_2024-03
PowerShell
terminal with: Start
⇒ M
⇒ Miniconda3
⇒ Anaconda Powershell Prompt
base
environment will be activated by defaultconda activate name_of_the_environment_you_want
TODO
menuinst
in order to directly initialize an existing environment (different from base
) from the Start
menu and from a Windows Terminal
When you open a terminal, your shell needs to know where to find the conda
command used to initialize an environment, or switch between existing environments. This can be configured in your shell configuration files (.bashrc
, .cshrc
, …)
You were asked the following question when installing Miniconda3: Do you wish to update your shell profile to automatically initialize conda? [yes|no]
conda
and the newly installed python
are probably directly available when you open a new terminal.conda init --reverse $SHELL
conda config --set auto_activate_base false
~/.bashrc
source /path_to_miniconda3/etc/profile.d/conda.sh alias pynit='conda activate base'
source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh alias pynit='conda activate base'
~/.cshrc
source /path_to_miniconda3/etc/profile.d/conda.csh'' alias pynit 'conda activate base'
source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh alias pynit 'conda activate base'
pynit
in any (new) terminal you open, when you need to use Pythonmy_power_env
, you can update the pynit
alias to initialize this environment instead of the base
environment (e.g. conda activate my_power_env
)pynit
!We need to be extra-careful in the case of python environments maintained by a single user, but used by several users, in order not to break anything
We could use the same configuration file as in the single-user case, but it is more efficient to have the users source an intermediate initialization file. The intermediate file will:
pynit
⇒ initialize the base environment (or the environment the central maintainer chooses)wp
⇒ which python
wconda
⇒ which python
ln -s
) to point from the unchanging central initialization file name to the latest initialization file./home/share/unix_files/cdat/conda_init_files/
:$ ls -l -rw-r--r-- Jun 14 16:37 conda3_jyp_2024-03.csh -rw-r--r-- Jun 14 17:48 conda3_jyp_2024-03.sh lrwxrwxrwx Jun 17 15:56 conda3_jyp_latest.csh -> conda3_jyp_2024-03.csh lrwxrwxrwx Jun 17 15:57 conda3_jyp_latest.sh -> conda3_jyp_2024-03.sh
bash
users should add to their ~/.bashrc
file something likesource /some_path_accessible_by_all_users/shared_conda_init_file.sh
shared_conda_init_file.sh
file looking like conda3_jyp_2024-03.sh.txtsource /home/share/unix_files/cdat/conda_init_files/conda3_jyp_2024-03.sh
source /home/share/unix_files/cdat/conda_init_files/conda3_jyp_latest.sh
(link to the latest version)spiritx
and spirit
:spiritx
are accessible in read-only mode on spirit
source /homedata/jypmce/conda_init_files/conda3_jyp_spiritx_2024-03.sh
source /homedata/jypmce/conda_init_files/conda3_jyp_spiritx_latest.sh
(link to the latest version)tcsh
users should add to their ~/.cshrc
file something likesource /some_path_accessible_by_all_users/shared_conda_init_file.csh
shared_conda_init_file.csh
file looking like conda3_jyp_2024-03.csh.txtsource /home/share/unix_files/cdat/conda_init_files/conda3_jyp_2024-03.csh
source /home/share/unix_files/cdat/conda_init_files/conda3_jyp_latest.csh
(link to the latest version)This is just a subset of some of the commands, and a subset of their options! For a complete reference, use the official conda website
conda -h
conda command --help
: help for a specific commandconda --version
conda info
conda config --show
conda env list
(base) $ conda env list # conda environments: # base * /homedata/jypmce/miniconda3_2024-03
conda env list
) or deactivate, in order to go back to the default Pythonconda activate existing_environment
conda activate
(without specifying an environment name) will activate the base
Miniconda3 environmentconda deactivate
which python
to determine where is the current python
executable you are using is(base) $ which python /homedata/jypmce/miniconda3_2024-03/bin/python (base) $ conda deactivate $ which python /usr/bin/python
4.6
, you had to type source activate existing_environment
and source deactivate
conda-forge
, pip
)conda list [-n existing_environment]
conda create -n new_environment_name package1 package2 packageN
conda remove -n crap_environment --all
conda
instead of pip
or mamba
, if it is available on a conda channelconda search package_exact_name
conda search '*package_partial_name*
'search
with wildcards fails in a tcsh shell, as of April 2024 (CSH install/search fail with '*' variable in an argument)conda search package_exact_name --info
mamba
-n target_environment
to make changes in another environment than the active oneconda install package1 package2 packageN
conda
should already be configured to use conda-forge by default if you have read this page carefully-c channel_url_or_alias
to install from a specific channelconda remove package1 package2 packageN
requests
packageconda update -n existing_environment installed_package
conda
package (not the full Miniconda3 base environment) in base
environmentconda update -n base conda
conda update -n existing_environment --all
conda
does not automatically clean the content of the pkgs
directoryconda clean
may remove packages that are not needed in an environment, but are still used in other environments, and you may end up with a broken installation…
You can get the current list of installed packages with conda list -n existing_environment
, but it may also be useful to know in which order the packages were installed:
conda list -n existing_environment --revisions
: display exactly what was installed and when (including the updates, and automatic dependencies installation)RR
revision number with: conda install -n existing_environment --revision RR
/path_to_miniconda/conda-meta/history
installation log files.base
history file: /path_to_miniconda/conda-meta/history
existing_environment
history file: /path_to_miniconda/envs/existing_environment/conda-meta/history
base
environment:$ egrep -e 'cmd:' -e '==>' /homedata/jypmce/miniconda3_2024-03/conda-meta/history ==> 2024-02-23 18:39:23 <== # cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW ==> 2024-03-27 16:10:01 <== # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all ==> 2024-03-28 11:38:08 <== [...] ==> 2024-04-24 17:47:32 <== # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all
Miniconda3 can be removed like a regular Windows application
Start
⇒ Settings
⇒ Apps
⇒ Apps & features
Miniconda3 py3xxxxx
Uninstall
and follow the instructions displayed by the uninstaller windowconda
shortcuts from the Start Menu
C:\Users\your_login\.condarc