Check if the Python package you are desperately seeking is not already installed, before installing your own Python on shared Linux servers, or on a shared cluster !!

• Ask other users who know the local environment

• Read the servers/cluster documentation

• Use module avail [mod_name] to find out the Python versions available, and then module load and conda list to get more information
  • e.g. on spiritx:

    $ module avail python
    python/meso-3.8  python/meso-3.9  python/meso-3.10  python/meso-3.11
    $ module avail anaconda
    anaconda-meso/2022.10    anaconda3-py/2020.11
    anaconda-meso/2023.09-0  anaconda3-py/2021.11

• Lots of Python packages do very similar things. Maybe you can use another installed package…

• If the packages you need are not available, it may be faster and easier to ask the local system administrators if they can install the packages, rather than doing everything yourself

If there is a jupyter notebook server connected to your system, but the Python you want to use is not available on the server, follow the using a non-standard kernel instructions, rather than starting your own local server

• Miniconda3 is a minimal/bootstrap Python distribution that can be used for creating more complex Python distributions.
  It will basically make a recent conda command available on your computer, and you can then use this conda executable to select an existing distribution, or create new and independent Python distributions.

• A conda environment is basically (the short name of the directory) where you install the Python that you will use (independently of Miniconda3 itself). You have to activate this environment in order to use it
  A python distribution is the collection of packages (and their dependencies) you have chosen to install together in a given environment.
  • Example: initializing an environment named cdatm_py3 (assuming it exists)

    $ source /path_to_miniconda/etc/profile.d/conda.sh
    
    $ which conda
    /path_to_miniconda/condabin/conda
    
    $ conda env list
    [...]
    cdatm_py3                /path_to_miniconda/envs/cdatm_py3
    some_other_env           /path_to_miniconda/envs/some_other_env
    
    $ conda activate cdatm_py3
    
    (cdatm_py3) $ which python
    /path_to_miniconda/envs/cdatm_py3/bin/python

  • The special environment where Miniconda3 itself is installed is called base (it also used to be called root). Note that the python binary of the special base environment is not located in the same directory hierarchy as the python of the other environments (i.e. there is no envs/ subdirectory

    $ conda activate base
    
    (base) $ which python
    /path_to_miniconda/bin/python

• You don't need to be (and you should not be) root when you install Miniconda3. You just need enough disk space on a disk where you have write access
  • WARNING: by default, Miniconda3 will install itself and subsequent Python environments in a sub-directory of your home directory (~/miniconda3 on Linux). Do not use the default installation location, if you want to avoid disk space related problems (disk full, quota exceeded, …)
    • WARNING: older versions of Miniconda3 used a hidden sub-directory of your home directory (~/.conda on Linux) for the installation
  • Choose carefully where you will install Miniconda3, because the size of the installation directory will start at a few Gb and will keep on growing

• You could also start with the full Anaconda (instead of Miniconda) installer that will install a much more complete python environment, ready for use.
  We choose not to use the full Anaconda installer because:
  • it requires more disk space than Miniconda3 during the initial installation,
  • all the Anaconda installer packages come from the default channel (or repository) provided by the conda repository.
    This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the conda-forge channel, in order to avoid complex package dependency problems

Some of the steps below are adapted from the Windows section of "Quick command line install" to install a pure Windows version of Miniconda3

• We use a download link that always point to the latest version of the Windows installer
  • https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe
  • The installer is based on Python 3.12.2 and Conda 24.3.0 as of April 2024

• Find some temporary space on Windows
  • e.g. C:\Scratch\your_login or whatever you use as a local Scratch folder

• Open a terminal, where you can type the required installation commands
  • Windows Powershell terminal: Start ⇒ W ⇒ Windows PowerShell ⇒ Windows PowerShell
  • PowerShell in a Windows Terminal

• Use curl to download the latest installer (78 Mb as of April 2024):

  PS C: cd C:\Scratch\your_login
  PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe
  PS C:\Scratch\your_login> dir miniconda.exe
  [...]
  -a----        19/04/2024     11:18       81274240 miniconda.exe

• Determine the folder where you will install Miniconda3 and ALL your future Python environments based on this version of Miniconda3
  We will later refer to this location as /path_to_miniconda3/ in the Linux sections
  • Remember that you will need a few Gb of available space
  • You should avoid installing Miniconda3 in your Windows home folder (C:\Users\your_login or <some_disk_different_from_c>:\Users\<your_login>), if you want to be able to easily make backups or your home, without backing up thousands of files related to your Python environments…
  • We add some date or version information at the end of the folder name, in order to differentiate it from other (older of future) installations
  • Example installation folder: C:\Utils\miniconda3_2024-03 (if we start installing in March 2024)

• Launch the Miniconda3 installer with

  PS C:\Scratch\your_login> .\miniconda.exe

• In the installer window:
  • (Review and) accept the License Agreement
  • Choose to install for Just Me (the recommended default choice)
  • Specify the Destination Folder selected above, e.g. C:\Utils\miniconda3_2024-03
  • Use the pre-selected Advanced Installation Options, i.e
    • Create start menu shortcutes
    • Register Miniconda3 as my default Python 3.12 ⇐ Useful?

• When the installer has finished running, you should have:
  • A new C:\Utils\miniconda3_2024-03 folder
    • This folder size will be ~650 Mb and will keep on growing (up to several Gb) as you add more Python environments and packages
      • ⇒ Be sure to have enough space on this disk!
  • Two new shortcuts in the Windows Start menu
    • Anaconda Prompt
    • Anaconda Powershell Prompt
      • We will now use this Anaconda PowerShell shortcut each time we need to use Miniconda3 or Python

• Remove the installer

  PS C:\Scratch\your_login> del .\miniconda.exe

• Open an Anaconda PowerShell Prompt, and type a few commands to make sure that conda is working

  (base) PS C:\Users\your_login> conda env list
  base                  *  C:\Utils\miniconda3_2024-03
  
  (base) PS C:\Users\your_login> conda list
  # packages in environment at C:\Utils\miniconda3_2024-03:
  # Name                    Version                   Build  Channel
  [...]
  conda                     24.3.0          py312haa95532_0
  [...]
  python                    3.12.2               h1d929f7_0
  [...]
  
  (base) PS C:\Users\your_login> conda deactivate
  
  PS C:\Users\your_login> conda env list
  base                     C:\Utils\miniconda3_2024-03

• Next: be sure to follow the steps in the Fine-tuning conda to use (only) conda-forge, and all the other Miniconda3 instructions
  • conda commands will work on all computers (Windows and Linux-like)
  • The .condarc is located in your Windows home folder
    ⇒ C:\Users\your_login\.condarc

• Important: do not install your own Python environment(s) on shared computers before you have made sure that the packages you need are not already installed, or cannot be easily installed in an existing environment (by the person who maintains this environment)!
  • Check the output of module avail, read your system documentation, ask other users…

• We use a download link that always point to the latest version of the Linux installer
  • For the Linux computers we commonly use, we need the Miniconda3 Linux 64-bit link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
    • Note: Mac users should use the latest macOS installer appropriate for their computer
  • The installer is based on Python 3.12.2 and Conda 24.3.0 as of April 2024

• Find some temporary space on Linux (or Windows, if you are using WSL)
  • e.g. Linux on the LSCE servers:
    $ cd /home/scratch01/$USER
  • e.g. Linux on the IPSL spiritx cluster:

    $ mkdir /homedata/$USER/Scratch
    $ cd /homedata/$USER/Scratch

  • e.g. Linux on a personal PC/Mac desktop/laptop: use a local scratch/temporary directory with enough space

• Use wget to download the latest installer (138 Mb as of 22 Mar 2024):
  

  $ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  $ ls -lh Miniconda3-latest-Linux-x86_64.sh
  [...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh

We assume below that we are in the directory where we have downloaded the installer

• Determine the directory where you will install miniconda3 and all the Python environments based on this version of miniconda3, but do not create the directory (the installation script will do it)
  We will later refer to this location as /path_to_miniconda3/
  • Remember that you will need a few Gb of available space
  • If you want to share this installation with other users, the directory has to be readable by these users
  • At LSCE, do not install in a sub-directory of your home directory!
    • On other servers and computers, you should also avoid installing in your home directory, if you want to be able to easily make backups or your home, without backing up thousands of files related to your Python installation…
  • We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations
  • e.g. Linux at LSCE: /home/share/unix_files/cdat/miniconda3_2024-03
  • e.g. Linux at spiritx: /homedata/$USER/miniconda3_2024-03
  • Linux (including WSL) on a personal PC/Mac desktop/laptop: find a directory with enough space (including space for adding more packages and their future updates)
    • You may want to use a directory that you are not backing up, if you want to avoid having to save thousands of files…

• Execute the installer with $ bash Miniconda3-latest-Linux-x86_64.sh
  • Review the license (type <SPACE> several times…) and accept it
  • When asked for the installation directory, specify the directory location chosen in the previous step, and not the default directory
    • e.g. specify
      /path_to_miniconda3/
      instead of the default
      $HOME/miniconda3
  • Answer no to the question Do you wish to update your shell profile to automatically initialize conda?.
    Otherwise the installation will make changes to your shell configuration files
    • If you forget to answer no, you can apparently:
      • remove the changes to your shell configuration files by typing later: conda init –reverse $SHELL
      • or at least disable the automatic activation of the environment (that might have side effect) by typing: conda config –set auto_activate_base false
    • The resulting miniconda3 directory size is 647 Mb (as of March 2024)
      

      $ du -sh miniconda3_2024-03
      647M    miniconda3_2024-03
      
      $ du -sh miniconda3_2024-03/*
      47M     miniconda3_2024-03/bin
      16K     miniconda3_2024-03/cmake
      8.0K    miniconda3_2024-03/compiler_compat
      32M     miniconda3_2024-03/_conda
      8.0K    miniconda3_2024-03/condabin
      976K    miniconda3_2024-03/conda-meta
      4.0K    miniconda3_2024-03/envs
      28K     miniconda3_2024-03/etc
      18M     miniconda3_2024-03/include
      317M    miniconda3_2024-03/lib
      92K     miniconda3_2024-03/LICENSE.txt
      1.1M    miniconda3_2024-03/man
      232M    miniconda3_2024-03/pkgs
      396K    miniconda3_2024-03/sbin
      1.5M    miniconda3_2024-03/share
      12K     miniconda3_2024-03/shell
      8.0K    miniconda3_2024-03/ssl
      8.0K    miniconda3_2024-03/x86_64-conda_cos7-linux-gnu
      8.0K    miniconda3_2024-03/x86_64-conda-linux-gnu

• Initialize the newly installed conda environment (this will initialize the environment only in the current terminal):
  • bash shell: source /path_to_miniconda3/etc/profile.d/conda.sh
    • e.g. (spiritx): source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
  • tcsh shell: source /path_to_miniconda3/etc/profile.d/conda.csh
    • e.g. (LSCE): source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh

• Check if you can use the conda command, and use it to initialize the base environment
  • e.g. on spiritx1:

    $ which conda
    $ which python
    /usr/bin/python
    
    $ source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
    
    $ which conda
    /homedata/jypmce/miniconda3_2024-03/condabin/conda
    $ which python
    /usr/bin/python
    
    $ conda activate base
    
    (base) $ which conda
    /homedata/jypmce/miniconda3_2024-03/bin/conda
    (base) $ which python
    /homedata/jypmce/miniconda3_2024-03/bin/python
    
    (base) $ conda deactivate
    
    $ which conda
    /homedata/jypmce/miniconda3_2024-03/condabin/conda
    $ which python
    /usr/bin/python

• Remove the Miniconda3 installer: $ rm Miniconda3-latest-Linux-x86_64.sh

• conda will probably work fine with the default settings if you create simple new environments with just one package and its dependencies.

• conda works much better and faster since the end of 2023 (conda versions starting at 23.10), now that it is using the mamba solver instead of the default solver.
  • If you have followed all the steps of this section in order to use only conda-forge packages, and there are still some problems with a recent version of conda, try to install and use mamba, as a drop-in replacement of conda
  • Historical note: before conda used libmamba, we had to use mamba (instead of conda) for dealing with our complex Python environments:
    • Making conda fast again
    • Solving environment: failed with initial frozen solve. Retrying with flexible solve.

• We are power users, and we intend to create complex Python environments combining lots (and lots) of packages available from conda-forge, with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge.
  • ⇒ The solution is to configure conda to always use conda-forge, and completely update Miniconda3 itself with packages coming only from conda-forge

The following steps will make sure that we only get packages from conda-forge (same thing as using the -c conda-forge option) by default unless the requested packages really don't exist on conda-forge. More details in Managing channels

$ cat ~/.condarc
cat: /home/jypmce/.condarc: No such file or directory

$ conda config --get channels

$ conda config --prepend channels conda-forge

$ conda config --set channel_priority strict

$ conda config --get channels
--add channels 'defaults'   # lowest priority
--add channels 'conda-forge'   # highest priority

$ cat ~/.condarc
channels:
  - conda-forge
  - defaults
channel_priority: strict

• conda config documentation
• Basic information: conda info
• Full configuration: conda config --show

We have to completely update Miniconda3 at least once after installing Miniconda3 and making conda-forge the highest priority channel

• Check the current version of conda

  • $ conda list -n base | grep conda
    # packages in environment at /path_to_miniconda:
    conda                     24.1.2          py312h06a4308_0
    conda-libmamba-solver     23.12.0            pyhd3eb1b0_1
    [... and other packages with 'conda' in their name]

• Completely update the Miniconda3 installation

  • $ conda update -n base --all
    Channels:
     - conda-forge
     - defaults
    Platform: linux-64
    Collecting package metadata (repodata.json): done
    Solving environment: done
    
    ## Package Plan ##
    
      environment location: /homedata/jypmce/miniconda3_2024-03
    
    
    The following packages will be downloaded:
    
        package                    |            build
        ---------------------------|-----------------
        conda-24.3.0               |  py312h7900ff3_0         1.1 MB  conda-forge
        conda-libmamba-solver-24.1.0|     pyhd8ed1ab_0          40 KB  conda-forge
        python-3.12.2              |hab00c5b_0_cpython        30.8 MB  conda-forge
    [...]
    The following NEW packages will be INSTALLED:
    [...]
    The following packages will be UPDATED:
    [...]
      conda              pkgs/main::conda-24.1.2-py312h06a4308~ --> conda-forge::conda-24.3.0-py312h7900ff3_0
    [...]
    The following packages will be SUPERSEDED by a higher-priority channel:
    [...]
    Proceed ([y]/n)? y
    [...]
    Preparing transaction: done
    Verifying transaction: done
    Executing transaction: done

• Check the updated version of conda. Note that (almost) all the packages should now specify that they are provided by conda-forge

  • $ conda list -n base
    conda                     24.3.0          py312h7900ff3_0    conda-forge
    conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    conda-forge
    python                    3.12.2          hab00c5b_0_cpython    conda-forge

  • You can use the following to determine if some packages are not provided by conda-forge

    • $ conda list -n base | grep -v conda-forge
      # Name                    Version                   Build  Channel
      libedit                   3.1.20230828         h5eee18b_0
      libffi                    3.4.4                h6a678d5_0
      xz                        5.4.5                h5eee18b_0

    • It should not be a major problem if a few packages are not provided by conda-forge.
      It only means that some packages provided by the anaconda channel were more recent than the same packages provided by the conda-forge channel when the packages were updated
      • libedit example: version 3.1.20191231 @ conda-forge and version 3.1.20230828 @ anaconda

• You can later update again the full Miniconda3, or just the conda command.
  This should not make any changes to the Python environment(s) have have installed with conda

  • $ conda update -n base --all
    [...]
    # All requested packages already installed.
    
    (base) $ conda update -n base conda
    [...]
    # All requested packages already installed.

• Note: during this installation, the miniconda3 directory size grew from 763 Mb to 1.6 Gb, and we have not installed any custom Python environment yet! This is the reason why you should install Miniconda3 on a (preferably non backed up) disk where you have enough space

  • $ du -sh /homedata/jypmce/miniconda3_2024-03
    736M    /homedata/jypmce/miniconda3_2024-03
    [... update]
    $ du -sh /homedata/jypmce/miniconda3_2024-03
    1.6G    /homedata/jypmce/miniconda3_2024-03

• You can clean the initial Miniconda3 installation to free up some disk space. The documentation specifies WARNING: This will break environments with packages installed using symlinks back to the package cache. and you should probably avoid using clean once you have installed new Python environments (unless you are desperate for disk space and know how to do a complete re-installation if something breaks…)

  • (base) $ conda clean --all
    Will remove 154 (180.9 MB) tarball(s).
    Proceed ([y]/n)? y
    
    Will remove 1 index cache(s).
    Proceed ([y]/n)? y
    
    Will remove 74 (378.0 MB) package(s).
    Proceed ([y]/n)? y
    
    There are no tempfile(s) to remove.
    There are no logfile(s) to remove.
    
    (base) $ du -sh /homedata/jypmce/miniconda3_2024-03
    457M    /homedata/jypmce/miniconda3_2024-03

When you open a terminal, your shell needs to know where to find the conda command used to initialize an environment, or switch between existing environments. This can be configured in the shell configuration files

You were asked the following question when installing Miniconda3: Do you wish to update your shell profile to automatically initialize conda? [yes|no]

• If you answered yes, the installer probably added some very complicated lines to your shell configuration files, and conda and the newly installer python are probably directly available when you open a new terminal.

• If you answered no (as suggested), use a text editor to add extra lines to the appropriate shell configuration file
  • bash shell user : add these lines to ~/.bashrc
    

    source /path_to_miniconda3/etc/profile.d/conda.sh
    alias pynit='conda activate base'

    • e.g. (spiritx):

      source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
      alias pynit='conda activate base'

  • tcsh shell user: add these lines to ~/.cshrc
    

    source /path_to_miniconda3/etc/profile.d/conda.csh''
    alias pynit 'conda activate base'

    • e.g. (LSCE):

      source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh
      alias pynit 'conda activate base'

  • When you have a custom Python environment installed (e.g. my_power_env, you can update the pynit alias to initialize this environment instead of the base environment (e.g. conda activate my_power_env)
  • You can also define other aliases, or use another name than pynit!

We choose not to directly add a conda activate env_name line to the shell configuration files. This would permanently initialize env_name and may cause future (and obscure) side effects.

When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type conda activate env_name or the pynit alias defined above

In the case of python environments maintained by a single user, but used by several users, we could do the same as in the General case, but it can be useful to have the users source an intermediate initialization file, that will then source the initialization file used in the general case. This makes it easier to maintain and change the environments, without asking users to make changes.

• ask bash users to add to ~/.bashrc something like
  source ~main_installer_login/.conda3_jyp.sh
  
  with a .conda3_jyp.sh file looking like conda3_jyp.sh.txt

• ask tcsh users to add to ~/.cshrc something like
  source ~main_installer_login/.conda3_jyp.csh
  
  with a .conda3_jyp.csh file looking like conda3_jyp.csh.txt

• Just type conda update -n base --all as shown in the Updating (Mini)conda section

• The base (miniconda3) and the other environments are all independent and can be updated independently

• It is up to you to decide if you want to update environments or not

• Official conda website
  • Cheat sheet (2 pages, for people in a hurry)
  • Commands' reference
• Package repositories
  • conda-forge ⇐ recommended source of packages
  • default

This is just a subset of some of the commands, and a subset of their options! For a complete reference, use the official conda website

• Get help
  • conda -h
    • conda command --help: help for a specific command

• Available environments
  • conda env list
  • The currently active environment has a “*” character on its line

    (base) $ conda env list
    # conda environments:
    #
    base                  *  /homedata/jypmce/miniconda3_2024-03

• Activate an existing environment (available in conda env list) or deactivate, in order to go back to the default Python
  • conda activate existing_environment
    • conda activate (without specifying an environment name) will activate the base Miniconda3 environment
  • Go back to the default Python available on the system
    • conda deactivate
  • You can type which python to determine where is the current python executable you are using is
    You should always know which Python you are using!

    • (base) $ which python
      /homedata/jypmce/miniconda3_2024-03/bin/python
      
      (base) $ conda deactivate
      
      $ which python
      /usr/bin/python

  • Note: before conda 4.6, you had to type source activate existing_environment and source deactivate

• Get the list of installed packages, their version, and where they come from (e.g. conda-forge, pip)
  • conda list [-n existing_environment]

• Create an environment and install a few packages and their dependencies
  • conda create -n new_environment_name package1 package2 packageN

• Delete an environment
  • conda remove -n crap_environment --all

• Find out if a package is available on the conda-forge channel (or other channels)
  • Note: you should always install a package with conda instead of pip, if it is available on a conda channel
  • conda search package_exact_name
conda search '*package_partial_name*'
    • Note: search with wildcards fails in a tcsh shell, as of April 2024 (CSH install/search fail with '*' variable in an argument)

• Install and remove packages
  Add -n target_environment to make changes in another environment than the active one
  • conda install package1 package2 packageN
    • conda should already be configured to use conda-forge by default if you have read this page carefully
    • Add -c channel_url_or_alias to install from a specific channel
  • conda remove package1 package2 packageN

• Update just conda (update the conda package, not the full Miniconda3 base environment)
  • conda update -n base conda

• Completely update an environment (all the packages)
  • conda update -n existing_environment --all

• Cleaning… Dangerous!
  If you have several environments, conda clean may remove packages that are not needed in an environment, but are still used in other environments, and you may end up with a broken installation…
  conda does not automatically clean the content of the pkgs directory!
  Use the following to Remove index cache, lock files, tarballs, unused cache packages, and source cache
  • conda clean --all

• Read the Conda vs. pip vs. virtualenv commands section

• Read the Using pip in an environment

You can get the current list of installed packages with conda list -n existing_environment, but it may also be useful to know in which order the packages were installed:

• conda list -n existing_environment --revisions: display exactly what was installed and when (including the updates, and automatic dependencies installation)
  • You can theoretically roll back to a previous RR revision number with: conda install -n existing_environment --revision RR
• There is more information in the /path_to_miniconda/conda-meta/history installation log files.
  • base history file: /path_to_miniconda/conda-meta/history
  • existing_environment history file: /path_to_miniconda/envs/existing_environment/conda-meta/history
  • e.g. installation commands for the base environment:
    

     $ egrep -e 'cmd:' -e '==>' /homedata/jypmce/miniconda3_2024-03/conda-meta/history
    
    ==> 2024-02-23 18:39:23 <==
    # cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW
    ==> 2024-03-27 16:10:01 <==
    # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all
    ==> 2024-03-28 11:38:08 <==
    [...]
    ==> 2024-04-24 17:47:32 <==
    # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all

  • You can also, use a script and the conda module available in the base environment: example

Miniconda3 can be removed like a regular Windows application

• Go to the Apps & features panel
  • Start ⇒ Settings ⇒ Apps ⇒ Apps & features
• Select Miniconda3 py3xxxxx
• Click on Uninstall and follow the instructions displayed by the uninstaller window
  • This will remove:
    • The (very big) directory that you specified when you installed Miniconda3
    • The conda shortcuts from the Start Menu
  • This will not remove your local configuration file C:\Users\your_login\.condarc

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]