PMIP3 wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== JYP steps for installing Miniconda3 ======

===== Before installing Python on a shared computer =====

==== Desperately seeking a specific package ====

Check if the Python package you are [[https://www.imdb.com/title/tt0089017/|desperately seeking]] is not already installed, before installing your own Python **on shared interactive Linux servers, or on batch computing clusters** !!

  * Ask other users who know the local environment

  * Read the servers/clusters documentation: [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting|Python at LSCE by JYP]], [[https://intranet.lsce.ipsl.fr/informatique/index.php|LSCE]], [[https://documentations.ipsl.fr/spirit/|ESPRI Computing and Data Center]], ...

  * Use ''[[other:newppl:starting#using_module_to_access_optional_programs|module avail [mod_name]]]'' to find out the Python versions available, and then ''module load'' and ''conda list'' to get more information on the available Python packages
    * e.g. on ''spiritx'' ([[https://documentations.ipsl.fr/spirit/common/module_command.html|modules on spirit]])): <code>$ module avail python
python/meso-3.8  python/meso-3.9  python/meso-3.10  python/meso-3.11

$ module avail anaconda
anaconda-meso/2022.10    anaconda3-py/2020.11
anaconda-meso/2023.09-0  anaconda3-py/2021.11

$ module avail pangeo
pangeo-meso/2023.04.15  pangeo-meso/2024.01.22

$ module load pangeo-meso/2024.01.22
(pangeo-meso-2024.01.22) $ which python
/net/nfs/tools/u20/Python/miniconda3_py311_23.11.0-2/envs/pangeo-meso-2024.01.22/bin/python

$ conda list | grep cartopy
cartopy                   0.22.0          py311h320fe9a_1    conda-forge
</code>

  * Lots of Python packages do very similar things. Maybe you can use another similar package that is already installed...

  * If the packages you need are not available, it may be faster and easier to ask the local system administrators if they can install the packages, rather than doing everything yourself

==== Using an unlisted Python distribution on a jupyter server ====

If there is a //jupyter notebook server// connected to your system, and a Python environment providing all the packages you need, but that is not directly available on the notebook server, follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local server
===== Why should you use Miniconda3 ? =====

  * **//Miniconda3// is a minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command/executable available on your computer, and you can then use this ''conda'' executable to select an existing distribution, or create new and independent Python distributions.

  * A **conda //environment//** is basically (the short name of the directory) where you install the Python that you will use (independently of //Miniconda3// itself). You have to ''activate'' this environment in order to use it\\ \\ A **Python //distribution//** is the collection of packages (and their dependencies) you have chosen to install together in a given environment.
    * Example: initializing an environment named ''cdatm_py3'' (assuming it exists) in a //bash// shell <code>$ source /path_to_miniconda/etc/profile.d/conda.sh

$ which conda
/path_to_miniconda/condabin/conda

$ conda env list
[...]
cdatm_py3                /path_to_miniconda/envs/cdatm_py3
some_other_env           /path_to_miniconda/envs/some_other_env

$ conda activate cdatm_py3

(cdatm_py3) $ which python
/path_to_miniconda/envs/cdatm_py3/bin/python
</code>
    * The //special// environment where //Miniconda3// itself is installed is called ''base'' (it also used to be called ''root''). Note that the ''python'' binary of the //special// ''base'' environment is not located in the same directory hierarchy as the ''python'' of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base

(base) $ which python
/path_to_miniconda/bin/python
</code>

  * You **don't need to be (and you should not be !!) //root//** when you install //Miniconda3//. You just **need //enough// disk space** on a disk where you have write access
    * **WARNING**: by default, //Miniconda3// will install itself and subsequent Python environments in a sub-directory of your home directory (''~/miniconda3'' on Linux). **Do not use the default installation location**, if you want to avoid disk space related problems (//disk full//, //quota exceeded//, ...) during the inittial installation or later
      * **WARNING**: older versions of //Miniconda3// used a **hidden sub-directory** of the home directory of the person installing //Miniconda3// (''~/.conda'' on Linux)
    * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing

  * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer, that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because:
    * It requires more disk space than //Miniconda3// during the initial installation,
    * All the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**


===== Downloading Miniconda3, and BASIC installation =====

==== Miniconda3 on a Windows computer ====

Some of the steps below are adapted from the [[https://docs.anaconda.com/free/miniconda/#quick-command-line-install|Windows section of "Quick command line install"]] to install a //pure Windows// version of Miniconda3

=== Downloading the Miniconda3 installer on Windows ===

  * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Windows installer]]
    * https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe
    * The installer is based on **Python 3.12.2 and Conda 24.3.0 as of April 2024**

  * Find some temporary disk space on Windows
    * e.g. ''C:\Scratch\your_login'' or whatever you use as a [[other:win10config#local_scratch_folder|local Scratch folder]]

  * Open a //terminal//, where you can type the required installation commands
    * Windows //Powershell terminal//: ''Start'' => ''W'' => ''Windows PowerShell'' => ''Windows PowerShell''
    * [[other:win10apps#windows_terminal|PowerShell in a Windows Terminal]]

  * Use ''curl'' to download the latest installer (78 Mb as of April 2024): <code>PS C: cd C:\Scratch\your_login
PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe
PS C:\Scratch\your_login> dir miniconda.exe
[...]
-a----        19/04/2024     11:18       81274240 miniconda.exe</code>

=== BASIC Miniconda3 installation on Windows ===

  * **Determine the folder where you will install Miniconda3 and ALL your future Python environments** based on this version of Miniconda3\\ **We will later refer to this installation location as ''/path_to_miniconda3/''** in the [[#miniconda3_on_a_linux-like_computer|Linux sections]] below
    * Remember that **you will need a few Gb of available space**
    * You should **avoid installing Miniconda3 in your Windows //home// folder** (''C:\Users\your_login'' or [[other:win10config#local_home_folder|<some_disk_different_from_c>:\Users\<your_login>]]), if you want to be able to easily make backups or your //home// folder, without backing up thousands of files related to your Python environments...
    * We add some date or version information at the end of the folder name, in order to differentiate it from other (older of future) installations
    * Example installation folder: ''C:\Utils\miniconda3_2024-03'' (if we start installing in March 2024)

  * Launch the Miniconda3 installer with<code>PS C:\Scratch\your_login> .\miniconda.exe</code>

  * In the installer window:
    * (Review and) accept the License Agreement
    * Choose to install for //Just Me// (the recommended default choice)
    * Specify the //Destination Folder// selected above, e.g. ''C:\Utils\miniconda3_2024-03''
    * Use the pre-selected //Advanced Installation Options//, i.e
      * //Create start menu shortcutes//
      * //Register Miniconda3 as my default Python 3.12// <= Useful?
    * There is no need to bookmark the web pages suggested by the installer at the end, since this web page gives all the required information

  * When the installer has finished running, you should have:
    * A new ''C:\Utils\miniconda3_2024-03'' folder
      * This folder size will be ~650 Mb and will keep on growing (up to several Gb) as you add more Python environments and packages
        * => **Be sure to have enough space on this disk!**
    * Two new shortcuts in the Windows ''Start''=>''M''=>''Miniconda3'' menu:
      * ''Anaconda Prompt''
      * ''Anaconda Powershell Prompt''
        * **We will now use this //Anaconda Powershell// shortcut each time we need to use Miniconda3 or Python**
      * It is **highly recommended to [[other:win10apps#windows_terminal|install Windows Terminal and make it the default terminal application]]**, so that the ''conda'' prompts will open in a modern (and highly configurable) terminal

  * Remove the installer<code>PS C:\Scratch\your_login> del .\miniconda.exe</code>

  * Open an //Anaconda PowerShell Prompt//, and type a few commands to make sure that ''conda'' is working
    * ''Start''=>''M''=>''Miniconda3'' => ''Anaconda Powershell Prompt''
    * <code>(base) PS C:\Users\your_login> conda env list
base                  *  C:\Utils\miniconda3_2024-03

(base) PS C:\Users\your_login> conda list
# packages in environment at C:\Utils\miniconda3_2024-03:
# Name                    Version                   Build  Channel
[...]
conda                     24.3.0          py312haa95532_0
[...]
python                    3.12.2               h1d929f7_0
[...]

(base) PS C:\Users\your_login> conda deactivate

PS C:\Users\your_login> conda env list
base                     C:\Utils\miniconda3_2024-03</code>

  * **Next**: be sure to <wrap hi>follow the steps in the [[other:uvcdat:cdat_conda:miniconda3_install#fine-tuning_conda_to_use_only_conda-forge|Fine-tuning conda to use (only) conda-forge]], and **all** the other Miniconda3 related instructions</wrap>
    * ''conda'' commands will work the same way on all Operating Systems (Windows and Linux-like)
    * The ''.condarc'' is located in your Windows //home// folder\\ => ''C:\Users\your_login\.condarc'' (or ''$HOME\.condarc'')

==== Miniconda3 on a Linux-like computer ====


<WRAP center round tip 60%>
By **//Linux-like//**, we mean:
  * A **Linux** desktop/laptop
    * The installation will be used by **one** person
  * A **Linux** server
    * The installation will be used by **one or more** persons
  * A **windows computer with [[other:win10wsl|WSL+Ubuntu installed]]**
    * You should use the [[#miniconda3_on_a_windows_computer|Windows installer]], if all the packages you need are available for Windows
    * The installation will be used by **one** person
  * A **Mac** where you can use Linux in a terminal
    * The installation will be used by **one** person
</WRAP>



=== Downloading the Miniconda3 installer on Linux ===

  * **Important**: <wrap hi>do not install your own Python environment(s) on shared computers before you have made sure that the packages you need are not already installed, or cannot be easily installed in an existing environment (by the person who maintains this environment)!</wrap>
    * Read again the [[#before_installing_python_on_a_shared_computer|Before installing Python on a shared computer]] section

  * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Linux installer]]
    * For the Linux computers we commonly use, we need the //Miniconda3 Linux 64-bit// link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
      * Note: Mac users should use the latest [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|macOS installer]] appropriate for their computer
    * The installer is based on **Python 3.12.2 and Conda 24.3.0 as of April 2024**


  * Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]])
    * e.g. Linux on the **LSCE** servers:\\ ''$ cd /home/scratch01/$USER''
    * e.g. Linux on the IPSL **spiritx** cluster:<code>$ mkdir /homedata/$USER/Scratch
$ cd /homedata/$USER/Scratch</code>
    * e.g. Linux on a **personal PC/Mac desktop/laptop**: use a local scratch/temporary directory with //enough// space

  * Use ''wget'' to download the latest installer (138 Mb as of 22 Mar 2024):\\ <code>$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ ls -lh Miniconda3-latest-Linux-x86_64.sh
[...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh</code>

=== Using the Miniconda3 installer on Linux ===

We assume below that we are in the [[#downloading_the_miniconda3_installer_on_linux|directory where we have downloaded the installer]]

  * **Determine the directory where you will install miniconda3 and all the Python environments** based on this version of miniconda3, but **do not create the directory** (the installation script will do it)\\ \\ **We will later refer to this location as ''/path_to_miniconda3/''**\\ \\ 
    * Remember that **you will need a few Gb of available space**
    * **If you want to share this installation with other users**, the directory has to be readable by these users
    * Always **avoid installing in your //home// directory**, if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python installation...
    * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations
      * e.g. //JYP// shared Linux installation at **LSCE**: ''/home/share/unix_files/cdat/miniconda3_2024-03''
      * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03''

  * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh''
    * Review the license (type ''<SPACE>'' several times...) and accept it
    * When asked for the installation directory, specify the directory location chosen in the previous step, and **not the default directory**
      * e.g. specify\\ ''/path_to_miniconda3/''\\ instead of the default\\ ''$HOME/miniconda3''
    * **Answer ''no'' to the question ''Do you wish to update your shell profile to automatically initialize conda?''**.\\ Otherwise the installation will make changes to your shell configuration files
      * If you forget to answer ''no'', you can apparently:
        * later remove the changes to your shell configuration files by typing: ''conda init %%--%%reverse $SHELL''
        * or at least disable the automatic activation of the environment (that might have side effect) by typing: ''conda config %%--%%set auto_activate_base false''
    * The resulting initial ''miniconda3'' directory size is 647 Mb (as of March 2024).\\ Remember that **this directory size will keep on growing**!\\ <code>$ du -sh miniconda3_2024-03
647M    miniconda3_2024-03

$ du -sh miniconda3_2024-03/*
47M     miniconda3_2024-03/bin
16K     miniconda3_2024-03/cmake
8.0K    miniconda3_2024-03/compiler_compat
32M     miniconda3_2024-03/_conda
8.0K    miniconda3_2024-03/condabin
976K    miniconda3_2024-03/conda-meta
4.0K    miniconda3_2024-03/envs
28K     miniconda3_2024-03/etc
18M     miniconda3_2024-03/include
317M    miniconda3_2024-03/lib
92K     miniconda3_2024-03/LICENSE.txt
1.1M    miniconda3_2024-03/man
232M    miniconda3_2024-03/pkgs
396K    miniconda3_2024-03/sbin
1.5M    miniconda3_2024-03/share
12K     miniconda3_2024-03/shell
8.0K    miniconda3_2024-03/ssl
8.0K    miniconda3_2024-03/x86_64-conda_cos7-linux-gnu
8.0K    miniconda3_2024-03/x86_64-conda-linux-gnu
</code>

  * Initialize the newly installed conda environment. This will initialize the environment **only in the current terminal**. When you have checked that things work properly, you can later make the initialization easier by following the steps in [[#initializing_conda_on_a_linux-like_computer|Initializing conda on a Linux-like computer]]
    * You have to execute a configuration script, and then activate the environment
      - Run the **init** script: ''source /some_path/conda_init_script''
      - **Activate** the base environment (these two commands have the same result):
        * ''conda activate''
        * or ''conda activate base''
    * **bash shell**: ''source /path_to_miniconda3/etc/profile.d/conda.sh''
      * e.g. (spiritx): ''source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh''
    * **tcsh shell**: ''source /path_to_miniconda3/etc/profile.d/conda.csh''
      * e.g. (LSCE): ''source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh''

  * After initializing the environment, check if you can access the ''conda'' command, and use it to initialize the //base// environment
    * e.g. on **spiritx1**: <code>$ which conda
/homedata/jypmce/miniconda3_2024-03/condabin/conda
$ which python
/usr/bin/python

$ conda activate base

(base) $ which conda
/homedata/jypmce/miniconda3_2024-03/bin/conda
(base) $ which python
/homedata/jypmce/miniconda3_2024-03/bin/python

(base) $ conda deactivate

$ which conda
/homedata/jypmce/miniconda3_2024-03/condabin/conda
$ which python
/usr/bin/python
</code>

  * Remove the //Miniconda3// installer: ''$ rm Miniconda3-latest-Linux-x86_64.sh''

===== Fine-tuning conda to use (only) conda-forge =====

<note important>Be sure to **also follow these steps on a Windows installation** !</note>

==== Why is it vital to use only conda-forge ? ====

  * ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies.

  * ''conda'' works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver.
    * Use ''conda %%--%%version'' version to determine which version you are using<code>$ conda --version
conda 24.5.0
</code>

  * We are power users, and **we intend to create complex Python environments** combining lots (and lots) of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both //conda// and //conda-forge//.
    * => **The sections below configure ''conda'' to always use //conda-forge//**, and completely update //Miniconda3// itself with packages coming only from //conda-forge//

  * Very unlikely situation: if you use a recent version of ''conda'' (''>= 23.10''), and you have followed all the steps below, in order to **use only //conda-forge// packages**, and there are still some package installation problems, try to [[#adding_mamba_optional|install and use mamba]], as a drop-in replacement of ''conda''
    * Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments:
      * [[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]]
      * [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]

==== Changing the .condarc file ====

The following steps will **make sure that we only get packages from //conda-forge//**  (same thing as using the ''-c conda-forge'' option) **by default**, unless the requested packages really do not exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]]

<code>$ cat ~/.condarc  # On Windows use: cat $HOME\.condarc
cat: /home/jypmce/.condarc: No such file or directory

$ conda config --get channels

$ conda config --prepend channels conda-forge

$ conda config --set channel_priority strict

$ conda config --get channels
--add channels 'defaults'   # lowest priority
--add channels 'conda-forge'   # highest priority

$ cat ~/.condarc  # On Windows use: cat $HOME\.condarc
channels:
  - conda-forge
  - defaults
channel_priority: strict
</code>

==== Updating Miniconda3 at least once! ====

Before you go further, <wrap hi>do not forget to [[#the_first_time_we_update_miniconda3|update conda at least once]] in order to take all the changes above into account!</wrap>
===== Adding mamba (optional) =====

<WRAP center round alert 60%>
  * Most people can (and should!) probably skip the ''mamba'' installation

  * Only install ''mamba'' if you are sure that you will need some features that are not already provided by ''conda'' (e.g. ''mamba repoquery'')
</WRAP>

[[https://mamba.readthedocs.io/|mamba]] is a fast, robust, and cross-platform package manager [...] fully compatible with conda packages, and supports most of conda’s commands

==== Installing mamba ====

  * ''mamba'' is used to manage environments, and **has to be installed in the //base// environment** (i.e. in the same environment as ''conda'')!
  * If :
    - You are sure that [[https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html#existing-conda-install-not-recommended|you need ''mamba'', even if the developers advise not to mix conda and mamba]]
    - You have configured ''conda'' to [[#fine-tuning_conda_to_use_only_conda-forge|use conda-forge by default]]
    - Then (and only then!) you can install ''mamba'':\\ => ''conda install -n base mamba''

==== Using mamba ====

  * The useful mamba commands (we need) are listed in the [[#useful_conda_commands|Useful conda commands section]]
    * We mostly need ''[[https://mamba.readthedocs.io/en/latest/user_guide/mamba.html#repoquery|mamba repoquery]]''
  * [[https://mamba.readthedocs.io/en/latest/user_guide/mamba.html|Mamba User Guide]]


===== Keeping (Mini)conda up-to-date =====

==== The update command ====

  * Just type ''conda update -n base %%--%%all'', (review) and accept the changes
    * Things should be OK if no package //downgrades// are requested

  * The ''base'' (//Miniconda3// itself) and the other Python environments installed with the ''conda'' executable provided by the ''base'' environment are all independent, and can be updated independently

  * It is up to you to decide if and when you want to update environments...
    * ...but <wrap hi>you have to **update the ''base'' environment at least once** in order to replace all of its packages by their equivalent ''conda-forge'' packages
</wrap>

  * You can later update again the full //Miniconda3//, or just the ''conda'' command.\\ This should not make any changes to the other Python environment(s) installed with //Miniconda3//
    * <code>$ conda update -n base --all
[...]
# All requested packages already installed.

(base) $ conda update -n base conda
[...]
# All requested packages already installed.
</code>

  * Note: during our initial installation, the ''miniconda3'' directory size grew from 763 Mb to 1.6 Gb, and we have not installed any custom Python environment yet! This is the reason why you should install //Miniconda3// on a (preferably non backed up) disk where you have //enough// free space
    * <code>$ du -sh /homedata/jypmce/miniconda3_2024-03
736M    /homedata/jypmce/miniconda3_2024-03
[... update]
$ du -sh /homedata/jypmce/miniconda3_2024-03
1.6G    /homedata/jypmce/miniconda3_2024-03
</code>

==== The first time we update Miniconda3... ====

We have to **completely update //Miniconda3// at least once after installing //Miniconda3//** and [[#fine-tuning_conda_to_use_only_conda-forge|making conda-forge the highest priority channel]]

  * Check the current version of ''conda''
    * <code>$ conda --version
conda 24.1.2</code>

  * Completely update the //Miniconda3// installation
    * <code>$ conda update -n base --all
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /homedata/jypmce/miniconda3_2024-03


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-24.3.0               |  py312h7900ff3_0         1.1 MB  conda-forge
    conda-libmamba-solver-24.1.0|     pyhd8ed1ab_0          40 KB  conda-forge
    python-3.12.2              |hab00c5b_0_cpython        30.8 MB  conda-forge
[...]
The following NEW packages will be INSTALLED:
[...]
The following packages will be UPDATED:
[...]
  conda              pkgs/main::conda-24.1.2-py312h06a4308~ --> conda-forge::conda-24.3.0-py312h7900ff3_0
[...]
The following packages will be SUPERSEDED by a higher-priority channel:
[...]
Proceed ([y]/n)? y
[...]
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
</code>
  * Check the updated version of ''conda'' and ''python''. Note that **(almost) all** the packages should now specify that they are provided by ''conda-forge''
    * <code>$ conda list -n base
[...]
conda                     24.3.0          py312h7900ff3_0    conda-forge
conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    conda-forge
[...]
python                    3.12.2          hab00c5b_0_cpython    conda-forge
[...]
</code>
    * You can use the following (on Linux) to determine if some packages were **not** available in ''conda-forge''
      * <code>$ conda list -n base | grep -v conda-forge
# Name                    Version                   Build  Channel
libedit                   3.1.20230828         h5eee18b_0
libffi                    3.4.4                h6a678d5_0
xz                        5.4.5                h5eee18b_0
</code>
      * It should not be a major problem if a few packages are not provided by ''conda-forge''.\\ It only means that some packages provided by the ''anaconda'' //channel// were more recent than the same packages provided by the ''conda-forge'' //channel// when the packages were updated
        * ''libedit'' example: [[https://anaconda.org/conda-forge/libedit|version 3.1.20191231 @ conda-forge]] and [[https://anaconda.org/anaconda/libedit|version 3.1.20230828 @ anaconda]]
==== Cleaning up Miniconda3... ====

<WRAP center round important 60%>
It seems **safe** to reclaim some disk space by cleaning up the initial (//base//) installation, **when no other (shared) environments have been installed yet**\\ \\ Use cleaning with caution after installing other environments (besides ''base''), especially environments **shared** by several users. But it should not break your installation
</WRAP>

  * ''conda clean %%--%%all''
    * [[https://docs.conda.io/projects/conda/en/latest/commands/clean.html|Documentation]]
    * There is **no ''-n'' option to specify an environment name**, because ''clean'' will //Remove **unused** packages and caches// shared by **all** the environments
      * The ''clean'' command will mostly deal with the content of the ''/path_to_miniconda/pkgs/'' directory
    * Example:<code>(base) $ conda clean --all
Will remove 154 (180.9 MB) tarball(s).
Proceed ([y]/n)? y

Will remove 1 index cache(s).
Proceed ([y]/n)? y

Will remove 74 (378.0 MB) package(s).
Proceed ([y]/n)? y

There are no tempfile(s) to remove.
There are no logfile(s) to remove.

(base) $ du -sh /homedata/jypmce/miniconda3_2024-03
457M    /homedata/jypmce/miniconda3_2024-03
</code>

  * It //should// be safe to use ''conda clean %%--%%all'' even in a multi-user installation with several environments, **if you don't use the ''-f'' option**
    * The ''-f''/''%%--%%force-pkgs-dirs'' documentation specifies //WARNING: This will break environments with packages installed using symlinks back to the package cache.//
      * ...but Linux installations probably use [[https://www.redhat.com/sysadmin/linking-linux-explained|hard links]] rather than soft/symbolic links
      * ...and you will probably never need this option anyway

  * Remember that **your Python installation will keep on growing and growing and growing, and you will need to do some cleaning at some point**



===== Initializing conda in terminals =====

==== Initializing conda on a Windows computer ====

  * Open a ''PowerShell'' terminal with: ''Start''=> ''M'' => ''Miniconda3'' => ''Anaconda Powershell Prompt''
  * The ''base'' environment will be activated by default
    * If you need another environment, just type: ''conda activate name_of_the_environment_you_want''

**TODO** FIXME

  * Find out how to use ''menuinst'' in order to directly initialize an existing environment (different from ''base'') from the ''Start'' menu and from a ''Windows Terminal''
    * [[https://docs.conda.io/projects/conda-build/en/latest/resources/add-win-start-menu-items.html|Adding Windows Start menu items]]
    * [[https://conda.github.io/menuinst/|menuinst documentation]]
    * [[https://github.com/conda/menuinst/issues/196|Add Windows Terminal profile option]]

==== Initializing conda on a Linux-like computer ====

When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between existing environments. This can be **configured in your //shell// configuration files** (''.bashrc'', ''.cshrc'', ...)

=== conda in a single-user Linux-like installation ===

You were asked the following question when installing //Miniconda3//: //Do you wish to update your shell profile to automatically initialize conda? [yes|no]//

  * **If you answered //yes//**, the installer probably added some very complicated lines to your shell configuration files, and ''conda'' and the newly installed ''python'' are probably directly available when you open a new terminal.\\ \\ **Warning!** The automatic activation might have **subtle side effects**. You can:
    * **Remove the changes** to your shell configuration files by typing: ''conda init %%--%%reverse $SHELL''
    * Or at least **disable the automatic activation** of the environment by typing: ''conda config %%--%%set auto_activate_base false''

  * **If you answered //no//** ([[#using_the_miniconda3_installer_on_linux|as suggested]]), use a [[other:newppl:starting#text_editors|text editor]] to add extra lines to the appropriate shell configuration file\\ The changes will apply to all the **new** terminals you open\\ \\ 
    * **bash shell** user : add these lines to ''~/.bashrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.sh
alias pynit='conda activate base'</code>
      * e.g. (spiritx):<code>source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
alias pynit='conda activate base'</code>
    * **tcsh shell** user: add these lines to ''~/.cshrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.csh''
alias pynit 'conda activate base'</code>
      * e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh
alias pynit 'conda activate base'</code>
    * After changing your configuration file, just **type ''pynit'' in any (new) terminal you open, when you need to use Python**
    * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'')
    * You can also define other aliases, or use another name than ''pynit''!

=== conda in a multi-user Linux installation ===

<WRAP center round info 60%>
We need to be extra-careful in the case of **python environments maintained by a single user, but used by several users**, in order not to break anything
</WRAP>

We could use the same configuration file as in the //single-user case//, but **it is more efficient to have the users //source// an intermediate initialization file**. This intermediate init file will:

  * **//source// the conda initialization file** used in the single-user case
  * **Define some aliases** that will be useful to all the people using these environments:
    * ''pynit'' => initialize the //base// environment (or another default environment chosen by the central maintainer)
    * ''wp'' => ''which python''
    * ''wconda'' => ''which python''
    * more aliases, if required
  * If changes have to be made later, it is very easy to update this central initialization file accordingly, without asking the other users to make changes
    * Do not forget to keep a copy of the previous version of the init file!
    * Use a symbolic link (''ln -s'') to point from the unchanging central initialization file name to the latest initialization file.\\ conda@LSCE example in ''/home/share/unix_files/cdat/conda_init_files/'':<code>$ ls -l
-rw-r--r-- Jun 14 16:37 conda3_jyp_2024-03.csh
-rw-r--r-- Jun 14 17:48 conda3_jyp_2024-03.sh
lrwxrwxrwx Jun 17 15:56 conda3_jyp_latest.csh -> conda3_jyp_2024-03.csh
lrwxrwxrwx Jun 17 15:57 conda3_jyp_latest.sh -> conda3_jyp_2024-03.sh
</code>

  * **''bash'' users** should add to their ''~/.bashrc'' file something like\\ ''source  /some_path_accessible_by_all_users/shared_conda_init_file.sh''\\ \\ with a ''shared_conda_init_file.sh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp_2024-03.sh.txt |}}\\ \\ Examples:\\ \\
    * **LSCE**:
      * **Add to ''~/.bashrc''**: ''source /home/share/unix_files/cdat/conda_init_files/conda3_jyp_**latest**.sh''
      * Note: ''conda3_jyp_**latest**.sh'' is a link to the **latest** version of the init file,\\ currently ''/home/share/unix_files/cdat/conda_init_files/conda3_jyp_2024-03.sh''
    * **''spiritx''** and **''spirit''**:
      * We use the fact that file systems on ''spiritx'' are accessible in //read-only// mode on ''spirit''
        * The python environments are installed on **local disks of ''spiritx''**, but they can also be used on ''spirit'', even if the remote disks access will add some latency
      * **Add to ''~/.bashrc''**: ''source /homedata/jypmce/conda_init_files/conda3_jyp_spiritx_**latest**.sh''
      * Note: ''conda3_jyp_spiritx_**latest**.sh'' is a link to the **latest** version of the init file,\\ currently ''/homedata/jypmce/conda_init_files/conda3_jyp_spiritx_2024-03.sh''

  * **''tcsh'' users** should add to their ''~/.cshrc'' file something like\\ ''source  /some_path_accessible_by_all_users/shared_conda_init_file.csh''\\ \\ with a ''shared_conda_init_file.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp_2024-03.csh.txt |}}\\ \\ Examples:
    * **LSCE**:
      * **Add to ''~/.cshrc''**: ''source /home/share/unix_files/cdat/conda_init_files/conda3_jyp_**latest**.csh''
      * Note: ''conda3_jyp_**latest**.csh'' is a link to the **latest** version of the init file,\\ currently  ''/home/share/unix_files/cdat/conda_init_files/conda3_jyp_2024-03.csh''



===== Using (mini)conda to create a new environment =====

  * Once ''conda'' is installed (in the ''base'' environment provided by //Miniconda3//), configured and updated, you can use it to create a new environment with the packages that you (or other users, in a shared environment) need
    * Use the commands described in [[#useful_conda_commands|Useful conda commands]]

  * Detailed example: the ''xcdat'' environment
    * FIXME: update/rewrite the [[other:uvcdat:cdat_conda:cdat_2024_03]] page


===== Creating Windows menu shortcuts to start environments =====

FIXME => Move this section somewhere else?

  * Use ''conda install'' and what is described in [[https://github.com/conda/menuinst/issues/246|Easily creating shortcuts with the powershell_shortcut_miniconda and console_shortcut_miniconda packages, and can this be replicated with menuinst ?]]

  * Can theoretically be done with [[https://conda.github.io/menuinst/|menuinst]]
    * [[https://github.com/conda/menuinst|menuinst @ GH]]

  * FIXME Find a way to add Windows Terminal profiles
    * See ''menuinst''+[[https://github.com/conda/menuinst/issues/196|Add Windows Terminal profile option]]
===== conda resources =====


==== Web sites ====

  * [[https://docs.conda.io/projects/conda/en/stable/|Official Documentation site]]
    * [[https://docs.conda.io/projects/conda/en/stable/user-guide/cheatsheet.html|Cheat sheet]] (2 pages, for people in a hurry)
    * [[https://docs.conda.io/projects/conda/en/stable/commands/|Commands' reference]]
    * [[https://docs.conda.io/projects/conda-build/en/stable/resources/package-spec.html#conda-package-specification|Conda package specification]]
      * This helps understand (and [[https://github.com/conda/conda/issues/14026|use]]) the content of the ''/path_to_miniconda/pkgs'' directory

  * [[https://conda.org/blog/|conda community site]]:
    * [[https://conda.org/blog/|conda Blog]]: ''conda'' new features, etc...

  * [[https://github.com/conda/conda|conda @ GitHub]]

  * Package repositories
    * [[https://conda-forge.org/packages/|conda-forge]] <= recommended source of packages
    * [[https://anaconda.org/|default]]


==== Useful conda commands ====

This is just a subset of some of the commands, and a subset of their options! For a complete reference, use the [[https://docs.conda.io/projects/conda/en/stable/|official conda website]]

  * **Get help**
    * ''conda -h''
      * ''conda //command// %%--%%help'': help for a specific //command//

  * **Version and configuration** information
    * Version information: ''conda %%--%%version''
    * Detailed information: ''conda info''
    * Full configuration: ''conda config %%--%%show''
      * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation

  * **Available environments**
    * ''conda env list''
    * The currently active environment has a "*****" character on its line<code>(base) $ conda env list
# conda environments:
#
base                  *  /homedata/jypmce/miniconda3_2024-03</code>

  * **Activate** an existing environment (available in ''conda env list'') or **deactivate**, in order to go back to the default Python
    * ''conda activate existing_environment''
      * ''conda activate'' (without specifying an environment name) will **activate the ''base'' Miniconda3 environment**
    * Go back to the default Python available on the system
      * ''conda deactivate''
    * You can type ''which python'' to determine where is the current ''python'' executable you are using is\\ You should **always know which Python you are using!**
      * On Linux:<code>(base) $ which python
/homedata/jypmce/miniconda3_2024-03/bin/python

(base) $ conda deactivate

$ which python
/usr/bin/python</code>
      * On Windows:<code>(base) PS C:\> where.exe conda
C:\Utils\miniconda3_2024-03\Library\bin\conda.bat
C:\Utils\miniconda3_2024-03\Scripts\conda.exe
C:\Utils\miniconda3_2024-03\condabin\conda.bat

(base) PS C:\> where.exe python
C:\Utils\miniconda3_2024-03\python.exe
C:\Users\jypeter\AppData\Local\Microsoft\WindowsApps\python.exe

(base) PS C:\Users\jypeter> conda deactivate

PS C:\> where.exe conda
C:\Utils\miniconda3_2024-03\condabin\conda.bat

PS C:\> where.exe python
C:\Users\jypeter\AppData\Local\Microsoft\WindowsApps\python.exe</code>
    * Note: //before conda ''4.6''//, you had to type ''**source** activate existing_environment'' and ''**source** deactivate''

  * Get the **list of installed packages**, their version, and where they come from (e.g. ''conda-forge'', ''pip'')
    * All the packages: ''conda list [-n existing_environment]''
    * Some packages: ''conda list [-n existing_environment] full_or_partial_name'' <code>(base) $ conda list libmamba
# Name                    Version                   Build  Channel
conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    conda-forge
libmamba                  1.5.8                had39da4_0    conda-forge
libmambapy                1.5.8           py312hd9e9ff6_0    conda-forge

(base) $ conda list lib
_libgcc_mutex             0.1                 conda_forge    conda-forge
conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    conda-forge
[...]
urllib3                   2.2.2              pyhd8ed1ab_1    conda-forge
zlib                      1.3.1                h4ab18f5_1    conda-forge
</code>

  * **Create an environment** and install a few packages and their dependencies
    * ''conda create -n new_environment_name package1 package2 packageN''

  * **Delete an environment**
    * ''conda remove -n crap_environment %%--%%all''

  * **Find out if a package is available** on the //conda-forge// channel (or other channels)
    * Note: you should always install a package with ''conda'' instead of ''pip'' or ''mamba'', if it is available on a //conda channel//
    * ''conda search package_exact_name''\\ ''conda search '*package_partial_name*'''
      * Note: ''search'' with wildcards fails in a //tcsh// shell, as of April 2024 ([[https://github.com/conda/conda/issues/9771|CSH install/search fail with '*' variable in an argument]])
    * Search **results can be filtered** by using the [[#the_conda_matchspec_format|conda MatchSpec format]]
    * **Get very detailed information about a package** (versions, size, dependencies) with:\\ ''conda search package_exact_name %%--%%info''
    * FIXME: find an option to get only the latest version available for the current platform
      * Did not find a way to do that efficiently (June 20th 2024), even with ''mamba'' :-(

  * **Install and remove packages**\\ Add ''-n target_environment'' to make changes in another environment than the active one
    * ''conda install package1 package2 packageN''
      * ''conda'' should already be [[#fine-tuning_conda_to_use_only_conda-forge|configured to use conda-forge by default]] if you have read this page carefully
      * Add ''-c channel_url_or_alias'' to install from a specific //channel//
    * **Specific versions can be specified** by using the [[#the_conda_matchspec_format|conda MatchSpec format]]
    * ''conda remove package1 package2 packageN''
    * Matching ''pip'' commands (**only** if the package is not available in conda!)
      * ''pip install package''
      * ''pip uninstall package''

  * **Update a package** in the current, or a specific environment
    * ''conda update -n existing_environment installed_package''
    * Update only the ''conda'' package (not the full Miniconda3 //base// environment) in the ''base'' environment
      * ''conda update -n base conda''
      * Changes and new ''conda'' features are detailed in the [[https://conda.org/blog/|conda Blog]]
    * Matching ''pip'' command (**only** if the package is not available in conda!)
      * ''pip install %%--%%upgrade package''

  * **Completely update an environment** (i.e. update **all** the packages)
    * ''conda update -n existing_environment %%--%%all''
    * [[#the_first_time_we_update_miniconda3|'base' environment update example]]

  * **Determine the relationship between installed packages**
    * Packages relationships can be listed with ''[[#adding_mamba_optional|mamba]] repoquery''
      * Note: ''mamba repoquery'' currently [[https://github.com/mamba-org/mamba/issues/3334|only works in the currently active environment]]
    * It can be useful to determine:
      * the packages that are required in order to use a given package => ''mamba repoquery **depends** package''
        * Without ''mamba repoquery depends'', you would have to follow the steps in [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-pkgs.html#listing-package-dependencies|Listing package dependencies]] :-(
      * the packages that need a given package => ''mamba repoquery **whoneeds** package''
    * Example: the ''[[https://github.com/tqdm/tqdm|tqdm]]'' package
      * ''conda'' needs ''tqdm'' to display package download and installation progress, and the base dependencies of ''tqdm'' are ''[[https://github.com/tartley/colorama|colorama]]'', and of course ''python''
      * Detailed use:<code> >mamba repoquery whoneeds --pretty tqdm
tqdm
  └─ conda[24.5.0]

 >mamba repoquery depends --pretty tqdm
tqdm[4.66.4]
  ├─ colorama[0.4.6]
  └─ python[3.12.3]
</code>
      * The packages that ''tqdm'' needs also need packages, and so on...<code> >mamba repoquery depends --tree tqdm
tqdm[4.66.4]
  ├─ colorama[0.4.6]
  │  └─ python[3.12.3]
  │     ├─ bzip2[1.0.8]
  │     │  └─ libgcc-ng[14.1.0]
  │     │     ├─ _libgcc_mutex[0.1]
  │     │     └─ _openmp_mutex[4.5]
  │     │        ├─ _libgcc_mutex already visited
  │     │        └─ libgomp[14.1.0]
  │     │           └─ _libgcc_mutex already visited
  │     ├─ libgcc-ng already visited
  │     ├─ ld_impl_linux-64[2.40]
  │     ├─ libexpat[2.6.2]
  │     │  └─ libgcc-ng already visited
  [...]
</code>

  * **Checking the //health// of an environment** with ''[[https://docs.conda.io/projects/conda/en/stable/commands/doctor.html|conda doctor]]''
    * Some details in [[https://conda.org/blog/2023-06-01-conda-doctor/|Detecting conda environment inconsistencies easily]]
      * ''conda doctor'' does not do much yet, and not necessarily in a useful way, but this will hopefully improve over time!
        * [[https://github.com/conda/conda/issues/14017|'conda doctor' output should be improved]]
        * [[https://github.com/conda/conda/issues/13574#issuecomment-2214291304|What exactly are altered files?]]
    * ''conda doctor -n base''
    * Example:<code> >conda doctor -n base
Environment Health Report for: /home/share/unix_files/cdat/miniconda3_2024-03

Altered Files:

python-3.12.3-hab00c5b_0_cpython: 1

Environment listed in environments.txt file: ✅

Missing Files:

✅ There are no packages with missing files.
</code>

  * **Cleaning...**
    * ''conda'' does not automatically clean the content of the ''/path_to_miniconda/pkgs'' directory and the size of ''/path_to_miniconda/'' will keep on growing, as you add new environments, or update existing ones
    * All the details are in the [[#cleaning_up_miniconda3|Cleaning up Miniconda3...]] section


==== conda, pip and virtualenv ====

  * Read the [[https://docs.conda.io/projects/conda/en/stable/commands/#conda-vs-pip-vs-virtualenv-commands|Conda vs. pip vs. virtualenv commands]] section

  * Read the [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-environments.html#using-pip-in-an-environment|Using pip in a conda environment]]
    * //Issues may arise when using pip and conda together//...
==== Installation history ====

You can get the current list of installed packages with ''conda list -n existing_environment'', but it may also be useful to know in which order the packages were installed:

  * ''conda list -n existing_environment %%--%%revisions'': display **exactly** what was installed and when (including the updates, and automatic dependencies installation)
    * You can theoretically roll back to a previous ''RR'' //revision// number with: ''conda install -n existing_environment %%--%%revision RR''
  * There is more information in the ''/path_to_miniconda/conda-meta/history'' installation log files.
    * ''base'' history file: ''/path_to_miniconda/conda-meta/history''
    * ''existing_environment'' history file: ''/path_to_miniconda/envs/existing_environment/conda-meta/history''
    * e.g. installation commands for the ''base'' environment:\\ <code> $ egrep -e 'cmd:' -e '==>' /homedata/jypmce/miniconda3_2024-03/conda-meta/history

==> 2024-02-23 18:39:23 <==
# cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW
==> 2024-03-27 16:10:01 <==
# cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all
==> 2024-03-28 11:38:08 <==
[...]
==> 2024-04-24 17:47:32 <==
# cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all</code>
    * You can also, use a script and the ''conda'' module available **in the ''base'' environment**: [[https://github.com/conda/conda/issues/4545#issuecomment-469984684|example]]

==== The conda MatchSpec format ====

  * The ''conda search'' command uses the //MatchSpec// format

  * ''conda install'' and ''conda search'' //probably// use the same syntax

  * The //MatchSpec// format is poorly documented...
    * Fortunately, you will usually only specify the package name, and want to install the latest version of a package

  * Using ''=='' (//exact version//) is **different** from ''='' (//"fuzzy" value//), which can be confusing
    * <code>$ conda search zlib==1.2
No match found for: zlib==1.2. Search: *zlib*==1.2
PackagesNotFoundError: The following packages are not available from current channels:
  - zlib==1.2

$ conda search zlib=1.2
# Name                       Version           Build  Channel
zlib                           1.2.8               0  conda-forge
zlib                           1.2.8               1  conda-forge
[...]
zlib                          1.2.13      h5eee18b_1  pkgs/main
zlib                          1.2.13      hd590300_5  conda-forge

$ conda search zlib==1.2.12
zlib                          1.2.12      h166bdaf_0  conda-forge
[...]
zlib                          1.2.12      h7f8727e_2  pkgs/main

</code>

  * Documentation:
    * [[https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#package-match-specifications|conda Package match specifications]]
    * [[https://conda.io/projects/conda/en/latest/dev-guide/api/conda/models/match_spec/index.html#conda.models.match_spec.MatchSpec|Documentation in the MatchSpec class]]
===== Uninstalling Miniconda3 =====

==== Windows computer ====

Miniconda3 can be removed like a regular Windows application

  * Go to the //Apps & features// panel
    * ''Start'' => ''Settings''  => ''Apps'' => ''Apps & features''
  * Select ''Miniconda3 py3xxxxx''
  * Click on ''Uninstall'' and follow the instructions displayed by the uninstaller window
    * This will remove:
      * The (very big) directory that you specified when you installed //Miniconda3//
      * The ''conda'' shortcuts from the ''Start Menu''
    * This will **not** remove your local configuration file ''C:\Users\your_login\.condarc''

==== Linux-like computer ====

FIXME


===== Improving conda and related tools =====

There is always room for improvement, and this is is some kind of //TODO// section

==== conda and mamba ideas ====

  * [[https://github.com/conda/conda/issues/14026|Add a conda command to find the package owning a file]]
    * The link also shows how to do it with standard Linux tools (find, grep), based on the [[https://docs.conda.io/projects/conda-build/en/stable/resources/package-spec.html#conda-package-specification|Conda package specification]] page
    * [[https://github.com/kelvinou01/conda-which|conda-which]] seems to work (see [[https://github.com/conda/conda/issues/14026#issuecomment-2268821608|test / example]]) !

/* standard page footer */

\\ \\ \\ 
----
[ [[pmip3:|PMIP3 Wiki Home]] ] -
[ [[pmip3:wiki_help|Help!]] ] -
[ [[wiki:syntax|Wiki syntax]] ]