This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:uvcdat:cdat_conda:miniconda3_install [2024/04/17 15:42] jypeter [Installing miniconda3 on a Linux-like computer] |
other:uvcdat:cdat_conda:miniconda3_install [2024/06/14 15:14] jypeter [Initializing conda on a Linux-like computer] Added example and actual init files |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== JYP steps for installing Miniconda3 ====== | ====== JYP steps for installing Miniconda3 ====== | ||
+ | ===== Before installing Python on a shared computer ===== | ||
+ | |||
+ | ==== Desperately seeking a specific package ==== | ||
+ | |||
+ | Check if the Python package you are [[https://www.imdb.com/title/tt0089017/|desperately seeking]] is not already installed, before installing your own Python **on shared interactive Linux servers, or on batch computing clusters** !! | ||
+ | |||
+ | * Ask other users who know the local environment | ||
+ | |||
+ | * Read the servers/clusters documentation: [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting|Python at LSCE by JYP]], [[https://intranet.lsce.ipsl.fr/informatique/index.php|LSCE]], [[https://documentations.ipsl.fr/spirit/|ESPRI Computing and Data Center]], ... | ||
+ | |||
+ | * Use ''[[other:newppl:starting#using_module_to_access_optional_programs|module avail [mod_name]]]'' to find out the Python versions available, and then ''module load'' and ''conda list'' to get more information on the available Python packages | ||
+ | * e.g. on ''spiritx'' ([[https://documentations.ipsl.fr/spirit/common/module_command.html|modules on spirit]])): <code>$ module avail python | ||
+ | python/meso-3.8 python/meso-3.9 python/meso-3.10 python/meso-3.11 | ||
+ | |||
+ | $ module avail anaconda | ||
+ | anaconda-meso/2022.10 anaconda3-py/2020.11 | ||
+ | anaconda-meso/2023.09-0 anaconda3-py/2021.11 | ||
+ | |||
+ | $ module avail pangeo | ||
+ | pangeo-meso/2023.04.15 pangeo-meso/2024.01.22 | ||
+ | |||
+ | $ module load pangeo-meso/2024.01.22 | ||
+ | (pangeo-meso-2024.01.22) $ which python | ||
+ | /net/nfs/tools/u20/Python/miniconda3_py311_23.11.0-2/envs/pangeo-meso-2024.01.22/bin/python | ||
+ | |||
+ | $ conda list | grep cartopy | ||
+ | cartopy 0.22.0 py311h320fe9a_1 conda-forge | ||
+ | </code> | ||
+ | |||
+ | * Lots of Python packages do very similar things. Maybe you can use another similar package that is already installed... | ||
+ | |||
+ | * If the packages you need are not available, it may be faster and easier to ask the local system administrators if they can install the packages, rather than doing everything yourself | ||
+ | |||
+ | ==== Using an unlisted Python distribution on a jupyter server ==== | ||
+ | |||
+ | If there is a //jupyter notebook server// connected to your system, and a Python environment providing all the packages you need, but that is not directly available on the notebook server, follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local server | ||
===== Why should you use Miniconda3 ? ===== | ===== Why should you use Miniconda3 ? ===== | ||
- | * **//Miniconda3// is a minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command available on your computer, and you can then use this ''conda'' executable to select an existing distribution, or create new and independent Python distributions. | + | * **//Miniconda3// is a minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command/executable available on your computer, and you can then use this ''conda'' executable to select an existing distribution, or create new and independent Python distributions. |
- | * A **conda //environment//** is basically (the short name of the directory) where you install the Python that you will use (independently of //Miniconda3// itself). You have to ''activate'' this environment in order to use it\\ A **python //distribution//** is the collection of packages (and their dependencies) you have chosen to install together in a given environment. | + | * A **conda //environment//** is basically (the short name of the directory) where you install the Python that you will use (independently of //Miniconda3// itself). You have to ''activate'' this environment in order to use it\\ \\ A **Python //distribution//** is the collection of packages (and their dependencies) you have chosen to install together in a given environment. |
- | * Example: initializing an environment named ''cdatm_py3'' (assuming it exists)<code>$ source /path_to_miniconda/etc/profile.d/conda.sh | + | * Example: initializing an environment named ''cdatm_py3'' (assuming it exists) in a //bash// shell <code>$ source /path_to_miniconda/etc/profile.d/conda.sh |
$ which conda | $ which conda | ||
Line 27: | Line 63: | ||
</code> | </code> | ||
- | * You **don't need to be (and you should not be) //root//** when you install //Miniconda3//. You just **need //enough// disk space** on a disk where you have write access | + | * You **don't need to be (and you should not be !!) //root//** when you install //Miniconda3//. You just **need //enough// disk space** on a disk where you have write access |
- | * **WARNING**: by default, //Miniconda3// will install itself and subsequent Python environments in a sub-directory of your home directory (''~/miniconda3'' on Linux). **Do not use the default installation location**, if you want to avoid disk space related problems (//disk full//, //quota exceeded//, ...) | + | * **WARNING**: by default, //Miniconda3// will install itself and subsequent Python environments in a sub-directory of your home directory (''~/miniconda3'' on Linux). **Do not use the default installation location**, if you want to avoid disk space related problems (//disk full//, //quota exceeded//, ...) during the inittial installation or later |
- | * **WARNING**: older versions of //Miniconda3// used a **hidden sub-directory** of your home directory (''~/.conda'' on Linux) for the installation | + | * **WARNING**: older versions of //Miniconda3// used a **hidden sub-directory** of the home directory of the person installing //Miniconda3// (''~/.conda'' on Linux) |
* Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing | * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing | ||
- | * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because: | + | * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer, that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because: |
- | * it requires more disk space than //Miniconda3// during the initial installation, | + | * It requires more disk space than //Miniconda3// during the initial installation, |
- | * all the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**. | + | * All the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems** |
+ | |||
+ | |||
+ | ===== Downloading Miniconda3, and BASIC installation ===== | ||
+ | |||
+ | ==== Miniconda3 on a Windows computer ==== | ||
+ | |||
+ | Some of the steps below are adapted from the [[https://docs.anaconda.com/free/miniconda/#quick-command-line-install|Windows section of "Quick command line install"]] to install a //pure Windows// version of Miniconda3 | ||
+ | |||
+ | === Downloading the Miniconda3 installer on Windows === | ||
+ | |||
+ | * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Windows installer]] | ||
+ | * https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe | ||
+ | * The installer is based on **Python 3.12.2 and Conda 24.3.0 as of April 2024** | ||
+ | |||
+ | * Find some temporary disk space on Windows | ||
+ | * e.g. ''C:\Scratch\your_login'' or whatever you use as a [[other:win10config#local_scratch_folder|local Scratch folder]] | ||
+ | |||
+ | * Open a //terminal//, where you can type the required installation commands | ||
+ | * Windows //Powershell terminal//: ''Start'' => ''W'' => ''Windows PowerShell'' => ''Windows PowerShell'' | ||
+ | * [[other:win10apps#windows_terminal|PowerShell in a Windows Terminal]] | ||
+ | |||
+ | * Use ''curl'' to download the latest installer (78 Mb as of April 2024): <code>PS C: cd C:\Scratch\your_login | ||
+ | PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe | ||
+ | PS C:\Scratch\your_login> dir miniconda.exe | ||
+ | [...] | ||
+ | -a---- 19/04/2024 11:18 81274240 miniconda.exe</code> | ||
+ | |||
+ | === BASIC Miniconda3 installation on Windows === | ||
+ | |||
+ | * **Determine the folder where you will install Miniconda3 and ALL your future Python environments** based on this version of Miniconda3\\ **We will later refer to this installation location as ''/path_to_miniconda3/''** in the [[#miniconda3_on_a_linux-like_computer|Linux sections]] below | ||
+ | * Remember that **you will need a few Gb of available space** | ||
+ | * You should **avoid installing Miniconda3 in your Windows //home// folder** (''C:\Users\your_login'' or [[other:win10config#local_home_folder|<some_disk_different_from_c>:\Users\<your_login>]]), if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python environments... | ||
+ | * We add some date or version information at the end of the folder name, in order to differentiate it from other (older of future) installations | ||
+ | * Example installation folder: ''C:\Utils\miniconda3_2024-03'' (if we start installing in March 2024) | ||
+ | |||
+ | * Launch the Miniconda3 installer with<code>PS C:\Scratch\your_login> .\miniconda.exe</code> | ||
+ | |||
+ | * In the installer window: | ||
+ | * (Review and) accept the License Agreement | ||
+ | * Choose to install for //Just Me// (the recommended default choice) | ||
+ | * Specify the //Destination Folder// selected above, e.g. ''C:\Utils\miniconda3_2024-03'' | ||
+ | * Use the pre-selected //Advanced Installation Options//, i.e | ||
+ | * //Create start menu shortcutes// | ||
+ | * //Register Miniconda3 as my default Python 3.12// <= Useful? | ||
+ | |||
+ | * When the installer has finished running, you should have: | ||
+ | * A new ''C:\Utils\miniconda3_2024-03'' folder | ||
+ | * This folder size will be ~650 Mb and will keep on growing (up to several Gb) as you add more Python environments and packages | ||
+ | * => **Be sure to have enough space on this disk!** | ||
+ | * Two new shortcuts in the Windows ''Start''=>''M''=>''Miniconda3'' menu: | ||
+ | * ''Anaconda Prompt'' | ||
+ | * ''Anaconda Powershell Prompt'' | ||
+ | * **We will now use this //Anaconda PowerShell// shortcut each time we need to use Miniconda3 or Python** | ||
+ | |||
+ | * Remove the installer<code>PS C:\Scratch\your_login> del .\miniconda.exe</code> | ||
+ | |||
+ | * Open an //Anaconda PowerShell Prompt//, and type a few commands to make sure that ''conda'' is working | ||
+ | * ''Start''=>''M''=>''Miniconda3'' => ''Anaconda Powershell Prompt'' | ||
+ | * <code>(base) PS C:\Users\your_login> conda env list | ||
+ | base * C:\Utils\miniconda3_2024-03 | ||
+ | |||
+ | (base) PS C:\Users\your_login> conda list | ||
+ | # packages in environment at C:\Utils\miniconda3_2024-03: | ||
+ | # Name Version Build Channel | ||
+ | [...] | ||
+ | conda 24.3.0 py312haa95532_0 | ||
+ | [...] | ||
+ | python 3.12.2 h1d929f7_0 | ||
+ | [...] | ||
+ | |||
+ | (base) PS C:\Users\your_login> conda deactivate | ||
+ | |||
+ | PS C:\Users\your_login> conda env list | ||
+ | base C:\Utils\miniconda3_2024-03</code> | ||
+ | |||
+ | * **Next**: be sure to <wrap hi>follow the steps in the [[other:uvcdat:cdat_conda:miniconda3_install#fine-tuning_conda_to_use_only_conda-forge|Fine-tuning conda to use (only) conda-forge]], and **all** the other Miniconda3 related instructions</wrap> | ||
+ | * ''conda'' commands will work the same way on all Operating Systems (Windows and Linux-like) | ||
+ | * The ''.condarc'' is located in your Windows //home// folder\\ => ''C:\Users\your_login\.condarc'' (or ''$HOME\.condarc'') | ||
+ | |||
+ | ==== Miniconda3 on a Linux-like computer ==== | ||
- | ===== Installing miniconda3 on a Linux-like or Windows computer ===== | ||
<WRAP center round tip 60%> | <WRAP center round tip 60%> | ||
By **//Linux-like//**, we mean: | By **//Linux-like//**, we mean: | ||
- | * A native Linux computer (or server) | + | * A **Linux** desktop/laptop |
- | * A windows 10 computer with [[other:win10wsl|WSL+Ubuntu installed]] | + | * The installation will be used by **one** person |
- | * A Mac where you can use Linux in a terminal | + | * A **Linux** server |
+ | * The installation will be used by **one or more** persons | ||
+ | * A **windows computer with [[other:win10wsl|WSL+Ubuntu installed]]** | ||
+ | * You should use the [[#miniconda3_on_a_windows_computer|Windows installer]], if all the packages you need are available for Windows | ||
+ | * The installation will be used by **one** person | ||
+ | * A **Mac** where you can use Linux in a terminal | ||
+ | * The installation will be used by **one** person | ||
</WRAP> | </WRAP> | ||
- | <WRAP center round info 60%> | ||
- | **//Pure Windows//** Note: you can also follow the steps listed in the [[https://docs.anaconda.com/free/miniconda/#quick-command-line-install|Windows section of "Quick command line install"]] to install a //pure Windows// version of miniconda3, and then follow the steps listed in the next sections of this page.\\ We assume below that the installer is downloaded in the ''C:\Scratch\your_login'' directory, and we install Miniconda3 in a ''C:\Utils\miniconda3_2024-03'' directory. <code>PS C: cd C:\Scratch\your_login | ||
- | PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe | ||
- | PS C:\Scratch\your_login> .\miniconda.exe | ||
- | PS C:\Scratch\your_login> del .\miniconda.exe | ||
- | </code> | ||
- | </WRAP> | ||
+ | === Downloading the Miniconda3 installer on Linux === | ||
- | ==== Downloading the Miniconda3 installer ==== | + | * **Important**: <wrap hi>do not install your own Python environment(s) on shared computers before you have made sure that the packages you need are not already installed, or cannot be easily installed in an existing environment (by the person who maintains this environment)!</wrap> |
+ | * Read again the [[#before_installing_python_on_a_shared_computer|Before installing Python on a shared computer]] section | ||
* We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Linux installer]] | * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Linux installer]] | ||
* For the Linux computers we commonly use, we need the //Miniconda3 Linux 64-bit// link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh | * For the Linux computers we commonly use, we need the //Miniconda3 Linux 64-bit// link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh | ||
* Note: Mac users should use the latest [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|macOS installer]] appropriate for their computer | * Note: Mac users should use the latest [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|macOS installer]] appropriate for their computer | ||
- | * The installer is based on **Python 3.12 as of March 2024** | + | * The installer is based on **Python 3.12.2 and Conda 24.3.0 as of April 2024** |
* Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]]) | * Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]]) | ||
- | * e.g. Linux at **LSCE**:\\ ''$ cd /home/scratch01/$USER'' | + | * e.g. Linux on the **LSCE** servers:\\ ''$ cd /home/scratch01/$USER'' |
- | * e.g. Linux at **spiritx**:<code>$ mkdir /homedata/$USER/Scratch | + | * e.g. Linux on the IPSL **spiritx** cluster:<code>$ mkdir /homedata/$USER/Scratch |
$ cd /homedata/$USER/Scratch</code> | $ cd /homedata/$USER/Scratch</code> | ||
- | * e.g. Linux on a **personal PC/Mac desktop/laptop**: use a scratch/temporary directory with //enough// space | + | * e.g. Linux on a **personal PC/Mac desktop/laptop**: use a local scratch/temporary directory with //enough// space |
* Use ''wget'' to download the latest installer (138 Mb as of 22 Mar 2024):\\ <code>$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh | * Use ''wget'' to download the latest installer (138 Mb as of 22 Mar 2024):\\ <code>$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh | ||
Line 72: | Line 189: | ||
[...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh</code> | [...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh</code> | ||
- | ==== Using the Miniconda3 installer ==== | + | === Using the Miniconda3 installer on Linux === |
- | We assume below that we are in the [[#downloading_the_installer|directory where we have downloaded the installer]] | + | We assume below that we are in the [[#downloading_the_miniconda3_installer_on_linux|directory where we have downloaded the installer]] |
- | * **Determine the directory where you will install miniconda3 and all the Python environments** based on this version of miniconda3, but **do not create the directory** (the installation script will do it)\\ **We will later refer to this location as ''/path_to_miniconda3/''** | + | * **Determine the directory where you will install miniconda3 and all the Python environments** based on this version of miniconda3, but **do not create the directory** (the installation script will do it)\\ \\ **We will later refer to this location as ''/path_to_miniconda3/''**\\ \\ |
* Remember that **you will need a few Gb of available space** | * Remember that **you will need a few Gb of available space** | ||
* **If you want to share this installation with other users**, the directory has to be readable by these users | * **If you want to share this installation with other users**, the directory has to be readable by these users | ||
- | * At LSCE, **do not** install in a sub-directory of your home directory! | + | * Always **avoid installing in your //home// directory**, if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python installation... |
- | * On other servers and computers, you should also **avoid installing in your //home// directory**, if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python installation... | + | |
* We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations | * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations | ||
- | * e.g. Linux at **LSCE**: ''/home/share/unix_files/cdat/miniconda3_2024-03'' | + | * e.g. //JYP// shared Linux installation at **LSCE**: ''/home/share/unix_files/cdat/miniconda3_2024-03'' |
- | * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03'' | + | * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03'' |
- | * Linux (including WSL) on a **personal PC/Mac desktop/laptop**: find a directory with enough space (including space for adding more packages and their future updates) | + | |
- | * You may want to use a directory that you are not backing up, if you want to avoid having to save thousands of files... | + | |
* Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh'' | * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh'' | ||
Line 93: | Line 207: | ||
* **Answer ''no'' to the question ''Do you wish to update your shell profile to automatically initialize conda?''**.\\ Otherwise the installation will make changes to your shell configuration files | * **Answer ''no'' to the question ''Do you wish to update your shell profile to automatically initialize conda?''**.\\ Otherwise the installation will make changes to your shell configuration files | ||
* If you forget to answer ''no'', you can apparently: | * If you forget to answer ''no'', you can apparently: | ||
- | * remove the changes to your shell configuration files by typing later: ''conda init --reverse $SHELL'' | + | * later remove the changes to your shell configuration files by typing: ''conda init %%--%%reverse $SHELL'' |
* or at least disable the automatic activation of the environment (that might have side effect) by typing: ''conda config --set auto_activate_base false'' | * or at least disable the automatic activation of the environment (that might have side effect) by typing: ''conda config --set auto_activate_base false'' | ||
- | * The resulting ''miniconda3'' directory size is 647 Mb (as of March 2024)\\ <code>$ du -sh miniconda3_2024-03 | + | * The resulting initial ''miniconda3'' directory size is 647 Mb (as of March 2024).\\ Remember that **this directory size will keep on growing**!\\ <code>$ du -sh miniconda3_2024-03 |
647M miniconda3_2024-03 | 647M miniconda3_2024-03 | ||
Line 120: | Line 234: | ||
</code> | </code> | ||
- | * Initialize the newly installed conda environment (this will initialize the environment **only in the current terminal**): | + | * Initialize the newly installed conda environment. This will initialize the environment **only in the current terminal**. When you have checked that things work properly, you can later make the initialization easier by following the steps in [[#initializing_conda_on_a_linux-like_computer|Initializing conda on a Linux-like computer]] |
+ | * You have to execute a configuration script, and then activate the environment | ||
+ | - Run the **init** script: ''source /some_path/conda_init_script'' | ||
+ | - **Activate** the base environment (these two commands have the same result): | ||
+ | * ''conda activate'' | ||
+ | * or ''conda activate base'' | ||
* **bash shell**: ''source /path_to_miniconda3/etc/profile.d/conda.sh'' | * **bash shell**: ''source /path_to_miniconda3/etc/profile.d/conda.sh'' | ||
* e.g. (spiritx): ''source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh'' | * e.g. (spiritx): ''source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh'' | ||
Line 126: | Line 245: | ||
* e.g. (LSCE): ''source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh'' | * e.g. (LSCE): ''source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh'' | ||
- | * Check if you can use the ''conda'' command, and use it to initialize the //base// environment | + | * After initializing the environment, check if you can access the ''conda'' command, and use it to initialize the //base// environment |
* e.g. on **spiritx1**: <code>$ which conda | * e.g. on **spiritx1**: <code>$ which conda | ||
- | $ which python | ||
- | /usr/bin/python | ||
- | |||
- | $ source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh | ||
- | |||
- | $ which conda | ||
/homedata/jypmce/miniconda3_2024-03/condabin/conda | /homedata/jypmce/miniconda3_2024-03/condabin/conda | ||
$ which python | $ which python | ||
Line 154: | Line 267: | ||
* Remove the //Miniconda3// installer: ''$ rm Miniconda3-latest-Linux-x86_64.sh'' | * Remove the //Miniconda3// installer: ''$ rm Miniconda3-latest-Linux-x86_64.sh'' | ||
- | ==== Fine-tuning conda to use (only) conda-forge ==== | + | |
+ | ===== Fine-tuning conda to use (only) conda-forge ===== | ||
+ | |||
+ | <note important>Be sure to **also follow these steps on a Windows installation** !</note> | ||
+ | |||
+ | ==== Why is it vital to use only conda-forge ? ==== | ||
* ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies. | * ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies. | ||
* ''conda'' works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver. | * ''conda'' works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver. | ||
- | * If you have followed all the steps of this section in order to **use only //conda-forge// packages**, and there are still some problems with a recent version of ''conda'', try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of conda | + | * Use ''conda %%--%%version'' version to determine which version you are using<code>$ conda --version |
+ | conda 24.5.0 | ||
+ | </code> | ||
+ | |||
+ | * We are power users, and **we intend to create complex Python environments** combining lots (and lots) of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both //conda// and //conda-forge//. | ||
+ | * => **The sections below configure ''conda'' to always use //conda-forge//**, and completely update //Miniconda3// itself with packages coming only from //conda-forge// | ||
+ | |||
+ | * Very unlikely situation: if you use a recent version of ''conda'' (''>= 23.10''), and you have followed all the steps below, in order to **use only //conda-forge// packages**, and there are still some package installation problems, try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of ''conda'' | ||
* Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments: | * Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments: | ||
* [[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]] | * [[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]] | ||
* [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]] | * [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]] | ||
- | * We are power users, and **we intend to create complex Python environments** combining lots (and lots) of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge. | + | ==== Changing the .condarc file ==== |
- | * => **The solution is to configure conda to always use conda-forge**, and completely update //Miniconda3// itself with packages coming only from conda-forge | + | |
- | + | ||
- | === Changing the .condarc file === | + | |
- | The following steps will **make sure that we only get packages from //conda-forge//** (same thing as using the ''-c conda-forge'' option) by default unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] | + | The following steps will **make sure that we only get packages from //conda-forge//** (same thing as using the ''-c conda-forge'' option) **by default**, unless the requested packages really do not exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] |
- | <code>$ cat ~/.condarc | + | <code>$ cat ~/.condarc # On Windows use: cat $HOME\.condarc |
cat: /home/jypmce/.condarc: No such file or directory | cat: /home/jypmce/.condarc: No such file or directory | ||
Line 184: | Line 306: | ||
--add channels 'conda-forge' # highest priority | --add channels 'conda-forge' # highest priority | ||
- | $ cat ~/.condarc | + | $ cat ~/.condarc # On Windows use: cat $HOME\.condarc |
channels: | channels: | ||
- conda-forge | - conda-forge | ||
Line 191: | Line 313: | ||
</code> | </code> | ||
- | === Getting conda configuration information === | + | ==== Updating Miniconda3 at least once! ==== |
- | * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation | + | <wrap hi>Do not forget to [[#the_first_time_we_update_miniconda3|update conda at least once]] in order to take all the changes above into account!</wrap> |
- | * Basic information: ''conda info'' | + | ===== Keeping (Mini)conda up-to-date ===== |
- | * Full configuration: ''conda config %%--%%show'' | + | |
- | ==== Updating (Mini)conda ==== | + | ==== The update command ==== |
+ | |||
+ | * Just type ''conda update -n base %%--%%all'', (review) and accept the changes | ||
+ | * Things should be OK if no package //downgrades// are requested | ||
+ | |||
+ | * The ''base'' (//Miniconda3// itself) and the other Python environments installed with the ''conda'' executable provided by the ''base'' environment are all independent, and can be updated independently | ||
+ | |||
+ | * It is up to you to decide if and when you want to update environments... | ||
+ | * ...but <wrap hi>you have to **update the ''base'' environment at least once** in order to replace all of its packages by their equivalent ''conda-forge'' packages | ||
+ | </wrap> | ||
+ | ==== The first time we update Miniconda3... ==== | ||
We have to **completely update //Miniconda3// at least once after installing //Miniconda3//** and [[#fine-tuning_conda_to_use_only_conda-forge|making conda-forge the highest priority channel]] | We have to **completely update //Miniconda3// at least once after installing //Miniconda3//** and [[#fine-tuning_conda_to_use_only_conda-forge|making conda-forge the highest priority channel]] | ||
* Check the current version of ''conda'' | * Check the current version of ''conda'' | ||
- | * <code>$ conda list -n base | grep conda | + | * <code>$ conda --version |
- | # packages in environment at /path_to_miniconda: | + | conda 24.1.2</code> |
- | conda 24.1.2 py312h06a4308_0 | + | |
- | conda-libmamba-solver 23.12.0 pyhd3eb1b0_1 | + | |
- | [... and other packages with 'conda' in their name] | + | |
- | </code> | + | |
* Completely update the //Miniconda3// installation | * Completely update the //Miniconda3// installation | ||
Line 245: | Line 372: | ||
Executing transaction: done | Executing transaction: done | ||
</code> | </code> | ||
- | * Check the updated version of ''conda''. Note that **(almost) all** the packages should now specify that they are provided by ''conda-forge'' | + | * Check the updated version of ''conda'' and ''python''. Note that **(almost) all** the packages should now specify that they are provided by ''conda-forge'' |
* <code>$ conda list -n base | * <code>$ conda list -n base | ||
+ | [...] | ||
conda 24.3.0 py312h7900ff3_0 conda-forge | conda 24.3.0 py312h7900ff3_0 conda-forge | ||
conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge | conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge | ||
+ | [...] | ||
python 3.12.2 hab00c5b_0_cpython conda-forge | python 3.12.2 hab00c5b_0_cpython conda-forge | ||
+ | [...] | ||
</code> | </code> | ||
- | * You can use the following to determine if some packages are not provided by ''conda-forge'' | + | * You can use the following to determine if some packages were **not** available in ''conda-forge'' |
* <code>$ conda list -n base | grep -v conda-forge | * <code>$ conda list -n base | grep -v conda-forge | ||
# Name Version Build Channel | # Name Version Build Channel | ||
Line 298: | Line 428: | ||
===== Initializing conda in terminals ===== | ===== Initializing conda in terminals ===== | ||
- | When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between existing environments. This can be **configured in the //shell// configuration files** | + | ==== Initializing conda on a Windows computer ==== |
- | ==== Single-user installation ==== | + | * Open a ''PowerShell'' terminal with: ''Start''=> ''M'' => ''Miniconda3'' => ''Anaconda Powershell Prompt'' |
+ | * The ''base'' environment will be activated by default | ||
+ | * If you need another environment, just type: ''conda activate name_of_the_environment_you_want'' | ||
+ | |||
+ | **TODO** FIXME | ||
+ | |||
+ | * Find out how to use ''menuinst'' in order to directly initialize an existing environment (different from ''base'') from the ''Start'' menu and from a ''Windows Terminal'' | ||
+ | * [[https://docs.conda.io/projects/conda-build/en/latest/resources/add-win-start-menu-items.html|Adding Windows Start menu items]] | ||
+ | * [[https://conda.github.io/menuinst/|menuinst documentation]] | ||
+ | * [[https://github.com/conda/menuinst/issues/196|Add Windows Terminal profile option]] | ||
+ | |||
+ | ==== Initializing conda on a Linux-like computer ==== | ||
+ | |||
+ | When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between existing environments. This can be **configured in your //shell// configuration files** (''.bashrc'', ''.cshrc'', ...) | ||
+ | |||
+ | === conda in a single-user Linux-like installation === | ||
You were asked the following question when installing //Miniconda3//: //Do you wish to update your shell profile to automatically initialize conda? [yes|no]// | You were asked the following question when installing //Miniconda3//: //Do you wish to update your shell profile to automatically initialize conda? [yes|no]// | ||
- | * **If you answered //yes//**, the installer probably added some very complicated lines to your shell configuration files, and ''conda'' and the newly installer ''python'' are probably directly available when you open a new terminal. | + | * **If you answered //yes//**, the installer probably added some very complicated lines to your shell configuration files, and ''conda'' and the newly installer ''python'' are probably directly available when you open a new terminal.\\ \\ **Warning!** The automatic activation might have **subtle side effects**, and you can: |
+ | * **Remove the changes** to your shell configuration files by typing: ''conda init %%--%%reverse $SHELL'' | ||
+ | * Or at least **disable the automatic activation** of the environment by typing: ''conda config --set auto_activate_base false'' | ||
- | * **If you answered //no//** ([[#using_the_miniconda3_installer|as suggested]]), use a [[other:newppl:starting#text_editors|text editor]] to add extra lines to the appropriate shell configuration file | + | * **If you answered //no//** ([[#using_the_miniconda3_installer_on_linux|as suggested]]), use a [[other:newppl:starting#text_editors|text editor]] to add extra lines to the appropriate shell configuration file\\ The changes will apply to all the **new** terminals you open\\ \\ |
* **bash shell** user : add these lines to ''~/.bashrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.sh | * **bash shell** user : add these lines to ''~/.bashrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.sh | ||
alias pynit='conda activate base'</code> | alias pynit='conda activate base'</code> | ||
Line 315: | Line 462: | ||
* e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh | * e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh | ||
alias pynit 'conda activate base'</code> | alias pynit 'conda activate base'</code> | ||
+ | * After changing your configuration file, just **type ''pynit'' in any (new) terminal you open, when you need to use Python** | ||
* When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'') | * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'') | ||
* You can also define other aliases, or use another name than ''pynit''! | * You can also define other aliases, or use another name than ''pynit''! | ||
- | We choose **not** to directly add a ''conda activate env_name'' line to the shell configuration files. This would permanently initialize ''env_name'' and may cause future (and obscure) side effects. | + | === conda in a multi-user Linux installation === |
- | When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type ''conda activate env_name'' or the ''pynit'' alias defined above | + | In the case of **python environments maintained by a single user, but used by several users**, we could do the same as in the //single-user case//. |
- | ==== Multi-user installation ==== | + | It is more efficient to have the users //source// an intermediate initialization file, that will then //source// the initialization file used in the single-user case. This makes it easier to maintain and change the environments, without asking users to make changes. We can also define other useful aliases (e.g a ''wp'' alias that will execute ''which python'') |
- | In the case of **python environments maintained by a single user, but used by several users**, we could do the same as in the //General case//, but it can be useful to have the users //source// an intermediate initialization file, that will then //source// the initialization file used in the general case. This makes it easier to maintain and change the environments, without asking users to make changes. | + | * ask //bash// users to add to their ''~/.bashrc'' file something like\\ ''source /some_path_accessible_by_all_users/shared_conda_init_file.sh''\\ \\ with a ''shared_conda_init_file.sh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp_2024-03.sh.txt |}} |
- | + | * LSCE //bash// example: ''source /home/share/unix_files/cdat/conda_init_files/conda3_jyp_2024-03.sh'' | |
- | * ask //bash// users to add to ''~/.bashrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.sh''\\ \\ with a ''.conda3_jyp.sh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.sh.txt |}} | + | * ''spiritx1'' //bash// example: |
* ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}} | * ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}} | ||
- | ===== Keeping Miniconda3 up-to-date ===== | ||
- | |||
- | * Just type ''conda update -n base %%--%%all'' as shown in the [[#updating_mini_conda|Updating (Mini)conda]] section | ||
- | |||
- | * The ''base'' (miniconda3) and the other environments are all independent and can be updated independently | ||
- | * It is up to you to decide if you want to update environments or not | ||
===== conda resources ===== | ===== conda resources ===== | ||
Line 359: | Line 501: | ||
* ''conda -h'' | * ''conda -h'' | ||
* ''conda //command// %%--%%help'': help for a specific //command// | * ''conda //command// %%--%%help'': help for a specific //command// | ||
+ | |||
+ | * **Version and configuration** information | ||
+ | * Version information: ''conda %%--%%version'' | ||
+ | * Detailed information: ''conda info'' | ||
+ | * Full configuration: ''conda config %%--%%show'' | ||
+ | * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation | ||
* **Available environments** | * **Available environments** | ||
Line 418: | Line 566: | ||
* Read the [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-environments.html#using-pip-in-an-environment|Using pip in an environment]] | * Read the [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-environments.html#using-pip-in-an-environment|Using pip in an environment]] | ||
==== Installation history ==== | ==== Installation history ==== | ||
+ | |||
+ | You can get the current list of installed packages with ''conda list -n existing_environment'', but it may also be useful to know in which order the packages were installed: | ||
+ | |||
+ | * ''conda list -n existing_environment %%--%%revisions'': display **exactly** what was installed and when (including the updates, and automatic dependencies installation) | ||
+ | * You can theoretically roll back to a previous ''RR'' //revision// number with: ''conda install -n existing_environment %%--%%revision RR'' | ||
+ | * There is more information in the ''/path_to_miniconda/conda-meta/history'' installation log files. | ||
+ | * ''base'' history file: ''/path_to_miniconda/conda-meta/history'' | ||
+ | * ''existing_environment'' history file: ''/path_to_miniconda/envs/existing_environment/conda-meta/history'' | ||
+ | * e.g. installation commands for the ''base'' environment:\\ <code> $ egrep -e 'cmd:' -e '==>' /homedata/jypmce/miniconda3_2024-03/conda-meta/history | ||
+ | |||
+ | ==> 2024-02-23 18:39:23 <== | ||
+ | # cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW | ||
+ | ==> 2024-03-27 16:10:01 <== | ||
+ | # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all | ||
+ | ==> 2024-03-28 11:38:08 <== | ||
+ | [...] | ||
+ | ==> 2024-04-24 17:47:32 <== | ||
+ | # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all</code> | ||
+ | * You can also, use a script and the ''conda'' module available **in the ''base'' environment**: [[https://github.com/conda/conda/issues/4545#issuecomment-469984684|example]] | ||
+ | ===== Uninstalling Miniconda3 ===== | ||
+ | |||
+ | ==== Windows computer ==== | ||
+ | |||
+ | Miniconda3 can be removed like a regular Windows application | ||
+ | |||
+ | * Go to the //Apps & features// panel | ||
+ | * ''Start'' => ''Settings'' => ''Apps'' => ''Apps & features'' | ||
+ | * Select ''Miniconda3 py3xxxxx'' | ||
+ | * Click on ''Uninstall'' and follow the instructions displayed by the uninstaller window | ||
+ | * This will remove: | ||
+ | * The (very big) directory that you specified when you installed //Miniconda3// | ||
+ | * The ''conda'' shortcuts from the ''Start Menu'' | ||
+ | * This will **not** remove your local configuration file ''C:\Users\your_login\.condarc'' | ||
+ | |||
+ | ==== Linux-like computer ==== | ||
+ | |||
+ | FIXME | ||