This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:uvcdat:cdat_conda:miniconda3_install [2021/03/08 15:29] jypeter |
other:uvcdat:cdat_conda:miniconda3_install [2024/05/29 13:57] jypeter [Updating (Mini)conda] |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== JYP steps for installing Miniconda3 ====== | ====== JYP steps for installing Miniconda3 ====== | ||
- | ===== What? Why? ===== | + | ===== Before installing Python on a shared computer ===== |
- | * //Miniconda3// is a minimal python environment/distribution that can be used for creating more complete working environments. It will basically make ''conda'' available on your computer, and then you can use ''conda'' to install and update more //stuff//. | + | ==== Desperately seeking a specific package ==== |
- | * A python //environment// is basically where and how you install python. A python //distribution// is more which packages you have chosen to install together in the environment. Hmmm, well, these are very similar things! | + | Check if the Python package you are [[https://www.imdb.com/title/tt0089017/|desperately seeking]] is not already installed, before installing your own Python on shared Linux servers, or on a shared cluster !! |
- | * You don't need to be (and **you should not be**) //root// when you install Miniconda3. You just need enough disk space where you have write access | + | * Ask other users who know the local environment |
- | * If you are installing Miniconda3 in a Linux environment on a [[other:win10wsl|Windows 10 computer using Windows Subsystem for Linux (WSL)]], pay special attention to the instructions on the **WSL** lines | + | * Read the servers/cluster documentation |
- | * The way you do things will depend on how you are going to use python? Are you installing the environment just for you, or for multiple users? | + | * Use ''[[other:newppl:starting#using_module_to_access_optional_programs|module avail [mod_name]]]'' to find out the Python versions available, and then ''module load'' and ''conda list'' to get more information |
+ | * e.g. on ''spiritx'': <code>$ module avail python | ||
+ | python/meso-3.8 python/meso-3.9 python/meso-3.10 python/meso-3.11 | ||
+ | $ module avail anaconda | ||
+ | anaconda-meso/2022.10 anaconda3-py/2020.11 | ||
+ | anaconda-meso/2023.09-0 anaconda3-py/2021.11 | ||
+ | </code> | ||
- | * We could also start with the //Anaconda// installer that will install a much more complete python environment, ready for use, but we choose not to do that because Anaconda requires more disk space at the beginning, with all its components coming from the ''default'' //channel// (or //repository//) provided by the [[https://docs.conda.io/projects/conda/en/latest/glossary.html#conda-repository|conda repository]]. It's not useful because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] channel. | + | * Lots of Python packages do very similar things. Maybe you can use another installed package... |
- | The steps on this page are adapted from the much older (but with lots of extra and possibly useful details) [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions | + | * If the packages you need are not available, it may be faster and easier to ask the local system administrators if they can install the packages, rather than doing everything yourself |
+ | |||
+ | ==== Using a specific Python on a jupyter server ==== | ||
+ | |||
+ | If there is a //jupyter notebook server// connected to your system, but the Python you want to use is not available on the server, follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local server | ||
+ | ===== Why should you use Miniconda3 ? ===== | ||
+ | |||
+ | * **//Miniconda3// is a minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command available on your computer, and you can then use this ''conda'' executable to select an existing distribution, or create new and independent Python distributions. | ||
+ | |||
+ | * A **conda //environment//** is basically (the short name of the directory) where you install the Python that you will use (independently of //Miniconda3// itself). You have to ''activate'' this environment in order to use it\\ A **python //distribution//** is the collection of packages (and their dependencies) you have chosen to install together in a given environment. | ||
+ | * Example: initializing an environment named ''cdatm_py3'' (assuming it exists)<code>$ source /path_to_miniconda/etc/profile.d/conda.sh | ||
+ | |||
+ | $ which conda | ||
+ | /path_to_miniconda/condabin/conda | ||
+ | |||
+ | $ conda env list | ||
+ | [...] | ||
+ | cdatm_py3 /path_to_miniconda/envs/cdatm_py3 | ||
+ | some_other_env /path_to_miniconda/envs/some_other_env | ||
+ | |||
+ | $ conda activate cdatm_py3 | ||
+ | |||
+ | (cdatm_py3) $ which python | ||
+ | /path_to_miniconda/envs/cdatm_py3/bin/python | ||
+ | </code> | ||
+ | * The //special// environment where //Miniconda3// itself is installed is called ''base'' (it also used to be called ''root''). Note that the ''python'' binary of the //special// ''base'' environment is not located in the same directory hierarchy as the ''python'' of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base | ||
+ | |||
+ | (base) $ which python | ||
+ | /path_to_miniconda/bin/python | ||
+ | </code> | ||
+ | |||
+ | * You **don't need to be (and you should not be) //root//** when you install //Miniconda3//. You just **need //enough// disk space** on a disk where you have write access | ||
+ | * **WARNING**: by default, //Miniconda3// will install itself and subsequent Python environments in a sub-directory of your home directory (''~/miniconda3'' on Linux). **Do not use the default installation location**, if you want to avoid disk space related problems (//disk full//, //quota exceeded//, ...) | ||
+ | * **WARNING**: older versions of //Miniconda3// used a **hidden sub-directory** of your home directory (''~/.conda'' on Linux) for the installation | ||
+ | * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing | ||
+ | |||
+ | * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because: | ||
+ | * it requires more disk space than //Miniconda3// during the initial installation, | ||
+ | * all the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems** | ||
+ | |||
+ | |||
+ | ===== Downloading Miniconda3, and BASIC installation ===== | ||
+ | |||
+ | ==== Miniconda3 on a Windows computer ==== | ||
+ | |||
+ | Some of the steps below are adapted from the [[https://docs.anaconda.com/free/miniconda/#quick-command-line-install|Windows section of "Quick command line install"]] to install a //pure Windows// version of Miniconda3 | ||
+ | |||
+ | === Downloading the Miniconda3 installer on Windows === | ||
+ | |||
+ | * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Windows installer]] | ||
+ | * https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe | ||
+ | * The installer is based on **Python 3.12.2 and Conda 24.3.0 as of April 2024** | ||
+ | |||
+ | * Find some temporary space on Windows | ||
+ | * e.g. ''C:\Scratch\your_login'' or whatever you use as a [[other:win10config#local_scratch_folder|local Scratch folder]] | ||
+ | |||
+ | * Open a //terminal//, where you can type the required installation commands | ||
+ | * Windows //Powershell terminal//: ''Start'' => ''W'' => ''Windows PowerShell'' => ''Windows PowerShell'' | ||
+ | * [[other:win10apps#windows_terminal|PowerShell in a Windows Terminal]] | ||
+ | |||
+ | * Use ''curl'' to download the latest installer (78 Mb as of April 2024): <code>PS C: cd C:\Scratch\your_login | ||
+ | PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe | ||
+ | PS C:\Scratch\your_login> dir miniconda.exe | ||
+ | [...] | ||
+ | -a---- 19/04/2024 11:18 81274240 miniconda.exe</code> | ||
+ | |||
+ | === BASIC Miniconda3 installation on Windows === | ||
+ | |||
+ | * **Determine the folder where you will install Miniconda3 and ALL your future Python environments** based on this version of Miniconda3\\ **We will later refer to this location as ''/path_to_miniconda3/''** in the Linux sections | ||
+ | * Remember that **you will need a few Gb of available space** | ||
+ | * You should **avoid installing Miniconda3 in your Windows //home// folder** (''C:\Users\your_login'' or [[other:win10config#local_home_folder|<some_disk_different_from_c>:\Users\<your_login>]]), if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python environments... | ||
+ | * We add some date or version information at the end of the folder name, in order to differentiate it from other (older of future) installations | ||
+ | * Example installation folder: ''C:\Utils\miniconda3_2024-03'' (if we start installing in March 2024) | ||
+ | |||
+ | * Launch the Miniconda3 installer with<code>PS C:\Scratch\your_login> .\miniconda.exe</code> | ||
+ | |||
+ | * In the installer window: | ||
+ | * (Review and) accept the License Agreement | ||
+ | * Choose to install for //Just Me// (the recommended default choice) | ||
+ | * Specify the //Destination Folder// selected above, e.g. ''C:\Utils\miniconda3_2024-03'' | ||
+ | * Use the pre-selected //Advanced Installation Options//, i.e | ||
+ | * //Create start menu shortcutes// | ||
+ | * //Register Miniconda3 as my default Python 3.12// <= Useful? | ||
+ | |||
+ | * When the installer has finished running, you should have: | ||
+ | * A new ''C:\Utils\miniconda3_2024-03'' folder | ||
+ | * This folder size will be ~650 Mb and will keep on growing (up to several Gb) as you add more Python environments and packages | ||
+ | * => **Be sure to have enough space on this disk!** | ||
+ | * Two new shortcuts in the Windows ''Start'' menu | ||
+ | * ''Anaconda Prompt'' | ||
+ | * ''Anaconda Powershell Prompt'' | ||
+ | * **We will now use this //Anaconda PowerShell// shortcut each time we need to use Miniconda3 or Python** | ||
+ | |||
+ | * Remove the installer<code>PS C:\Scratch\your_login> del .\miniconda.exe</code> | ||
+ | |||
+ | * Open an //Anaconda PowerShell Prompt//, and type a few commands to make sure that ''conda'' is working<code>(base) PS C:\Users\your_login> conda env list | ||
+ | base * C:\Utils\miniconda3_2024-03 | ||
+ | |||
+ | (base) PS C:\Users\your_login> conda list | ||
+ | # packages in environment at C:\Utils\miniconda3_2024-03: | ||
+ | # Name Version Build Channel | ||
+ | [...] | ||
+ | conda 24.3.0 py312haa95532_0 | ||
+ | [...] | ||
+ | python 3.12.2 h1d929f7_0 | ||
+ | [...] | ||
+ | |||
+ | (base) PS C:\Users\your_login> conda deactivate | ||
+ | |||
+ | PS C:\Users\your_login> conda env list | ||
+ | base C:\Utils\miniconda3_2024-03</code> | ||
+ | |||
+ | * Next: be sure to <wrap hi>follow the steps in the [[other:uvcdat:cdat_conda:miniconda3_install#fine-tuning_conda_to_use_only_conda-forge|Fine-tuning conda to use (only) conda-forge]], and all the other Miniconda3 instructions</wrap> | ||
+ | * ''conda'' commands will work on all computers (Windows and Linux-like) | ||
+ | * The ''.condarc'' is located in your Windows //home// folder\\ => ''C:\Users\your_login\.condarc'' | ||
+ | |||
+ | ==== Miniconda3 on a Linux-like computer ==== | ||
- | ===== Installing miniconda3 on a Linux-like computer ===== | ||
<WRAP center round tip 60%> | <WRAP center round tip 60%> | ||
- | By //Linux-like//, we mean: | + | By **//Linux-like//**, we mean: |
- | * A native Linux computer | + | * A **Linux** desktop/laptop |
- | * A windows 10 computer with [[other:win10wsl|WSL+Ubuntu installed]] | + | * The installation will be used by **one** person |
- | * A mac where you can use Linux in a terminal | + | * A **Linux** server |
+ | * The installation will be used by **one or more** persons | ||
+ | * A **windows computer with [[other:win10wsl|WSL+Ubuntu installed]]** | ||
+ | * You should use the [[#miniconda3_on_a_windows_computer|Windows installer]], if all the packages you need are available for Windows | ||
+ | * The installation will be used by **one** person | ||
+ | * A **Mac** where you can use Linux in a terminal | ||
+ | * The installation will be used by **one** person | ||
</WRAP> | </WRAP> | ||
- | * Get the [[https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh|Python 3.8 Linux 64-bit (bash installer)]] (unless there is a [[https://conda.io/miniconda.html|more recent version?]]) | ||
- | * Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]]) | ||
- | * e.g. Linux at LSCE:\\ ''cd /home/scratch01/<your_login>'' | ||
- | * e.g. WSL, assuming that there is a ''C:\Scratch\<your_login>'' directory:\\ ''cd /mnt/c/Scratch/<your_login>'' | ||
- | * Use ''wget'' to download the installer:\\ ''wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'' (90 Mb, 26 Feb 2021) | ||
- | * Execute the installer | + | === Downloading the Miniconda3 installer === |
- | * ''bash Miniconda3-latest-Linux-x86_64.sh'' | + | |
- | * Accept the license | + | |
- | * Note: at the end of the installation (next step), **answer //no// to the following question**, so that the installer does **not** change your existing shell initialization files!\\ ''Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]'' => **NO** | + | |
- | * Specify an explicit installation path **outside of your //home// directory**, with enough disk space (more than 3 Gb if you are going to install CDAT and some extra packages), preferably on a disk that is not backed up: | + | |
- | * Installations by //JYP//: | + | |
- | * Linux at LSCE: ''/home/share/unix_files/cdat/miniconda3'', or a ''miniconda3<some_version>'' subdirectory of ''/home/share/unix_files/cdat/'' | + | |
- | * Linux on ciclad: ''/data/jypmce/cdat/miniconda3'', or a ''miniconda3<some_version>'' sub-directory of ''/data/jypmce/cdat/'' | + | |
- | * WSL: installing to a directory that is not in ''/home/'' <wrap hi>does not work</wrap> (e.g. ''/mnt/h/CDAT/miniconda3'',assuming there is a ''H:\CDAT\'' directory, does not work)\\ You need to accept the installation in the default location: ''/home/<your_login>/miniconda3'' | + | |
- | * The resulting ''miniconda3'' directory size is 342M\\ <code> > du -sh miniconda3 | + | |
- | 342M miniconda3 | + | |
- | > cd miniconda3 | + | * Important: <wrap hi>do not install your own Python environment(s) on shared computers before you have made sure that the packages you need are not already installed, or cannot be easily installed in an existing environment (by the person who maintains this environment)!</wrap> |
+ | * Check the output of ''module avail'', read your system documentation, ask other users... | ||
- | > du -sh * | + | * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Linux installer]] |
- | 20M bin | + | * For the Linux computers we commonly use, we need the //Miniconda3 Linux 64-bit// link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh |
- | 0 compiler_compat | + | * Note: Mac users should use the latest [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|macOS installer]] appropriate for their computer |
- | 4.0K condabin | + | * The installer is based on **Python 3.12.2 and Conda 24.3.0 as of April 2024** |
- | 684K conda-meta | + | |
- | 0 envs | + | |
- | 16K etc | + | |
- | 5.5M include | + | |
- | 4.0K info | + | |
- | 198M lib | + | |
- | 12K LICENSE.txt | + | |
- | 114M pkgs | + | |
- | 604K share | + | |
- | 4.0K shell | + | |
- | 0 ssl | + | |
- | 0 x86_64-conda_cos6-linux-gnu</code> | + | |
- | * Initialize the newly installed conda environment (this will initialize the environment only in the current terminal): | + | |
- | * bash shell: ''source <installation_path>/miniconda3/etc/profile.d/conda.sh'' | + | * Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]]) |
- | * tcsh shell: ''source <installation_path>/miniconda3/etc/profile.d/conda.csh'' | + | * e.g. Linux on the **LSCE** servers:\\ ''$ cd /home/scratch01/$USER'' |
+ | * e.g. Linux on the IPSL **spiritx** cluster:<code>$ mkdir /homedata/$USER/Scratch | ||
+ | $ cd /homedata/$USER/Scratch</code> | ||
+ | * e.g. Linux on a **personal PC/Mac desktop/laptop**: use a local scratch/temporary directory with //enough// space | ||
+ | |||
+ | * Use ''wget'' to download the latest installer (138 Mb as of 22 Mar 2024):\\ <code>$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh | ||
+ | $ ls -lh Miniconda3-latest-Linux-x86_64.sh | ||
+ | [...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh</code> | ||
+ | |||
+ | === Using the Miniconda3 installer === | ||
+ | |||
+ | We assume below that we are in the [[#downloading_the_installer|directory where we have downloaded the installer]] | ||
+ | |||
+ | * **Determine the directory where you will install miniconda3 and all the Python environments** based on this version of miniconda3, but **do not create the directory** (the installation script will do it)\\ **We will later refer to this location as ''/path_to_miniconda3/''** | ||
+ | * Remember that **you will need a few Gb of available space** | ||
+ | * **If you want to share this installation with other users**, the directory has to be readable by these users | ||
+ | * At LSCE, **do not** install in a sub-directory of your home directory! | ||
+ | * On other servers and computers, you should also **avoid installing in your //home// directory**, if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python installation... | ||
+ | * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations | ||
+ | * e.g. Linux at **LSCE**: ''/home/share/unix_files/cdat/miniconda3_2024-03'' | ||
+ | * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03'' | ||
+ | * Linux (including WSL) on a **personal PC/Mac desktop/laptop**: find a directory with enough space (including space for adding more packages and their future updates) | ||
+ | * You may want to use a directory that you are not backing up, if you want to avoid having to save thousands of files... | ||
+ | |||
+ | * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh'' | ||
+ | * Review the license (type ''<SPACE>'' several times...) and accept it | ||
+ | * When asked for the installation directory, specify the directory location chosen in the previous step, and **not the default directory** | ||
+ | * e.g. specify\\ ''/path_to_miniconda3/''\\ instead of the default\\ ''$HOME/miniconda3'' | ||
+ | * **Answer ''no'' to the question ''Do you wish to update your shell profile to automatically initialize conda?''**.\\ Otherwise the installation will make changes to your shell configuration files | ||
+ | * If you forget to answer ''no'', you can apparently: | ||
+ | * remove the changes to your shell configuration files by typing later: ''conda init --reverse $SHELL'' | ||
+ | * or at least disable the automatic activation of the environment (that might have side effect) by typing: ''conda config --set auto_activate_base false'' | ||
+ | * The resulting ''miniconda3'' directory size is 647 Mb (as of March 2024)\\ <code>$ du -sh miniconda3_2024-03 | ||
+ | 647M miniconda3_2024-03 | ||
+ | |||
+ | $ du -sh miniconda3_2024-03/* | ||
+ | 47M miniconda3_2024-03/bin | ||
+ | 16K miniconda3_2024-03/cmake | ||
+ | 8.0K miniconda3_2024-03/compiler_compat | ||
+ | 32M miniconda3_2024-03/_conda | ||
+ | 8.0K miniconda3_2024-03/condabin | ||
+ | 976K miniconda3_2024-03/conda-meta | ||
+ | 4.0K miniconda3_2024-03/envs | ||
+ | 28K miniconda3_2024-03/etc | ||
+ | 18M miniconda3_2024-03/include | ||
+ | 317M miniconda3_2024-03/lib | ||
+ | 92K miniconda3_2024-03/LICENSE.txt | ||
+ | 1.1M miniconda3_2024-03/man | ||
+ | 232M miniconda3_2024-03/pkgs | ||
+ | 396K miniconda3_2024-03/sbin | ||
+ | 1.5M miniconda3_2024-03/share | ||
+ | 12K miniconda3_2024-03/shell | ||
+ | 8.0K miniconda3_2024-03/ssl | ||
+ | 8.0K miniconda3_2024-03/x86_64-conda_cos7-linux-gnu | ||
+ | 8.0K miniconda3_2024-03/x86_64-conda-linux-gnu | ||
+ | </code> | ||
+ | |||
+ | * Initialize the newly installed conda environment (this will initialize the environment **only in the current terminal**): | ||
+ | * **bash shell**: ''source /path_to_miniconda3/etc/profile.d/conda.sh'' | ||
+ | * e.g. (spiritx): ''source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh'' | ||
+ | * **tcsh shell**: ''source /path_to_miniconda3/etc/profile.d/conda.csh'' | ||
+ | * e.g. (LSCE): ''source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh'' | ||
* Check if you can use the ''conda'' command, and use it to initialize the //base// environment | * Check if you can use the ''conda'' command, and use it to initialize the //base// environment | ||
- | * <code>$ which conda | + | * e.g. on **spiritx1**: <code>$ which conda |
- | <installation_path>/miniconda3/condabin/conda | + | |
$ which python | $ which python | ||
/usr/bin/python | /usr/bin/python | ||
- | $ conda activate | + | |
+ | $ source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh | ||
+ | |||
+ | $ which conda | ||
+ | /homedata/jypmce/miniconda3_2024-03/condabin/conda | ||
+ | $ which python | ||
+ | /usr/bin/python | ||
+ | |||
+ | $ conda activate base | ||
+ | |||
+ | (base) $ which conda | ||
+ | /homedata/jypmce/miniconda3_2024-03/bin/conda | ||
(base) $ which python | (base) $ which python | ||
- | <installation_path>/miniconda3/bin/python</code> | + | /homedata/jypmce/miniconda3_2024-03/bin/python |
- | * Update the new installation | + | |
- | * <code>$ conda update --all | + | |
- | [...]</code> | + | |
- | * During the update, the ''miniconda3'' directory size goes from 432 Mb to 581 Mb. This directory will keep on growing, which is the reason why you should put it on a (preferably non backed up) disk where you have //enough// space | + | |
- | * Make sure we have the latest ''conda'' package (just in case we did not get it with the //update//)\\ ''conda update -n base conda'' | + | (base) $ conda deactivate |
- | * Remove the installer later, when you have tester your installation:\\ ''rm Miniconda3-latest-Linux-x86_64.sh'' | + | $ which conda |
+ | /homedata/jypmce/miniconda3_2024-03/condabin/conda | ||
+ | $ which python | ||
+ | /usr/bin/python | ||
+ | </code> | ||
- | ===== Initializing conda in new terminals ===== | + | * Remove the //Miniconda3// installer: ''$ rm Miniconda3-latest-Linux-x86_64.sh'' |
- | When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between environments | + | ===== Fine-tuning conda to use (only) conda-forge ===== |
- | ==== General case ==== | + | * ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies. |
- | You were asked the following question when installing miniconda3: //Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]// | + | * ''conda'' works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver. |
+ | * If you have followed all the steps of this section in order to **use only //conda-forge// packages**, and there are still some problems with a recent version of ''conda'', try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of conda | ||
+ | * Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments: | ||
+ | * [[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]] | ||
+ | * [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]] | ||
- | * If you answered //yes//, the installer probably added some very complicated lines to your shell configuration files, but you probably have ''conda'' directly available when you open a new terminal | + | * We are power users, and **we intend to create complex Python environments** combining lots (and lots) of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge. |
- | * if you answered //no// (as suggested), use a text editor to add an extra line to the appropriate configuration file | + | * => **The solution is to configure conda to always use conda-forge**, and completely update //Miniconda3// itself with packages coming only from conda-forge |
- | * //bash// user: add this line to ''~/.bashrc''\\ ''source <installation_path>/miniconda3/etc/profile.d/conda.sh'' | + | |
- | * //tcsh// user: add this line to ''~/.cshrc''\\ ''source <installation_path>/miniconda3/etc/profile.d/conda.csh'' | + | |
- | We choose **not** to add a ''conda activate env_name'' line to the shell configuration files, in order to avoid side effects. When we open a new terminal, we get the default python available on the system. When we need a specific python environment, we just open a new window and then explicitly type: ''conda activate env_name'' | + | === Changing the .condarc file === |
+ | |||
+ | The following steps will **make sure that we only get packages from //conda-forge//** (same thing as using the ''-c conda-forge'' option) by default unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] | ||
+ | |||
+ | <code>$ cat ~/.condarc | ||
+ | cat: /home/jypmce/.condarc: No such file or directory | ||
+ | |||
+ | $ conda config --get channels | ||
+ | |||
+ | $ conda config --prepend channels conda-forge | ||
+ | |||
+ | $ conda config --set channel_priority strict | ||
+ | |||
+ | $ conda config --get channels | ||
+ | --add channels 'defaults' # lowest priority | ||
+ | --add channels 'conda-forge' # highest priority | ||
+ | |||
+ | $ cat ~/.condarc | ||
+ | channels: | ||
+ | - conda-forge | ||
+ | - defaults | ||
+ | channel_priority: strict | ||
+ | </code> | ||
+ | |||
+ | === Getting conda configuration information === | ||
+ | |||
+ | * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation | ||
+ | * Basic information: ''conda info'' | ||
+ | * Full configuration: ''conda config %%--%%show'' | ||
+ | |||
+ | ==== Updating (Mini)conda ==== | ||
+ | |||
+ | We have to **completely update //Miniconda3// at least once after installing //Miniconda3//** and [[#fine-tuning_conda_to_use_only_conda-forge|making conda-forge the highest priority channel]] | ||
+ | |||
+ | * Check the current version of ''conda'' | ||
+ | * <code>$ conda list -n base | grep '^conda' | ||
+ | # packages in environment at /path_to_miniconda: | ||
+ | conda 24.1.2 py312h06a4308_0 | ||
+ | conda-libmamba-solver 23.12.0 pyhd3eb1b0_1 | ||
+ | [... and other packages with 'conda' in their name] | ||
+ | </code> | ||
+ | |||
+ | * Completely update the //Miniconda3// installation | ||
+ | * <code>$ conda update -n base --all | ||
+ | Channels: | ||
+ | - conda-forge | ||
+ | - defaults | ||
+ | Platform: linux-64 | ||
+ | Collecting package metadata (repodata.json): done | ||
+ | Solving environment: done | ||
+ | |||
+ | ## Package Plan ## | ||
+ | |||
+ | environment location: /homedata/jypmce/miniconda3_2024-03 | ||
+ | |||
+ | |||
+ | The following packages will be downloaded: | ||
+ | |||
+ | package | build | ||
+ | ---------------------------|----------------- | ||
+ | conda-24.3.0 | py312h7900ff3_0 1.1 MB conda-forge | ||
+ | conda-libmamba-solver-24.1.0| pyhd8ed1ab_0 40 KB conda-forge | ||
+ | python-3.12.2 |hab00c5b_0_cpython 30.8 MB conda-forge | ||
+ | [...] | ||
+ | The following NEW packages will be INSTALLED: | ||
+ | [...] | ||
+ | The following packages will be UPDATED: | ||
+ | [...] | ||
+ | conda pkgs/main::conda-24.1.2-py312h06a4308~ --> conda-forge::conda-24.3.0-py312h7900ff3_0 | ||
+ | [...] | ||
+ | The following packages will be SUPERSEDED by a higher-priority channel: | ||
+ | [...] | ||
+ | Proceed ([y]/n)? y | ||
+ | [...] | ||
+ | Preparing transaction: done | ||
+ | Verifying transaction: done | ||
+ | Executing transaction: done | ||
+ | </code> | ||
+ | * Check the updated version of ''conda''. Note that **(almost) all** the packages should now specify that they are provided by ''conda-forge'' | ||
+ | * <code>$ conda list -n base | ||
+ | conda 24.3.0 py312h7900ff3_0 conda-forge | ||
+ | conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge | ||
+ | python 3.12.2 hab00c5b_0_cpython conda-forge | ||
+ | </code> | ||
+ | * You can use the following to determine if some packages are not provided by ''conda-forge'' | ||
+ | * <code>$ conda list -n base | grep -v conda-forge | ||
+ | # Name Version Build Channel | ||
+ | libedit 3.1.20230828 h5eee18b_0 | ||
+ | libffi 3.4.4 h6a678d5_0 | ||
+ | xz 5.4.5 h5eee18b_0 | ||
+ | </code> | ||
+ | * It should not be a major problem if a few packages are not provided by ''conda-forge''.\\ It only means that some packages provided by the ''anaconda'' //channel// were more recent than the same packages provided by the ''conda-forge'' //channel// when the packages were updated | ||
+ | * ''libedit'' example: [[https://anaconda.org/conda-forge/libedit|version 3.1.20191231 @ conda-forge]] and [[https://anaconda.org/anaconda/libedit|version 3.1.20230828 @ anaconda]] | ||
+ | |||
+ | * You can later update again the full //Miniconda3//, or just the ''conda'' command.\\ This should not make any changes to the Python environment(s) have have installed with ''conda'' | ||
+ | * <code>$ conda update -n base --all | ||
+ | [...] | ||
+ | # All requested packages already installed. | ||
+ | |||
+ | (base) $ conda update -n base conda | ||
+ | [...] | ||
+ | # All requested packages already installed. | ||
+ | </code> | ||
+ | |||
+ | * Note: during this installation, the ''miniconda3'' directory size grew from 763 Mb to 1.6 Gb, and we have not installed any custom Python environment yet! This is the reason why you should install //Miniconda3// on a (preferably non backed up) disk where you have //enough// space | ||
+ | * <code>$ du -sh /homedata/jypmce/miniconda3_2024-03 | ||
+ | 736M /homedata/jypmce/miniconda3_2024-03 | ||
+ | [... update] | ||
+ | $ du -sh /homedata/jypmce/miniconda3_2024-03 | ||
+ | 1.6G /homedata/jypmce/miniconda3_2024-03 | ||
+ | </code> | ||
+ | |||
+ | * You can [[https://docs.conda.io/projects/conda/en/latest/commands/clean.html|clean]] the initial //Miniconda3// installation to free up some disk space. The documentation specifies //WARNING: This will break environments with packages installed using symlinks back to the package cache.// and **you should probably avoid using ''clean'' once you have installed new Python environments** (unless you are desperate for disk space and know how to do a complete re-installation if something breaks...) | ||
+ | * <code>(base) $ conda clean --all | ||
+ | Will remove 154 (180.9 MB) tarball(s). | ||
+ | Proceed ([y]/n)? y | ||
+ | |||
+ | Will remove 1 index cache(s). | ||
+ | Proceed ([y]/n)? y | ||
+ | |||
+ | Will remove 74 (378.0 MB) package(s). | ||
+ | Proceed ([y]/n)? y | ||
+ | |||
+ | There are no tempfile(s) to remove. | ||
+ | There are no logfile(s) to remove. | ||
+ | |||
+ | (base) $ du -sh /homedata/jypmce/miniconda3_2024-03 | ||
+ | 457M /homedata/jypmce/miniconda3_2024-03 | ||
+ | </code> | ||
+ | ===== Initializing conda in terminals ===== | ||
+ | <WRAP center round tip 60%> | ||
+ | |||
+ | Reminder: if you have installed Miniconda3 on a Windows computer, you only have to open a terminal with | ||
+ | |||
+ | * ''Start'' menu => ''Anaconda Powershell Prompt'' | ||
+ | * ''conda activate the_environment_you_want'' | ||
+ | </WRAP> | ||
+ | |||
+ | |||
+ | When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between existing environments. This can be **configured in the //shell// configuration files** | ||
+ | |||
+ | ==== Single-user installation ==== | ||
+ | |||
+ | You were asked the following question when installing //Miniconda3//: //Do you wish to update your shell profile to automatically initialize conda? [yes|no]// | ||
+ | |||
+ | * **If you answered //yes//**, the installer probably added some very complicated lines to your shell configuration files, and ''conda'' and the newly installer ''python'' are probably directly available when you open a new terminal. | ||
+ | |||
+ | * **If you answered //no//** ([[#using_the_miniconda3_installer|as suggested]]), use a [[other:newppl:starting#text_editors|text editor]] to add extra lines to the appropriate shell configuration file | ||
+ | * **bash shell** user : add these lines to ''~/.bashrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.sh | ||
+ | alias pynit='conda activate base'</code> | ||
+ | * e.g. (spiritx):<code>source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh | ||
+ | alias pynit='conda activate base'</code> | ||
+ | * **tcsh shell** user: add these lines to ''~/.cshrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.csh'' | ||
+ | alias pynit 'conda activate base'</code> | ||
+ | * e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh | ||
+ | alias pynit 'conda activate base'</code> | ||
+ | * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'') | ||
+ | * You can also define other aliases, or use another name than ''pynit''! | ||
+ | |||
+ | We choose **not** to directly add a ''conda activate env_name'' line to the shell configuration files. This would permanently initialize ''env_name'' and may cause future (and obscure) side effects. | ||
+ | |||
+ | When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type ''conda activate env_name'' or the ''pynit'' alias defined above | ||
==== Multi-user installation ==== | ==== Multi-user installation ==== | ||
- | In the case of **python environments maintained by a single user, but used by several users**, we could do the same as above, but it can be useful to have the users //source// an intermediate initialization file, that will then //source// the initialization file used in the general case. This makes it easier to maintain and change the environments, without asking users to make changes. | + | In the case of **python environments maintained by a single user, but used by several users**, we could do the same as in the //General case//, but it can be useful to have the users //source// an intermediate initialization file, that will then //source// the initialization file used in the general case. This makes it easier to maintain and change the environments, without asking users to make changes. |
+ | |||
+ | * ask //bash// users to add to ''~/.bashrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.sh''\\ \\ with a ''.conda3_jyp.sh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.sh.txt |}} | ||
+ | |||
+ | * ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}} | ||
+ | |||
+ | ===== Keeping Miniconda3 up-to-date ===== | ||
+ | |||
+ | * Just type ''conda update -n base %%--%%all'' as shown in the [[#updating_mini_conda|Updating (Mini)conda]] section | ||
+ | |||
+ | * The ''base'' (miniconda3) and the other environments are all independent and can be updated independently | ||
+ | |||
+ | * It is up to you to decide if you want to update environments or not | ||
+ | ===== conda resources ===== | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ==== Web sites ==== | ||
+ | |||
+ | * [[https://docs.conda.io/projects/conda/en/stable/|Official conda website]] | ||
+ | * [[https://docs.conda.io/projects/conda/en/stable/user-guide/cheatsheet.html|Cheat sheet]] (2 pages, for people in a hurry) | ||
+ | * [[https://docs.conda.io/projects/conda/en/stable/commands/|Commands' reference]] | ||
+ | * Package repositories | ||
+ | * [[https://conda-forge.org/packages/|conda-forge]] <= recommended source of packages | ||
+ | * [[https://anaconda.org/|default]] | ||
+ | |||
+ | |||
+ | ==== Useful conda commands ==== | ||
+ | |||
+ | This is just a subset of some of the commands, and a subset of their options! For a complete reference, use the [[https://docs.conda.io/projects/conda/en/stable/|official conda website]] | ||
+ | |||
+ | * **Get help** | ||
+ | * ''conda -h'' | ||
+ | * ''conda //command// %%--%%help'': help for a specific //command// | ||
+ | |||
+ | * **Available environments** | ||
+ | * ''conda env list'' | ||
+ | * The currently active environment has a "*****" character on its line<code>(base) $ conda env list | ||
+ | # conda environments: | ||
+ | # | ||
+ | base * /homedata/jypmce/miniconda3_2024-03</code> | ||
+ | |||
+ | * **Activate** an existing environment (available in ''conda env list'') or **deactivate**, in order to go back to the default Python | ||
+ | * ''conda activate existing_environment'' | ||
+ | * ''conda activate'' (without specifying an environment name) will **activate the ''base'' Miniconda3 environment** | ||
+ | * Go back to the default Python available on the system | ||
+ | * ''conda deactivate'' | ||
+ | * You can type ''which python'' to determine where is the current ''python'' executable you are using is\\ You should **always know which Python you are using!** | ||
+ | * <code>(base) $ which python | ||
+ | /homedata/jypmce/miniconda3_2024-03/bin/python | ||
+ | |||
+ | (base) $ conda deactivate | ||
+ | |||
+ | $ which python | ||
+ | /usr/bin/python</code> | ||
+ | * Note: //before conda ''4.6''//, you had to type ''**source** activate existing_environment'' and ''**source** deactivate'' | ||
+ | |||
+ | * Get the **list of installed packages**, their version, and where they come from (e.g. ''conda-forge'', ''pip'') | ||
+ | * ''conda list [-n existing_environment]'' | ||
+ | |||
+ | * **Create an environment** and install a few packages and their dependencies | ||
+ | * ''conda create -n new_environment_name package1 package2 packageN'' | ||
+ | |||
+ | * **Delete an environment** | ||
+ | * ''conda remove -n crap_environment %%--%%all'' | ||
+ | |||
+ | * **Find out if a package is available** on the //conda-forge// channel (or other channels) | ||
+ | * Note: you should always install a package with ''conda'' instead of ''pip'', if it is available on a //conda channel// | ||
+ | * ''conda search package_exact_name''\\ ''conda search '*package_partial_name*''' | ||
+ | * Note: ''search'' with wildcards fails in a //tcsh// shell, as of April 2024 ([[https://github.com/conda/conda/issues/9771|CSH install/search fail with '*' variable in an argument]]) | ||
+ | |||
+ | * **Install and remove packages**\\ Add ''-n target_environment'' to make changes in another environment than the active one | ||
+ | * ''conda install package1 package2 packageN'' | ||
+ | * ''conda'' should already be [[#fine-tuning_conda_to_use_only_conda-forge|configured to use conda-forge by default]] if you have read this page carefully | ||
+ | * Add ''-c channel_url_or_alias'' to install from a specific //channel// | ||
+ | * ''conda remove package1 package2 packageN'' | ||
+ | |||
+ | * **Update** just ''conda'' (update the //conda// package, not the full Miniconda3 //base// environment) | ||
+ | * ''conda update -n base conda'' | ||
+ | |||
+ | * **Completely update an environment** (all the packages) | ||
+ | * ''conda update -n existing_environment %%--%%all'' | ||
+ | |||
+ | * **Cleaning... <wrap em>Dangerous</wrap>**!\\ If you have several environments, ''conda clean'' may remove packages that are not needed in an environment, but are still used in other environments, and **you may end up with a broken installation...**\\ ''conda'' does not automatically clean the content of the ''pkgs'' directory!\\ Use the following to //Remove index cache, lock files, tarballs, unused cache packages, and source cache// | ||
+ | * ''conda clean %%--%%all'' | ||
+ | |||
+ | |||
+ | ==== conda, pip and virtualenv ==== | ||
+ | |||
+ | * Read the [[https://docs.conda.io/projects/conda/en/stable/commands/#conda-vs-pip-vs-virtualenv-commands|Conda vs. pip vs. virtualenv commands]] section | ||
+ | |||
+ | * Read the [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-environments.html#using-pip-in-an-environment|Using pip in an environment]] | ||
+ | ==== Installation history ==== | ||
+ | |||
+ | You can get the current list of installed packages with ''conda list -n existing_environment'', but it may also be useful to know in which order the packages were installed: | ||
+ | |||
+ | * ''conda list -n existing_environment %%--%%revisions'': display **exactly** what was installed and when (including the updates, and automatic dependencies installation) | ||
+ | * You can theoretically roll back to a previous ''RR'' //revision// number with: ''conda install -n existing_environment %%--%%revision RR'' | ||
+ | * There is more information in the ''/path_to_miniconda/conda-meta/history'' installation log files. | ||
+ | * ''base'' history file: ''/path_to_miniconda/conda-meta/history'' | ||
+ | * ''existing_environment'' history file: ''/path_to_miniconda/envs/existing_environment/conda-meta/history'' | ||
+ | * e.g. installation commands for the ''base'' environment:\\ <code> $ egrep -e 'cmd:' -e '==>' /homedata/jypmce/miniconda3_2024-03/conda-meta/history | ||
+ | |||
+ | ==> 2024-02-23 18:39:23 <== | ||
+ | # cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW | ||
+ | ==> 2024-03-27 16:10:01 <== | ||
+ | # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all | ||
+ | ==> 2024-03-28 11:38:08 <== | ||
+ | [...] | ||
+ | ==> 2024-04-24 17:47:32 <== | ||
+ | # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all</code> | ||
+ | * You can also, use a script and the ''conda'' module available **in the ''base'' environment**: [[https://github.com/conda/conda/issues/4545#issuecomment-469984684|example]] | ||
+ | ===== Uninstalling Miniconda3 ===== | ||
+ | |||
+ | ==== Windows computer ==== | ||
+ | |||
+ | Miniconda3 can be removed like a regular Windows application | ||
+ | |||
+ | * Go to the //Apps & features// panel | ||
+ | * ''Start'' => ''Settings'' => ''Apps'' => ''Apps & features'' | ||
+ | * Select ''Miniconda3 py3xxxxx'' | ||
+ | * Click on ''Uninstall'' and follow the instructions displayed by the uninstaller window | ||
+ | * This will remove: | ||
+ | * The (very big) directory that you specified when you installed //Miniconda3// | ||
+ | * The ''conda'' shortcuts from the ''Start Menu'' | ||
+ | * This will **not** remove your local configuration file ''C:\Users\your_login\.condarc'' | ||
+ | |||
+ | ==== Linux-like computer ==== | ||
- | * ask //bash// users to add something like to ''~/.bashrc''\\ ''source ~main_installer_login/.conda3_jyp.sh''\\ \\ with a ''.conda3_jyp.sh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.sh.txt |}} | + | FIXME |
- | * ask //tcsh// users to add something like to ''~/.cshrc''\\ ''source ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}} | ||
/* standard page footer */ | /* standard page footer */ |