Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:miniconda3_install [2024/03/26 10:41]
jypeter [Downloading the installer]
+++ other:uvcdat:cdat_conda:miniconda3_install [2024/04/12 14:26]
jypeter Added the "conda resources" section
@@ Line 21: / Line 21: @@
 /path_to_miniconda/envs/cdatm_py3/bin/python
 </code>
-    * The special environment where //Miniconda3// itself is installed is called ''base''. Note that the ''python'' binary of the ''base'' environment is not located in the same directory hierarchy of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base
+    * The //special// environment where //Miniconda3// itself is installed is called ''base''. Note that the ''python'' binary of the //special// ''base'' environment is not located in the same directory hierarchy as the ''python'' of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base
 (base) $ which python
@@ Line 32: / Line 32: @@
     * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing
-  * If you are installing //Miniconda3// in a Linux environment on a [[other:win10wsl|Windows 10 computer using Windows Subsystem for Linux (WSL)]], pay special attention to the specific instructions on the **WSL** lines
+  * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because:
+    * it requires more disk space than //Miniconda3// during the initial installation,
-  * You could also start with the full //Anaconda// installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full Anaconda installer** because it requires more disk space than //Miniconda3// at the beginning, and all its packages come from the ''default'' //channel// (or //repository//) provided by the [[https://docs.conda.io/projects/conda/en/latest/glossary.html#conda-repository|conda repository]]. This is not very useful (and could even cause complex dependency problems) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**.
+    * all the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**.
-Note: some extra details are available on the much older page //Installing and maintaining UV-CDAT with conda//. You can check later the [[other:uvcdat:conda_notes#useful_conda_commands|Useful conda commands]], but the [[https://docs.conda.io/projects/conda/en/latest/index.html|official conda documentation]] and the [[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html|Cheat sheet]] are probably more up-to-date.
 ===== Installing miniconda3 on a Linux-like computer =====
 <WRAP center round tip 60%>
-By //Linux-like//, we mean:
+By **//Linux-like//**, we mean:
   * A native Linux computer (or server)
   * A windows 10 computer with [[other:win10wsl|WSL+Ubuntu installed]]
   * A Mac where you can use Linux in a terminal
 </WRAP>
+<WRAP center round info 60%>
+**//Pure Windows//** Note: you can also follow the steps listed in the [[https://docs.anaconda.com/free/miniconda/#quick-command-line-install|Windows section of "Quick command line install"]] to install a //pure Windows// version of miniconda3, and then follow the steps listed in the next sections of this page.\\ We assume below that the installer is downloaded in the ''C:\Scratch\your_login'' directory, and we install Miniconda3 in a ''C:\Utils\miniconda3_2024-03'' directory. <code>PS C: cd C:\Scratch\your_login
+PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe
+PS C:\Scratch\your_login> .\miniconda.exe
+PS C:\Scratch\your_login> del .\miniconda.exe
+</code>
+</WRAP>
-==== Downloading the Minconda3 installer ====
+==== Downloading the Miniconda3 installer ====
   * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Linux installer]]
@@ Line 60: / Line 67: @@
 $ cd /homedata/$USER/Scratch</code>
     * e.g. Linux on a **personal PC/Mac desktop/laptop**: use a scratch/temporary directory with //enough// space
-    * e.g. **WSL**, assuming that there is a ''C:\Scratch\<your_login>'' directory:\\ ''cd /mnt/c/Scratch/<your_login>''
   * Use ''wget'' to download the latest installer (138 Mb as of 22 Mar 2024):\\ <code>$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
@@ Line 66: / Line 72: @@
 [...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh</code>
-==== Using the installer ====
+==== Using the Miniconda3 installer ====
 We assume below that we are in the [[#downloading_the_installer|directory where we have downloaded the installer]]
   * **Determine the directory where you will install miniconda3 and all the Python environments** based on this version of miniconda3, but **do not create the directory** (the installation script will do it)\\ **We will later refer to this location as ''/path_to_miniconda3/''**
-    * Remember that you will need a few Gb of available space
+    * Remember that **you will need a few Gb of available space**
-    * At LSCE, do not install in a subdirectory of your home directory!
+    * **If you want to share this installation with other users**, the directory has to be readable by these users
+    * At LSCE, **do not** install in a sub-directory of your home directory!
       * On other servers and computers, you should also **avoid installing in your //home// directory**, if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python installation...
     * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations
@@ Line 78: / Line 85: @@
     * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03''
     * Linux (including WSL) on a **personal PC/Mac desktop/laptop**: find a directory with enough space (including space for adding more packages and their future updates)
+      * You may want to use a directory that you are not backing up, if you want to avoid having to save thousands of files...
   * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh''
@@ Line 145: / Line 153: @@
 </code>
+  * Remove the //Miniconda3// installer: ''$ rm Miniconda3-latest-Linux-x86_64.sh''
+==== Fine-tuning conda to use (only) conda-forge ====
-==== Fine-tuning conda to use conda-forge ====
+  * ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies.
-conda will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies. Besides, conda works much better and faster since the end of 2023, now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver.
+  * ''conda'' works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver.
+    * If you have followed all the steps of this section in order to **use only //conda-forge// packages**, and there are still some problems with a recent version of ''conda'', try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of conda
+    * Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments:
+      * [[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]]
+      * [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]
-We are power users, and **we intend to create complex environments** combining lots of packages available in ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge. **The solution is to configure conda to always use conda-forge**, and completely update Miniconda3 itself with packages coming from conda-forge
+  * We are power users, and **we intend to create complex Python environments** combining lots (and lots) of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge.
+    * => **The solution is to configure conda to always use conda-forge**, and completely update //Miniconda3// itself with packages coming only from conda-forge
 === Changing the .condarc file ===
-Most of the packages we will install will be provided by the //conda-forge// channel (''-c conda-forge'' option). The installation steps will also add dependencies that will theoretically come from //conda-forge//, but could also come from the //defaults// channel.
+The following steps will **make sure that we only get packages from //conda-forge//**  (same thing as using the ''-c conda-forge'' option) by default unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]]
-The following will **make sure that we only get packages from //conda-forge//** unless the requested packages really don't exist on //conda-forge// (see [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] and [[https://conda-forge.org/docs/user/tipsandtricks.html#how-to-fix-it|Using multiple channels => How to fix it]])
-<code>$ conda config --add channels conda-forge
-$ conda config --set channel_priority strict
-$ cat ~/.condarc
-channels:
-  - conda-forge
-  - defaults
-channel_priority: strict</code>
 <code>$ cat ~/.condarc
 cat: /home/jypmce/.condarc: No such file or directory
 $ conda config --get channels
 $ conda config --prepend channels conda-forge
 $ conda config --set channel_priority strict
 $ conda config --get channels
 --add channels 'defaults'   # lowest priority
 --add channels 'conda-forge'   # highest priority
 $ cat ~/.condarc
 channels:
@@ Line 185: / Line 195: @@
   * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation
   * Basic information: ''conda info''
-  * Full configuration: ''conda config --show''
+  * Full configuration: ''conda config %%--%%show''
-==== Updating conda ====
+==== Updating (Mini)conda ====
-  * Update the new installation
+We have to **completely update //Miniconda3// at least once after installing //Miniconda3//** and [[#fine-tuning_conda_to_use_only_conda-forge|making conda-forge the highest priority channel]]
-    * <code>$ conda update --all
-[...]</code>
-    * During the update, the ''miniconda3'' directory size goes from 432 Mb to 581 Mb. This directory will keep on growing, which is the reason why you should put it on a (preferably non backed up) disk where you have //enough// space
-  * Make sure we have the latest ''conda'' package (just in case we did not get it with the //update//)\\ ''conda update -n base conda''
+  * Check the current version of ''conda''
+    * <code>$ conda list -n base | grep conda
+# packages in environment at /path_to_miniconda:
+conda                     24.1.2          py312h06a4308_0
+conda-libmamba-solver     23.12.0            pyhd3eb1b0_1
+[... and other packages with 'conda' in their name]
+</code>
-  * Remove the installer later, when you have tester your installation:\\ ''rm Miniconda3-latest-Linux-x86_64.sh''
+  * Completely update the //Miniconda3// installation
+    * <code>$ conda update -n base --all
+Channels:
+ - conda-forge
+ - defaults
+Platform: linux-64
+Collecting package metadata (repodata.json): done
+Solving environment: done
-===== Initializing conda in new terminals =====
+## Package Plan ##
-When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between environments
+  environment location: /homedata/jypmce/miniconda3_2024-03
-==== General case ====
-You were asked the following question when installing miniconda3: //Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]//
+The following packages will be downloaded:
-  * If you answered //yes//, the installer probably added some very complicated lines to your shell configuration files, but you probably have ''conda'' directly available when you open a new terminal
+    package                    |            build
-  * if you answered //no// (as suggested), use a text editor to add an extra line to the appropriate configuration file
+    ---------------------------|-----------------
-    * //bash// user: add this line to ''~/.bashrc''\\ ''source <installation_path>/miniconda3/etc/profile.d/conda.sh''
+    conda-24.3.0               |  py312h7900ff3_0         1.1 MB  conda-forge
-    * //tcsh// user: add this line to ''~/.cshrc''\\ ''source <installation_path>/miniconda3/etc/profile.d/conda.csh''
+    conda-libmamba-solver-24.1.0|     pyhd8ed1ab_0          40 KB  conda-forge
+    python-3.12.2              |hab00c5b_0_cpython        30.8 MB  conda-forge
+[...]
+The following NEW packages will be INSTALLED:
+[...]
+The following packages will be UPDATED:
+[...]
+  conda              pkgs/main::conda-24.1.2-py312h06a4308~ --> conda-forge::conda-24.3.0-py312h7900ff3_0
+[...]
+The following packages will be SUPERSEDED by a higher-priority channel:
+[...]
+Proceed ([y]/n)? y
+[...]
+Preparing transaction: done
+Verifying transaction: done
+Executing transaction: done
+</code>
+  * Check the updated version of ''conda''. Note that **(almost) all** the packages should now specify that they are provided by ''conda-forge''
+    * <code>$ conda list -n base
+conda                     24.3.0          py312h7900ff3_0    conda-forge
+conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    conda-forge
+python                    3.12.2          hab00c5b_0_cpython    conda-forge
+</code>
+    * You can use the following to determine if some packages are not provided by ''conda-forge''
+      * <code>$ conda list -n base | grep -v conda-forge
+# Name                    Version                   Build  Channel
+libedit                   3.1.20230828         h5eee18b_0
+libffi                    3.4.4                h6a678d5_0
+xz                        5.4.5                h5eee18b_0
+</code>
+      * It should not be a major problem if a few packages are not provided by ''conda-forge''.\\ It only means that some packages provided by the ''anaconda'' //channel// were more recent than the same packages provided by the ''conda-forge'' //channel// when the packages were updated
+        * ''libedit'' example: [[https://anaconda.org/conda-forge/libedit|version 3.1.20191231 @ conda-forge]] and [[https://anaconda.org/anaconda/libedit|version 3.1.20230828 @ anaconda]]
-We choose **not** to add a ''conda activate env_name'' line to the shell configuration files, in order to avoid side effects. When we open a new terminal, we get the default python available on the system. When we need a specific python environment, we just open a new window and then explicitly type: ''conda activate env_name''
+  * You can later update again the full //Miniconda3//, or just the ''conda'' command.\\ This should not make any changes to the Python environment(s) have have installed with ''conda''
+    * <code>$ conda update -n base --all
+[...]
+# All requested packages already installed.
+(base) $ conda update -n base conda
+[...]
+# All requested packages already installed.
+</code>
+  * Note: during this installation, the ''miniconda3'' directory size grew from 763 Mb to 1.6 Gb, and we have not installed any custom Python environment yet! This is the reason why you should install //Miniconda3// on a (preferably non backed up) disk where you have //enough// space
+    * <code>$ du -sh /homedata/jypmce/miniconda3_2024-03
+M    /homedata/jypmce/miniconda3_2024-03
+[... update]
+$ du -sh /homedata/jypmce/miniconda3_2024-03
+.6G    /homedata/jypmce/miniconda3_2024-03
+</code>
+  * You can [[https://docs.conda.io/projects/conda/en/latest/commands/clean.html|clean]] the initial //Miniconda3// installation to free up some disk space. The documentation specifies //WARNING: This will break environments with packages installed using symlinks back to the package cache.// and **you should probably avoid using ''clean'' once you have installed new Python environments** (unless you are desperate for disk space and know how to do a complete re-installation if something breaks...)
+    * <code>(base) $ conda clean --all
+Will remove 154 (180.9 MB) tarball(s).
+Proceed ([y]/n)? y
+Will remove 1 index cache(s).
+Proceed ([y]/n)? y
+Will remove 74 (378.0 MB) package(s).
+Proceed ([y]/n)? y
+There are no tempfile(s) to remove.
+There are no logfile(s) to remove.
+(base) $ du -sh /homedata/jypmce/miniconda3_2024-03
+M    /homedata/jypmce/miniconda3_2024-03
+</code>
+===== Initializing conda in terminals =====
+When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between existing environments. This can be **configured in the //shell// configuration files**
+==== Single-user installation ====
+You were asked the following question when installing //Miniconda3//: //Do you wish to update your shell profile to automatically initialize conda? [yes|no]//
+  * **If you answered //yes//**, the installer probably added some very complicated lines to your shell configuration files, and ''conda'' and the newly installer ''python'' are probably directly available when you open a new terminal.
+  * **If you answered //no//** ([[#using_the_miniconda3_installer|as suggested]]), use a [[other:newppl:starting#text_editors|text editor]] to add extra lines to the appropriate shell configuration file
+    * **bash shell** user : add these lines to ''~/.bashrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.sh
+alias pynit='conda activate base'</code>
+      * e.g. (spiritx):<code>source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
+alias pynit='conda activate base'</code>
+    * **tcsh shell** user: add these lines to ''~/.cshrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.csh''
+alias pynit 'conda activate base'</code>
+      * e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh
+alias pynit 'conda activate base'</code>
+    * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'')
+    * You can also define other aliases, or use another name than ''pynit''!
+We choose **not** to directly add a ''conda activate env_name'' line to the shell configuration files. This would permanently initialize ''env_name'' and may cause future (and obscure) side effects.
+When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type ''conda activate env_name'' or the ''pynit'' alias defined above
 ==== Multi-user installation ====
@@ Line 221: / Line 330: @@
   * ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source  ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}}
-==== Using mamba instead of conda ====
+===== Keeping miniconda3 up-to-date =====
+===== conda resources =====
+Note: some extra details are available on the much older page //Installing and maintaining UV-CDAT with conda//. You can check later the [[other:uvcdat:conda_notes#useful_conda_commands|Useful conda commands]], but the [[https://docs.conda.io/projects/conda/en/latest/index.html|official conda documentation]] and the [[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html|Cheat sheet]] are probably more up-to-date.
+===== FIXME Using mamba instead of conda =====
 Even when using a proper ''.condarc'' in order to get packages coming only from //conda-forge//, ''conda install'' may fail to solve the dependencies after meditating during an enormous time (e.g. [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]), when we try to deal with **complex environments** (i.e. we already have lots of installed packages and we want to add even more packages).

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools