Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:miniconda3_install [2024/07/04 16:28] – [Useful conda commands] Added 'conda doctor' jypeter
+++ other:uvcdat:cdat_conda:miniconda3_install [2025/09/17 14:24] (current) – [Useful conda commands] More export details jypeter
@@ Line 1: / Line 1: @@
 ====== JYP steps for installing Miniconda3 ======
-===== Before installing Python on a shared computer =====
+===== Before installing Python on a shared Linux server =====
 ==== Desperately seeking a specific package ====
-Check if the Python package you are [[https://www.imdb.com/title/tt0089017/|desperately seeking]] is not already installed, before installing your own Python **on shared interactive Linux servers, or on batch computing clusters** !!
+Check if the Python package you are [[https://www.imdb.com/title/tt0089017/|desperately seeking]] is not already installed, before installing your own Python environment **on shared interactive Linux servers, or on batch computing clusters** !!
   * Ask other users who know the local environment
-  * Read the servers/clusters documentation: [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting|Python at LSCE by JYP]], [[https://intranet.lsce.ipsl.fr/informatique/index.php|LSCE]], [[https://documentations.ipsl.fr/spirit/|ESPRI Computing and Data Center]], ...
+  * Read the servers/clusters Python-related documentation: [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting|Python at LSCE by JYP]], [[https://intranet.lsce.ipsl.fr/informatique/index.php|LSCE]], [[https://documentations.ipsl.fr/spirit/spirit_clusters/python_modules.html|IPSL Mesocenter]], ...
   * Use ''[[other:newppl:starting#using_module_to_access_optional_programs|module avail [mod_name]]]'' to find out the Python versions available, and then ''module load'' and ''conda list'' to get more information on the available Python packages
-    * e.g. on ''spiritx'' ([[https://documentations.ipsl.fr/spirit/common/module_command.html|modules on spirit]])): <code>$ module avail python
+    * e.g. on ''spiritx'' ([[https://documentations.ipsl.fr/spirit/common/module_command.html|modules on spirit]])): <code>$ module avail -t python
-python/meso-3.8  python/meso-3.9  python/meso-3.10  python/meso-3.11
+python/meso-3.8
+python/meso-3.9
+python/meso-3.10
+python/meso-3.11
-$ module avail anaconda
+jypmce@spiritx1:~$ module avail -l 2>&1 | grep meso
-anaconda-meso/2022.10    anaconda3-py/2020.11
+anaconda-meso/2022.10                                       2023/04/21 18:02:35
-anaconda-meso/2023.09-0  anaconda3-py/2021.11
+anaconda-meso/2023.09-0                                     2024/01/25 15:07:50
+pangeo-meso/2023.04.15                                      2024/10/18 15:11:30
+pangeo-meso/2024.01.22                                      2024/10/29 15:16:33
+pangeo-meso/2025.01.24                                      2025/02/28 15:45:21
+python/meso-3.8                                             2024/10/18 15:12:15
+python/meso-3.9                                             2024/10/18 15:12:01
+python/meso-3.10                                            2024/10/18 15:11:30
+python/meso-3.11                                            2024/10/29 15:16:33
-$ module avail pangeo
+jypmce@spiritx1:~$ module load pangeo-meso/2025.01.24
-pangeo-meso/2023.04.15  pangeo-meso/2024.01.22
-$ module load pangeo-meso/2024.01.22
+(pangeo-meso-2025.01.24) jypmce@spiritx1:~$ which python
-(pangeo-meso-2024.01.22) $ which python
+/net/nfs/tools/u20/Python/miniforge_24.11.3-0/envs/pangeo-meso-2025.01.24/bin/python
-/net/nfs/tools/u20/Python/miniconda3_py311_23.11.0-2/envs/pangeo-meso-2024.01.22/bin/python
-$ conda list | grep cartopy
+(pangeo-meso-2025.01.24) jypmce@spiritx1:~$ conda list | grep cartopy
-cartopy                   0.22.0          py311h320fe9a_1    conda-forge
+cartopy                   0.24.0          py312hf9745cd_0    conda-forge
 </code>
   * Lots of Python packages do very similar things. Maybe you can use another similar package that is already installed...
-  * If the packages you need are not available, it may be faster and easier to ask the local system administrators if they can install the packages, rather than doing everything yourself
+  * If the packages you need are not available, it may be faster and easier to ask JYP or the local system administrators if they can install the packages in an existing environment, rather than doing everything yourself
 ==== Using an unlisted Python distribution on a jupyter server ====
-If there is a //jupyter notebook server// connected to your system, and a Python environment providing all the packages you need, but that is not directly available on the notebook server, follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local server
+In the case when:
+  * there is a //jupyter notebook server// connected to your Linux cluster (e.g. https://jupyter.lsce.ipsl.fr/)
+  * and an existing Python environment on this cluster providing all the packages you need
+but this Python environment is not directly available on the notebook server (in the listed //kernels//), you should follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local notebook server (on this multi-user computer)
 ===== Why should you use Miniconda3 ? =====
+FIXME: try using [[https://conda-forge.org/download/|Miniforge3]] instead of Miniconda3 ?
   * **//Miniconda3// is a minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command/executable available on your computer, and you can then use this ''conda'' executable to select an existing distribution, or create new and independent Python distributions.
@@ Line 124: / Line 137: @@
       * ''Anaconda Prompt''
       * ''Anaconda Powershell Prompt''
-        * **We will now use this //Anaconda PowerShell// shortcut each time we need to use Miniconda3 or Python**
+        * **We will now use this //Anaconda Powershell// shortcut each time we need to use Miniconda3 or Python**
+      * It is **highly recommended to [[other:win10apps#windows_terminal|install Windows Terminal and make it the default terminal application]]**, so that the ''conda'' prompts will open in a modern (and highly configurable) terminal
   * Remove the installer<code>PS C:\Scratch\your_login> del .\miniconda.exe</code>
@@ Line 438: / Line 452: @@
 <WRAP center round important 60%>
-It seems **safe** to reclaim some disk space by cleaning up the initial (//base//) installation, **when no other environments have been installed yet**\\ \\ Use cleaning with caution after installing other environments...
+It seems **safe** to reclaim some disk space by cleaning up the initial (//base//) installation, **when no other (shared) environments have been installed yet**\\ \\ Use cleaning with caution after installing other environments (besides ''base''), especially environments **shared** by several users. But it should not break your installation
 </WRAP>
-You can [[https://docs.conda.io/projects/conda/en/latest/commands/clean.html|clean]] the initial //Miniconda3// installation to free up some disk space.
+  * ''conda clean %%--%%all''
+    * [[https://docs.conda.io/projects/conda/en/latest/commands/clean.html|Documentation]]
-The documentation specifies //WARNING: This will break environments with packages installed using symlinks back to the package cache.//
+    * There is **no ''-n'' option to specify an environment name**, because ''clean'' will //Remove **unused** packages and caches// shared by **all** the environments
+      * The ''clean'' command will mostly deal with the content of the ''/path_to_miniconda/pkgs/'' directory
-**You should probably avoid using ''clean'' once you have used //Miniconda3// to install new Python environments** (unless you are desperate for disk space, and know how to do a complete re-installation if something breaks...)
+    * Example:<code>(base) $ conda clean --all
-<code>(base) $ conda clean --all
 Will remove 154 (180.9 MB) tarball(s).
 Proceed ([y]/n)? y
@@ Line 463: / Line 475: @@
 M    /homedata/jypmce/miniconda3_2024-03
 </code>
+  * It //should// be safe to use ''conda clean %%--%%all'' even in a multi-user installation with several environments, **if you don't use the ''-f'' option**
+    * The ''-f''/''%%--%%force-pkgs-dirs'' documentation specifies //WARNING: This will break environments with packages installed using symlinks back to the package cache.//
+      * ...but Linux installations probably use [[https://www.redhat.com/sysadmin/linking-linux-explained|hard links]] rather than soft/symbolic links
+      * ...and you will probably never need this option anyway
+  * Remember that **your Python installation will keep on growing and growing and growing, and you will need to do some cleaning at some point**
 ===== Initializing conda in terminals =====
@@ Line 551: / Line 573: @@
     * FIXME: update/rewrite the [[other:uvcdat:cdat_conda:cdat_2024_03]] page
+===== Creating Windows menu shortcuts to start environments =====
+FIXME => Move this section somewhere else?
+  * Use ''conda install'' and what is described in [[https://github.com/conda/menuinst/issues/246|Easily creating shortcuts with the powershell_shortcut_miniconda and console_shortcut_miniconda packages, and can this be replicated with menuinst ?]]
+  * Can theoretically be done with [[https://conda.github.io/menuinst/|menuinst]]
+    * [[https://github.com/conda/menuinst|menuinst @ GH]]
+  * FIXME Find a way to add Windows Terminal profiles
+    * See ''menuinst''+[[https://github.com/conda/menuinst/issues/196|Add Windows Terminal profile option]]
 ===== conda resources =====
@@ Line 556: / Line 590: @@
 ==== Web sites ====
-  * [[https://docs.conda.io/projects/conda/en/stable/|Official conda website]]
+  * [[https://docs.conda.io/projects/conda/en/stable/|Official Documentation site]]
     * [[https://docs.conda.io/projects/conda/en/stable/user-guide/cheatsheet.html|Cheat sheet]] (2 pages, for people in a hurry)
     * [[https://docs.conda.io/projects/conda/en/stable/commands/|Commands' reference]]
-  * Package repositories
+    * [[https://docs.conda.io/projects/conda-build/en/stable/resources/package-spec.html#conda-package-specification|Conda package specification]]
+      * This helps understand (and [[https://github.com/conda/conda/issues/14026|use]]) the content of the ''/path_to_miniconda/pkgs'' directory
+  * [[https://conda.org/blog/|conda community site]]:
+    * [[https://conda.org/blog/|conda Blog]]: ''conda'' new features, etc...
+  * [[https://github.com/conda/conda|conda @ GitHub]]
+  * [[https://conda.org/blog/2024-08-14-conda-ecosystem-explained#conda-package-repository-and-channels|Package repositories]]
     * [[https://conda-forge.org/packages/|conda-forge]] <= recommended source of packages
     * [[https://anaconda.org/|default]]
@@ Line 580: / Line 622: @@
   * **Available environments**
     * ''conda env list''
+      * Same as ''conda info %%--%%envs''
     * The currently active environment has a "*****" character on its line<code>(base) $ conda env list
 # conda environments:
@@ Line 637: / Line 680: @@
   * **Delete an environment**
     * ''conda remove -n crap_environment %%--%%all''
+      * This will remove all packages of ''crap_environment'' and the ''crap_environment'' environment itself
   * **Find out if a package is available** on the //conda-forge// channel (or other channels)
@@ Line 653: / Line 697: @@
     * **Specific versions can be specified** by using the [[#the_conda_matchspec_format|conda MatchSpec format]]
     * ''conda remove package1 package2 packageN''
+      * Same as ''conda //uninstall// package1 package2 packageN''
-  * Display the **dependencies of a package**, and the **packages that need a specific packages**
+    * Matching ''pip'' commands (**only** if the package is not available in conda!)
-    * e.g. the ''requests'' package
+      * ''pip install package''
+      * ''pip uninstall package''
   * **Update a package** in the current, or a specific environment
     * ''conda update -n existing_environment installed_package''
-    * Update only the ''conda'' package (not the full Miniconda3 //base// environment) in the ''base'' environment\\ ''conda update -n base conda''
+    * Update only the ''conda'' package (not the full Miniconda3 //base// environment) in the ''base'' environment
+      * ''conda update -n base conda''
+      * Changes and new ''conda'' features are detailed in the [[https://conda.org/blog/|conda Blog]]
+    * Matching ''pip'' command (**only** if the package is not available in conda!)
+      * ''pip install %%--%%upgrade package''
   * **Completely update an environment** (i.e. update **all** the packages)
     * ''conda update -n existing_environment %%--%%all''
     * [[#the_first_time_we_update_miniconda3|'base' environment update example]]
+  * **Export the description of an environment** (as a checkpoint, or to replicate the environment somewhere else)
+    * There are lots of [[https://docs.conda.io/projects/conda/en/stable/commands/export.html#|export options]] available, but we recommend the following combination of options.
+      * The options used below will create a text file ([[https://en.wikipedia.org/wiki/YAML|YAML]] format) listing the names of the Python packages that were installed with ''conda install -n existing_environment_name package_name''. This file does not specify the packages' version and their dependencies and can be used later to easily reproduce an environment (using the latest available version of the requested packages)
+      * ''conda export -n existing_environment_name %%--%%format yml %%--%%from-history -f some_path/existing_environment_yyyy-mm-dd.yml''
+        * Windows example:\\ ''conda export -n base %%--%%format yml %%--%%from-history -f C:\Users\your_login\Desktop\base_yyyy-mm-dd.yml''
+        * The environment can later be replicated (with the latest version of the requested packages) with:\\ ''conda env create -n existing_environment_clone -f some_path/existing_environment_yyyy-mm-dd.yml''
+    * <wrap hi>''pip'' WARNING!</wrap>
+      * ''conda export'' will only export information about packages installed with ''conda install''. You should also check if you have installed packages with ''pip'' in your environment, and write it down...
+      * Linux/Mac example:
+        * <code>(base) > conda list -n test_env | grep pypi
+sbck                                    1.4.0            pypi_0                   pypi
+xsbck                                   1.0.0            pypi_0                   pypi
+(base) > conda list -n test_env | grep pypi > some_path/test_env_pypi_yyyy-mm-dd.txt</code>
+      * Windows example:
+        * <code>(base) PS C:\> conda list -n test_env | findstr pypi
+brotli                           1.1.0               pypi_0                   pypi
+(base) PS C:\> conda list -n test_env | findstr pypi >  C:\Users\your_login\Desktop\test_env_pypi_yyyy-mm-dd.txt</code>
   * **Determine the relationship between installed packages**
@@ Line 669: / Line 737: @@
       * Note: ''mamba repoquery'' currently [[https://github.com/mamba-org/mamba/issues/3334|only works in the currently active environment]]
     * It can be useful to determine:
-      * the packages that are required in order to use a given package => ''depends''
+      * the packages that are required in order to use a given package => ''mamba repoquery **depends** package''
-      * the packages that need a given package => ''whoneeds''
+        * Without ''mamba repoquery depends'', you would have to follow the steps in [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-pkgs.html#listing-package-dependencies|Listing package dependencies]] :-(
+      * the packages that need a given package => ''mamba repoquery **whoneeds** package''
     * Example: the ''[[https://github.com/tqdm/tqdm|tqdm]]'' package
       * ''conda'' needs ''tqdm'' to display package download and installation progress, and the base dependencies of ''tqdm'' are ''[[https://github.com/tartley/colorama|colorama]]'', and of course ''python''
@@ Line 700: / Line 769: @@
 </code>
-  * Checking the health of an environment
+  * **Checking the //health// of an environment** with ''[[https://docs.conda.io/projects/conda/en/stable/commands/doctor.html|conda doctor]]''
     * Some details in [[https://conda.org/blog/2023-06-01-conda-doctor/|Detecting conda environment inconsistencies easily]]
+      * ''conda doctor'' does not do much yet, and not necessarily in a useful way, but this will hopefully improve over time!
+        * [[https://github.com/conda/conda/issues/14017|'conda doctor' output should be improved]]
+        * [[https://github.com/conda/conda/issues/13574#issuecomment-2214291304|What exactly are altered files?]]
     * ''conda doctor -n base''
     * Example:<code> >conda doctor -n base
@@ Line 717: / Line 789: @@
 </code>
-  * **Cleaning... <wrap em>Dangerous</wrap>**!
+  * **Cleaning...**
-    * ''conda'' does not automatically clean the content of the ''pkgs'' directory
+    * ''conda'' does not automatically clean the content of the ''/path_to_miniconda/pkgs'' directory and the size of ''/path_to_miniconda/'' will keep on growing, as you add new environments, or update existing ones
-    * If you have several environments, ''conda clean'' may remove packages that are not needed in an environment, but are still used in other environments, and **you may end up with a broken installation...**
+    * All the details are in the [[#cleaning_up_miniconda3|Cleaning up Miniconda3...]] section
-    * More details in [[#cleaning_up_miniconda3|Cleaning up Miniconda3...]]
@@ Line 727: / Line 798: @@
   * Read the [[https://docs.conda.io/projects/conda/en/stable/commands/#conda-vs-pip-vs-virtualenv-commands|Conda vs. pip vs. virtualenv commands]] section
-  * Read the [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-environments.html#using-pip-in-an-environment|Using pip in an environment]]
+  * Read the [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-environments.html#using-pip-in-an-environment|Using pip in a conda environment]]
+    * //Issues may arise when using pip and conda together//...
 ==== Installation history ====
@@ Line 737: / Line 809: @@
     * ''base'' history file: ''/path_to_miniconda/conda-meta/history''
     * ''existing_environment'' history file: ''/path_to_miniconda/envs/existing_environment/conda-meta/history''
-    * e.g. installation commands for the ''base'' environment:\\ <code> $ egrep -e 'cmd:' -e '==>' /homedata/jypmce/miniconda3_2024-03/conda-meta/history
+    * e.g. installation commands for the ''base'' environment:\\ <code>$egrep -e 'cmd:' -e '==>' /home/share/unix_files/cdat/miniconda3_2024-03/conda-meta/history
 ==> 2024-02-23 18:39:23 <==
 # cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW
-==> 2024-03-27 16:10:01 <==
+==> 2024-03-27 16:13:08 <==
-# cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all
+# cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda update -n base --all
-==> 2024-03-28 11:38:08 <==
-[...]
+[ ... ]
-==> 2024-04-24 17:47:32 <==
-# cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all</code>
+==> 2025-07-24 17:10:51 <==
+# cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda update -n base --all
+==> 2025-07-28 17:05:44 <==
+# cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda update -n base --all</code>
+    * e.g. installation commands for a more complex //work// environment (once the ''base'' environment is available):\\ <code>$ egrep -e 'cmd:' -e '==>' /home/share/unix_files/cdat/miniconda3_2024-03/envs/bauer_env/conda-meta/history
+==> 2024-06-18 11:16:12 <==
+# cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda create -n bauer_env xcdat
+==> 2024-06-18 11:25:14 <==
+# cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda install cartopy cmocean
+[ ... lots of extra packages and updates ... ]
+# cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda update -n bauer_env --all numpy<2.3
+==> 2025-07-29 16:18:44 <==
+# cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda install -n bauer_env rpy2</code>
     * You can also, use a script and the ''conda'' module available **in the ''base'' environment**: [[https://github.com/conda/conda/issues/4545#issuecomment-469984684|example]]
@@ Line 801: / Line 886: @@
 FIXME
+===== Improving conda and related tools =====
+There is always room for improvement, and this is is some kind of //TODO// section
+==== conda and mamba ideas ====
+  * [[https://github.com/conda/conda/issues/14026|Add a conda command to find the package owning a file]]
+    * The link also shows how to do it with standard Linux tools (find, grep), based on the [[https://docs.conda.io/projects/conda-build/en/stable/resources/package-spec.html#conda-package-specification|Conda package specification]] page
+    * [[https://github.com/kelvinou01/conda-which|conda-which]] seems to work (see [[https://github.com/conda/conda/issues/14026#issuecomment-2268821608|test / example]]) !
+      * Note: ''conda-which'' **has to be (obviously) installed in the ''base'' environment**
 /* standard page footer */