Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:miniconda3_install [2024/03/28 10:40]
jypeter [Updating (Mini)conda] Checking if some packages are not provided by conda-forge
+++ other:uvcdat:cdat_conda:miniconda3_install [2024/04/19 13:49]
jypeter [Miniconda3 on a Windows computer] Improved
@@ Line 21: / Line 21: @@
 /path_to_miniconda/envs/cdatm_py3/bin/python
 </code>
-    * The special environment where //Miniconda3// itself is installed is called ''base''. Note that the ''python'' binary of the ''base'' environment is not located in the same directory hierarchy of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base
+    * The //special// environment where //Miniconda3// itself is installed is called ''base'' (it also used to be called ''root''). Note that the ''python'' binary of the //special// ''base'' environment is not located in the same directory hierarchy as the ''python'' of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base
 (base) $ which python
@@ Line 32: / Line 32: @@
     * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing
-  * You could also start with the full //Anaconda// installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full Anaconda installer** because it requires more disk space than //Miniconda3// at the beginning, and all its packages come from the ''default'' //channel// (or //repository//) provided by the [[https://docs.conda.io/projects/conda/en/latest/glossary.html#conda-repository|conda repository]]. This is not very useful (and could even cause complex dependency problems) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**.
+  * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because:
+    * it requires more disk space than //Miniconda3// during the initial installation,
+    * all the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**
-Note: some extra details are available on the much older page //Installing and maintaining UV-CDAT with conda//. You can check later the [[other:uvcdat:conda_notes#useful_conda_commands|Useful conda commands]], but the [[https://docs.conda.io/projects/conda/en/latest/index.html|official conda documentation]] and the [[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html|Cheat sheet]] are probably more up-to-date.
-===== Installing miniconda3 on a Linux-like computer =====
+===== Downloading Miniconda3, and BASIC installation =====
+==== Miniconda3 on a Windows computer ====
+Some of the steps below are adapted from the [[https://docs.anaconda.com/free/miniconda/#quick-command-line-install|Windows section of "Quick command line install"]] to install a //pure Windows// version of miniconda3
+=== Downloading the Miniconda3 installer on Windows ===
+  * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Windows installer]]
+    * https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe
+    * The installer is based on **Python 3.12.2 and Conda 24.3.0 as of April 2024**
+  * Find some temporary space on Windows
+    * e.g. ''C:\Scratch\your_login'' or whatever you use as a [[other:win10config#local_scratch_folder|local Scratch folder]]
+  * Open a //terminal//, where you can type the required installation commands
+    * Windows //Powershell terminal//: ''Start'' => ''W'' => ''Windows PowerShell'' => ''Windows PowerShell''
+    * [[other:win10apps#windows_terminal|Windows Terminal]]
+  * Use ''curl'' to download the latest installer (78 Mb as of April 2024): <code>PS C: cd C:\Scratch\your_login
+PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe
+PS C:\Scratch\your_login> dir miniconda.exe
+[...]
+-a----        19/04/2024     11:18       81274240 miniconda.exe</code>
+=== BASIC Miniconda3 installation on Windows ===
+  * **Determine the directory where you will install Miniconda3 and all your future Python environments** based on this version of Miniconda3\\ **We will later refer to this location as ''/path_to_miniconda3/''** in the Linux sections
+    * Remember that **you will need a few Gb of available space**
+    * You should **avoid installing Miniconda3 in your Windows //home// directory** (''C:\Users\your_login'' or [[other:win10config#local_home_folder|<some_disk_different_from_c>:\Users\<your_login>]]), if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python environments...
+    * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations
+    * e.g. ''C:\Utils\miniconda3_2024-03''
+  * Launch the installer with<code>PS C:\Scratch\your_login> .\miniconda.exe</code>
+  * In the installer window:
+    * (Review and) accept the License Agreement
+    * Choose to install for //Just Me// (the recommended default choice)
+    * Specify the //Destination Folder// selected above, e.g. ''C:\Utils\miniconda3_2024-03''
+    * Use the pre-selected //Advanced Installation Options//, i.e
+      * //Create start menu shortcutes//
+      * //Register Miniconda3 as my default Python 3.12// <= Useful?
+  * When the installer has finished running, you should have two new shortcuts in the Windows ''Start menu''
+    * ''Anaconda Prompt''
+    * ''Anaconda Powershell Prompt''
+      * **We will now use this //Anaconda PowerShell// shortcut each time we need to use Miniconda3 or Python**
+  * Remove the installer<code>PS C:\Scratch\your_login> del .\miniconda.exe</code>
+==== Miniconda3 on a Linux-like computer ====
 <WRAP center round tip 60%>
-By //Linux-like//, we mean:
+By **//Linux-like//**, we mean:
-  * A native Linux computer (or server)
+  * A **Linux** desktop/laptop
-  * A windows 10 computer with [[other:win10wsl|WSL+Ubuntu installed]]
+    * The installation will be used by **one** person
-  * A Mac where you can use Linux in a terminal
+  * A **Linux** server
+    * The installation will be used by **one or more** persons
+  * A **windows computer with [[other:win10wsl|WSL+Ubuntu installed]]**
+    * You should use the [[#miniconda3_on_a_windows_computer|Windows installer]], if all the packages you need are available for Windows
+    * The installation will be used by **one** person
+  * A **Mac** where you can use Linux in a terminal
+    * The installation will be used by **one** person
 </WRAP>
-==== Downloading the Miniconda3 installer ====
+=== Downloading the Miniconda3 installer ===
   * We use a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the Linux installer]]
     * For the Linux computers we commonly use, we need the //Miniconda3 Linux 64-bit// link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
       * Note: Mac users should use the latest [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|macOS installer]] appropriate for their computer
-    * The installer is based on **Python 3.12 as of March 2024**
+    * The installer is based on **Python 3.12.2 and Conda 24.3.0 as of April 2024**
   * Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]])
@@ Line 63: / Line 122: @@
 [...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh</code>
-==== Using the Miniconda3 installer ====
+=== Using the Miniconda3 installer ===
 We assume below that we are in the [[#downloading_the_installer|directory where we have downloaded the installer]]
   * **Determine the directory where you will install miniconda3 and all the Python environments** based on this version of miniconda3, but **do not create the directory** (the installation script will do it)\\ **We will later refer to this location as ''/path_to_miniconda3/''**
-    * Remember that you will need a few Gb of available space
+    * Remember that **you will need a few Gb of available space**
-    * At LSCE, do not install in a subdirectory of your home directory!
+    * **If you want to share this installation with other users**, the directory has to be readable by these users
+    * At LSCE, **do not** install in a sub-directory of your home directory!
       * On other servers and computers, you should also **avoid installing in your //home// directory**, if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python installation...
     * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations
@@ Line 75: / Line 135: @@
     * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03''
     * Linux (including WSL) on a **personal PC/Mac desktop/laptop**: find a directory with enough space (including space for adding more packages and their future updates)
+      * You may want to use a directory that you are not backing up, if you want to avoid having to save thousands of files...
   * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh''
@@ Line 143: / Line 204: @@
   * Remove the //Miniconda3// installer: ''$ rm Miniconda3-latest-Linux-x86_64.sh''
-==== Fine-tuning conda to use (only) conda-forge ====
+===== Fine-tuning conda to use (only) conda-forge =====
   * ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies.
-  * Luckily, ''conda'' also works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver.
+  * ''conda'' works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver.
     * If you have followed all the steps of this section in order to **use only //conda-forge// packages**, and there are still some problems with a recent version of ''conda'', try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of conda
     * Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments:
@@ Line 153: / Line 215: @@
       * [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]
-  * We are power users, and **we intend to create complex Python environments** combining lots of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge.
+  * We are power users, and **we intend to create complex Python environments** combining lots (and lots) of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge.
     * => **The solution is to configure conda to always use conda-forge**, and completely update //Miniconda3// itself with packages coming only from conda-forge
 === Changing the .condarc file ===
-The following will **make sure that we only get packages from //conda-forge//**  (same thing as using the ''-c conda-forge'' option) unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]]
+The following steps will **make sure that we only get packages from //conda-forge//**  (same thing as using the ''-c conda-forge'' option) by default unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]]
 <code>$ cat ~/.condarc
@@ Line 184: / Line 246: @@
   * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation
   * Basic information: ''conda info''
-  * Full configuration: ''conda config --show''
+  * Full configuration: ''conda config %%--%%show''
 ==== Updating (Mini)conda ====
@@ Line 285: / Line 347: @@
 M    /homedata/jypmce/miniconda3_2024-03
 </code>
-===== Initializing conda in new terminals =====
+===== Initializing conda in terminals =====
-When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between environments
+When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between existing environments. This can be **configured in the //shell// configuration files**
-==== General case ====
+==== Single-user installation ====
-You were asked the following question when installing miniconda3: //Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]//
+You were asked the following question when installing //Miniconda3//: //Do you wish to update your shell profile to automatically initialize conda? [yes|no]//
-  * If you answered //yes//, the installer probably added some very complicated lines to your shell configuration files, but you probably have ''conda'' directly available when you open a new terminal
+  * **If you answered //yes//**, the installer probably added some very complicated lines to your shell configuration files, and ''conda'' and the newly installer ''python'' are probably directly available when you open a new terminal.
-  * if you answered //no// (as suggested), use a text editor to add an extra line to the appropriate configuration file
-    * //bash// user: add this line to ''~/.bashrc''\\ ''source <installation_path>/miniconda3/etc/profile.d/conda.sh''
-    * //tcsh// user: add this line to ''~/.cshrc''\\ ''source <installation_path>/miniconda3/etc/profile.d/conda.csh''
-We choose **not** to add a ''conda activate env_name'' line to the shell configuration files, in order to avoid side effects. When we open a new terminal, we get the default python available on the system. When we need a specific python environment, we just open a new window and then explicitly type: ''conda activate env_name''
+  * **If you answered //no//** ([[#using_the_miniconda3_installer|as suggested]]), use a [[other:newppl:starting#text_editors|text editor]] to add extra lines to the appropriate shell configuration file
+    * **bash shell** user : add these lines to ''~/.bashrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.sh
+alias pynit='conda activate base'</code>
+      * e.g. (spiritx):<code>source /homedata/jypmce/miniconda3_2024-03/etc/profile.d/conda.sh
+alias pynit='conda activate base'</code>
+    * **tcsh shell** user: add these lines to ''~/.cshrc''\\ <code>source /path_to_miniconda3/etc/profile.d/conda.csh''
+alias pynit 'conda activate base'</code>
+      * e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh
+alias pynit 'conda activate base'</code>
+    * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'')
+    * You can also define other aliases, or use another name than ''pynit''!
+We choose **not** to directly add a ''conda activate env_name'' line to the shell configuration files. This would permanently initialize ''env_name'' and may cause future (and obscure) side effects.
+When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type ''conda activate env_name'' or the ''pynit'' alias defined above
 ==== Multi-user installation ====
@@ Line 308: / Line 381: @@
   * ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source  ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}}
-==== Using mamba instead of conda ====
+===== Keeping Miniconda3 up-to-date =====
+  * Just type ''conda update -n base %%--%%all'' as shown in the [[#updating_mini_conda|Updating (Mini)conda]] section
+  * The ''base'' (miniconda3) and the other environments are all independent and can be updated independently
+  * It is up to you to decide if you want to update environments or not
+===== conda resources =====
+==== Web sites ====
+  * [[https://docs.conda.io/projects/conda/en/stable/|Official conda website]]
+    * [[https://docs.conda.io/projects/conda/en/stable/user-guide/cheatsheet.html|Cheat sheet]] (2 pages, for people in a hurry)
+    * [[https://docs.conda.io/projects/conda/en/stable/commands/|Commands' reference]]
+  * Package repositories
+    * [[https://conda-forge.org/packages/|conda-forge]] <= recommended source of packages
+    * [[https://anaconda.org/|default]]
+==== Useful conda commands ====
+This is just a subset of some of the commands, and a subset of their options! For a complete reference, use the [[https://docs.conda.io/projects/conda/en/stable/|official conda website]]
+  * **Get help**
+    * ''conda -h''
+      * ''conda //command// %%--%%help'': help for a specific //command//
+  * **Available environments**
+    * ''conda env list''
+    * The currently active environment has a "*****" character on its line<code>(base) $ conda env list
+# conda environments:
+#
+base                  *  /homedata/jypmce/miniconda3_2024-03</code>
+  * **Activate** an existing environment (available in ''conda env list'') or **deactivate**, in order to go back to the default Python
+    * ''conda activate existing_environment''
+      * ''conda activate'' (without specifying an environment name) will **activate the ''base'' Miniconda3 environment**
+    * Go back to the default Python available on the system
+      * ''conda deactivate''
+    * You can type ''which python'' to determine where is the current ''python'' executable you are using is\\ You should **always know which Python you are using!**
+      * <code>(base) $ which python
+/homedata/jypmce/miniconda3_2024-03/bin/python
+(base) $ conda deactivate
+$ which python
+/usr/bin/python</code>
+    * Note: //before conda ''4.6''//, you had to type ''**source** activate existing_environment'' and ''**source** deactivate''
+  * Get the **list of installed packages**, their version, and where they come from (e.g. ''conda-forge'', ''pip'')
+    * ''conda list [-n existing_environment]''
+  * **Create an environment** and install a few packages and their dependencies
+    * ''conda create -n new_environment_name package1 package2 packageN''
+  * **Delete an environment**
+    * ''conda remove -n crap_environment %%--%%all''
+  * **Find out if a package is available** on the //conda-forge// channel (or other channels)
+    * Note: you should always install a package with ''conda'' instead of ''pip'', if it is available on a //conda channel//
+    * ''conda search package_exact_name''\\ ''conda search '*package_partial_name*'''
+      * Note: ''search'' with wildcards fails in a //tcsh// shell, as of April 2024 ([[https://github.com/conda/conda/issues/9771|CSH install/search fail with '*' variable in an argument]])
+  * **Install and remove packages**\\ Add ''-n target_environment'' to make changes in another environment than the active one
+    * ''conda install package1 package2 packageN''
+      * ''conda'' should already be [[#fine-tuning_conda_to_use_only_conda-forge|configured to use conda-forge by default]] if you have read this page carefully
+      * Add ''-c channel_url_or_alias'' to install from a specific //channel//
+    * ''conda remove package1 package2 packageN''
+  * **Update** just ''conda'' (update the //conda// package, not the full Miniconda3 //base// environment)
+    * ''conda update -n base conda''
+  * **Completely update an environment** (all the packages)
+    * ''conda update -n existing_environment %%--%%all''
+  * **Cleaning... <wrap em>Dangerous</wrap>**!\\ If you have several environments, ''conda clean'' may remove packages that are not needed in an environment, but are still used in other environments, and **you may end up with a broken installation...**\\ ''conda'' does not automatically clean the content of the ''pkgs'' directory!\\ Use the following to //Remove index cache, lock files, tarballs, unused cache packages, and source cache//
+    * ''conda clean %%--%%all''
+==== conda, pip and virtualenv ====
+  * Read the [[https://docs.conda.io/projects/conda/en/stable/commands/#conda-vs-pip-vs-virtualenv-commands|Conda vs. pip vs. virtualenv commands]] section
+  * Read the [[https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-environments.html#using-pip-in-an-environment|Using pip in an environment]]
+==== Installation history ====
+===== Uninstalling Miniconda3 =====
+==== Windows computer ====
-Even when using a proper ''.condarc'' in order to get packages coming only from //conda-forge//, ''conda install'' may fail to solve the dependencies after meditating during an enormous time (e.g. [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]), when we try to deal with **complex environments** (i.e. we already have lots of installed packages and we want to add even more packages).
+Miniconda3 can be removed like a regular Windows application
-Using CDAT as a base environment, and adding a lot of extra packages is a **complex environment**.
+  * Go to the //Apps & features// panel
+    * ''Start'' => ''Settings''  => ''Apps'' => ''Apps & features''
+  * Select ''Miniconda3 py3xxxxx''
+  * Click on ''Uninstall'' and follow the instructions displayed by the uninstaller window\\ This will remove:
+    * The (very big) directory that you specified when you installed //Miniconda3//
+    * The ''conda'' shortcuts from the ''Start Menu''
-The only solution seems to replace ''conda'' with [[https://github.com/mamba-org/mamba|mamba]], a reimplementation of the conda package manager in C++ ([[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]]):
+==== Linux-like computer ====
-  * Install ''mamba'' in the //base// (i.e. //miniconda//) environment\\ ''conda install mamba -n base -c conda-forge''
-  * Use ''mamba'' instead of ''conda'' when installing packages.\\ Theoretically, you don't need to specify ''install -c conda forge'' any more, because ''conda-forge'' is now the default source of new packages\\ ''mamba install lots_of_packages''

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools