This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:uvcdat:cdat_conda:miniconda3_install [2024/04/10 14:59] jypeter [Using the Miniconda3 installer] |
other:uvcdat:cdat_conda:miniconda3_install [2024/04/17 09:43] jypeter [Useful commands] Improved |
||
---|---|---|---|
Line 32: | Line 32: | ||
* Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing | * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing | ||
- | * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer that will install a much more complete python environment, ready for use.\\ \\ **We choose not to use the full Anaconda installer** because: | + | * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because: |
- | * it requires more disk space than //Miniconda3// at the beginning, | + | * it requires more disk space than //Miniconda3// during the initial installation, |
- | * and all its packages come from the ''default'' //channel// (or //repository//) provided by the [[https://docs.conda.io/projects/conda/en/latest/glossary.html#conda-repository|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**. | + | * all the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**. |
- | + | ||
- | Note: some extra details are available on the much older page //Installing and maintaining UV-CDAT with conda//. You can check later the [[other:uvcdat:conda_notes#useful_conda_commands|Useful conda commands]], but the [[https://docs.conda.io/projects/conda/en/latest/index.html|official conda documentation]] and the [[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html|Cheat sheet]] are probably more up-to-date. | + | |
===== Installing miniconda3 on a Linux-like computer ===== | ===== Installing miniconda3 on a Linux-like computer ===== | ||
Line 87: | Line 85: | ||
* e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03'' | * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03'' | ||
* Linux (including WSL) on a **personal PC/Mac desktop/laptop**: find a directory with enough space (including space for adding more packages and their future updates) | * Linux (including WSL) on a **personal PC/Mac desktop/laptop**: find a directory with enough space (including space for adding more packages and their future updates) | ||
+ | * You may want to use a directory that you are not backing up, if you want to avoid having to save thousands of files... | ||
* Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh'' | * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh'' | ||
Line 159: | Line 158: | ||
* ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies. | * ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies. | ||
- | * Luckily, ''conda'' also works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver. | + | * ''conda'' works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver. |
* If you have followed all the steps of this section in order to **use only //conda-forge// packages**, and there are still some problems with a recent version of ''conda'', try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of conda | * If you have followed all the steps of this section in order to **use only //conda-forge// packages**, and there are still some problems with a recent version of ''conda'', try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of conda | ||
* Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments: | * Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments: | ||
Line 165: | Line 164: | ||
* [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]] | * [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]] | ||
- | * We are power users, and **we intend to create complex Python environments** combining lots of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge. | + | * We are power users, and **we intend to create complex Python environments** combining lots (and lots) of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge. |
* => **The solution is to configure conda to always use conda-forge**, and completely update //Miniconda3// itself with packages coming only from conda-forge | * => **The solution is to configure conda to always use conda-forge**, and completely update //Miniconda3// itself with packages coming only from conda-forge | ||
=== Changing the .condarc file === | === Changing the .condarc file === | ||
- | The following will **make sure that we only get packages from //conda-forge//** (same thing as using the ''-c conda-forge'' option) unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] | + | The following steps will **make sure that we only get packages from //conda-forge//** (same thing as using the ''-c conda-forge'' option) by default unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] |
<code>$ cat ~/.condarc | <code>$ cat ~/.condarc | ||
Line 196: | Line 195: | ||
* [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation | * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation | ||
* Basic information: ''conda info'' | * Basic information: ''conda info'' | ||
- | * Full configuration: ''conda config --show'' | + | * Full configuration: ''conda config %%--%%show'' |
==== Updating (Mini)conda ==== | ==== Updating (Mini)conda ==== | ||
Line 316: | Line 315: | ||
* e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh | * e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh | ||
alias pynit 'conda activate base'</code> | alias pynit 'conda activate base'</code> | ||
- | * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'' | + | * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'') |
* You can also define other aliases, or use another name than ''pynit''! | * You can also define other aliases, or use another name than ''pynit''! | ||
- | We choose **not** to add a ''conda activate env_name'' line to the shell configuration files, in order to avoid future side effects. | + | We choose **not** to directly add a ''conda activate env_name'' line to the shell configuration files. This would permanently initialize ''env_name'' and may cause future (and obscure) side effects. |
When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type ''conda activate env_name'' or the ''pynit'' alias defined above | When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type ''conda activate env_name'' or the ''pynit'' alias defined above | ||
Line 331: | Line 330: | ||
* ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}} | * ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}} | ||
- | ==== Using mamba instead of conda ==== | + | ===== Keeping Miniconda3 up-to-date ===== |
+ | |||
+ | * Just type ''conda update -n base %%--%%all'' as shown in the [[#updating_mini_conda|Updating (Mini)conda]] section | ||
+ | |||
+ | * The ''base'' (miniconda3) and the other environments are all independent and can be updated independently | ||
+ | |||
+ | * It is up to you to decide if you want to update environments or not | ||
+ | ===== conda resources ===== | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ==== Web sites ==== | ||
+ | |||
+ | * [[https://docs.conda.io/projects/conda/en/stable/|Official conda website]] | ||
+ | * [[https://docs.conda.io/projects/conda/en/stable/user-guide/cheatsheet.html|Cheat sheet]] (2 pages, for people in a hurry) | ||
+ | * [[https://docs.conda.io/projects/conda/en/stable/commands/|Commands' reference]] | ||
+ | * Package repositories | ||
+ | * [[https://conda-forge.org/packages/|conda-forge]] <= recommended source of packages | ||
+ | * [[https://anaconda.org/|default]] | ||
+ | |||
+ | |||
+ | ==== Useful commands ==== | ||
+ | |||
+ | * Get help | ||
+ | * ''conda -h'' | ||
+ | * ''conda //command// %%--%%help'': help for a specific //command// | ||
+ | * Available environments | ||
+ | * ''conda env list'' | ||
+ | * Activate an existing environment (available in ''conda env list'') or go back to the default Python | ||
+ | * ''conda activate existing_environment'' | ||
+ | * ''conda activate'' (without specifying an environment name) will **activate the ''base'' Miniconda3 environment** | ||
+ | * Go back to the default Python available on the system | ||
+ | * ''conda deactivate'' | ||
+ | * You can type ''which python'' to determine where is the current ''python'' executable you are using is\\ You should **always know which Python you are using!** | ||
+ | * <code>(base) $ which python | ||
+ | /homedata/jypmce/miniconda3_2024-03/bin/python | ||
+ | |||
+ | (base) $ conda deactivate | ||
+ | |||
+ | $ which python | ||
+ | /usr/bin/python</code> | ||
+ | * Note: //before conda ''4.6''//, you had to type ''**source** activate existing_environment'' and ''**source** deactivate'' | ||
+ | * Get the list of installed packages, their version, and where they come from (e.g. ''conda-forge'', ''pip'') | ||
+ | * ''conda list [-n existing_environment]'' | ||
+ | * Create an environment and install a few packages and their dependencies | ||
+ | * ''conda create -n new_environment_name package1 package2 packageN'' | ||
+ | * Delete all packages from an environment **and the environment** itself | ||
+ | * ''conda remove -n crap_environment %%--%%all'' | ||
+ | * Look for a package | ||
+ | * ''conda search package_exact_name''\\ ''conda search '*package_partial_name*''' | ||
+ | * Note: ''search'' with wildcards seems to fail in a tcsh shell, as of April 2024 ([[https://github.com/conda/conda/issues/9771|CSH install/search fail with '*' variable in an argument]]) | ||
+ | * Installing and removing packages\\ Add ''-n target_environment'' to act on another environment than the active one | ||
+ | * ''conda install package1 package2 packageN'' | ||
+ | * ''conda'' should already be [[#fine-tuning_conda_to_use_only_conda-forge|configured to use conda-forge by default]] if you have read this page carefully | ||
+ | * Add ''-c channel_url_or_alias'' to install from a specific //channel// | ||
+ | * ''conda remove package1 package2 packageN'' | ||
+ | * Updating conda (update the //conda// package, not the full Miniconda) | ||
+ | * ''conda update -n base -c defaults conda'' | ||
+ | * Downgrading conda (required for installing UV-CDAT 2.10) | ||
+ | * <code>bash-4.1$ conda install -n root "conda<4.3.13" | ||
+ | Fetching package metadata ......... | ||
+ | Solving package specifications: . | ||
+ | |||
+ | Package plan for installation in environment /home/share/unix_files/cdat/miniconda2: | ||
+ | |||
+ | The following packages will be DOWNGRADED due to dependency conflicts: | ||
+ | |||
+ | conda: 4.3.13-py27_0 --> 4.3.11-py27_0 | ||
+ | |||
+ | Proceed ([y]/n)? y | ||
+ | |||
+ | conda-4.3.11-p 100% |###############################################################| Time: 0:00:00 4.28 MB/s | ||
+ | |||
+ | bash-4.1$ conda info | ||
+ | Current conda install: | ||
- | Even when using a proper ''.condarc'' in order to get packages coming only from //conda-forge//, ''conda install'' may fail to solve the dependencies after meditating during an enormous time (e.g. [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]), when we try to deal with **complex environments** (i.e. we already have lots of installed packages and we want to add even more packages). | + | platform : linux-64 |
+ | conda version : 4.3.11 | ||
+ | conda is private : False | ||
+ | conda-env version : 4.3.11</code> | ||
- | Using CDAT as a base environment, and adding a lot of extra packages is a **complex environment**. | + | * conda does not automatically clean the content of the ''pkgs'' directory! Use the following to //Remove index cache, lock files, tarballs, unused cache packages, and source cache// |
+ | * ''conda clean %%--%%all'' | ||
- | The only solution seems to replace ''conda'' with [[https://github.com/mamba-org/mamba|mamba]], a reimplementation of the conda package manager in C++ ([[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]]): | + | ==== Installation history ==== |
- | * Install ''mamba'' in the //base// (i.e. //miniconda//) environment\\ ''conda install mamba -n base -c conda-forge'' | + | |
- | * Use ''mamba'' instead of ''conda'' when installing packages.\\ Theoretically, you don't need to specify ''install -c conda forge'' any more, because ''conda-forge'' is now the default source of new packages\\ ''mamba install lots_of_packages'' | + | |