other:uvcdat:cdat_conda:miniconda3_install
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
other:uvcdat:cdat_conda:miniconda3_install [2024/03/29 15:51] jypeter [Using the Miniconda3 installer] |
other:uvcdat:cdat_conda:miniconda3_install [2024/04/16 15:27] jypeter [Useful commands] Kept on updating |
||
|---|---|---|---|
| Line 21: | Line 21: | ||
| /path_to_miniconda/envs/cdatm_py3/bin/python | /path_to_miniconda/envs/cdatm_py3/bin/python | ||
| </code> | </code> | ||
| - | * The special environment where //Miniconda3// itself is installed is called ''base''. Note that the ''python'' binary of the ''base'' environment is not located in the same directory hierarchy of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base | + | * The //special// environment where //Miniconda3// itself is installed is called ''base''. Note that the ''python'' binary of the //special// ''base'' environment is not located in the same directory hierarchy as the ''python'' of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base |
| (base) $ which python | (base) $ which python | ||
| Line 32: | Line 32: | ||
| * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing | * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing | ||
| - | * You could also start with the full //Anaconda// installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full Anaconda installer** because it requires more disk space than //Miniconda3// at the beginning, and all its packages come from the ''default'' //channel// (or //repository//) provided by the [[https://docs.conda.io/projects/conda/en/latest/glossary.html#conda-repository|conda repository]]. This is not very useful (and could even cause complex dependency problems) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**. | + | * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because: |
| - | + | * it requires more disk space than //Miniconda3// during the initial installation, | |
| - | Note: some extra details are available on the much older page //Installing and maintaining UV-CDAT with conda//. You can check later the [[other:uvcdat:conda_notes#useful_conda_commands|Useful conda commands]], but the [[https://docs.conda.io/projects/conda/en/latest/index.html|official conda documentation]] and the [[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html|Cheat sheet]] are probably more up-to-date. | + | * all the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems**. |
| ===== Installing miniconda3 on a Linux-like computer ===== | ===== Installing miniconda3 on a Linux-like computer ===== | ||
| <WRAP center round tip 60%> | <WRAP center round tip 60%> | ||
| - | By //Linux-like//, we mean: | + | By **//Linux-like//**, we mean: |
| * A native Linux computer (or server) | * A native Linux computer (or server) | ||
| * A windows 10 computer with [[other:win10wsl|WSL+Ubuntu installed]] | * A windows 10 computer with [[other:win10wsl|WSL+Ubuntu installed]] | ||
| * A Mac where you can use Linux in a terminal | * A Mac where you can use Linux in a terminal | ||
| </WRAP> | </WRAP> | ||
| + | |||
| + | <WRAP center round info 60%> | ||
| + | **//Pure Windows//** Note: you can also follow the steps listed in the [[https://docs.anaconda.com/free/miniconda/#quick-command-line-install|Windows section of "Quick command line install"]] to install a //pure Windows// version of miniconda3, and then follow the steps listed in the next sections of this page.\\ We assume below that the installer is downloaded in the ''C:\Scratch\your_login'' directory, and we install Miniconda3 in a ''C:\Utils\miniconda3_2024-03'' directory. <code>PS C: cd C:\Scratch\your_login | ||
| + | PS C:\Scratch\your_login> curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe | ||
| + | PS C:\Scratch\your_login> .\miniconda.exe | ||
| + | PS C:\Scratch\your_login> del .\miniconda.exe | ||
| + | </code> | ||
| + | </WRAP> | ||
| + | |||
| Line 70: | Line 79: | ||
| * Remember that **you will need a few Gb of available space** | * Remember that **you will need a few Gb of available space** | ||
| * **If you want to share this installation with other users**, the directory has to be readable by these users | * **If you want to share this installation with other users**, the directory has to be readable by these users | ||
| - | * At LSCE, do not install in a subdirectory of your home directory! | + | * At LSCE, **do not** install in a sub-directory of your home directory! |
| * On other servers and computers, you should also **avoid installing in your //home// directory**, if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python installation... | * On other servers and computers, you should also **avoid installing in your //home// directory**, if you want to be able to easily make backups or your //home//, without backing up thousands of files related to your Python installation... | ||
| * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations | * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations | ||
| Line 76: | Line 85: | ||
| * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03'' | * e.g. Linux at **spiritx**: ''/homedata/$USER/miniconda3_2024-03'' | ||
| * Linux (including WSL) on a **personal PC/Mac desktop/laptop**: find a directory with enough space (including space for adding more packages and their future updates) | * Linux (including WSL) on a **personal PC/Mac desktop/laptop**: find a directory with enough space (including space for adding more packages and their future updates) | ||
| + | * You may want to use a directory that you are not backing up, if you want to avoid having to save thousands of files... | ||
| * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh'' | * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh'' | ||
| Line 148: | Line 158: | ||
| * ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies. | * ''conda'' will probably work fine with the default settings **if you create simple new environments** with just one package and its dependencies. | ||
| - | * Luckily, ''conda'' also works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver. | + | * ''conda'' works much better and faster since the end of 2023 (''conda'' versions starting at ''23.10''), now that it is [[https://conda.org/blog/2023-07-05-conda-libmamba-solver-rollout/|using the mamba solver]] instead of the default solver. |
| * If you have followed all the steps of this section in order to **use only //conda-forge// packages**, and there are still some problems with a recent version of ''conda'', try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of conda | * If you have followed all the steps of this section in order to **use only //conda-forge// packages**, and there are still some problems with a recent version of ''conda'', try to install and use [[https://mamba.readthedocs.io/|mamba]], as a drop-in replacement of conda | ||
| * Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments: | * Historical note: before ''conda'' used ''libmamba'', we had to use ''mamba'' (instead of ''conda'') for dealing with our complex Python environments: | ||
| Line 154: | Line 164: | ||
| * [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]] | * [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]] | ||
| - | * We are power users, and **we intend to create complex Python environments** combining lots of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge. | + | * We are power users, and **we intend to create complex Python environments** combining lots (and lots) of packages available from ''conda-forge'', with complex dependencies, and we may run into dependency problems when combining packages coming from both conda and conda-forge. |
| * => **The solution is to configure conda to always use conda-forge**, and completely update //Miniconda3// itself with packages coming only from conda-forge | * => **The solution is to configure conda to always use conda-forge**, and completely update //Miniconda3// itself with packages coming only from conda-forge | ||
| === Changing the .condarc file === | === Changing the .condarc file === | ||
| - | The following will **make sure that we only get packages from //conda-forge//** (same thing as using the ''-c conda-forge'' option) unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] | + | The following steps will **make sure that we only get packages from //conda-forge//** (same thing as using the ''-c conda-forge'' option) by default unless the requested packages really don't exist on //conda-forge//. More details in [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] |
| <code>$ cat ~/.condarc | <code>$ cat ~/.condarc | ||
| Line 185: | Line 195: | ||
| * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation | * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation | ||
| * Basic information: ''conda info'' | * Basic information: ''conda info'' | ||
| - | * Full configuration: ''conda config --show'' | + | * Full configuration: ''conda config %%--%%show'' |
| ==== Updating (Mini)conda ==== | ==== Updating (Mini)conda ==== | ||
| Line 305: | Line 315: | ||
| * e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh | * e.g. (LSCE):<code>source /home/share/unix_files/cdat/miniconda3_2024-03/etc/profile.d/conda.csh | ||
| alias pynit 'conda activate base'</code> | alias pynit 'conda activate base'</code> | ||
| - | * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'' | + | * When you have a custom Python environment installed (e.g. ''my_power_env'', you can update the ''pynit'' alias to initialize this environment instead of the ''base'' environment (e.g. ''conda activate my_power_env'') |
| * You can also define other aliases, or use another name than ''pynit''! | * You can also define other aliases, or use another name than ''pynit''! | ||
| - | We choose **not** to add a ''conda activate env_name'' line to the shell configuration files, in order to avoid future side effects. | + | We choose **not** to directly add a ''conda activate env_name'' line to the shell configuration files. This would permanently initialize ''env_name'' and may cause future (and obscure) side effects. |
| When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type ''conda activate env_name'' or the ''pynit'' alias defined above | When we open a new terminal, we get the default Python available on the system. If we need a specific Python environment, we just open a new window and then explicitly type ''conda activate env_name'' or the ''pynit'' alias defined above | ||
| Line 320: | Line 330: | ||
| * ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}} | * ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}} | ||
| - | ==== Using mamba instead of conda ==== | + | ===== Keeping Miniconda3 up-to-date ===== |
| + | |||
| + | * Just type ''conda update -n base %%--%%all'' as shown in the [[#updating_mini_conda|Updating (Mini)conda]] section | ||
| + | |||
| + | * The ''base'' (miniconda3) and the other environments are all independent and can be updated independently | ||
| + | |||
| + | * It is up to you to decide if you want to update environments or not | ||
| + | ===== conda resources ===== | ||
| + | |||
| + | |||
| + | |||
| + | |||
| + | ==== Web sites ==== | ||
| + | |||
| + | * [[https://docs.conda.io/projects/conda/en/stable/|Official conda website]] | ||
| + | * [[https://docs.conda.io/projects/conda/en/stable/user-guide/cheatsheet.html|Cheat sheet]] (2 pages for people in a hurry) | ||
| + | * [[https://docs.conda.io/projects/conda/en/stable/commands/|Commands' reference]] | ||
| + | |||
| + | |||
| + | ==== Useful commands ==== | ||
| + | |||
| + | * Get help | ||
| + | * ''conda -h'' | ||
| + | * ''conda //command// %%--%%help'': help for a specific //command// | ||
| + | * Available environments | ||
| + | * ''conda env list'' | ||
| + | * Activate an existing environment (available in ''conda env list'') or go back to the default Python | ||
| + | * ''conda activate existing_environment'' | ||
| + | * ''conda activate'' (without specifying an environment name) will activate the ''base'' environment | ||
| + | * Go back to the default Python available on the system | ||
| + | * ''conda deactivate'' | ||
| + | * You can type ''which python'' to determine where is the current Python you are using | ||
| + | * <code>(base) $ which python | ||
| + | /homedata/jypmce/miniconda3_2024-03/bin/python | ||
| + | |||
| + | (base) $ conda deactivate | ||
| + | |||
| + | $ which python | ||
| + | /usr/bin/python</code> | ||
| + | * Note: //before conda ''4.6''//, you had to type ''**source** activate existing_environment'' and ''**source** deactivate'' | ||
| + | * Get the list of installed packages, their version, and where they come from (e.g. ''conda-forge'', ''pip'') | ||
| + | * ''conda list [-n existing_environment]'' | ||
| + | * Create an environment | ||
| + | * ''conda create -n new_environment_name package1 package2 packageN'' | ||
| + | * Delete all packages from an environment **and the environment** itself | ||
| + | * ''conda remove -n crap_environment %%--%%all'' | ||
| + | * Look for a package | ||
| + | * ''conda search package_exact_name''\\ ''conda search '*package_partial_name*''' | ||
| + | * Note: ''search'' with wildcards seems to fail in a tcsh shell, as of April 2024 ([[https://github.com/conda/conda/issues/9771|CSH install/search fail with '*' variable in an argument]]) | ||
| + | * Installing and removing packages\\ Add ''-n target_environment'' to act on another environment than the active one | ||
| + | * ''conda install package1 package2 packageN'' | ||
| + | * Add ''%%--%%channel channel_url_or_alias'' to install from a specific //channel// | ||
| + | * If there is no URL, conda will use the value of ''channel_alias'' that defaults to https://conda.anaconda.org/ | ||
| + | * UV-CDAT relies on packages maintained in [[https://conda-forge.github.io|conda-forge]] ([[http://wesmckinney.com/blog/conda-forge-centos-moment/|why conda-forge?]]) and the installation commands should include the following channel option:\\ ''%%--%%channel conda-forge'' | ||
| + | * ''conda remove package1 package2 packageN'' | ||
| + | * Updating conda (update the //conda// package, not the full Miniconda) | ||
| + | * ''conda update -n base -c defaults conda'' | ||
| + | * Downgrading conda (required for installing UV-CDAT 2.10) | ||
| + | * <code>bash-4.1$ conda install -n root "conda<4.3.13" | ||
| + | Fetching package metadata ......... | ||
| + | Solving package specifications: . | ||
| + | |||
| + | Package plan for installation in environment /home/share/unix_files/cdat/miniconda2: | ||
| + | |||
| + | The following packages will be DOWNGRADED due to dependency conflicts: | ||
| + | |||
| + | conda: 4.3.13-py27_0 --> 4.3.11-py27_0 | ||
| + | |||
| + | Proceed ([y]/n)? y | ||
| + | |||
| + | conda-4.3.11-p 100% |###############################################################| Time: 0:00:00 4.28 MB/s | ||
| + | |||
| + | bash-4.1$ conda info | ||
| + | Current conda install: | ||
| - | Even when using a proper ''.condarc'' in order to get packages coming only from //conda-forge//, ''conda install'' may fail to solve the dependencies after meditating during an enormous time (e.g. [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]), when we try to deal with **complex environments** (i.e. we already have lots of installed packages and we want to add even more packages). | + | platform : linux-64 |
| + | conda version : 4.3.11 | ||
| + | conda is private : False | ||
| + | conda-env version : 4.3.11</code> | ||
| - | Using CDAT as a base environment, and adding a lot of extra packages is a **complex environment**. | + | * conda does not automatically clean the content of the ''pkgs'' directory! Use the following to //Remove index cache, lock files, tarballs, unused cache packages, and source cache// |
| + | * ''conda clean %%--%%all'' | ||
| - | The only solution seems to replace ''conda'' with [[https://github.com/mamba-org/mamba|mamba]], a reimplementation of the conda package manager in C++ ([[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]]): | + | ==== Installation history ==== |
| - | * Install ''mamba'' in the //base// (i.e. //miniconda//) environment\\ ''conda install mamba -n base -c conda-forge'' | + | |
| - | * Use ''mamba'' instead of ''conda'' when installing packages.\\ Theoretically, you don't need to specify ''install -c conda forge'' any more, because ''conda-forge'' is now the default source of new packages\\ ''mamba install lots_of_packages'' | + | |
other/uvcdat/cdat_conda/miniconda3_install.txt · Last modified: 2024/07/08 13:28 by jypeter
