other:uvcdat:cdat_conda:miniconda3_install
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
other:uvcdat:cdat_conda:miniconda3_install [2021/03/09 10:38] jypeter [Fine-tuning conda] |
other:uvcdat:cdat_conda:miniconda3_install [2024/03/22 16:04] jypeter [Using the installer] Updated for 2024-03 installation |
||
|---|---|---|---|
| Line 3: | Line 3: | ||
| ===== What? Why? ===== | ===== What? Why? ===== | ||
| - | * //Miniconda3// is a minimal python environment/distribution that can be used for creating more complete work environments. It will basically make the ''conda'' command available on your computer, and you can then use ''conda'' to install and update more python packages and their dependencies. | + | * //Miniconda3// is a minimal/bootstrap Python distribution that can be used for creating more complex Python distributions. It will basically make a recent ''conda'' command available on your computer, and you can then use this ''conda'' to create independent Python distributions, where you can install and update Python packages and their dependencies. |
| - | * A python //environment// is basically where you install python. A python //distribution// is the collection of packages you have chosen to install together in the environment. | + | * A conda //environment// is basically where you install the Python that you will use (independently of //Miniconda3// itself). A python //distribution// is the collection of packages you have chosen to install together in this environment. |
| + | * Example: initializing an environment named ''cdatm_py3''<code> >which conda | ||
| + | conda: aliased to source /path_to_miniconda/etc/profile.d/conda.csh | ||
| - | * You don't need to be (and **you should not be**) //root// when you install Miniconda3. You just need enough disk space where you have write access | + | > conda activate cdatm_py3 |
| - | * If you are installing Miniconda3 in a Linux environment on a [[other:win10wsl|Windows 10 computer using Windows Subsystem for Linux (WSL)]], pay special attention to the instructions on the **WSL** lines | + | (cdatm_py3) > which python |
| + | /path_to_miniconda/envs/cdatm_py3/bin/python | ||
| + | </code> | ||
| + | * The special environment where //miniconda// itself is installed is called ''base''<code> > conda activate base | ||
| + | |||
| + | (base) > which python | ||
| + | /path_to_miniconda/bin/python | ||
| + | </code> | ||
| + | |||
| + | * You don't need to be (and **you should not be**) //root// when you install Miniconda3. You just need enough disk space where you have write access | ||
| + | * **WARNING**: by default, miniconda will install itself and subsequent Python environments in a hidden sub-directory of your home (''~/.conda'' on Linux). **Do not use the default installation location**, if you want to avoid disk space related problems (//disk full//, q//uota exceeded//, ...) | ||
| + | * Choose carefully where you will install miniconda, because the size of the installation directory will start at a few Gb and will keep on growing | ||
| - | * The way you do things will depend on how you are going to use python. There may be some slight differences if you are installing the environment just for you, or if you need a more stable environment because you are installing for multiple users. | + | * If you are installing Miniconda3 in a Linux environment on a [[other:win10wsl|Windows 10 computer using Windows Subsystem for Linux (WSL)]], pay special attention to the specific instructions on the **WSL** lines |
| - | * You could also start with the //Anaconda// installer that will install a much more complete python environment, ready for use, but we choose not to do that because Anaconda requires more disk space at the beginning, and all its packages come from the ''default'' //channel// (or //repository//) provided by the [[https://docs.conda.io/projects/conda/en/latest/glossary.html#conda-repository|conda repository]]. This is not very useful because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] channel. | + | * You could also start with the full //Anaconda// installer that will install a much more complete python environment, ready for use. We choose not to do that because Anaconda requires more disk space at the beginning, and all its packages come from the ''default'' //channel// (or //repository//) provided by the [[https://docs.conda.io/projects/conda/en/latest/glossary.html#conda-repository|conda repository]]. This is not very useful because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//. |
| Note: some extra details are available on the much older page //Installing and maintaining UV-CDAT with conda//. You can check later the [[other:uvcdat:conda_notes#useful_conda_commands|Useful conda commands]], but the [[https://docs.conda.io/projects/conda/en/latest/index.html|official conda documentation]] and the [[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html|Cheat sheet]] are probably more up-to-date. | Note: some extra details are available on the much older page //Installing and maintaining UV-CDAT with conda//. You can check later the [[other:uvcdat:conda_notes#useful_conda_commands|Useful conda commands]], but the [[https://docs.conda.io/projects/conda/en/latest/index.html|official conda documentation]] and the [[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html|Cheat sheet]] are probably more up-to-date. | ||
| Line 27: | Line 40: | ||
| - | * Get the [[https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh|Python 3.8 Linux 64-bit (bash installer)]] (unless there is a [[https://conda.io/miniconda.html|more recent version?]]) | + | ==== Downloading the installer ==== |
| - | * Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]]) | + | |
| - | * e.g. Linux at LSCE:\\ ''cd /home/scratch01/<your_login>'' | + | |
| - | * e.g. WSL, assuming that there is a ''C:\Scratch\<your_login>'' directory:\\ ''cd /mnt/c/Scratch/<your_login>'' | + | |
| - | * Use ''wget'' to download the installer:\\ ''wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'' (90 Mb, 26 Feb 2021) | + | |
| - | * Execute the installer | + | * We will work with a download link that always point to the [[https://docs.anaconda.com/free/miniconda/#latest-miniconda-installer-links|latest version of the installer]] |
| - | * ''bash Miniconda3-latest-Linux-x86_64.sh'' | + | * For the Linux computers we commonly use, we need the //Miniconda3 Linux 64-bit// link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh |
| - | * Accept the license | + | * The installer is based on **Python 3.12 as of March 2024** |
| - | * Note: at the end of the installation, **answer //no// to the following question**, so that the installer does **not** change your existing shell configuration files!\\ ''Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]'' => **NO** | + | |
| - | * Specify an explicit installation path **outside of your //home// directory**, with enough disk space (more than 3 Gb if you are going to install CDAT and some extra packages), preferably on a disk that is not backed up: | + | |
| - | * Installations by //JYP//: | + | |
| - | * Linux at LSCE: ''/home/share/unix_files/cdat/miniconda3<possibly_some_version_here>/'' | + | |
| - | * Linux on ciclad: ''/data/jypmce/cdat/miniconda3<possibly_some_version_here>/'' | + | |
| - | * WSL: installing outside of your ''/home/your_login/'' directory <wrap hi>does not work</wrap>. You need to accept the installation in the default location: ''/home/<your_login>/miniconda3'' | + | |
| - | * The resulting ''miniconda3'' directory size is 342M\\ <code> > du -sh miniconda3 | + | |
| - | 342M miniconda3 | + | |
| - | > cd miniconda3 | + | * Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]]) |
| + | * e.g. Linux at LSCE:\\ ''$ cd /home/scratch01/$USER'' | ||
| + | * e.g. Linux at spiritx:<code>$ mkdir /homedata/$USER/Scratch | ||
| + | $ cd /homedata/$USER/Scratch | ||
| + | </code> | ||
| + | * e.g. WSL, assuming that there is a ''C:\Scratch\<your_login>'' directory:\\ ''cd /mnt/c/Scratch/<your_login>'' | ||
| - | > du -sh * | + | * Use ''wget'' to download the latest installer (138 Mb as of 22 Mar 2024):\\ <code>$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh |
| - | 20M bin | + | $ ls -lh Miniconda3-latest-Linux-x86_64.sh |
| - | 0 compiler_compat | + | [...] 138M Feb 27 20:40 Miniconda3-latest-Linux-x86_64.sh</code> |
| - | 4.0K condabin | + | |
| - | 684K conda-meta | + | |
| - | 0 envs | + | |
| - | 16K etc | + | |
| - | 5.5M include | + | |
| - | 4.0K info | + | |
| - | 198M lib | + | |
| - | 12K LICENSE.txt | + | |
| - | 114M pkgs | + | |
| - | 604K share | + | |
| - | 4.0K shell | + | |
| - | 0 ssl | + | |
| - | 0 x86_64-conda_cos6-linux-gnu</code> | + | |
| - | * Initialize the newly installed conda environment (this will initialize the environment only in the current terminal): | + | ==== Using the installer ==== |
| - | * bash shell: ''source <installation_path>/miniconda3/etc/profile.d/conda.sh'' | + | |
| - | * tcsh shell: ''source <installation_path>/miniconda3/etc/profile.d/conda.csh'' | + | We assume below that we are in the [[#downloading_the_installer|directory where we have downloaded the installer]] |
| + | |||
| + | * Determine the directory where you will install miniconda3 and all the Python environments based on this version of miniconda3, but **do not create the directory** (the installation script will do it) | ||
| + | * Remember that you will need a few Gb of available space | ||
| + | * At LSCE, do not install in a subdirectory of your home directory! | ||
| + | * We add some date or version information at the end of the directory, in order to differentiate it from other (older of future) installations | ||
| + | * e.g. Linux at LSCE: ''/home/share/unix_files/cdat/miniconda3_2024-03'' | ||
| + | * e.g. Linux at spiritx: ''/homedata/$USER/miniconda3_2024-03'' | ||
| + | * e.g. WSL FIXME | ||
| + | |||
| + | * Execute the installer with ''$ bash Miniconda3-latest-Linux-x86_64.sh'' | ||
| + | * Review the license (type ''<SPACE>'' several times...) and accept it | ||
| + | * When asked for the installation directory, use the directory name chosen above, and **not the default directory** | ||
| + | * **Answer ''no'' to the question ''Do you wish to update your shell profile to automatically initialize conda?''**. Otherwise the installation will make changes to your shell configuration files | ||
| + | * If you forget to answer ''no'', you can apparently: | ||
| + | * remove the changes to your shell configuration files by typing later: ''conda init --reverse $SHELL'' | ||
| + | * or at least disable the automatic activation of the environment (that might have side effect) by typing: ''conda config --set auto_activate_base false'' | ||
| + | * The resulting ''miniconda3'' directory size is 647 Mb (as of March 2024)\\ <code>$ du -sh miniconda3_2024-03 | ||
| + | 647M miniconda3_2024-03 | ||
| + | |||
| + | $ cd miniconda3_2024-03 | ||
| + | |||
| + | $ du -sh * | ||
| + | 47M bin | ||
| + | 16K cmake | ||
| + | 8.0K compiler_compat | ||
| + | 32M _conda | ||
| + | 8.0K condabin | ||
| + | 968K conda-meta | ||
| + | 4.0K envs | ||
| + | 28K etc | ||
| + | 18M include | ||
| + | 317M lib | ||
| + | 92K LICENSE.txt | ||
| + | 1.1M man | ||
| + | 231M pkgs | ||
| + | 396K sbin | ||
| + | 1.4M share | ||
| + | 12K shell | ||
| + | 8.0K ssl | ||
| + | 8.0K x86_64-conda_cos7-linux-gnu | ||
| + | 8.0K x86_64-conda-linux-gnu | ||
| + | </code> | ||
| + | |||
| + | * Initialize the newly installed conda environment (this will initialize the environment **only in the current terminal**): | ||
| + | * bash shell: ''source <installation_path>/miniconda3_2024-03/etc/profile.d/conda.sh'' | ||
| + | * tcsh shell: ''source <installation_path>/miniconda3_2024-03/etc/profile.d/conda.csh'' | ||
| * Check if you can use the ''conda'' command, and use it to initialize the //base// environment | * Check if you can use the ''conda'' command, and use it to initialize the //base// environment | ||
| Line 124: | Line 163: | ||
| Most of the packages we will install will be provided by the //conda-forge// channel (''-c conda-forge'' option). The installation steps will also add dependencies that will theoretically come from //conda-forge//, but could also come from the //defaults// channel. | Most of the packages we will install will be provided by the //conda-forge// channel (''-c conda-forge'' option). The installation steps will also add dependencies that will theoretically come from //conda-forge//, but could also come from the //defaults// channel. | ||
| - | The following will **make sure that we only get packages from //conda-forge//** (see [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] and [[https://conda-forge.org/docs/user/tipsandtricks.html#how-to-fix-it|Using multiple channels => How to fix it]]) | + | The following will **make sure that we only get packages from //conda-forge//** unless the requested packages really don't exist on //conda-forge// (see [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] and [[https://conda-forge.org/docs/user/tipsandtricks.html#how-to-fix-it|Using multiple channels => How to fix it]]) |
| <code>$ conda config --add channels conda-forge | <code>$ conda config --add channels conda-forge | ||
| $ conda config --set channel_priority strict | $ conda config --set channel_priority strict | ||
| - | $ cat .condarc | + | $ cat ~/.condarc |
| channels: | channels: | ||
| - conda-forge | - conda-forge | ||
| Line 136: | Line 175: | ||
| ==== Using mamba instead of conda ==== | ==== Using mamba instead of conda ==== | ||
| - | Even when using a proper ''.condarc'' in order to get packages coming only from //conda-forge//, ''conda install'' may fail to solve the dependencies after meditating during an enormous time, when we specify **complex environments**. Using CDAT as a base environment, and adding a lot of extra packages is a complex environment. | + | Even when using a proper ''.condarc'' in order to get packages coming only from //conda-forge//, ''conda install'' may fail to solve the dependencies after meditating during an enormous time (e.g. [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]), when we try to deal with **complex environments** (i.e. we already have lots of installed packages and we want to add even more packages). |
| + | |||
| + | Using CDAT as a base environment, and adding a lot of extra packages is a **complex environment**. | ||
| - | The only solution seems to replace ''conda'' with [[https://github.com/mamba-org/mamba|mamba]], a reimplementation of the conda package manager in C++ ([[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]]) | + | The only solution seems to replace ''conda'' with [[https://github.com/mamba-org/mamba|mamba]], a reimplementation of the conda package manager in C++ ([[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]]): |
| + | * Install ''mamba'' in the //base// (i.e. //miniconda//) environment\\ ''conda install mamba -n base -c conda-forge'' | ||
| + | * Use ''mamba'' instead of ''conda'' when installing packages.\\ Theoretically, you don't need to specify ''install -c conda forge'' any more, because ''conda-forge'' is now the default source of new packages\\ ''mamba install lots_of_packages'' | ||
other/uvcdat/cdat_conda/miniconda3_install.txt · Last modified: 2024/07/09 15:54 by jypeter
