| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| other:uvcdat:cdat_conda:miniforge_install [2025/10/06 16:18] – [Miniforge3 on a Windows computer] Improved jypeter | other:uvcdat:cdat_conda:miniforge_install [2026/10/19 11:32] (current) – [Useful conda commands] jypeter |
|---|
| * and an existing Python environment on this cluster providing all the packages you need | * and an existing Python environment on this cluster providing all the packages you need |
| **but** this Python environment is not directly available on the notebook server (i.e. not listed in the //kernels// pull-down menu), you should follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local notebook server (on this multi-user computer) | **but** this Python environment is not directly available on the notebook server (i.e. not listed in the //kernels// pull-down menu), you should follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local notebook server (on this multi-user computer) |
| ===== Why should you use Miniconda3 ? ===== | ===== Why should you use Miniforge3 ? ===== |
| |
| FIXME: try using [[https://conda-forge.org/download/|Miniforge3]] instead of Miniconda3 ? | <note tip> |
| | * **You should use [[https://conda-forge.org/download/|Miniforge3]]** instead of ''Miniconda3'' or ''Anaconda3'' because it is not provided by the Anaconda company and is **completely free** |
| | * Check the [[https://www.anaconda.com/legal|Anaconda Legal]] page for more details |
| | * People with an existing ''Miconconda3'' based installation can have a look at the old [[other:uvcdat:cdat_conda:miniconda3_install|JYP steps for installing Miniconda3]] page |
| | </note> |
| |
| * **//Miniconda3// is a minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command/executable available on your computer, and you can then use this ''conda'' executable to select an existing distribution, or create new and independent Python distributions. | * **//Miniforge3// provides a free minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command/executable available on your computer, and you can then use this ''conda'' executable to create new and independent Python distributions, or select an existing distribution. |
| |
| * A **conda //environment//** is basically (the short name of the directory) where you install the Python that you will use (independently of //Miniconda3// itself). You have to ''activate'' this environment in order to use it\\ \\ A **Python //distribution//** is the collection of packages (and their dependencies) you have chosen to install together in a given environment. | * A **conda //environment//** is basically (the short name of the directory) where you install the Python that you will use (independently of //Miniforge3// itself). You have to ''activate'' this environment in order to use it\\ \\ A **Python //distribution//** is the collection of packages (and their dependencies) you have chosen to install together in a given environment. |
| * Example: initializing an environment named ''cdatm_py3'' (assuming it exists) in a //bash// shell <code>$ source /path_to_miniconda/etc/profile.d/conda.sh | * Example: initializing an environment named ''cdatm_py3'' (assuming it exists) in a //bash// shell <code>$ source /path_to_miniforge/etc/profile.d/conda.sh |
| |
| $ which conda | $ which conda |
| /path_to_miniconda/condabin/conda | /path_to_miniforge/condabin/conda |
| |
| $ conda env list | $ conda env list |
| [...] | [...] |
| cdatm_py3 /path_to_miniconda/envs/cdatm_py3 | cdatm_py3 /path_to_miniforge/envs/cdatm_py3 |
| some_other_env /path_to_miniconda/envs/some_other_env | some_other_env /path_to_miniforge/envs/some_other_env |
| |
| $ conda activate cdatm_py3 | $ conda activate cdatm_py3 |
| |
| (cdatm_py3) $ which python | (cdatm_py3) $ which python |
| /path_to_miniconda/envs/cdatm_py3/bin/python | /path_to_miniforge/envs/cdatm_py3/bin/python |
| </code> | </code> |
| * The //special// environment where //Miniconda3// itself is installed is called ''base'' (it also used to be called ''root''). Note that the ''python'' binary of the //special// ''base'' environment is not located in the same directory hierarchy as the ''python'' of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base | * The //special// environment where //Miniforge3// itself is installed is called ''base'' (it also used to be called ''root''). Note that the ''python'' binary of the //special// ''base'' environment is not located in the same directory hierarchy as the ''python'' of the other environments (i.e. there is no ''envs/'' subdirectory<code>$ conda activate base |
| |
| (base) $ which python | (base) $ which python |
| /path_to_miniconda/bin/python | /path_to_miniforge/bin/python |
| </code> | </code> |
| |
| * You **don't need to be (and you should not be !!) //root//** when you install //Miniconda3//. You just **need //enough// disk space** on a disk where you have write access | * You **don't need to be (and you should not be !!) //root//** when you install //Miniforge3//. You just **need //enough// disk space** on a disk where you have write access |
| * **WARNING**: by default, //Miniconda3// will install itself and subsequent Python environments in a sub-directory of your home directory (''~/miniconda3'' on Linux). **Do not use the default installation location**, if you want to avoid disk space related problems (//disk full//, //quota exceeded//, ...) during the inittial installation or later | * **WARNING**: by default, //Miniforge3// will install itself and subsequent Python environments in a sub-directory of your home directory (''~/miniforge3'' on Linux). **Do not use the default installation location**, if you want to avoid disk space related problems (//disk full//, //quota exceeded//, ...) during the initial installation or later |
| * **WARNING**: older versions of //Miniconda3// used a **hidden sub-directory** of the home directory of the person installing //Miniconda3// (''~/.conda'' on Linux) | * **WARNING**: older versions of //Miniforge3// used a **hidden sub-directory** of the home directory of the person installing //Miniforge3// (''~/.conda'' on Linux) |
| * Choose carefully where you will install //Miniconda3//, because the size of the installation directory will start at a few Gb and will keep on growing | * Choose carefully where you will install //Miniforge3//, because the size of the installation directory will start at a few Gb and will keep on growing |
| | |
| * You //could// also start with the full //Anaconda// (instead of //Miniconda//) installer, that will install a much more complete python environment, ready for use.\\ **We choose not to use the full //Anaconda// installer** because: | |
| * It requires more disk space than //Miniconda3// during the initial installation, | |
| * All the //Anaconda// installer packages come from the ''default'' //channel// (or //repository//) provided by the [[https://anaconda.org/|conda repository]].\\ This is not very useful for us (and could even cause complex dependency problems later) because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] //channel//, **in order to avoid complex package dependency problems** | |
| |
| |
| |
| <WRAP center round important 60%> | <WRAP center round important 60%> |
| It seems **safe** to reclaim some disk space by cleaning up the initial (//base//) installation, **when no other (shared) environments have been installed yet**\\ \\ Use cleaning with caution after installing other environments (besides ''base''), especially environments **shared** by several users. But it should not break your installation | * **''conda'' does not remove old/unused packages**, and a distribution with lots of packages may grow a lot, and exceed disk and/or files quotas, if you do not clean it from time to time |
| | |
| | * Use cleaning with caution after installing other environments (besides ''base''), especially environments **shared** by several users. But it should not break your installation |
| </WRAP> | </WRAP> |
| |
| * There is **no ''-n'' option to specify an environment name**, because ''clean'' will //Remove **unused** packages and caches// shared by **all** the environments | * There is **no ''-n'' option to specify an environment name**, because ''clean'' will //Remove **unused** packages and caches// shared by **all** the environments |
| * The ''clean'' command will mostly deal with the content of the ''/path_to_miniconda/pkgs/'' directory | * The ''clean'' command will mostly deal with the content of the ''/path_to_miniconda/pkgs/'' directory |
| * Example:<code>(base) $ conda clean --all | * Example 1: cleaning during the initial installation<code>(base) $ conda clean --all |
| Will remove 154 (180.9 MB) tarball(s). | Will remove 154 (180.9 MB) tarball(s). |
| Proceed ([y]/n)? y | Proceed ([y]/n)? y |
| (base) $ du -sh /homedata/jypmce/miniconda3_2024-03 | (base) $ du -sh /homedata/jypmce/miniconda3_2024-03 |
| 457M /homedata/jypmce/miniconda3_2024-03 | 457M /homedata/jypmce/miniconda3_2024-03 |
| | </code> |
| | * Example 2: **exceeding files quotas** (in the same installation) after forgetting to clean regularly<code>$ quotas |
| | Filesystem used quota limit grace files quota limit grace |
| | /homedata 16.59G 1T 1.1T - 339671* 300000 350000 5d39m28s |
| | |
| | $ conda clean --all |
| | Will remove 563 (1.85 GB) tarball(s). |
| | Proceed ([y]/n)? y |
| | |
| | Will remove 1 index cache(s). |
| | Proceed ([y]/n)? y |
| | |
| | Will remove 592 (8.25 GB) package(s). |
| | Proceed ([y]/n)? y |
| | |
| | There are no tempfile(s) to remove. |
| | There are no logfile(s) to remove. |
| | |
| | $ quotas |
| | Filesystem used quota limit grace files quota limit grace |
| | /homedata 5.291G 1T 1.1T - 141557 300000 350000 - |
| | |
| | $ du -sh /homedata/jypmce/miniconda3_2024-03 |
| | 5.2G /homedata/jypmce/miniconda3_2024-03 |
| </code> | </code> |
| |
| * Same as ''conda //uninstall// package1 package2 packageN'' | * Same as ''conda //uninstall// package1 package2 packageN'' |
| * Matching ''pip'' commands (**only** if the package is not available in conda!) | * Matching ''pip'' commands (**only** if the package is not available in conda!) |
| * ''pip install package'' | * **First**: |
| * ''pip uninstall package'' | * Try: ''conda pypi install package'' |
| | * More details why you should do that in the [[#conda_pip_and_virtualenv|conda vs pip]] section |
| | * Otherwise: |
| | * ''pip install package'' |
| | * ''pip uninstall package'' |
| |
| * **Update a package** in the current, or a specific environment | * **Update a package** in the current, or a specific environment |
| |
| ==== conda, pip and virtualenv ==== | ==== conda, pip and virtualenv ==== |
| | |
| | * May 2026: [[https://conda.org/blog/2026-05-07-conda-and-pip-ecosystems|Conda and pip are two ecosystems, not just tools]] |
| | |
| | * June 2026: [[https://conda.org/blog/2026-06-25-bridging-conda-and-pypi-ecosystems/|Bridging conda and PyPI ecosystems: Install PyPI packages with conda install]] |
| |
| * Read the [[https://docs.conda.io/projects/conda/en/stable/commands/#conda-vs-pip-vs-virtualenv-commands|Conda vs. pip vs. virtualenv commands]] section | * Read the [[https://docs.conda.io/projects/conda/en/stable/commands/#conda-vs-pip-vs-virtualenv-commands|Conda vs. pip vs. virtualenv commands]] section |
