| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| other:uvcdat:cdat_conda:miniconda3_install [2025/07/23 14:28] – [Before installing Python on a shared computer] jypeter | other:uvcdat:cdat_conda:miniconda3_install [2025/07/29 16:36] (current) – [Installation history] Updated the examples jypeter |
|---|
| ==== Desperately seeking a specific package ==== | ==== Desperately seeking a specific package ==== |
| |
| Check if the Python package you are [[https://www.imdb.com/title/tt0089017/|desperately seeking]] is not already installed, before installing your own Python **on shared interactive Linux servers, or on batch computing clusters** !! | Check if the Python package you are [[https://www.imdb.com/title/tt0089017/|desperately seeking]] is not already installed, before installing your own Python environment **on shared interactive Linux servers, or on batch computing clusters** !! |
| |
| * Ask other users who know the local environment | * Ask other users who know the local environment |
| |
| * Read the servers/clusters documentation: [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting|Python at LSCE by JYP]], [[https://intranet.lsce.ipsl.fr/informatique/index.php|LSCE]], [[https://documentations.ipsl.fr/spirit/|ESPRI Computing and Data Center]], ... | * Read the servers/clusters Python-related documentation: [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting|Python at LSCE by JYP]], [[https://intranet.lsce.ipsl.fr/informatique/index.php|LSCE]], [[https://documentations.ipsl.fr/spirit/spirit_clusters/python_modules.html|IPSL Mesocenter]], ... |
| |
| * Use ''[[other:newppl:starting#using_module_to_access_optional_programs|module avail [mod_name]]]'' to find out the Python versions available, and then ''module load'' and ''conda list'' to get more information on the available Python packages | * Use ''[[other:newppl:starting#using_module_to_access_optional_programs|module avail [mod_name]]]'' to find out the Python versions available, and then ''module load'' and ''conda list'' to get more information on the available Python packages |
| * e.g. on ''spiritx'' ([[https://documentations.ipsl.fr/spirit/common/module_command.html|modules on spirit]])): <code>$ module avail python | * e.g. on ''spiritx'' ([[https://documentations.ipsl.fr/spirit/common/module_command.html|modules on spirit]])): <code>$ module avail -t python |
| python/meso-3.8 python/meso-3.9 python/meso-3.10 python/meso-3.11 | python/meso-3.8 |
| | python/meso-3.9 |
| | python/meso-3.10 |
| | python/meso-3.11 |
| |
| $ module avail anaconda | jypmce@spiritx1:~$ module avail -l 2>&1 | grep meso |
| anaconda-meso/2022.10 anaconda3-py/2020.11 | anaconda-meso/2022.10 2023/04/21 18:02:35 |
| anaconda-meso/2023.09-0 anaconda3-py/2021.11 | anaconda-meso/2023.09-0 2024/01/25 15:07:50 |
| | pangeo-meso/2023.04.15 2024/10/18 15:11:30 |
| | pangeo-meso/2024.01.22 2024/10/29 15:16:33 |
| | pangeo-meso/2025.01.24 2025/02/28 15:45:21 |
| | python/meso-3.8 2024/10/18 15:12:15 |
| | python/meso-3.9 2024/10/18 15:12:01 |
| | python/meso-3.10 2024/10/18 15:11:30 |
| | python/meso-3.11 2024/10/29 15:16:33 |
| |
| $ module avail pangeo | jypmce@spiritx1:~$ module load pangeo-meso/2025.01.24 |
| pangeo-meso/2023.04.15 pangeo-meso/2024.01.22 | |
| |
| $ module load pangeo-meso/2024.01.22 | (pangeo-meso-2025.01.24) jypmce@spiritx1:~$ which python |
| (pangeo-meso-2024.01.22) $ which python | /net/nfs/tools/u20/Python/miniforge_24.11.3-0/envs/pangeo-meso-2025.01.24/bin/python |
| /net/nfs/tools/u20/Python/miniconda3_py311_23.11.0-2/envs/pangeo-meso-2024.01.22/bin/python | |
| |
| $ conda list | grep cartopy | (pangeo-meso-2025.01.24) jypmce@spiritx1:~$ conda list | grep cartopy |
| cartopy 0.22.0 py311h320fe9a_1 conda-forge | cartopy 0.24.0 py312hf9745cd_0 conda-forge |
| </code> | </code> |
| |
| * Lots of Python packages do very similar things. Maybe you can use another similar package that is already installed... | * Lots of Python packages do very similar things. Maybe you can use another similar package that is already installed... |
| |
| * If the packages you need are not available, it may be faster and easier to ask the local system administrators if they can install the packages, rather than doing everything yourself | * If the packages you need are not available, it may be faster and easier to ask JYP or the local system administrators if they can install the packages in an existing environment, rather than doing everything yourself |
| |
| ==== Using an unlisted Python distribution on a jupyter server ==== | ==== Using an unlisted Python distribution on a jupyter server ==== |
| |
| If there is a //jupyter notebook server// connected to your system, and a Python environment providing all the packages you need, but that is not directly available on the notebook server, follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local server | In the case when: |
| | * there is a //jupyter notebook server// connected to your Linux cluster (e.g. https://jupyter.lsce.ipsl.fr/) |
| | * and an existing Python environment on this cluster providing all the packages you need |
| | but this Python environment is not directly available on the notebook server (in the listed //kernels//), you should follow the [[other:python:starting#using_a_non-standard_kernel|using a non-standard kernel]] instructions, rather than starting your own local notebook server (on this multi-user computer) |
| ===== Why should you use Miniconda3 ? ===== | ===== Why should you use Miniconda3 ? ===== |
| | |
| | FIXME: try using [[https://conda-forge.org/download/|Miniforge3]] instead of Miniconda3 ? |
| |
| * **//Miniconda3// is a minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command/executable available on your computer, and you can then use this ''conda'' executable to select an existing distribution, or create new and independent Python distributions. | * **//Miniconda3// is a minimal/bootstrap Python distribution** that can be used for creating more complex Python distributions.\\ It will basically make a recent ''conda'' command/executable available on your computer, and you can then use this ''conda'' executable to select an existing distribution, or create new and independent Python distributions. |
| * [[https://github.com/conda/conda|conda @ GitHub]] | * [[https://github.com/conda/conda|conda @ GitHub]] |
| |
| * Package repositories | * [[https://conda.org/blog/2024-08-14-conda-ecosystem-explained#conda-package-repository-and-channels|Package repositories]] |
| * [[https://conda-forge.org/packages/|conda-forge]] <= recommended source of packages | * [[https://conda-forge.org/packages/|conda-forge]] <= recommended source of packages |
| * [[https://anaconda.org/|default]] | * [[https://anaconda.org/|default]] |
| * **Available environments** | * **Available environments** |
| * ''conda env list'' | * ''conda env list'' |
| | * Same as ''conda info --envs'' |
| * The currently active environment has a "*****" character on its line<code>(base) $ conda env list | * The currently active environment has a "*****" character on its line<code>(base) $ conda env list |
| # conda environments: | # conda environments: |
| * **Delete an environment** | * **Delete an environment** |
| * ''conda remove -n crap_environment %%--%%all'' | * ''conda remove -n crap_environment %%--%%all'' |
| | * This will remove all packages of ''crap_environment'' and the ''crap_environment'' environment itself |
| |
| * **Find out if a package is available** on the //conda-forge// channel (or other channels) | * **Find out if a package is available** on the //conda-forge// channel (or other channels) |
| * **Specific versions can be specified** by using the [[#the_conda_matchspec_format|conda MatchSpec format]] | * **Specific versions can be specified** by using the [[#the_conda_matchspec_format|conda MatchSpec format]] |
| * ''conda remove package1 package2 packageN'' | * ''conda remove package1 package2 packageN'' |
| | * Same as ''conda //uninstall// package1 package2 packageN'' |
| * Matching ''pip'' commands (**only** if the package is not available in conda!) | * Matching ''pip'' commands (**only** if the package is not available in conda!) |
| * ''pip install package'' | * ''pip install package'' |
| * ''base'' history file: ''/path_to_miniconda/conda-meta/history'' | * ''base'' history file: ''/path_to_miniconda/conda-meta/history'' |
| * ''existing_environment'' history file: ''/path_to_miniconda/envs/existing_environment/conda-meta/history'' | * ''existing_environment'' history file: ''/path_to_miniconda/envs/existing_environment/conda-meta/history'' |
| * e.g. installation commands for the ''base'' environment:\\ <code> $ egrep -e 'cmd:' -e '==>' /homedata/jypmce/miniconda3_2024-03/conda-meta/history | * e.g. installation commands for the ''base'' environment:\\ <code>$egrep -e 'cmd:' -e '==>' /home/share/unix_files/cdat/miniconda3_2024-03/conda-meta/history |
| ==> 2024-02-23 18:39:23 <== | ==> 2024-02-23 18:39:23 <== |
| # cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW | # cmd: constructor /tmp/tmp.4um5tGKcqW/miniconda3/ --output-dir /tmp/tmp.4um5tGKcqW |
| ==> 2024-03-27 16:10:01 <== | ==> 2024-03-27 16:13:08 <== |
| # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all | # cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda update -n base --all |
| ==> 2024-03-28 11:38:08 <== | |
| [...] | [ ... ] |
| ==> 2024-04-24 17:47:32 <== | |
| # cmd: /homedata/jypmce/miniconda3_2024-03/bin/conda update -n base --all</code> | ==> 2025-07-24 17:10:51 <== |
| | # cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda update -n base --all |
| | ==> 2025-07-28 17:05:44 <== |
| | # cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda update -n base --all</code> |
| | * e.g. installation commands for a more complex //work// environment (once the ''base'' environment is available):\\ <code>$ egrep -e 'cmd:' -e '==>' /home/share/unix_files/cdat/miniconda3_2024-03/envs/bauer_env/conda-meta/history |
| | ==> 2024-06-18 11:16:12 <== |
| | # cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda create -n bauer_env xcdat |
| | ==> 2024-06-18 11:25:14 <== |
| | # cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda install cartopy cmocean |
| | |
| | [ ... lots of extra packages and updates ... ] |
| | |
| | # cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda update -n bauer_env --all numpy<2.3 |
| | ==> 2025-07-29 16:18:44 <== |
| | # cmd: /home/share/unix_files/cdat/miniconda3_2024-03/bin/conda install -n bauer_env rpy2</code> |
| * You can also, use a script and the ''conda'' module available **in the ''base'' environment**: [[https://github.com/conda/conda/issues/4545#issuecomment-469984684|example]] | * You can also, use a script and the ''conda'' module available **in the ''base'' environment**: [[https://github.com/conda/conda/issues/4545#issuecomment-469984684|example]] |
| |
