PMIP3 wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== JYP steps for installing Miniconda3 ======

===== What? Why? =====

  * //Miniconda3// is a minimal python environment/distribution that can be used for creating more complete work environments. It will basically make the ''conda'' command available on your computer, and you can then use ''conda'' to install and update more python packages and their dependencies.

  * A python //environment// is basically where you install python. A python //distribution// is the collection of packages you have chosen to install together in the environment.

  * You don't need to be (and **you should not be**) //root// when you install Miniconda3. You just need enough disk space where you have write access

  * If you are installing Miniconda3 in a Linux environment on a [[other:win10wsl|Windows 10 computer using Windows Subsystem for Linux (WSL)]], pay special attention to the instructions on the **WSL** lines

  * The way you do things will depend on how you are going to use python. There may be some slight differences if you are installing the environment just for you, or if you need a more stable environment because you are installing for multiple users.

  * You could also start with the //Anaconda// installer that will install a much more complete python environment, ready for use, but we choose not to do that because Anaconda requires more disk space at the beginning, and all its packages come from the ''default'' //channel// (or //repository//) provided by the [[https://docs.conda.io/projects/conda/en/latest/glossary.html#conda-repository|conda repository]]. This is not very useful because we will be mostly using (the same) packages provided by the [[https://conda-forge.org/feedstock-outputs/|conda-forge]] channel.

Note: some extra details are available on the much older page //Installing and maintaining UV-CDAT with conda//. You can check later the [[other:uvcdat:conda_notes#useful_conda_commands|Useful conda commands]], but the [[https://docs.conda.io/projects/conda/en/latest/index.html|official conda documentation]] and the [[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html|Cheat sheet]] are probably more up-to-date.

===== Installing miniconda3 on a Linux-like computer =====

<WRAP center round tip 60%>
By //Linux-like//, we mean:
  * A native Linux computer
  * A windows 10 computer with [[other:win10wsl|WSL+Ubuntu installed]]
  * A mac where you can use Linux in a terminal
</WRAP>


  * Get the [[https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh|Python 3.8 Linux 64-bit (bash installer)]] (unless there is a [[https://conda.io/miniconda.html|more recent version?]])
    * Find some temporary space on Linux (or Windows, if you are using [[other:win10wsl|WSL]])
      * e.g. Linux at LSCE:\\ ''cd /home/scratch01/<your_login>''
      * e.g. WSL, assuming that there is a ''C:\Scratch\<your_login>'' directory:\\ ''cd /mnt/c/Scratch/<your_login>''
    * Use ''wget'' to download the installer:\\ ''wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'' (90 Mb, 26 Feb 2021)

  * Execute the installer
    * ''bash Miniconda3-latest-Linux-x86_64.sh''
      * Accept the license
      * Note: at the end of the installation, **answer //no// to the following question**, so that the installer does **not** change your existing shell configuration files!\\ ''Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]'' => **NO**
      * Specify an explicit installation path **outside of your //home// directory**, with enough disk space (more than 3 Gb if you are going to install CDAT and some extra packages), preferably on a disk that is not backed up:
        * Installations by //JYP//:
          * Linux at LSCE: ''/home/share/unix_files/cdat/miniconda3<possibly_some_version_here>/''
          * Linux on ciclad: ''/data/jypmce/cdat/miniconda3<possibly_some_version_here>/''
        * WSL: installing outside of your ''/home/your_login/'' directory <wrap hi>does not work</wrap>. You need to accept the installation in the default location: ''/home/<your_login>/miniconda3''
      * The resulting ''miniconda3'' directory size is 342M\\ <code> > du -sh miniconda3
342M    miniconda3

 > cd miniconda3

 > du -sh *
20M     bin
0       compiler_compat
4.0K    condabin
684K    conda-meta
0       envs
16K     etc
5.5M    include
4.0K    info
198M    lib
12K     LICENSE.txt
114M    pkgs
604K    share
4.0K    shell
0       ssl
0       x86_64-conda_cos6-linux-gnu</code>

  * Initialize the newly installed conda environment (this will initialize the environment only in the current terminal):
    * bash shell: ''source <installation_path>/miniconda3/etc/profile.d/conda.sh''
    * tcsh shell: ''source <installation_path>/miniconda3/etc/profile.d/conda.csh''

  * Check if you can use the ''conda'' command, and use it to initialize the //base// environment
    * <code>$ which conda
<installation_path>/miniconda3/condabin/conda
$ which python
/usr/bin/python
$ conda activate
(base) $ which python
<installation_path>/miniconda3/bin/python</code>
  * Update the new installation
    * <code>$ conda update --all
[...]</code>
    * During the update, the ''miniconda3'' directory size goes from 432 Mb to 581 Mb. This directory will keep on growing, which is the reason why you should put it on a (preferably non backed up) disk where you have //enough// space

  * Make sure we have the latest ''conda'' package (just in case we did not get it with the //update//)\\ ''conda update -n base conda''

  * Remove the installer later, when you have tester your installation:\\ ''rm Miniconda3-latest-Linux-x86_64.sh''

===== Initializing conda in new terminals =====

When you open a terminal, your shell needs to know where to find the ''conda'' command used to initialize an environment, or switch between environments

==== General case ====

You were asked the following question when installing miniconda3: //Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]//

  * If you answered //yes//, the installer probably added some very complicated lines to your shell configuration files, but you probably have ''conda'' directly available when you open a new terminal
  * if you answered //no// (as suggested), use a text editor to add an extra line to the appropriate configuration file
    * //bash// user: add this line to ''~/.bashrc''\\ ''source <installation_path>/miniconda3/etc/profile.d/conda.sh''
    * //tcsh// user: add this line to ''~/.cshrc''\\ ''source <installation_path>/miniconda3/etc/profile.d/conda.csh''

We choose **not** to add a ''conda activate env_name'' line to the shell configuration files, in order to avoid side effects. When we open a new terminal, we get the default python available on the system. When we need a specific python environment, we just open a new window and then explicitly type: ''conda activate env_name''

==== Multi-user installation ====

In the case of **python environments maintained by a single user, but used by several users**, we could do the same as in the //General case//, but it can be useful to have the users //source// an intermediate initialization file, that will then //source// the initialization file used in the general case. This makes it easier to maintain and change the environments, without asking users to make changes.

  * ask //bash// users to add to ''~/.bashrc'' something like\\ ''source  ~main_installer_login/.conda3_jyp.sh''\\ \\ with a ''.conda3_jyp.sh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.sh.txt |}}

  * ask //tcsh// users to add to ''~/.cshrc'' something like\\ ''source  ~main_installer_login/.conda3_jyp.csh''\\ \\ with a ''.conda3_jyp.csh'' file looking like {{ :other:uvcdat:cdat_conda:conda3_jyp.csh.txt |}}

===== Fine-tuning conda =====

conda will probably work fine with the default settings for creating new environments with just one package (e.g. [[https://github.com/NOAA-PMEL/PyFerret/#anaconda-installation---linux-os-x-and-windows-10bash|just PyFerret]]).

We intend to **create complex environments** (combining lots of packages, with complex dependencies) and it is highly recommended to use the specific settings described in this section

==== Getting conda configuration information ====

  * [[https://docs.conda.io/projects/conda/en/latest/commands/config.html|conda config]] documentation
  * Basic information: ''conda info''
  * Full configuration: ''conda config --show''

==== Changing the .condarc file ====

Most of the packages we will install will be provided by the //conda-forge// channel (''-c conda-forge'' option). The installation steps will also add dependencies that will theoretically come from //conda-forge//, but could also come from the //defaults// channel.

The following will **make sure that we only get packages from //conda-forge//** unless the requested packages really don't exist on //conda-forge// (see [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html|Managing channels]] and [[https://conda-forge.org/docs/user/tipsandtricks.html#how-to-fix-it|Using multiple channels => How to fix it]])

<code>$ conda config --add channels conda-forge
$ conda config --set channel_priority strict
$ cat ~/.condarc
channels:
  - conda-forge
  - defaults
channel_priority: strict</code>

==== Using mamba instead of conda ====

Even when using a proper ''.condarc'' in order to get packages coming only from //conda-forge//, ''conda install'' may fail to solve the dependencies after meditating during an enormous time (e.g. [[https://github.com/conda/conda/issues/9367#issuecomment-793968239|Solving environment: failed with initial frozen solve. Retrying with flexible solve.]]), when we try to deal with **complex environments** (i.e. we already have lots of installed packages and we want to add even more packages).

Using CDAT as a base environment, and adding a lot of extra packages is a **complex environment**.

The only solution seems to replace ''conda'' with [[https://github.com/mamba-org/mamba|mamba]], a reimplementation of the conda package manager in C++ ([[https://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7|Making conda fast again]]):
  * Install ''mamba'' in the //base// (i.e. //miniconda//) environment\\ ''conda install mamba -n base -c conda-forge''
  * Use ''mamba'' instead of ''conda'' when installing packages.\\ Theoretically, you don't need to specify ''install -c conda forge'' any more, because ''conda-forge'' is now the default source of new packages\\ ''mamba install lots_of_packages''


/* standard page footer */

\\ \\ \\ 
----
[ [[pmip3:|PMIP3 Wiki Home]] ] -
[ [[pmip3:wiki_help|Help!]] ] -
[ [[wiki:syntax|Wiki syntax]] ]
PMIP3 wiki

User Tools

Site Tools

Page Tools
