Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:cdat_8_2_1 [2021/03/03 16:39] – [Packages that have no dependency problems] jypeter
+++ other:uvcdat:cdat_conda:cdat_8_2_1 [2021/03/08 11:55] – [CDAT 8.2.1 installation notes] Changed main title jypeter
@@ Line 1: / Line 1: @@
-====== CDAT 8.2.1 installation notes ======
+====== Python distribution based on CDAT 8.2.1 notes ======
 [ [[.:index|Back to all versions]] ]
@@ Line 11: / Line 11: @@
 ===== Using JYP version =====
-If you mostly want to use the CDAT 8.2.1 installed by JYP, just use the following steps and then read the //What should I do now?// section below
+If you mostly want to use the **python distribution based on CDAT 8.2.1** installed by JYP, just use the following steps and then read the [[#what_should_i_do_now|What should I do now?]] section below
   * Initialize //conda// with:
@@ Line 23: / Line 23: @@
     * python 3.x: ''conda activate cdatm_py3''
-  * Type ''which python'' (or the ''wp'' alias) and make sure you get something like\\ ''[...]/miniconda3/envs/env_name/bin/python''
+  * Type ''which python'' (or the ''wp'' alias) and make sure you get something looking like
+    * ''[...]/miniconda3<possibly_some_version>/envs/<some_env_name>/bin/python''
+    * e.g. ''cdatm_py3'' environment at LSCE:\\ ''/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python''
@@ Line 94: / Line 96: @@
 The installation line below is based on information from the [[https://github.com/CDAT/cdat/wiki/install#for-linux-python|CDAT wiki page]]. See also [[https://github.com/CDAT/cdat/issues/2265|Some questions and notes about 8.2.1 installation]]
+<code>
+bash-4.2$ conda config --env --add channels conda-forge
+bash-4.2$ conda config --set channel_priority strict
+bash-4.2$ cat .condarc
+channels:
+  - conda-forge
+  - defaults
+$ conda config --describe channel_priority
+$ conda create -n cdat-8.2.1_nompi_py3 -c conda-forge -c cdat/label/v8.2.1 cdat "libnetcdf=*=nompi_*" "mesalib=18.3.1" "python=3"</code>
 <code>$ conda create -n cdat-8.2.1_py3 -c conda-forge -c cdat/label/v8.2.1 cdat "libnetcdf=*=mpi_openmpi_*" "mesalib=18.3.1" "python=3"</code>
 Resulting list of installed packages
-  * ''conda list -n cdat-8.2.1_py3 > cdat_8.2.1_installed_packages.txt''
+  * ''conda list -n cdat-8.2.1_nompi_py3 > cdat_8.2.1_nompi_21-03-05.txt''
-  * {{ :other:uvcdat:cdat_conda:cdat_8.2.1_installed_packages.txt |}}
+  * no mpi: {{ :other:uvcdat:cdat_conda:cdat_8.2.1_nompi_21-03-05.txt |}}
+  * with mpî: {{ :other:uvcdat:cdat_conda:cdat_8.2.1_installed_packages.txt |}}
 [[https://github.com/CDAT/cdat/issues/2265|Some extra installation notes]]
@@ Line 165: / Line 178: @@
 </code>
-===== Customizing UV-CDAT for LSCE =====
+===== Customizing CDAT for LSCE =====
 ==== A quick test of cdms2 and vcs ====
@@ Line 196: / Line 209: @@
 ==== Packages that have no dependency problems ====
-After [[#cloning_the_base_cdat_environment_before_adding_specific_packages_for_lsce|cloning]], we are ready to install some extra packages that may be requested by LSCE users
+After [[#cloning_the_base_cdat_environment_before_adding_specific_packages_for_lsce|cloning]], we are ready to install some extra (e.g. not directly related to CDAT) packages that may be useful to LSCE users
-  * We first try to install together as many packages as possible that don't require other channels than //conda-forge//, and that don't request a downgrade of what is already installed
+  * We first install together as many packages as possible that don't require other channels than //conda-forge//, and that don't request a significant downgrade of what is already installed
-  * We then install individual extra packages with ''conda install'' or ''pip install''
+    * We use the //dry-run// option of conda in order to check the version (theoretically the latest) and origin (theoretically //conda-forge//) of the packages that would be installed. Or maybe the result will be that there are too many conflicts and the installation cannot proceed with the requested packages combination
+    * conda may take a very long time to report that there are too many conflicts and we have started using [[https://github.com/mamba-org/mamba|mamba]] that uses different solving libraries and is much (**much!**) faster than conda
+    * if the dry-run reports too many conflicts or requests too many downgrades, we try to remove some (not really important) packages and check if it works better
+      * the last resort is too create a new environment with a smaller set of packages if a user really needs a conflicting package
+  * We later install individual extra packages with ''conda install'', ''mamba install'' or ''pip install''
-<code># You can use the following to keep a trace of what will be installed
+=== Pre-installation check with the dry-run option ===
-#$ conda install --dry-run -n cdatm19_py3 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm  > somewhere/extra_packages.txt
-# Install...
+<code>/usr/bin/time mamba install -n cdatm19_nompi_py3 --dry-run -c conda-forge basemap basemap-data-hires cartopy cmocean cmor ferret_datasets gdal gdal glances gsw iris joblib netcdf4 palettable pandas pillow pyferret rpy2 scikit-image scikit-learn seaborn seawater shapely spyder statsmodels windspharm wrf-python xarray xlsxwriter
-$ conda install -n cdatm19_py3 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret ferret_datasets basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
-[...]
-# Use a similar install line if you want to install the same packages in the python3 version</code>
+[... lots of information about what would be done followed by some information about how long it took ...]
+  Summary:
+  Install: 167 packages
+  Downgrade: 3 packages
+  Total download: 474 MB
+DryRunExit: Dry run. Exiting.
+.31user 0.51system 0:11.45elapsed 77%CPU</code>
+=== Actual installation ===
+<code>/usr/bin/time mamba install -n cdatm19_nompi_py3 -c conda-forge basemap basemap-data-hires cartopy cmocean cmor ferret_datasets gdal gdal glances gsw iris joblib netcdf4 palettable pandas pillow pyferret rpy2 scikit-image scikit-learn seaborn seawater shapely spyder statsmodels windspharm wrf-python xarray xlsxwriter</code>
+Resulting list of installed packages
+  * ''conda list -n cdatm19_nompi_py3 > cdatm19_nompi_py3_21-03-06.txt''
+  * no mpi: {{ :other:uvcdat:cdat_conda:cdatm19_nompi_py3_21-03-06.txt |}}
+=== Extra packages installed with pip ===
-List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_190304.txt |}}
-Packages installed with //pip//
   * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API
     * ''conda activate cdatm19_py3''\\ ''pip install dreqPy''
@@ Line 222: / Line 255: @@
     * ''conda activate cdatm19_py3''\\ ''pip install ferretmagic''
-The following packages have no dependency problems and were installed (or updated) //later//
+=== Packages with no dependency problems and were added (or updated) later ===
   * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files