Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:cdat_8_0_py2 [2019/03/04 16:55]
jypeter
+++ other:uvcdat:cdat_conda:cdat_8_0_py2 [2021/02/27 13:55] (current)
jypeter [Installing Miniconda3]
@@ Line 18: / Line 18: @@
 ==== Installing Miniconda3 ====
-We have installed a new version of Miniconda3 (steps below adapted from the [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions)
+  * You don't need to be (and **you should not be**) //root// when you install Miniconda3. You just need enough disk space where you have write access
+  * If you are installing Miniconda3 in a Linux environment on a [[other:win10wsl|Windows 10 computer using Windows Subsystem for Linux (WSL)]], pay special attention to the instructions on the **WSL** lines
+The steps below are adapted from the older (but with lots of extra and possibly useful details) [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions
+  * Get the [[https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh|Python 3.8 Linux 64-bit (bash installer)]] (unless there is a [[https://conda.io/miniconda.html|more recent version?]])
+    * Find some temporary space on Linux (or Windows, if you are using WSL)
+      * e.g. Linux at LSCE:\\ ''cd /home/scratch01/<your_login>''
+      * e.g. WSL, assuming that there is a ''C:\Scratch\<your_login>'' directory:\\ ''cd /mnt/c/Scratch/<your_login>''
+    * Use ''wget'' to download the installer:\\ ''wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'' (90 Mb, 26 Feb 2021)
-  * Get the [[https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh|Python 3.7 Linux 64-bit (bash installer)]] (unless there is a [[https://conda.io/miniconda.html|more recent version?]])
-    * Find some temporary space on Linux or Windows
-      * Linux:''cd /home/scratch01/jypeter''
-      * WSL: ''cd /mnt/h/Scratch/jypeter/'' (assuming that there is a ''H:\Scratch\jypeter'' directory)
-    * ''wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh'' (67 Mo, 15 Feb 2019)
   * Execute the installer
     * ''bash Miniconda3-latest-Linux-x86_64.sh''
       * Accept the license
-      * Specify an explicit installation path **outside of the home directory**, with enough disk space (more than 3G if you are going to install UV-CDAT and some extra packages), preferably on a disk that is not backed up:
+      * Note: at the end of the installation (next step), **answer //no// to the following question**, so that the installer does not change your existing shell initialization files!\\ ''Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]'' => **NO**
-        * Linux LSCE: ''/home/share/unix_files/cdat/miniconda3''
+      * Specify an explicit installation path **outside of your //home// directory**, with enough disk space (more than 3 Gb if you are going to install CDAT and some extra packages), preferably on a disk that is not backed up:
-        * Linux ciclad: ''/data/jypmce/cdat/miniconda3''
+        * Installations by //JYP//:
-        * WSL: installing to a directory that is not in ''/home/'' <wrap hi>does not work</wrap> (e.g. ''/mnt/h/CDAT/miniconda3'',assuming there is a ''H:\CDAT\'' directory, does not work)\\ Need to accept the installation in the default location: ''/home/jypeter/miniconda3''
+          * Linux at LSCE: ''/home/share/unix_files/cdat/miniconda3'', or another subdirectory of ''/home/share/unix_files/cdat/''
-        * Resulting ''miniconda3'' directory size is 302M\\ <code> >du -sh miniconda3
+          * Linux on ciclad: ''/data/jypmce/cdat/miniconda3'', or another sub-directory of ''/data/jypmce/cdat/''
-M    miniconda3
+        * WSL: installing to a directory that is not in ''/home/'' <wrap hi>does not work</wrap> (e.g. ''/mnt/h/CDAT/miniconda3'',assuming there is a ''H:\CDAT\'' directory, does not work)\\ You need to accept the installation in the default location: ''/home/<your_login>/miniconda3''
- >cd miniconda3
+      * The resulting ''miniconda3'' directory size is 342M\\ <code> > du -sh miniconda3
-/home/share/unix_files/cdat/miniconda3
+M    miniconda3
- >du -sh *
-.3M    bin
+ > cd miniconda3
-.0M    compiler_compat
-K    conda-meta
+ > du -sh *
-.0K    envs
+M     bin
-K     etc
+       compiler_compat
-.2M    include
+.0K    condabin
-M    lib
+K    conda-meta
-.0K    LICENSE.txt
+       envs
-M    pkgs
+K     etc
-K    share
+.5M    include
-K     ssl
+.0K    info
-K     x86_64-conda_cos6-linux-gnu</code>
+M    lib
-      * Accept the automatic update of the //PATH// variable in ''.bashrc'' (note: the installer automatically creates a backup of ''.bashrc''). This will some extra lines at the end of ''.bashrc'' used by the //old-style conda initialization// (we will later switch to the //new-style//). The added lines should look like:
+K     LICENSE.txt
-        * ''# added by Miniconda3 installer''\\ ''export PATH="/home/share/unix_files/cdat/miniconda3/bin:$PATH"''
+M    pkgs
-        * ''# added by Miniconda4.xxx installer''\\ ''[some really fancy bash shell commands]''
+K    share
-  * Remove the installer: ''rm Miniconda3-latest-Linux-x86_64.sh''
+.0K    shell
-  * Open a new terminal (if you use bash) or open a new terminal and type ''bash''
+       ssl
-    * check if the python from the conda distribution has been correctly installed\\ <code> > bash
+       x86_64-conda_cos6-linux-gnu</code>
+  * Initialize the newly installed conda environment:
+    * bash shell: ''source <installation_path>/miniconda3/etc/profile.d/conda.sh''
+    * tcsh shell: ''source <installation_path>/miniconda3/etc/profile.d/conda.csh''
+  * Check if you can use the ''conda'' command, and use it to initialize the //base// environment
+    * <code>$ which conda
+<installation_path>/miniconda3/condabin/conda
 $ which python
-/home/share/unix_files/cdat/miniconda3/bin/python
+/usr/bin/python
-$ which conda
+$ conda activate
-/home/share/unix_files/cdat/miniconda3/bin/conda</code>
+(base) $ which python
-    * Update and clean the new installation\\ <code>$ conda update --all
+<installation_path>/miniconda3/bin/python</code>
-$ du -sh /home/share/unix_files/cdat/miniconda3
+  * Update the new installation
-M    /home/share/unix_files/cdat/miniconda3
+    * <code>$ conda update --all
-$ conda clean --all
-$ du -sh /home/share/unix_files/cdat/miniconda3
-M    /home/share/unix_files/cdat/miniconda3
-$ conda list
-# packages in environment at /home/share/unix_files/cdat/miniconda3:
-#
-# Name                    Version                   Build  Channel
-[...]
-python                    3.7.1                h0371630_3
-readline                  7.0                  h7b6447c_5
-requests                  2.20.0                   py37_0
 [...]</code>
+    * During the update, the ''miniconda3'' directory size goes from 432 Mb to 581 Mb. This directory will keep on growing, which is the reason why you should put it on a (preferably non backed up) disk where you have //enough// space
+  * Make sure we have the latest ''conda'' (just in case we did not get it with the //update//)\\ ''conda update -n base -c defaults conda''
+  * Remove the installer:\\ ''rm Miniconda3-latest-Linux-x86_64.sh''
 ==== Post-Miniconda3 installation ====
@@ Line 372: / Line 378: @@
     * Update with: ''pip install %%--%%upgrade dreqPy''
       * Get version number with:\\ <code>$ drq -v
-dreqPy version 01.00.06 [Version 01.00.06]</code>
+dreqPy version 01.00.29 [Version 01.00.29]</code>
 The following packages have no dependency problems and were installed (or updated) //later//
-  * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology
-    * ''conda install -n cdatm15 -c conda-forge iris''
+  * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
+    * ''conda install -n cdatm17_py2 -c conda-forge -c vacumm vacumm''
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
-    * ''conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib''
+    * ''conda install -n cdatm17_py2 -c stefraynaud -c conda-forge spanlib''
     * Test: ''python -c 'from spanlib.analyzer import Analyzer' ''
-  * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
+  * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework
-    * ''conda install -n cdatm15 -c conda-forge wrf-python''
+    * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]]
- * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'')
-    * ''conda install -n cdatm15 -c conda-forge glances''
-    * [[https://github.com/nicolargo/glances|glances@github]]
 ==== TODO ====
-//Add here packages that would be useful but have some problems that prevent their installation//
+//Add here packages that would be useful and have not been installed yet, or have some problems that prevent their installation//
-  * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
+  * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology
-    * As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])!
+    * ''conda install -n cdatxxx -c conda-forge iris''
-    * <wrap hi>Problem solved</wrap> with **vacumm 3.4.1-1** on August 31st! Installation was OK with\\ ''conda install -n cdatm15 -c conda-forge -c vacumm vacumm''
+  * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
+    * ''conda install -n cdatxxx -c conda-forge wrf-python''
+  * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'')
+    * ''conda install -n cdatxxx -c conda-forge glances''
+    * [[https://github.com/nicolargo/glances|glances@github]]
 ==== Other packages ====
@@ Line 408: / Line 417: @@
   * [[https://cmor.llnl.gov/|CMOR]]
-    * Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor''
+    * Update with: ''conda update -n cdatxxx -c conda-forge -c pcmdi -c uvcdat cmor''
       * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' ''
@@ Line 414: / Line 423: @@
     * Update with: ''pip install %%--%%upgrade dreqPy''
       * Get version number with: ''drq -v''
+==== Installing CLIMAF ====
+The [[https://climaf.readthedocs.io/|CLIMAF]] installation is a special case, because we may want to use it with the Python installed for CDAT, but it is not installed using conda
+We follow the steps for the [[https://climaf.readthedocs.io/en/latest/installing.html#installing-climaf-if-you-are-not-at-cnrm-on-ciclad-or-climserv|installation of a non-standard server]]
+  * Go to the appropriate installation directory (we choose the directory where we have also installed miniconda3):
+    * LSCE: ''cd /home/share/unix_files/cdat''
+    * ciclad: ''/data/jypmce/cdat/''
+  * ''git clone https://github.com/senesis/climaf climaf''
+  * Add CLIMAF to the PYTHONPATH environment variable
+    * LSCE & tcsh: ''setenv PYTHONPATH ${PYTHONPATH}:/home/share/unix_files/cdat/climaf''
+    * LSCE & bash: ''export PYTHONPATH=${PYTHONPATH}:/home/share/unix_files/cdat/climaf''
+    * ciclad: same as above, using ''/data/jypmce/cdat/climaf''
+  * Test the installation
+    * make sure the [[https://climaf.readthedocs.io/en/latest/requirements.html#requirements|Requirements]] are available:
+      * LSCE: ''module load netcdf/4 nco cdo ncl ncview''
+      * ciclad: ''module load netcdf4 nco cdo ncl/6.3.0''
+      * optional: start an X server
+      * <code>cd climaf/testing
+./test_install.sh</code>
+  * Once the tests are OK, add the required //module load// and environment variables definitions to the ''.basrc'' and ''.cshrc'' files. <wrap hi>Note: maybe these lines should go to the ''.profile'' and ''.login'' files...(check [[https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html|Bash Startup Files]])</wrap>
+    * LSCE: add to your .cshrc
+      * <code># Prerequisites for using CLIMAF
+module load netcdf/4 nco cdo ncl ncview
+# CLIMAF environment variables
+setenv CLIMAF /home/share/unix_files/cdat/climaf
+setenv PYTHONPATH "${PYTHONPATH}:${CLIMAF}"
+#setenv PYTHONPATH $CLIMAF
+setenv PATH "${PATH}:${CLIMAF}/bin"
+setenv CLIMAF_CACHE "/home/scratch01/${USER}/climaf_cache"
+setenv TMPDIR $CLIMAF_CACHE
+setenv CLIMAF_LOG_DIR $CLIMAF_CACHE
+#conda activate cdatm-nightly_py2</code>
+    * ciclad: add to your .bashrc
+      * <code># Prerequisites for using CLIMAF when not using the
+# default 'module load climaf'
+module load netcdf4 nco cdo ncl/6.3.0
+# CLIMAF environment variables
+export CLIMAF=/data/jypmce/cdat/climaf
+#export PYTHONPATH=$PYTHONPATH:$CLIMAF
+export PYTHONPATH=$CLIMAF
+export PATH=$PATH:$CLIMAF/bin
+export CLIMAF_CACHE=/data/jypmce/climaf_cache
+export TMPDIR=$CLIMAF_CACHE
+export CLIMAF_LOG_DIR=$CLIMAF_CACHE
+#conda activate cdatm-nightly_py2
+</code>
+  * Open a new shell and <wrap em>run the tests again</wrap>. If there is an error that was not there, check what you have added to the shell init files
+  * [[other:uvcdat:cdat_conda:climaf_test|Test CLIMAF on ciclad]]
 ==== Cleaning up things ====
@@ Line 420: / Line 484: @@
 We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions
-<code>(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls
+<code> >find /home/share/unix_files/cdat/miniconda3/envs \! -perm /g+r,o+r -ls
-[...]
+# Everything OK!</code>
-(cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp*
-(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls</code>
@@ Line 449: / Line 511: @@
   * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python
   * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography
-  * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology
   * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
     * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]]
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
-  * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model
-  * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'')
   * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python
@@ Line 466: / Line 525: @@
 ^  Environment\\ name  ^  Server  ^  ''conda list''  ^
-|  uvcdat-2.10  |  LSCE  |  {{  |package list}}  |
+| cdat-8.0_py2 |  LSCE  | |
-|  cdatm15  |  LSCE  |  {{ |package list}}  |
+| cdatm17_py2 |  LSCE | |
+| cdat-nightly_py2 |  LSCE\\ ciclad  | |
+| cdatm-nightly_py2 |  LSCE\\ ciclad  | |
 /* standard page footer */

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools