User Tools

Site Tools


other:uvcdat:cdat_conda:cdat_8_0_py2

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
other:uvcdat:cdat_conda:cdat_8_0_py2 [2019/03/04 16:55]
jypeter
other:uvcdat:cdat_conda:cdat_8_0_py2 [2021/02/26 21:11]
jypeter [Installing Miniconda3] Many updates
Line 18: Line 18:
 ==== Installing Miniconda3 ====  ==== Installing Miniconda3 ==== 
  
-We have installed a new version of Miniconda3 (steps below adapted from the [[other:​uvcdat:​conda_notes#​installing_miniconda|installing miniconda]] instructions)+  * You don't need to be (and **you should not be**) //root// when you install Miniconda3. You just need enough disk space where you have write access 
 +  * If you are installing ​Miniconda3 ​in a Linux environment on a [[other:​win10wsl|Windows 10 computer using Windows Subsystem for Linux (WSL)]], pay special attention to the instructions on the **WSL** lines 
 + 
 +The steps below are adapted from the older (but with lots of extra and possibly useful details) ​[[other:​uvcdat:​conda_notes#​installing_miniconda|installing miniconda]] instructions 
 + 
 +  * Get the [[https://​repo.anaconda.com/​miniconda/​Miniconda3-latest-Linux-x86_64.sh|Python 3.8 Linux 64-bit (bash installer)]] (unless there is a [[https://​conda.io/​miniconda.html|more recent version?​]]) 
 +    * Find some temporary space on Linux (or Windows, if you are using WSL) 
 +      * e.g. Linux at LSCE:\\ ''​cd /​home/​scratch01/<​your_login>''​ 
 +      * e.g. WSL, assuming that there is a ''​C:​\Scratch\<​your_login>''​ directory:​\\ ''​cd /​mnt/​c/​Scratch/<​your_login>''​ 
 +    * Use ''​wget''​ to download the installer:​\\ ''​wget https://​repo.anaconda.com/​miniconda/​Miniconda3-latest-Linux-x86_64.sh''​ (90 Mb, 26 Feb 2021)
  
-  * Get the [[https://​repo.continuum.io/​miniconda/​Miniconda3-latest-Linux-x86_64.sh|Python 3.7 Linux 64-bit (bash installer)]] (unless there is a [[https://​conda.io/​miniconda.html|more recent version?]]) 
-    * Find some temporary space on Linux or Windows 
-      * Linux:''​cd /​home/​scratch01/​jypeter''​ 
-      * WSL: ''​cd /​mnt/​h/​Scratch/​jypeter/''​ (assuming that there is a ''​H:​\Scratch\jypeter''​ directory) 
-    * ''​wget https://​repo.continuum.io/​miniconda/​Miniconda3-latest-Linux-x86_64.sh''​ (67 Mo, 15 Feb 2019) 
   * Execute the installer   * Execute the installer
     * ''​bash Miniconda3-latest-Linux-x86_64.sh''​     * ''​bash Miniconda3-latest-Linux-x86_64.sh''​
       * Accept the license       * Accept the license
-      * Specify an explicit installation path **outside of the home directory**,​ with enough disk space (more than 3G if you are going to install ​UV-CDAT and some extra packages), preferably on a disk that is not backed up: +      ​* Note: at the end of the installation (next step), **answer //no// to the following question**, so that the installer does not change your existing shell initialization files!\\ ''​Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no]''​ => **NO** 
-        * Linux LSCE: ''/​home/​share/​unix_files/​cdat/​miniconda3''​ +      ​* Specify an explicit installation path **outside of your //home// directory**,​ with enough disk space (more than 3 Gb if you are going to install CDAT and some extra packages), preferably on a disk that is not backed up: 
-        * Linux ciclad: ''/​data/​jypmce/​cdat/​miniconda3''​ +        ​* Installations by //JYP//: 
-        * WSL: installing to a directory that is not in ''/​home/''​ <wrap hi>does not work</​wrap>​ (e.g. ''/​mnt/​h/​CDAT/​miniconda3'',​assuming there is a ''​H:​\CDAT\''​ directory, does not work)\\ ​Need to accept the installation in the default location: ''/​home/​jypeter/​miniconda3''​ +          ​* Linux at LSCE: ''/​home/​share/​unix_files/​cdat/​miniconda3'',​ or another subdirectory of ''/​home/​share/​unix_files/​cdat/​''​  
-        Resulting ​''​miniconda3''​ directory size is 302M\\ <​code>​ >du -sh miniconda3 +          * Linux on ciclad: ''/​data/​jypmce/​cdat/​miniconda3'',​ or another sub-directory of ''/​data/​jypmce/​cdat/​''​ 
-302M    ​miniconda3 +        * WSL: installing to a directory that is not in ''/​home/''​ <wrap hi>does not work</​wrap>​ (e.g. ''/​mnt/​h/​CDAT/​miniconda3'',​assuming there is a ''​H:​\CDAT\''​ directory, does not work)\\ ​You need to accept the installation in the default location: ''/​home/​<​your_login>​/​miniconda3''​ 
- >​cd miniconda3 +      The resulting ​''​miniconda3''​ directory size is 342M\\ <​code>​ > du -sh miniconda3 
-/​home/​share/​unix_files/​cdat/​miniconda3 +342M    ​miniconda3 
- >​du -sh * + 
-6.3M    ​bin + > cd miniconda3 
-2.0M    compiler_compat + 
-524K    ​conda-meta + > du -sh * 
-4.0K    ​envs +20M     bin 
-28K     etc +0       ​compiler_compat 
-5.2M    ​include +4.0K    condabin 
-146M    lib +684K    ​conda-meta 
-8.0K    LICENSE.txt +0       envs 
-142M    ​pkgs +16K     etc 
-928K    ​share +5.5M    ​include 
-92K     ssl +4.0K    ​info 
-12K     x86_64-conda_cos6-linux-gnu</​code>​ +198M    lib 
-      Accept ​the automatic update of the //PATH// variable in ''​.bashrc''​ (note: the installer automatically creates a backup of ''​.bashrc''​). This will some extra lines at the end of ''​.bashrc''​ used by the //​old-style ​conda initialization//​ (we will later switch to the //​new-style//​). The added lines should look like+12K     LICENSE.txt 
-        * ''​# added by Miniconda3 installer''​\\ ''​export PATH="/home/share/unix_files/cdat/​miniconda3/​bin:​$PATH"''​ +114M    ​pkgs 
-        * ''#​ added by Miniconda4.xxx installer''​\\ ''​[some really fancy bash shell commands]''​ +604K    ​share 
-  Remove the installer: ''​rm Miniconda3-latest-Linux-x86_64.sh''​ +4.0K    shell 
-  * Open a new terminal (if you use bash) or open a new terminal and type ''​bash''​ +0       ssl 
-    * check if the python from the conda distribution has been correctly installed\\ ​<​code>​ > bash+0       x86_64-conda_cos6-linux-gnu</​code>​ 
 + 
 +  ​Initialize ​the newly installed ​conda environment
 +    bash shell: ​''​source <​installation_path>​/miniconda3/etc/profile.d/conda.sh''​ 
 +    tcsh shell: ''​source <​installation_path>/​miniconda3/​etc/​profile.d/conda.csh''​ 
 + 
 +  * Check if you can use the ''​conda'' ​command, and use it to initialize the //base// environment 
 +    * <​code>​$ which conda 
 +<​installation_path>/​miniconda3/​condabin/​conda
 $ which python $ which python
-/home/​share/​unix_files/​cdat/​miniconda3/​bin/​python +/usr/​bin/​python 
-which conda +$ conda activate 
-/​home/​share/​unix_files/​cdat/​miniconda3/​bin/​conda</​code>​ +(base) $ which python 
-    * Update ​and clean the new installation\\ <​code>​$ conda update --all +<​installation_path>​/​miniconda3/​bin/​python</​code>​ 
-$ du -sh /​home/​share/​unix_files/​cdat/​miniconda3 +  * Update the new installation 
-515M    /​home/​share/​unix_files/​cdat/​miniconda3 +    * <​code>​$ conda update --all
-$ conda clean --all +
-$ du -sh /​home/​share/​unix_files/​cdat/​miniconda3 +
-270M    /​home/​share/​unix_files/​cdat/​miniconda3 +
-$ conda list +
-# packages in environment at /​home/​share/​unix_files/​cdat/​miniconda3:​ +
-+
-# Name                    Version ​                  ​Build ​ Channel +
-[...] +
-python ​                   3.7.1                h0371630_3 +
-readline ​                 7.0                  h7b6447c_5 +
-requests ​                 2.20.0 ​                  ​py37_0+
 [...]</​code>​ [...]</​code>​
 +    * During the update, the ''​miniconda3''​ directory size goes from 432 Mb to 581 Mb. This directory will keep on growing, which is the reason why you should put it on a (preferably non backed up) disk where you have //enough// space
 +  * Remove the installer: ''​rm Miniconda3-latest-Linux-x86_64.sh''​
  
 ==== Post-Miniconda3 installation ==== ==== Post-Miniconda3 installation ====
Line 372: Line 375:
     * Update with: ''​pip install %%--%%upgrade dreqPy''​     * Update with: ''​pip install %%--%%upgrade dreqPy''​
       * Get version number with:\\ <​code>​$ drq -v       * Get version number with:\\ <​code>​$ drq -v
-dreqPy version 01.00.06 [Version 01.00.06]</​code>​+dreqPy version 01.00.29 [Version 01.00.29]</​code>​
  
 The following packages have no dependency problems and were installed (or updated) //later// The following packages have no dependency problems and were installed (or updated) //later//
-  ​* [[http://scitools.org.uk/iris/index.html|iris]]: Python ​library for Meteorology ​and Climatology + 
-    * ''​conda install -n cdatm15 ​-c conda-forge ​iris''​+  ​* [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python ​to validate and analyze Multi-Model outputs, ​and compare them to observations 
 +    * ''​conda install -n cdatm17_py2 ​-c conda-forge ​-c vacumm vacumm''​
  
   * [[https://​github.com/​stefraynaud/​spanlib|spanlib]]:​ Spectral Analysis Library   * [[https://​github.com/​stefraynaud/​spanlib|spanlib]]:​ Spectral Analysis Library
-    * ''​conda install -n cdatm15 ​-c stefraynaud -c conda-forge spanlib''​+    * ''​conda install -n cdatm17_py2 ​-c stefraynaud -c conda-forge spanlib''​
     * Test: ''​python -c 'from spanlib.analyzer import Analyzer'​ ''​     * Test: ''​python -c 'from spanlib.analyzer import Analyzer'​ ''​
- +  ​ 
-  * [[http://wrf-python.readthedocs.io/​en/​latest/​|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) ​Model +  * [[https://climaf.readthedocs.io/​en/​latest/​|CliMAF]]: a Climate ​Model Assessment Framework 
-    ​* ''​conda install -n cdatm15 -c conda-forge wrf-python''​ +    * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]]
-  +
- * [[https://​glances.readthedocs.io/​en/​stable/​index.html|glances]]a cross-platform monitoring tool (similar to ''​top''​) +
-    * ''​conda install -n cdatm15 -c conda-forge glances''​ +
-    * [[https://​github.com/​nicolargo/​glances|glances@github]]+
  
 ==== TODO ==== ==== TODO ====
  
-//Add here packages that would be useful ​but have some problems that prevent their installation//​+//Add here packages that would be useful ​and have not been installed yet, or have some problems that prevent their installation//​
  
-  * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python ​to validate ​and analyze Multi-Model outputs, and compare them to observations +  * [[http://scitools.org.uk/iris/index.html|iris]]: Python ​library for Meteorology ​and Climatology 
-    * As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])! +    * ''​conda install -n cdatxxx -c conda-forge iris''​ 
-    * <wrap hi>​Problem solved</​wrap>​ with **vacumm 3.4.1-1*on August 31st! Installation was OK with\\ ​''​conda install -n cdatm15 ​-c conda-forge ​-c vacumm vacumm''​+ 
 +  * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARWModel 
 +    * ''​conda install -n cdatxxx -c conda-forge wrf-python''​ 
 +  
 +  ​[[https://​glances.readthedocs.io/​en/​stable/​index.html|glances]]:​ a cross-platform monitoring tool (similar to ''​top''​) 
 +    ​* ''​conda install -n cdatxxx ​-c conda-forge ​glances''​ 
 +    * [[https://​github.com/​nicolargo/​glances|glances@github]]
  
 ==== Other packages ==== ==== Other packages ====
Line 408: Line 414:
  
   * [[https://​cmor.llnl.gov/​|CMOR]]   * [[https://​cmor.llnl.gov/​|CMOR]]
-    * Update with: ''​conda update -n cdatm15 ​-c conda-forge -c pcmdi -c uvcdat cmor''​+    * Update with: ''​conda update -n cdatxxx ​-c conda-forge -c pcmdi -c uvcdat cmor''​
       * Get version number with: ''​python -c 'from cmor import *; print CMOR_VERSION_MAJOR,​ CMOR_VERSION_MINOR,​ CMOR_VERSION_PATCH'​ ''​       * Get version number with: ''​python -c 'from cmor import *; print CMOR_VERSION_MAJOR,​ CMOR_VERSION_MINOR,​ CMOR_VERSION_PATCH'​ ''​
  
Line 414: Line 420:
     * Update with: ''​pip install %%--%%upgrade dreqPy''​     * Update with: ''​pip install %%--%%upgrade dreqPy''​
       * Get version number with: ''​drq -v''​       * Get version number with: ''​drq -v''​
 +
 +==== Installing CLIMAF ====
 +
 +The [[https://​climaf.readthedocs.io/​|CLIMAF]] installation is a special case, because we may want to use it with the Python installed for CDAT, but it is not installed using conda
 +
 +We follow the steps for the [[https://​climaf.readthedocs.io/​en/​latest/​installing.html#​installing-climaf-if-you-are-not-at-cnrm-on-ciclad-or-climserv|installation of a non-standard server]]
 +
 +  * Go to the appropriate installation directory (we choose the directory where we have also installed miniconda3):​
 +    * LSCE: ''​cd /​home/​share/​unix_files/​cdat''​
 +    * ciclad: ''/​data/​jypmce/​cdat/''​
 +  * ''​git clone https://​github.com/​senesis/​climaf climaf''​
 +  * Add CLIMAF to the PYTHONPATH environment variable
 +    * LSCE & tcsh: ''​setenv PYTHONPATH ${PYTHONPATH}:/​home/​share/​unix_files/​cdat/​climaf''​
 +    * LSCE & bash: ''​export PYTHONPATH=${PYTHONPATH}:/​home/​share/​unix_files/​cdat/​climaf''​
 +    * ciclad: same as above, using ''/​data/​jypmce/​cdat/​climaf''​
 +  * Test the installation
 +    * make sure the [[https://​climaf.readthedocs.io/​en/​latest/​requirements.html#​requirements|Requirements]] are available:
 +      * LSCE: ''​module load netcdf/4 nco cdo ncl ncview''​
 +      * ciclad: ''​module load netcdf4 nco cdo ncl/​6.3.0''​
 +      * optional: start an X server
 +      * <​code>​cd climaf/​testing
 +./​test_install.sh</​code>​
 +  * Once the tests are OK, add the required //module load// and environment variables definitions to the ''​.basrc''​ and ''​.cshrc''​ files. <wrap hi>Note: maybe these lines should go to the ''​.profile''​ and ''​.login''​ files...(check [[https://​www.gnu.org/​software/​bash/​manual/​html_node/​Bash-Startup-Files.html|Bash Startup Files]])</​wrap> ​
 +    * LSCE: add to your .cshrc
 +      * <​code>#​ Prerequisites for using CLIMAF
 +module load netcdf/4 nco cdo ncl ncview
 +
 +# CLIMAF environment variables
 +setenv CLIMAF /​home/​share/​unix_files/​cdat/​climaf
 +setenv PYTHONPATH "​${PYTHONPATH}:​${CLIMAF}"​
 +#setenv PYTHONPATH $CLIMAF
 +setenv PATH "​${PATH}:​${CLIMAF}/​bin"​
 +setenv CLIMAF_CACHE "/​home/​scratch01/​${USER}/​climaf_cache"​
 +setenv TMPDIR $CLIMAF_CACHE
 +setenv CLIMAF_LOG_DIR $CLIMAF_CACHE
 +
 +#conda activate cdatm-nightly_py2</​code>​
 +    * ciclad: add to your .bashrc
 +      * <​code>#​ Prerequisites for using CLIMAF when not using the 
 +# default '​module load climaf'​
 +module load netcdf4 nco cdo ncl/6.3.0
 +
 +# CLIMAF environment variables
 +export CLIMAF=/​data/​jypmce/​cdat/​climaf
 +#export PYTHONPATH=$PYTHONPATH:​$CLIMAF
 +export PYTHONPATH=$CLIMAF
 +export PATH=$PATH:​$CLIMAF/​bin
 +export CLIMAF_CACHE=/​data/​jypmce/​climaf_cache
 +export TMPDIR=$CLIMAF_CACHE
 +export CLIMAF_LOG_DIR=$CLIMAF_CACHE
 +
 +#conda activate cdatm-nightly_py2
 +</​code>​
 +  * Open a new shell and <wrap em>run the tests again</​wrap>​. If there is an error that was not there, check what you have added to the shell init files
 +  * [[other:​uvcdat:​cdat_conda:​climaf_test|Test CLIMAF on ciclad]]
  
 ==== Cleaning up things ==== ==== Cleaning up things ====
Line 420: Line 481:
  
 We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions
-<​code>​(cdatm15) bash-4.2$ ​find cdatm15 \! -perm /g+r,o+r -ls +<​code> ​>find /home/share/unix_files/cdat/miniconda3/​envs ​\! -perm /g+r,o+r -ls 
-[...] +# Everything OK!</​code>​
-(cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp* +
-(cdatm15) bash-4.2$ find cdatm15 ​\! -perm /g+r,o+r -ls</​code>​+
  
  
Line 449: Line 508:
   * [[https://​rpy2.readthedocs.io/​en/​version_2.8.x/​|rpy2]]:​ providing simple and robust access to R from within Python   * [[https://​rpy2.readthedocs.io/​en/​version_2.8.x/​|rpy2]]:​ providing simple and robust access to R from within Python
   * [[http://​matplotlib.org/​cmocean/​|cmocean]]:​ beautiful colormaps for oceanography   * [[http://​matplotlib.org/​cmocean/​|cmocean]]:​ beautiful colormaps for oceanography
-  * [[http://​scitools.org.uk/​iris/​index.html|iris]]:​ A Python library for Meteorology and Climatology 
   * [[http://​www.gdal.org/​|OSGeo/​GDAL]]:​ Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats   * [[http://​www.gdal.org/​|OSGeo/​GDAL]]:​ Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats
     * [[https://​pcjericks.github.io/​py-gdalogr-cookbook/​|Python GDAL/OGR Cookbook]]     * [[https://​pcjericks.github.io/​py-gdalogr-cookbook/​|Python GDAL/OGR Cookbook]]
   * [[https://​github.com/​stefraynaud/​spanlib|spanlib]]:​ Spectral Analysis Library   * [[https://​github.com/​stefraynaud/​spanlib|spanlib]]:​ Spectral Analysis Library
-  * [[http://​wrf-python.readthedocs.io/​en/​latest/​|wrf-python]]:​ A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model 
-  * [[https://​glances.readthedocs.io/​en/​stable/​index.html|glances]]:​ a cross-platform monitoring tool (similar to ''​top''​) 
   * [[https://​ajdawson.github.io/​windspharm/​latest/​|windspharm]]:​ spherical harmonic wind analysis in Python   * [[https://​ajdawson.github.io/​windspharm/​latest/​|windspharm]]:​ spherical harmonic wind analysis in Python
  
Line 466: Line 522:
  
 ^  Environment\\ name  ^  Server ​ ^  ''​conda list'' ​ ^ ^  Environment\\ name  ^  Server ​ ^  ''​conda list'' ​ ^
- ​uvcdat-2.10  ​|  LSCE  |  ​{{  ​|package list}} ​ | +cdat-8.0_py2 |  LSCE  | | 
- ​cdatm15  ​|  LSCE  |  ​{{ ​|package list}} ​ |+| cdatm17_py2 |  LSCE | 
 +cdat-nightly_py2 ​|  LSCE\\ ciclad ​ | | 
 +| cdatm-nightly_py2 |  LSCE\\ ciclad ​ | |
  
 /* standard page footer */ /* standard page footer */
other/uvcdat/cdat_conda/cdat_8_0_py2.txt · Last modified: 2021/02/27 13:55 by jypeter