Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision |
other:uvcdat:cdat_conda:cdat_8_0_py2 [2019/03/04 17:55] – jypeter | other:uvcdat:cdat_conda:cdat_8_0_py2 [2021/02/26 19:31] – Improved... jypeter |
---|
==== Installing Miniconda3 ==== | ==== Installing Miniconda3 ==== |
| |
We have installed a new version of Miniconda3 (steps below adapted from the [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions) | * You don't need to be (and **you should not be**) //root// when you install Miniconda3. You just need enough disk space where you have write access |
| * If you are installing Miniconda3 in a Linux environment on a [[other:win10wsl|Windows 10 computer using Windows Subsystem for Linux (WSL)]], pay special attention to the instructions on the **WSL** lines |
| |
* Get the [[https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh|Python 3.7 Linux 64-bit (bash installer)]] (unless there is a [[https://conda.io/miniconda.html|more recent version?]]) | The steps below are adapted from the older (but with lots of extra and possibly useful details) [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions |
* Find some temporary space on Linux or Windows | |
* Linux:''cd /home/scratch01/jypeter'' | * Get the [[https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh|Python 3.8 Linux 64-bit (bash installer)]] (unless there is a [[https://conda.io/miniconda.html|more recent version?]]) |
* WSL: ''cd /mnt/h/Scratch/jypeter/'' (assuming that there is a ''H:\Scratch\jypeter'' directory) | * Find some temporary space on Linux (or Windows, if you are using WSL) |
* ''wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh'' (67 Mo, 15 Feb 2019) | * e.g. Linux at LSCE:\\ ''cd /home/scratch01/<your_login>'' |
| * e.g. WSL, assuming that there is a ''C:\Scratch\<your_login>'' directory:\\ ''cd /mnt/c/Scratch/<your_login>'' |
| * Use ''wget'' to download the installer:\\ ''wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'' (90 Mb, 26 Feb 2021) |
* Execute the installer | * Execute the installer |
* ''bash Miniconda3-latest-Linux-x86_64.sh'' | * ''bash Miniconda3-latest-Linux-x86_64.sh'' |
* Accept the license | * Accept the license |
* Specify an explicit installation path **outside of the home directory**, with enough disk space (more than 3G if you are going to install UV-CDAT and some extra packages), preferably on a disk that is not backed up: | * Specify an explicit installation path **outside of your //home// directory**, with enough disk space (more than 3 Gb if you are going to install CDAT and some extra packages), preferably on a disk that is not backed up: |
* Linux LSCE: ''/home/share/unix_files/cdat/miniconda3'' | * Installations by //JYP//: |
* Linux ciclad: ''/data/jypmce/cdat/miniconda3'' | * Linux at LSCE: ''/home/share/unix_files/cdat/miniconda3'', or another subdirectory of ''/home/share/unix_files/cdat/'' |
| * Linux on ciclad: ''/data/jypmce/cdat/miniconda3'', or another subdirectory of ''/data/jypmce/cdat/'' |
* WSL: installing to a directory that is not in ''/home/'' <wrap hi>does not work</wrap> (e.g. ''/mnt/h/CDAT/miniconda3'',assuming there is a ''H:\CDAT\'' directory, does not work)\\ Need to accept the installation in the default location: ''/home/jypeter/miniconda3'' | * WSL: installing to a directory that is not in ''/home/'' <wrap hi>does not work</wrap> (e.g. ''/mnt/h/CDAT/miniconda3'',assuming there is a ''H:\CDAT\'' directory, does not work)\\ Need to accept the installation in the default location: ''/home/jypeter/miniconda3'' |
* Resulting ''miniconda3'' directory size is 302M\\ <code> >du -sh miniconda3 | * Resulting ''miniconda3'' directory size is 302M\\ <code> >du -sh miniconda3 |
* Update with: ''pip install %%--%%upgrade dreqPy'' | * Update with: ''pip install %%--%%upgrade dreqPy'' |
* Get version number with:\\ <code>$ drq -v | * Get version number with:\\ <code>$ drq -v |
dreqPy version 01.00.06 [Version 01.00.06]</code> | dreqPy version 01.00.29 [Version 01.00.29]</code> |
| |
The following packages have no dependency problems and were installed (or updated) //later// | The following packages have no dependency problems and were installed (or updated) //later// |
* [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology | |
* ''conda install -n cdatm15 -c conda-forge iris'' | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations |
| * ''conda install -n cdatm17_py2 -c conda-forge -c vacumm vacumm'' |
| |
* [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library |
* ''conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib'' | * ''conda install -n cdatm17_py2 -c stefraynaud -c conda-forge spanlib'' |
* Test: ''python -c 'from spanlib.analyzer import Analyzer' '' | * Test: ''python -c 'from spanlib.analyzer import Analyzer' '' |
| |
* [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework |
* ''conda install -n cdatm15 -c conda-forge wrf-python'' | * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]] |
| |
* [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'') | |
* ''conda install -n cdatm15 -c conda-forge glances'' | |
* [[https://github.com/nicolargo/glances|glances@github]] | |
| |
==== TODO ==== | ==== TODO ==== |
| |
//Add here packages that would be useful but have some problems that prevent their installation// | //Add here packages that would be useful and have not been installed yet, or have some problems that prevent their installation// |
| |
* [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology |
* As of August 08 2017, installing vacumm in 2.10 requires downgrading to 2.10 (see [[https://github.com/VACUMM/vacumm/issues/9|github issue]])! | * ''conda install -n cdatxxx -c conda-forge iris'' |
* <wrap hi>Problem solved</wrap> with **vacumm 3.4.1-1** on August 31st! Installation was OK with\\ ''conda install -n cdatm15 -c conda-forge -c vacumm vacumm'' | |
| * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model |
| * ''conda install -n cdatxxx -c conda-forge wrf-python'' |
| |
| * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'') |
| * ''conda install -n cdatxxx -c conda-forge glances'' |
| * [[https://github.com/nicolargo/glances|glances@github]] |
| |
==== Other packages ==== | ==== Other packages ==== |
| |
* [[https://cmor.llnl.gov/|CMOR]] | * [[https://cmor.llnl.gov/|CMOR]] |
* Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor'' | * Update with: ''conda update -n cdatxxx -c conda-forge -c pcmdi -c uvcdat cmor'' |
* Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' '' | * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' '' |
| |
* Update with: ''pip install %%--%%upgrade dreqPy'' | * Update with: ''pip install %%--%%upgrade dreqPy'' |
* Get version number with: ''drq -v'' | * Get version number with: ''drq -v'' |
| |
| ==== Installing CLIMAF ==== |
| |
| The [[https://climaf.readthedocs.io/|CLIMAF]] installation is a special case, because we may want to use it with the Python installed for CDAT, but it is not installed using conda |
| |
| We follow the steps for the [[https://climaf.readthedocs.io/en/latest/installing.html#installing-climaf-if-you-are-not-at-cnrm-on-ciclad-or-climserv|installation of a non-standard server]] |
| |
| * Go to the appropriate installation directory (we choose the directory where we have also installed miniconda3): |
| * LSCE: ''cd /home/share/unix_files/cdat'' |
| * ciclad: ''/data/jypmce/cdat/'' |
| * ''git clone https://github.com/senesis/climaf climaf'' |
| * Add CLIMAF to the PYTHONPATH environment variable |
| * LSCE & tcsh: ''setenv PYTHONPATH ${PYTHONPATH}:/home/share/unix_files/cdat/climaf'' |
| * LSCE & bash: ''export PYTHONPATH=${PYTHONPATH}:/home/share/unix_files/cdat/climaf'' |
| * ciclad: same as above, using ''/data/jypmce/cdat/climaf'' |
| * Test the installation |
| * make sure the [[https://climaf.readthedocs.io/en/latest/requirements.html#requirements|Requirements]] are available: |
| * LSCE: ''module load netcdf/4 nco cdo ncl ncview'' |
| * ciclad: ''module load netcdf4 nco cdo ncl/6.3.0'' |
| * optional: start an X server |
| * <code>cd climaf/testing |
| ./test_install.sh</code> |
| * Once the tests are OK, add the required //module load// and environment variables definitions to the ''.basrc'' and ''.cshrc'' files. <wrap hi>Note: maybe these lines should go to the ''.profile'' and ''.login'' files...(check [[https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html|Bash Startup Files]])</wrap> |
| * LSCE: add to your .cshrc |
| * <code># Prerequisites for using CLIMAF |
| module load netcdf/4 nco cdo ncl ncview |
| |
| # CLIMAF environment variables |
| setenv CLIMAF /home/share/unix_files/cdat/climaf |
| setenv PYTHONPATH "${PYTHONPATH}:${CLIMAF}" |
| #setenv PYTHONPATH $CLIMAF |
| setenv PATH "${PATH}:${CLIMAF}/bin" |
| setenv CLIMAF_CACHE "/home/scratch01/${USER}/climaf_cache" |
| setenv TMPDIR $CLIMAF_CACHE |
| setenv CLIMAF_LOG_DIR $CLIMAF_CACHE |
| |
| #conda activate cdatm-nightly_py2</code> |
| * ciclad: add to your .bashrc |
| * <code># Prerequisites for using CLIMAF when not using the |
| # default 'module load climaf' |
| module load netcdf4 nco cdo ncl/6.3.0 |
| |
| # CLIMAF environment variables |
| export CLIMAF=/data/jypmce/cdat/climaf |
| #export PYTHONPATH=$PYTHONPATH:$CLIMAF |
| export PYTHONPATH=$CLIMAF |
| export PATH=$PATH:$CLIMAF/bin |
| export CLIMAF_CACHE=/data/jypmce/climaf_cache |
| export TMPDIR=$CLIMAF_CACHE |
| export CLIMAF_LOG_DIR=$CLIMAF_CACHE |
| |
| #conda activate cdatm-nightly_py2 |
| </code> |
| * Open a new shell and <wrap em>run the tests again</wrap>. If there is an error that was not there, check what you have added to the shell init files |
| * [[other:uvcdat:cdat_conda:climaf_test|Test CLIMAF on ciclad]] |
| |
==== Cleaning up things ==== | ==== Cleaning up things ==== |
| |
We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions | We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions |
<code>(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls | <code> >find /home/share/unix_files/cdat/miniconda3/envs \! -perm /g+r,o+r -ls |
[...] | # Everything OK!</code> |
(cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp* | |
(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls</code> | |
| |
| |
* [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python | * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python |
* [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography |
* [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology | |
* [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats |
* [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] | * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] |
* [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library |
* [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | |
* [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'') | |
* [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python |
| |
| |
^ Environment\\ name ^ Server ^ ''conda list'' ^ | ^ Environment\\ name ^ Server ^ ''conda list'' ^ |
| uvcdat-2.10 | LSCE | {{ |package list}} | | | cdat-8.0_py2 | LSCE | | |
| cdatm15 | LSCE | {{ |package list}} | | | cdatm17_py2 | LSCE | | |
| | cdat-nightly_py2 | LSCE\\ ciclad | | |
| | cdatm-nightly_py2 | LSCE\\ ciclad | | |
| |
/* standard page footer */ | /* standard page footer */ |