====== UV-CDAT 2.8.0 installation notes ====== [ [[.:index|Back to all versions]] ] Follow the instructions about the [[other:python:starting#conda-based_versions_of_uv-cdat|conda-based versions of UV-CDAT initialization]] for actually using an installed version of ''2.8.0'' ===== What's New? ===== * [[https://github.com/UV-CDAT/uvcdat/releases/tag/v2.8|2.8 announcement]] and features summary ([[https://github.com/UV-CDAT/uvcdat/releases|all versions]]) * Full [[https://uv-cdat.llnl.gov/changelog.html#2.8|Change log]] ===== Installation with Anaconda2 ===== [[#installation_with_miniconda2|Installing with Miniconda2]] rather than Anaconda2 is **much better** because it will save disk space! # Keep a trace of what will be installed, with/without conda-forge conda create --dry-run -n uvcdat-2.8.0 -c uvcdat uvcdat=2.8.0 > /home/scratch01/jypeter/uvcdat-2.8.0_install_dry_no-forge_160220.txt conda create --dry-run -n uvcdat-2.8.0 -c conda-forge -c uvcdat uvcdat=2.8.0 > /home/scratch01/jypeter/uvcdat-2.8.0_install_dry_with-forge_160220.txt # Do the actual installation (with conda-forge) conda create -n uvcdat-2.8.0 -c conda-forge -c uvcdat uvcdat=2.8.0 Lists of installed packages: {{ :other:uvcdat:uvcdat-2.8.0_install_dry_with-forge_160220.txt |with conda-forge}} or {{ :other:uvcdat:uvcdat-2.8.0_install_dry_no-forge_160220.txt |without conda-forge}} Size of ''/home/share/unix_files/cdat/anaconda2'': * before installation: **2.1G** * after installation (with conda-forge): **4.6G** ===== Installation with Miniconda2 ===== \\ We assume that [[other:uvcdat:conda_notes#miniconda|Miniconda2]] is already installed Note: after installation, we use **//only// 2.7G** (details below), compared to the 4.6G of the Anaconda2 installation # Keep a trace of what will be installed (with conda-forge) conda create --dry-run -n uvcdat-2.8.0 -c conda-forge -c uvcdat uvcdat=2.8.0 > /home/scratch01/jypeter/uvcdat-2.8.0_install_dry_with-forge_160220.txt # Do the actual installation conda create -n uvcdat-2.8.0 -c conda-forge -c uvcdat uvcdat=2.8.0 Lists of installed packages: {{ :other:uvcdat:uvcdat-2.8.0_install_dry_with-forge_160221.txt |List of installed packages}} bash-4.1$ conda info --envs # conda environments: # uvcdat-2.8.0 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 root * /home/share/unix_files/cdat/miniconda2 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 2.7G /home/share/unix_files/cdat/miniconda2 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/pkgs 2.6G /home/share/unix_files/cdat/miniconda2/pkgs bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/* 4.4M /home/share/unix_files/cdat/miniconda2/bin 4.0K /home/share/unix_files/cdat/miniconda2/conda-bld 276K /home/share/unix_files/cdat/miniconda2/conda-meta 2.0G /home/share/unix_files/cdat/miniconda2/envs 20K /home/share/unix_files/cdat/miniconda2/etc 3.8M /home/share/unix_files/cdat/miniconda2/include 99M /home/share/unix_files/cdat/miniconda2/lib 4.0K /home/share/unix_files/cdat/miniconda2/LICENSE.txt 665M /home/share/unix_files/cdat/miniconda2/pkgs 16K /home/share/unix_files/cdat/miniconda2/share 8.0K /home/share/unix_files/cdat/miniconda2/ssl bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/* 56M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/bin 1.9M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/conda-meta 48K /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/etc 420K /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/imports 67M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/include 1.7G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/lib 24K /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/man 3.1M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/mkspecs 16M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/mod 3.5M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/plugins 57M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/share 4.3M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/ssl 2.7M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/tests 12K /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/var 107M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0/vistrails ===== Downgrading conda ===== Note: on June 1st 2017, we used the following step in order to **down**grade conda and be ready to install UV-CDAT 2.10 that required an older version of conda (strictly older than 4.3.13) than the one used for 2.8 bash-4.1$ conda install -n root "conda<4.3.13" Fetching package metadata ......... Solving package specifications: . Package plan for installation in environment /home/share/unix_files/cdat/miniconda2: The following packages will be DOWNGRADED due to dependency conflicts: conda: 4.3.13-py27_0 --> 4.3.11-py27_0 Proceed ([y]/n)? y conda-4.3.11-p 100% |###############################################################| Time: 0:00:00 4.28 MB/s bash-4.1$ conda info Current conda install: platform : linux-64 conda version : 4.3.11 conda is private : False conda-env version : 4.3.11 ===== Cloning uvcdat to add specific packages for LSCE ===== This section is about the creation of the **cdatm14** environment Notes about [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat|actually using the cdatm14 conda-based python]] Note: using hard links, cloning a full environment only adds an extra 100M of disk space bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 2.7G /home/share/unix_files/cdat/miniconda2 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 bash-4.1$ conda create -n cdatm14 --clone uvcdat-2.8.0 [...] bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 2.8G /home/share/unix_files/cdat/miniconda2 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* 2.0G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 82M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 ==== Downloading cdms2/vcs test data ==== You should download the test data and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...) (cdatm14) bash-4.1$ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data' Finished downloading sample data to /home/share/unix_files/cdat/miniconda2/envs/cdatm14/share/uvcdat/sample_data (cdatm14) bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/cdatm14/share/uvcdat/sample_data 174M /home/share/unix_files/cdat/miniconda2/envs/cdatm14/share/uvcdat/sample_data ==== Packages that have no dependency problems ==== After cloning, we are ready to install some extra packages that may be requested by LSCE users # Keep a trace of what will be installed bash-4.1$ conda install --dry-run -n cdatm14 -c conda-forge -c uvcdat -c vacumm pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret vacumm > /home/scratch01/jypeter/lsce-extra_01_install_160223.txt # Install... bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat -c vacumm pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret vacumm # Check the disk space after installation bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 3.6G /home/share/unix_files/cdat/miniconda2 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* 2.4G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 82M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 # Check the disk space again after cleaning bash-4.1$ conda clean all bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 2.9G /home/share/unix_files/cdat/miniconda2 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* 2.4G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 82M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 Lists of installed packages: {{ :other:uvcdat:lsce-extra_01_install_160223.txt |}} The following packages have no dependency problems and were installed //later// * High resolution data for ''basemap'' * ''conda install -n cdatm14 -c conda-forge basemap-data-hires'' * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP) * ''conda install -n cdatm14 -c conda-forge -c pcmdi pcmdi_metrics'' * Note: There was a [[https://github.com/PCMDI/pcmdi_metrics/issues/496|read permission problem on the files]] - Solved in 2.10 * [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files * ''conda install -n cdatm14 xlsxwriter'' * Note: this is a dependency of ''dreqPy'' * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API * ''pip install dreqPy'' * Update with: ''pip install %%--%%upgrade dreqPy'' * Get version number with:\\ $ drq -v dreqPy version 01.00.06 [Version 01.00.06] * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files * ''conda install -n cdatm14 -c conda-forge -c pcmdi -c uvcdat cmor'' * Did not complete the installation yet, because it requested a downgrade of libnetcdf (see [[https://github.com/PCMDI/cmor/issues/138|github issue]]) * Installation issue was solved in ''cmor-3.2.3'' * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module * ''pip install dtw'' * Requested by Sentia Goursaud ==== TODO ==== //Add here packages that would be useful but have some problems that prevent their installation// * [[https://earthsystemcog.org/projects/esmpy/|ESMPy]]: ESMF Python Regridding Interface - Works in [[.:cdat_2_10|2.10]]! * ''conda install -n cdatm14 -c conda-forge esmpy'' * Installation of esmpy ''7.0.0'' did not report errors, **but** esmpy seemed to have libraries based importation problems and has been removed * ''conda -n cdatm14 remove esmpy'' * [[http://www.gdal.org/|GDAL]]: Geospatial Data Abstraction Library - Works in [[.:cdat_2_10|2.10]]! * Cannot be installed due to too many dependency problems, and because it requires the downgrade of geos and rpy2 ==== Other packages ==== There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them... * ''shapely'' request a downgrade of ''geos'' when installed from //conda-forge// (see [[https://github.com/Toblerity/Shapely/issues/472|github issue]]) but it will work when installed from the default repo * ''conda install -n cdatm14 shapely'' * We get a slightly older version (//1.5.**16**-py27_0// instead of //1.5.**17**-np111py27_2//) * Note: ''cartopy'' (below) requests the same downgrade of ''geos''... * ''cartopy'' requests a downgrade of ''geos'' and ''matplotlib'' (see [[https://github.com/SciTools/cartopy/issues/853#issuecomment-282023158|github issue]]) * We install the requested dependencies and then we **force** the installation of ''cartopy''\\ (uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge -c uvcdat libxslt lxml owslib pyepsg [...] The following NEW packages will be INSTALLED: libxslt: 1.1.29-3 conda-forge lxml: 3.7.2-py27_1 conda-forge owslib: 0.14.0-py27_0 conda-forge pyepsg: 0.3.1-py27_0 conda-forge [...] (uvcdat-2.8.0) bash-4.1$ conda install -n cdatm14 -c conda-forge cartopy [...] The following NEW packages will be INSTALLED: cartopy: 0.15.0-np111py27_0 conda-forge The following packages will be DOWNGRADED due to dependency conflicts: geos: 3.6.0-0 conda-forge --> 3.5.1-1 conda-forge matplotlib: 2.0.0-np111py27_2 conda-forge --> 1.5.3-np111py27_7 conda-forge Proceed ([y]/n)? n (uvcdat-2.8.0) bash-4.1$ conda install --force -n cdatm14 -c conda-forge cartopy [...] The following NEW packages will be INSTALLED: cartopy: 0.15.0-np111py27_0 conda-forge Proceed ([y]/n)? y * trying to install ''rpy2'' will trigger the installation of ''R''! * We force the **installation of only rpy2** in order to use the ''R'' already installed at LSCE * ''conda install %%--%%force -n cdatm14 -c conda-forge -c uvcdat rpy2'' ===== Extra packages list ===== * [[https://python-pillow.org/|pillow]]: the friendly PIL (//Python Imaging Library//) fork * [[http://pandas.pydata.org/|pandas]]: Python Data Analysis Library * [[http://statsmodels.sourceforge.net/|statsmodels]]: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests * [[https://seaborn.pydata.org/|seaborn]]: statistical data visualization * [[http://scikit-image.org/|scikit-image]]: image processing in Python * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox * [[https://pypi.python.org/pypi/gsw/|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in Python * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations * [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: high resolution data for ''basemap'' * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests * [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) * [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python ==== Removed packages ==== * [[https://pypi.python.org/pypi/wget|wget]]: pure python download utility * Used ''conda install -n cdatm14 -c daler wget=3.2'' in order to get version **3.2**, the same as the one on Pypi * **REMOVED** because it did not work correctly in the script where it was needed: it emulated wget with //ftp// and failed in our case after downloading the first file ===== Environments summary ===== After following the steps above, we get the following environments. Use the ''conda info --envs'' command to get the up-to-date list of available environments ^ Environment\\ name ^ Server ^ ''conda list'' ^ | uvcdat-2.8.0 | LSCE\\ ciclad | {{ :other:uvcdat:cdat_conda:conda_list_lsce_uvcdat-2.8.0_170310.txt |package list}} | | cdatm14 | LSCE\\ ciclad | {{ :other:uvcdat:cdat_conda:conda_list_lsce_cdatm14_170310.txt|package list}} | /* standard page footer */ \\ \\ \\ ---- [ [[pmip3:|PMIP3 Wiki Home]] ] - [ [[pmip3:wiki_help|Help!]] ] - [ [[wiki:syntax|Wiki syntax]] ]