====== CDAT 8.1 installation notes ====== [ [[.:index|Back to all versions]] ] If you have questions about Python, read: * [[other:python:starting|Working with Python]] * [[other:python:jyp_steps|JYP's recommended steps for learning python]]! ===== Using JYP version ===== If you mostly want to use the CDAT 8.1 installed by JYP, just use the following steps and then read the //What should I do now?// section below * Initialize //conda// with: ^ Server ^ tcsh ^ bash ^ | LSCE | ''source ~jypeter/.conda3_jyp.csh'' | ''source ~jypeter/.conda3_jyp.sh'' | | ciclad | ''source ~jypmce/.conda3_jyp.csh'' | ''source ~jypmce/.conda3_jyp.sh'' | * Choose one of the [[#environments_summary|installed environments]] and activate it with: ''conda activate env_name'' * python 2.7.x: ''conda activate cdatm_py2'' * python 3.x: ''conda activate cdatm_py3'' (not available on ciclad, but could be installed) * Type ''which python'' (or the ''wp'' alias) and make sure you get something like\\ ''[...]/miniconda3/envs/env_name/bin/python'' ===== What should I do now? ===== ==== You just want to use the installed version ==== You are ready if you have typed the ''conda activate'' command specified in the previous section! * You may want to add the ''source'' line to your shell configuration file so that you don't have to remember it and type it each time you open a new terminal * Note: it's better to type the ''activate'' command in a terminal only when you want to use this specific version of python, rather than having the ''activate'' in your shell configuration file. This may have some side effects in some cases * Have a look at the [[other:uvcdat:cdat_conda:cdat_8_1#extra_packages_list|Extra package list]] at the bottom of this page to see what is available in this distribution * If you have not done it yet, read (or at least have a look at the list of sections): * [[other:python:starting|Working with Python]] * [[other:python:jyp_steps|JYP's recommended steps for learning python]] ==== You want to replicate this installation on your own server or desktop/laptop ==== Read carefully the following sections of this page! 8-) ===== What's New in CDAT 8.1? ===== * [[https://github.com/cdat/cdat/releases/tag/v8.1|8.1 announcement]] and features summary ([[https://github.com/UV-CDAT/uvcdat/releases|all versions]]) * Full [[https://uvcdat.llnl.gov/changelog.html#8.1|Change log]] Note: this particular CDAT installation at LSCE (and on other machines/servers) provides **many extra (non-CDAT) packages**. You can jump directly to the [[other:uvcdat:cdat_conda:cdat_8_1#extra_packages_list|Extra package list]] at the bottom of this page to see what is available ===== Installation with Miniconda3 ===== We assume that Miniconda3 is already installed. Otherwise, follow the the [[other:uvcdat:cdat_conda:cdat_8_0_py2#installing_miniconda3|Installing Miniconda]] steps (and the //Post-Miniconda3 installation// section) we followed when installing CDAT 8.0 ==== Updating conda ==== The conda package itself can be updated (if need be) with\\ ''conda update -n base -c defaults conda'' ==== Installing CDAT 8.1 ==== We first check that we have indeed access to a conda installation, and we assume that we have a write-access to the conda hierarchy, and a few Gb of disk space available >conda activate (base) >conda env list | grep base base * /home/share/unix_files/cdat/miniconda3 (base) >df -h /home/share/ Filesystem Size Used Avail Use% Mounted on prolix3:/share 917G 171G 700G 20% /home/share (base) >du -sh /home/share/unix_files/cdat/miniconda3 7.8G /home/share/unix_files/cdat/miniconda3 We can also use ''conda env list'' and remove some obsolete versions with ''conda remove -n obsolete_name --all'' to get some space === Python 2.7 version === $ conda create -n cdat-8.1_py2 -c cdat/label/v81 -c conda-forge python=2.7 cdat # Generate the list of installed packages $ conda list -n cdat-8.1_py2 > /home/scratch01/jypeter/cdat-8.1_py2_list_190307.txt List of installed packages: {{:other:uvcdat:cdat_conda:cdat-8.1_py2_list_190307.txt}} === Python 3.6 version === $ conda create -n cdat-8.1_py3 -c cdat/label/v81 -c conda-forge python=3.6 cdat ==== Installing CDAT nightly ==== Notes: * This page is about CDAT 8.1, but we have added a short nightly section here as a convenient shortcut. This should probably be moved to a stand-alone nightly page later * The nightly version is **not automatically updated**, and may well be out-of-date * [[https://github.com/CDAT/cdat/wiki/Obtain-CDAT-nightly-packages|Official nightly page]] === Python 2.7 nightly version === $ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge python=2.7 cdat Not tested! Can this be updated with ''conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge --all'' ??? === Python 3.6 nightly version === Not installed yet! $ conda create -n cdat-nightly_py3 -c cdat/label/nightly -c conda-forge python=3.6 cdat ===== Cloning cdat to add specific packages for LSCE ===== This section is about the creation of the **cdatm18** environment Notes about [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat|actually using the cdatm18 conda-based python]] Note: using hard links, //cloning// a full environment uses less disk space than making a real copy $ conda create -n cdatm18_py2 --clone cdat-8.1_py2 $ du -sh cdat-8.1_py2 cdatm18_py2 1.9G cdat-8.1_py2 448M cdatm18_py2 ==== cdat nightly case ==== ''conda create -n cdatm-nightly_py2 %%--%%clone cdat-nightly_py2'' ===== Getting ready for a moving default CDAT ===== This step should probably be listed at the **end**, especially in a multi-user environment! If there is already a ''cdatm'' link, make sure that the new version is stable and working correctly before updating the ''cdatm'' link We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version $ cd /home/share/unix_files/cdat/miniconda3/envs $ ln -s cdatm18_py2 cdatm_py2 ===== Customizing UV-CDAT for LSCE ===== ==== Testing vcs ==== Start python and paste the following lines import numpy as np, vcs id100 = np.identity(100) x = vcs.init() x.plot(id100) If you get the kind of errors described in [[https://github.com/CDAT/vcs/issues/393|x.plot crashes with OpenGL2 error]], the drivers on your server are probably not compatible with an interactive use of vcs... The workaround is to **install mesalib and work in //headless mode//** (e.g. without a display)... Install //mesalib// with ''conda install -c cdat/label/v81 -c conda-forge mesalib vtk-cdat'' and try pasting again the few code lines above in python. If things work as expected, no canvas will be opened, but you'll be able to save the canvas to a file with x.png('test_mesalib') x.pdf('test_mesalib') Note: using //mesalib// for headless mode work makes it possible to work without an X server, and with no DISPLAY variable defined ==== Downloading cdms2/vcs test data ==== You should download the test data (174M of data...) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...) $ conda activate cdatm18_py2 (cdatm18_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)' [...] Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data (cdatm18_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data 174M /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data ==== Packages that have no dependency problems ==== After cloning, we are ready to install some extra packages that may be requested by LSCE users * We first try to install together as many packages as possible that don't require other channels than //conda-forge//, and that don't request a downgrade of what is already installed * We then install individual extra packages with ''conda install'' or ''pip install'' # You can use the following to keep a trace of what will be installed #$ conda install --dry-run -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm > somewhere/extra_packages.txt # Install... $ conda install -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret ferret_datasets basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm [...] # Use a similar install line if you want to install the same packages in the python3 version List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_190304.txt |}} Packages installed with //pip// * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API * ''conda activate cdatm18_py2''\\ ''pip install dreqPy'' * Update with: ''pip install %%--%%upgrade dreqPy'' * Get version number with:\\ $ drq -v dreqPy version 01.00.29 [Version 01.00.29] * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret * ''conda activate cdatm18_py2''\\ ''pip install ferretmagic'' The following packages have no dependency problems and were installed (or updated) //later// * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files * ''conda install -n cdatm18_py2 -c conda-forge cmor'' * Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' '' * Warning! [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]] * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations * ''conda install -n cdatm18_py2 -c conda-forge -c vacumm vacumm'' * Warning! [[https://github.com/VACUMM/vacumm/issues/15|standard install of vacumm does not work for Python3]] * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library * ''conda install -n cdatm18_py2 -c stefraynaud -c conda-forge spanlib'' * Test: ''python -c 'from spanlib.analyzer import Analyzer' '' * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses * ''conda install -n cdatm18_py2 -c conda-forge cartopy'' * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs * ''conda install -n cdatm18_py2 -c conda-forge joblib'' * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]] * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python * ''conda install -n cdatm18_py2 -c conda-forge palettable'' ==== TODO ==== //Add here packages that would be useful and have not been installed yet, or have some problems that prevent their installation// * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology * ''conda install -n cdatxxx -c conda-forge iris'' * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model * ''conda install -n cdatm15 -c conda-forge wrf-python'' * [[https://glances.readthedocs.io/en/stable/index.html|glances]]: a cross-platform monitoring tool (similar to ''top'') * ''conda install -n cdatm15 -c conda-forge glances'' * [[https://github.com/nicolargo/glances|glances@github]] ==== Other packages ==== There is no warranty that the packages listed below will work correctly, because it was required to bypass the compatibility checks in order to install them... * NO such packages now! ==== Updating some packages ==== Some packages change more often than others, and can be easily updated the following way: * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]] * Update with: ''pip install %%--%%upgrade dreqPy'' * Get version number with: ''drq -v'' ==== Cleaning up things ==== Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg [[https://github.com/PCMDI/pcmdi_metrics/issues/496|pcmdi-metrics]] in 2.8.0 and [[https://github.com/UV-CDAT/cdp/issues/21|cdp]] in 2.10) We check if some of the installed files are missing read access for the //group// or //other//, and we manually change the permissions >find /home/share/unix_files/cdat/miniconda3/envs \! -perm /g+r,o+r -ls # Everything OK! ===== Extra packages list ===== * [[https://python-pillow.org/|pillow]]: the friendly PIL (//Python Imaging Library//) fork * [[http://pandas.pydata.org/|pandas]]: Python Data Analysis Library * [[http://statsmodels.sourceforge.net/|statsmodels]]: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests * [[https://seaborn.pydata.org/|seaborn]]: statistical data visualization * [[http://scikit-image.org/|scikit-image]]: image processing in Python * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox * [[https://pypi.python.org/pypi/gsw/|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in Python * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret * [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: high resolution data for ''basemap'' * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests * [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files * Note: this is a dependency of ''dreqPy'' * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python ==== Removed packages ==== * NO removed packages! ===== Environments summary ===== After following the steps above, we get the following environments. Use the ''conda env list'' command (same result as ''conda info %%--%%envs'') to get the up-to-date list of available environments. Use ''conda list -n env_name'' to get a detailed list of which packages/modules are installed, or check the [[other:uvcdat:conda_notes#installation_history|conda Installation history]] section to get more details ^ Environment\\ name ^ Server ^ Summary ^ | cdat-8.1_py2 | LSCE\\ ciclad | CDAT 8.1 & Python 2.7 | | cdat-8.1_py3 | LSCE | CDAT 8.1 & Python 3.6 | | cdatm18_py2\\ or //cdatm_py2// | LSCE\\ ciclad | CDAT 8.1 & P 2.7 //JYP version// | | cdatm18_py3\\ or //cdatm_py3// | LSCE | CDAT 8.1 & P 3.7 //JYP version// | | cdat-nightly_py2 | LSCE\\ ciclad | CDAT nightly & Python 2.7 | | cdatm-nightly_py2 | LSCE\\ ciclad | CDAT nightly & P 2.7 //JYP version// | /* standard page footer */ \\ \\ \\ ---- [ [[pmip3:|PMIP3 Wiki Home]] ] - [ [[pmip3:wiki_help|Help!]] ] - [ [[wiki:syntax|Wiki syntax]] ]