This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:uvcdat:cdat_conda:cdat_8_2_1 [2021/03/06 20:02] jypeter [Packages that have no dependency problems] Lots of changes, including using mamba |
other:uvcdat:cdat_conda:cdat_8_2_1 [2022/03/23 14:51] jypeter [Extra packages list] Mention ESMValTool |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== CDAT 8.2.1 installation notes ====== | + | ====== Python distribution based on CDAT 8.2.1 notes ====== |
[ [[.:index|Back to all versions]] ] | [ [[.:index|Back to all versions]] ] | ||
Line 11: | Line 11: | ||
===== Using JYP version ===== | ===== Using JYP version ===== | ||
- | If you mostly want to use the CDAT 8.2.1 installed by JYP, just use the following steps and then read the //What should I do now?// section below | + | If you mostly want to use the **python distribution based on CDAT 8.2.1** installed by JYP, just use the following steps and then read the [[#what_should_i_do_now|What should I do now?]] section below |
* Initialize //conda// with: | * Initialize //conda// with: | ||
Line 23: | Line 23: | ||
* python 3.x: ''conda activate cdatm_py3'' | * python 3.x: ''conda activate cdatm_py3'' | ||
- | * Type ''which python'' (or the ''wp'' alias) and make sure you get something like\\ ''[...]/miniconda3/envs/env_name/bin/python'' | + | * Type ''which python'' (or the ''wp'' alias) and make sure you get something looking like |
+ | * ''[...]/miniconda3<possibly_some_version>/envs/<some_env_name>/bin/python'' | ||
+ | * e.g. ''cdatm_py3'' environment at LSCE:\\ ''/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python'' | ||
Line 40: | Line 42: | ||
==== You want to replicate this installation on your own server or desktop/laptop ==== | ==== You want to replicate this installation on your own server or desktop/laptop ==== | ||
+ | Read the next sections of this web page | ||
+ | |||
+ | There are some slight differences if you want to install this distribution: | ||
+ | * just for you. This is the usual case. Well, if you break something you can always remove everything and start from scratch | ||
+ | * for sharing a stable version with several other people | ||
+ | |||
+ | The notes at the beginning of each section will tell you if you can skip the section if you are just installing for you | ||
===== What's New in CDAT 8.2.1? ===== | ===== What's New in CDAT 8.2.1? ===== | ||
Line 45: | Line 54: | ||
* Full [[https://uvcdat.llnl.gov/changelog.html#8.2.1|Change log]] | * Full [[https://uvcdat.llnl.gov/changelog.html#8.2.1|Change log]] | ||
- | Note: this particular CDAT installation at LSCE (and on other machines/servers) provides **many extra (non-CDAT) packages**. You can jump directly to the [[#extra_packages_list|Extra package list]] at the bottom of this page to see what is available | + | Note: this particular CDAT installation at LSCE (and on other machines/servers) provides **many extra (non-CDAT) packages**. You can jump directly to the [[#extra_packages_list|Extra package list]] at the bottom of this page, if you want to see what is available |
===== Installation with Miniconda3 ===== | ===== Installation with Miniconda3 ===== | ||
- | We assume that Miniconda3 is already installed. Otherwise, follow the the [[other:uvcdat:cdat_conda:cdat_8_0_py2#installing_miniconda3|Installing Miniconda]] steps (and the //Post-Miniconda3 installation// section) we followed when installing CDAT 8.0 | + | ==== Installing Miniconda3 ==== |
+ | |||
+ | If //Miniconda3// is not already installed, or if you want to start over with a brand new version, read [[other:uvcdat:cdat_conda:miniconda3_install|JYP steps for installing Miniconda3]]. | ||
- | ==== Updating conda ==== | ||
- | The conda package itself can be updated (if need be) with\\ ''conda update -n base -c defaults conda'' | ||
==== Installing CDAT 8.2.1 ==== | ==== Installing CDAT 8.2.1 ==== | ||
Line 108: | Line 117: | ||
Resulting list of installed packages | Resulting list of installed packages | ||
- | * ''conda list -n cdat-8.2.1_py3 > cdat_8.2.1_installed_packages.txt'' | + | * ''conda list -n cdat-8.2.1_nompi_py3 > cdat_8.2.1_nompi_21-03-05.txt'' |
- | * {{ :other:uvcdat:cdat_conda:cdat_8.2.1_installed_packages.txt |}} | + | * no mpi: {{ :other:uvcdat:cdat_conda:cdat_8.2.1_nompi_21-03-05.txt |}} |
+ | * with mpî: {{ :other:uvcdat:cdat_conda:cdat_8.2.1_installed_packages.txt |}} | ||
[[https://github.com/CDAT/cdat/issues/2265|Some extra installation notes]] | [[https://github.com/CDAT/cdat/issues/2265|Some extra installation notes]] | ||
Line 175: | Line 185: | ||
</code> | </code> | ||
- | ===== Customizing UV-CDAT for LSCE ===== | + | ===== Customizing CDAT for LSCE ===== |
==== A quick test of cdms2 and vcs ==== | ==== A quick test of cdms2 and vcs ==== | ||
Line 217: | Line 227: | ||
=== Pre-installation check with the dry-run option === | === Pre-installation check with the dry-run option === | ||
- | <code>/usr/bin/time mamba install -n cdatm19_nompi_py3 --dry-run -c conda-forge basemap basemap-data-hires cartopy cmocean cmor ferret_datasets gdal gdal glances gsw iris joblib netcdf4 palettable pandas pillow pyferret rpy2 scikit-image scikit-learn seaborn seawater shapely spyder statsmodels windspharm wrf-python xarray xlsxwriter | + | <code>/usr/bin/time mamba install -n cdatm19_nompi_py3 --dry-run -c conda-forge basemap basemap-data basemap-data-hires cartopy cmocean cmor ferret_datasets gdal gdal geopandas glances gsw iris joblib netcdf4 palettable pandas pillow pyferret rpy2 scikit-image scikit-learn seaborn seawater shapely spyder statsmodels windspharm wrf-python xarray xlsxwriter |
[... lots of information about what would be done followed by some information about how long it took ...] | [... lots of information about what would be done followed by some information about how long it took ...] | ||
Line 234: | Line 244: | ||
=== Actual installation === | === Actual installation === | ||
- | <code>/usr/bin/time mamba install -n cdatm19_nompi_py3 -c conda-forge basemap basemap-data-hires cartopy cmocean cmor ferret_datasets gdal gdal glances gsw iris joblib netcdf4 palettable pandas pillow pyferret rpy2 scikit-image scikit-learn seaborn seawater shapely spyder statsmodels windspharm wrf-python xarray xlsxwriter</code> | + | <code>/usr/bin/time mamba install -n cdatm19_nompi_py3 -c conda-forge basemap basemap-data basemap-data-hires cartopy cmocean cmor ferret_datasets gdal gdal geopandas glances gsw iris joblib netcdf4 palettable pandas pillow pyferret rpy2 scikit-image scikit-learn seaborn seawater shapely spyder statsmodels windspharm wrf-python xarray xlsxwriter</code> |
- | List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_190304.txt |}} | + | Resulting list of installed packages |
+ | * ''conda list -n cdatm19_nompi_py3 > cdatm19_nompi_py3_21-03-06.txt'' | ||
+ | * no mpi: {{ :other:uvcdat:cdat_conda:cdatm19_nompi_py3_21-03-06.txt |}} | ||
=== Extra packages installed with pip === | === Extra packages installed with pip === | ||
Line 257: | Line 269: | ||
* Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' '' | * Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' '' | ||
* <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]] | * <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]] | ||
- | |||
- | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | ||
- | * ''conda install -n cdatm19_py3 -c conda-forge -c vacumm vacumm'' | ||
- | * <wrap hi>Warning!</wrap> [[https://github.com/VACUMM/vacumm/issues/15|standard install of vacumm does not work for Python3]] | ||
- | |||
- | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
- | * ''conda install -n cdatm19_py3 -c stefraynaud -c conda-forge spanlib'' | ||
- | * Test: ''python -c 'from spanlib.analyzer import Analyzer' '' | ||
* [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | ||
Line 274: | Line 278: | ||
* [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | ||
* [[other:uvcdat:cdat_conda:climaf_install|Installation notes]] | * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]] | ||
- | |||
- | * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python | ||
- | * ''conda install -n cdatm19_py3 -c conda-forge palettable'' | ||
==== TODO ==== | ==== TODO ==== | ||
Line 318: | Line 319: | ||
===== Extra packages list ===== | ===== Extra packages list ===== | ||
- | * [[https://python-pillow.org/|pillow]]: the friendly PIL (//Python Imaging Library//) fork | + | FIXME: need to clean the list below, and move the packages to the ordered list further down |
- | * [[http://pandas.pydata.org/|pandas]]: Python Data Analysis Library | + | |
- | * [[http://statsmodels.sourceforge.net/|statsmodels]]: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests | + | |
- | * [[https://seaborn.pydata.org/|seaborn]]: statistical data visualization | + | |
- | * [[http://scikit-image.org/|scikit-image]]: image processing in Python | + | |
- | * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox | + | |
- | * [[https://pypi.python.org/pypi/gsw/|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater | + | |
- | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | + | |
* [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library | * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library | ||
- | * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in Python | ||
* [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret | * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret | ||
- | * [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: high resolution data for ''basemap'' | ||
* [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests | * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests | ||
* [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files | * [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files | ||
Line 335: | Line 328: | ||
* [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API | * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API | ||
* [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | ||
- | * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | ||
* [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) | * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) | ||
- | * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | ||
- | * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python | ||
- | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography | ||
- | * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | ||
- | * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] | ||
- | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
* [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | ||
* [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | ||
+ | |||
+ | <WRAP center round tip 60%> | ||
+ | The extra packages below are (more or less) listed in package name alphabetical order. These are the extra packages that we explicitly installed, but there are many more packages installed (e.g. {{ :other:uvcdat:cdat_conda:cdatm19_nompi_py3_21-03-06.txt |}})! | ||
+ | |||
+ | Initialize the environment and type ''conda list'' if you need an up-to-date list, including the packages' version and where they came from (conda-forge for most packages | ||
+ | </WRAP> | ||
+ | |||
+ | |||
+ | * [[https://matplotlib.org/basemap/|basemap]]: a library for plotting 2D data on maps in Python | ||
+ | * [[missing|basemap-data]] and [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: (high resolution) data for ''basemap'' | ||
+ | * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | ||
+ | * see also ''iris'' | ||
+ | * [[https://github.com/Try2Code/cdo-bindings|python-cdo]]: Python scripting interface of [[https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo#Documentation|cdo]] | ||
+ | * ''python-cdo'' will install the ''cdo'' package (providing the ''cdo ''executable) as a dependency | ||
+ | * see also [[https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo%7Brbpy%7D|Using CDO from python or ruby]] | ||
+ | * [[https://github.com/ecmwf/cfgrib|cfgrib]]: Python interface to map GRIB files to the Unidata's Common Data Model v4 following the CF Conventions | ||
+ | * see also ''eccodes'' and ''pygrib'' | ||
+ | * [[https://clustergram.readthedocs.io/|clustergram]]: visualization and diagnostics for cluster analysis | ||
+ | * [[https://www.fabiocrameri.ch/colourmaps/|cmcrameri]]: Crameri's Scientific colour maps ''[color]'' | ||
+ | * the colormaps are also available in [[https://jiffyclub.github.io/palettable/scientific/|palettable.scientific]] | ||
+ | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography ''[color]'' | ||
+ | * [[https://dynamictimewarping.github.io/|python-dtw]]: implementation of Dynamic Time Warping-type (DTW) | ||
+ | * Note: older installations provided [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | ||
+ | * [[https://confluence.ecmwf.int/display/ECC/|ecCodes]]: ecCodes is a package developed by ECMWF which provides an application programming interface and a set of tools for decoding and encoding GRIB and BUFR messages | ||
+ | * need to install ''eccodes'' and ''python-eccodes'' ([[https://github.com/ecmwf/eccodes-python/issues/56|details]]) | ||
+ | * see also ''cfgrib'' and ''pygrib'' | ||
+ | * [[https://docs.esmvaltool.org/en/latest/|ESMValTool]]: Earth System Model Evaluation Tool | ||
+ | * not installed yet (March 2022) because it required download of 302 Mb extra packages and a downgrade of gdal | ||
+ | * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]] and ''ferret_datasets'': Ferret encapsulated in Python | ||
+ | * OSGeo/[[http://www.gdal.org/|gdal]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | ||
+ | * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] | ||
+ | * [[https://github.com/TEOS-10/GSW-python|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater | ||
+ | * see also //seawater// | ||
+ | * [[https://intake.readthedocs.io/|intake]]: a lightweight package for finding, investigating, loading and disseminating data | ||
+ | * [[https://intake-esm.readthedocs.io/|intake-esm]]: data cataloging utility built on top of intake, pandas, and xarray | ||
+ | * [[https://scitools-iris.readthedocs.io/en/stable/|iris]]: a powerful, format-agnostic, community-driven Python package for analysing and visualising Earth science data | ||
+ | * see also ''cartopy'' | ||
+ | * install sample data with ''conda install -c conda-forge iris-sample-data'' | ||
* [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs | * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs | ||
- | * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python | + | * [[https://opencv.org/|OpenCV]]: OpenCV (Open Source Computer Vision Library) is an open-source library that includes several hundreds of computer vision algorithms. See also [[https://www.geeksforgeeks.org/opencv-python-tutorial/|OpenCV Python Tutorial]] and **scikit-image** |
+ | * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python ''[color]'' | ||
+ | * [[http://pandas.pydata.org/|pandas]]: Python Data Analysis Library | ||
+ | * [[https://geopandas.org/|geopandas]]: an open source project to make working with geospatial data in python easier | ||
+ | * [[https://peakutils.readthedocs.io/|PeakUtils]]: utilities related to the detection of peaks on 1D data | ||
+ | * [[https://python-pillow.org/|pillow]]: the friendly PIL (//Python Imaging Library//) fork | ||
+ | * [[https://proplot.readthedocs.io/en/latest/|proplot]]: a lightweight **matplotlib wrapper** for making beautiful, publication-quality graphics | ||
+ | * [[https://github.com/jswhit/pygrib|pygrib]]: high-level interface to the ECWMF ECCODES C library for reading GRIB files | ||
+ | * see also ''eccodes'' and ''cfgrib'' | ||
+ | * [[https://pyleoclim-util.readthedocs.io/|pyleoclim]]: a Python package designed for the analysis of paleoclimate data | ||
+ | * **Wait** till it can be installed with ''conda'' ([[https://github.com/LinkedEarth/Pyleoclim_util/discussions/205|Why I'm not installing Pyleoclim yet]]) | ||
+ | * [[https://rpy2.github.io/|rpy2]]: an interface to R running embedded in a Python process | ||
+ | * [[http://scikit-image.org/|scikit-image]]: a collection of algorithms for image processing in Python | ||
+ | * [[https://scikit-learn.org/|scikit-learn]]: an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection and evaluation, and many other utilities. | ||
+ | * [[https://seaborn.pydata.org/|seaborn]]: statistical data visualization | ||
+ | * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox | ||
+ | * see also //gsw// | ||
+ | * [[https://www.statsmodels.org/|statsmodels]]: a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration. | ||
+ | * [[https://github.com/SatAgro/suntime|suntime]]: simple sunset and sunrise time calculation python library | ||
+ | * **Warning**: not available in conda, use ''pip install suntime'' | ||
+ | * [[https://www.tensorflow.org/|tensorflow-mkl]]: an end-to-end open source machine learning platform | ||
+ | * [[https://anaconda.org/anaconda/tensorflow-mkl|tensorflow-mkl]] will install the **CPU**-based (**not GPU**) version | ||
+ | * [[https://xcdat.readthedocs.io/|xcdat]]: Xarray Extended with Climate Data Analysis Tools | ||
+ | * [[https://xoa.readthedocs.io/en/latest/|xoa]]: xarray-based ocean analysis library | ||
+ | * ''xoa'' is the successor of [[http://www.ifremer.fr/vacumm/|vacumm]] (vacumm does **not** support Python3) | ||
==== Removed packages ==== | ==== Removed packages ==== | ||
Line 352: | Line 400: | ||
* NO removed packages! | * NO removed packages! | ||
+ | ==== Packages NOT supported in Python3 ==== | ||
+ | |||
+ | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
+ | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | ||
+ | * Check the ''xoa'' replacement package | ||
===== Environments summary ===== | ===== Environments summary ===== | ||