This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:uvcdat:cdat_conda:cdat_8_2_1 [2021/03/01 11:02] jypeter [cdat nightly case] Removed this section |
other:uvcdat:cdat_conda:cdat_8_2_1 [2023/07/12 12:49] jypeter [Extra packages list] Added eofs |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== CDAT 8.2.1 installation notes ====== | + | ====== Python distribution based on CDAT 8.2.1 notes ====== |
[ [[.:index|Back to all versions]] ] | [ [[.:index|Back to all versions]] ] | ||
Line 11: | Line 11: | ||
===== Using JYP version ===== | ===== Using JYP version ===== | ||
- | If you mostly want to use the CDAT 8.2.1 installed by JYP, just use the following steps and then read the //What should I do now?// section below | + | If you mostly want to use the **python distribution based on CDAT 8.2.1** installed by JYP, just use the following steps and then read the [[#what_should_i_do_now|What should I do now?]] section below |
* Initialize //conda// with: | * Initialize //conda// with: | ||
Line 23: | Line 23: | ||
* python 3.x: ''conda activate cdatm_py3'' | * python 3.x: ''conda activate cdatm_py3'' | ||
- | * Type ''which python'' (or the ''wp'' alias) and make sure you get something like\\ ''[...]/miniconda3/envs/env_name/bin/python'' | + | * Type ''which python'' (or the ''wp'' alias) and make sure you get something looking like |
+ | * ''[...]/miniconda3<possibly_some_version>/envs/<some_env_name>/bin/python'' | ||
+ | * e.g. ''cdatm_py3'' environment at LSCE:\\ ''/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python'' | ||
Line 40: | Line 42: | ||
==== You want to replicate this installation on your own server or desktop/laptop ==== | ==== You want to replicate this installation on your own server or desktop/laptop ==== | ||
+ | Read the next sections of this web page | ||
+ | |||
+ | There are some slight differences if you want to install this distribution: | ||
+ | * just for you. This is the usual case. Well, if you break something you can always remove everything and start from scratch | ||
+ | * for sharing a stable version with several other people | ||
+ | |||
+ | The notes at the beginning of each section will tell you if you can skip the section if you are just installing for you | ||
===== What's New in CDAT 8.2.1? ===== | ===== What's New in CDAT 8.2.1? ===== | ||
Line 45: | Line 54: | ||
* Full [[https://uvcdat.llnl.gov/changelog.html#8.2.1|Change log]] | * Full [[https://uvcdat.llnl.gov/changelog.html#8.2.1|Change log]] | ||
- | Note: this particular CDAT installation at LSCE (and on other machines/servers) provides **many extra (non-CDAT) packages**. You can jump directly to the [[#extra_packages_list|Extra package list]] at the bottom of this page to see what is available | + | Note: this particular CDAT installation at LSCE (and on other machines/servers) provides **many extra (non-CDAT) packages**. You can jump directly to the [[#extra_packages_list|Extra package list]] at the bottom of this page, if you want to see what is available |
===== Installation with Miniconda3 ===== | ===== Installation with Miniconda3 ===== | ||
- | We assume that Miniconda3 is already installed. Otherwise, follow the the [[other:uvcdat:cdat_conda:cdat_8_0_py2#installing_miniconda3|Installing Miniconda]] steps (and the //Post-Miniconda3 installation// section) we followed when installing CDAT 8.0 | + | ==== Installing Miniconda3 ==== |
+ | |||
+ | If //Miniconda3// is not already installed, or if you want to start over with a brand new version, read [[other:uvcdat:cdat_conda:miniconda3_install|JYP steps for installing Miniconda3]]. | ||
- | ==== Updating conda ==== | ||
- | The conda package itself can be updated (if need be) with\\ ''conda update -n base -c defaults conda'' | ||
==== Installing CDAT 8.2.1 ==== | ==== Installing CDAT 8.2.1 ==== | ||
Line 72: | Line 81: | ||
577M /home/share/unix_files/cdat/miniconda3_21-02</code> | 577M /home/share/unix_files/cdat/miniconda3_21-02</code> | ||
- | We can also use ''conda env list'' and remove some obsolete versions with ''conda remove -n obsolete_name --all'' to get some space | + | We can also use ''conda env list'' and remove some obsolete versions with ''conda remove -n obsolete_environment %%--%%all'', in order to free up some space |
=== Python 2.7 version === | === Python 2.7 version === | ||
- | Python 2 is deprecated in CDAT, starting with CDAT 8.2.1. If you still need to use python 2.7, please use [[other:uvcdat:cdat_conda:cdat_8_1|CDAT 8.1]] | + | <wrap hi>Python 2.x is deprecated in CDAT, starting with CDAT 8.2.1</wrap>. |
+ | |||
+ | If you still need to use python 2.7, please use [[other:uvcdat:cdat_conda:cdat_8_1|CDAT 8.1]] | ||
=== Python 3.8 version === | === Python 3.8 version === | ||
<WRAP center round important 60%> | <WRAP center round important 60%> | ||
- | It's possible that, when using the tcsh shell and trying to use the ''conda create -n cdat-8.2.1_py3 lots_of_parameters_with_stars'' line, you will get the following error message: | + | It's possible that, when using the //tcsh// shell and trying to use the ''conda create -n cdat-8.2.1_py3 lots_of_parameters_with_wildcards'' line, you will get the following error message: |
**conda: No match** | **conda: No match** | ||
- | This may be due to [[https://gitmemory.com/issue/conda/conda/9771/608209654|CSH install/search fail with '*' variable in an argument]]. In that case, just type ''bash'' and run the installation command in a //bash// shell | + | Apparently the //tcsh// version of conda does not like wildcards too much ([[https://gitmemory.com/issue/conda/conda/9771/608209654|CSH install/search fail with '*' variable in an argument]])... |
+ | |||
+ | In that case, just type ''bash'' and run the installation command in a //bash// shell | ||
</WRAP> | </WRAP> | ||
The installation line below is based on information from the [[https://github.com/CDAT/cdat/wiki/install#for-linux-python|CDAT wiki page]]. See also [[https://github.com/CDAT/cdat/issues/2265|Some questions and notes about 8.2.1 installation]] | The installation line below is based on information from the [[https://github.com/CDAT/cdat/wiki/install#for-linux-python|CDAT wiki page]]. See also [[https://github.com/CDAT/cdat/issues/2265|Some questions and notes about 8.2.1 installation]] | ||
+ | <code> | ||
+ | bash-4.2$ conda config --env --add channels conda-forge | ||
+ | bash-4.2$ conda config --set channel_priority strict | ||
+ | bash-4.2$ cat .condarc | ||
+ | channels: | ||
+ | - conda-forge | ||
+ | - defaults | ||
+ | $ conda config --describe channel_priority | ||
+ | |||
+ | $ conda create -n cdat-8.2.1_nompi_py3 -c conda-forge -c cdat/label/v8.2.1 cdat "libnetcdf=*=nompi_*" "mesalib=18.3.1" "python=3"</code> | ||
<code>$ conda create -n cdat-8.2.1_py3 -c conda-forge -c cdat/label/v8.2.1 cdat "libnetcdf=*=mpi_openmpi_*" "mesalib=18.3.1" "python=3"</code> | <code>$ conda create -n cdat-8.2.1_py3 -c conda-forge -c cdat/label/v8.2.1 cdat "libnetcdf=*=mpi_openmpi_*" "mesalib=18.3.1" "python=3"</code> | ||
- | ===== Cloning CDAT before adding specific packages for LSCE ===== | + | Resulting list of installed packages |
+ | * ''conda list -n cdat-8.2.1_nompi_py3 > cdat_8.2.1_nompi_21-03-05.txt'' | ||
+ | * no mpi: {{ :other:uvcdat:cdat_conda:cdat_8.2.1_nompi_21-03-05.txt |}} | ||
+ | * with mpî: {{ :other:uvcdat:cdat_conda:cdat_8.2.1_installed_packages.txt |}} | ||
+ | |||
+ | [[https://github.com/CDAT/cdat/issues/2265|Some extra installation notes]] | ||
+ | ===== Cloning the base CDAT environment before adding specific packages for LSCE ===== | ||
<WRAP center round alert 60%> | <WRAP center round alert 60%> | ||
Line 109: | Line 138: | ||
Notes: | Notes: | ||
- | * Why //cloning//? The initial CDAT environment is strictly the one created at PCMDI and certified by PCMDI. Rather than making changes directly in there, we keep it as it is, clone it, and work on the clone | + | * Why //cloning//? The initial CDAT environment is strictly the one created at PCMDI and certified by PCMDI. Rather than making changes directly in there, we keep it as it is, clone it, and make changes in the cloned environment |
- | * Carefully working on different python environments (possibly cloned) is safer in a multi-user environment | + | * Carefully working on different python environments (possibly cloned) is safer in a multi-user environment (i.e. you can have people use a specific environment, and make tests in another environment) |
* Cloning a full environment uses Linux hard links and requires less disk space than making a real copy | * Cloning a full environment uses Linux hard links and requires less disk space than making a real copy | ||
Line 123: | Line 152: | ||
- | ===== Getting ready for a moving default CDAT ===== | + | ===== Getting ready for a moving default CDAT environment ===== |
- | <note warning>This step should probably be listed at the **end**, especially in a multi-user environment! If there is already a ''cdatm'' link, make sure that the new version is stable and working correctly before updating the ''cdatm'' link</note> | + | <WRAP center round alert 60%> |
+ | You can skip this section if you are **installing CDAT just for one user (yourself)** on a Linux machine (or a Windows 10 machine with WSL) | ||
- | We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version | + | You will directly make changes and install packages in **your** main python environment |
+ | </WRAP> | ||
- | <code>$ cd /home/share/unix_files/cdat/miniconda3/envs | + | <WRAP center round tip 60%> |
- | $ ln -s cdatm19_py2 cdatm_py2</code> | + | This step could probably be listed at the **end**, especially in a multi-user environment! |
- | ===== Customizing UV-CDAT for LSCE ===== | + | If there is already a ''cdatm_py3'' link (pointing to an older environment), make sure that the new CDAT environment is stable and working correctly before updating the ''cdatm_py3'' link |
+ | </WRAP> | ||
- | ==== Testing vcs ==== | + | We create a **//cdatm_py3// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default (and hopefully stable) CDAT environment. In that case, most users can just activate this //cdatm_py3// environment and always get the latest stable version. |
- | Start python and paste the following lines | + | <code>$ cd /home/share/unix_files/cdat/miniconda3_21-02/envs |
+ | $ ln -s cdatm19_py3 cdatm_py3 | ||
- | <code>import numpy as np, vcs | + | conda env list |
- | id100 = np.identity(100) | + | # conda environments: |
- | x = vcs.init() | + | # |
- | x.plot(id100) | + | base * /home/share/unix_files/cdat/miniconda3_21-02 |
+ | cdat-8.2.1_py3 /home/share/unix_files/cdat/miniconda3_21-02/envs/cdat-8.2.1_py3 | ||
+ | cdatm19_py3 /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_py3 | ||
+ | cdatm_py3 /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3 | ||
+ | |||
+ | $ ls -l /home/share/unix_files/cdat/miniconda3_21-02/envs/ | ||
+ | drwxr-xr-x [...] cdat-8.2.1_py3/ | ||
+ | drwxr-xr-x [...] cdatm19_py3/ | ||
+ | lrwxrwxrwx [...] cdatm_py3 -> cdatm19_py3/ | ||
</code> | </code> | ||
- | If you get the kind of errors described in [[https://github.com/CDAT/vcs/issues/393|x.plot crashes with OpenGL2 error]], the drivers on your server are probably not compatible with an interactive use of vcs... The workaround is to **install mesalib and work in //headless mode//** (e.g. without a display)... | + | ===== Customizing CDAT for LSCE ===== |
- | Install //mesalib// with ''conda install -c cdat/label/v81 -c conda-forge mesalib vtk-cdat'' and try pasting again the few code lines above in python. If things work as expected, no canvas will be opened, but you'll be able to save the canvas to a file with | + | ==== A quick test of cdms2 and vcs ==== |
- | <code>x.png('test_mesalib') | + | You can use the quick test shown in [[https://github.com/CDAT/cdat/wiki/install#to-test-your-cdat-installation|To Test Your CDAT Installation]], and check if you get the expected png file |
- | x.pdf('test_mesalib')</code> | + | |
- | Note: using //mesalib// for headless mode work makes it possible to work without an X server, and with no DISPLAY variable defined | + | You can also check the [[other:uvcdat:cdat_conda:cdat_8_1#testing_vcs|Testing vcs]] section of the LSCE installation of CDAT 8.1 |
==== Downloading cdms2/vcs test data ==== | ==== Downloading cdms2/vcs test data ==== | ||
- | You should download the test data (174M of data...) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...) | + | You should download the test data (174M of data...) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...). The downloaded data files will be available in: ''vcs.sample_data'' |
- | <code>$ conda activate cdatm19_py2 | + | <code>$ conda activate cdatm19_py3 |
- | (cdatm19_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)' | + | (cdatm19_py3) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)' |
[...] | [...] | ||
- | Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data | + | Finished downloading sample data to /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_py3/share/cdat/sample_data |
- | (cdatm19_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data | + | (cdatm19_py3) $ du -sh /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_py3/share/cdat/sample_data |
- | 174M /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data | + | 174M /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_py3/share/cdat/sample_data |
- | </code> | + | |
+ | $ python | ||
+ | >>> import os, vcs | ||
+ | >>> vcs.sample_data | ||
+ | '/home/jypeter/miniconda3_21-02/envs/cdat-8.2.1_py3/share/cdat/sample_data' | ||
+ | |||
+ | >>> os.listdir(vcs.sample_data) | ||
+ | ['BlueMarble.ppm', 'clt.nc', 'geo.1deg.ctl', 'geo.1deg.gmp', 'geo.1deg.grb', ...]</code> | ||
==== Packages that have no dependency problems ==== | ==== Packages that have no dependency problems ==== | ||
- | After cloning, we are ready to install some extra packages that may be requested by LSCE users | + | After [[#cloning_the_base_cdat_environment_before_adding_specific_packages_for_lsce|cloning]], we are ready to install some extra (e.g. not directly related to CDAT) packages that may be useful to LSCE users |
- | * We first try to install together as many packages as possible that don't require other channels than //conda-forge//, and that don't request a downgrade of what is already installed | + | * We first install together as many packages as possible that don't require other channels than //conda-forge//, and that don't request a significant downgrade of what is already installed |
- | * We then install individual extra packages with ''conda install'' or ''pip install'' | + | * We use the //dry-run// option of conda in order to check the version (theoretically the latest) and origin (theoretically //conda-forge//) of the packages that would be installed. Or maybe the result will be that there are too many conflicts and the installation cannot proceed with the requested packages combination |
+ | * conda may take a very long time to report that there are too many conflicts and we have started using [[https://github.com/mamba-org/mamba|mamba]] that uses different solving libraries and is much (**much!**) faster than conda | ||
+ | * if the dry-run reports too many conflicts or requests too many downgrades, we try to remove some (not really important) packages and check if it works better | ||
+ | * the last resort is too create a new environment with a smaller set of packages if a user really needs a conflicting package | ||
+ | * We later install individual extra packages with ''conda install'', ''mamba install'' or ''pip install'' | ||
- | <code># You can use the following to keep a trace of what will be installed | + | === Pre-installation check with the dry-run option === |
- | #$ conda install --dry-run -n cdatm19_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm > somewhere/extra_packages.txt | + | |
- | # Install... | + | <code>/usr/bin/time mamba install -n cdatm19_nompi_py3 --dry-run -c conda-forge basemap basemap-data basemap-data-hires cartopy cmocean cmor ferret_datasets gdal gdal geopandas glances gsw iris joblib netcdf4 palettable pandas pillow pyferret rpy2 scikit-image scikit-learn seaborn seawater shapely spyder statsmodels windspharm wrf-python xarray xlsxwriter |
- | $ conda install -n cdatm19_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret ferret_datasets basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm | + | |
- | [...] | + | |
- | # Use a similar install line if you want to install the same packages in the python3 version</code> | + | [... lots of information about what would be done followed by some information about how long it took ...] |
+ | |||
+ | Summary: | ||
+ | |||
+ | Install: 167 packages | ||
+ | Downgrade: 3 packages | ||
+ | |||
+ | Total download: 474 MB | ||
+ | |||
+ | DryRunExit: Dry run. Exiting. | ||
+ | |||
+ | 8.31user 0.51system 0:11.45elapsed 77%CPU</code> | ||
+ | |||
+ | === Actual installation === | ||
+ | |||
+ | <code>/usr/bin/time mamba install -n cdatm19_nompi_py3 -c conda-forge basemap basemap-data basemap-data-hires cartopy cmocean cmor ferret_datasets gdal gdal geopandas glances gsw iris joblib netcdf4 palettable pandas pillow pyferret rpy2 scikit-image scikit-learn seaborn seawater shapely spyder statsmodels windspharm wrf-python xarray xlsxwriter</code> | ||
+ | |||
+ | Resulting list of installed packages | ||
+ | * ''conda list -n cdatm19_nompi_py3 > cdatm19_nompi_py3_21-03-06.txt'' | ||
+ | * no mpi: {{ :other:uvcdat:cdat_conda:cdatm19_nompi_py3_21-03-06.txt |}} | ||
+ | |||
+ | === Extra packages installed with pip === | ||
- | List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_190304.txt |}} | ||
- | Packages installed with //pip// | ||
* [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API | * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API | ||
- | * ''conda activate cdatm19_py2''\\ ''pip install dreqPy'' | + | * ''conda activate cdatm19_py3''\\ ''pip install dreqPy'' |
* Update with: ''pip install %%--%%upgrade dreqPy'' | * Update with: ''pip install %%--%%upgrade dreqPy'' | ||
* Get version number with:\\ <code>$ drq -v | * Get version number with:\\ <code>$ drq -v | ||
dreqPy version 01.00.29 [Version 01.00.29]</code> | dreqPy version 01.00.29 [Version 01.00.29]</code> | ||
- | * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret | + | * ''ipython_ferretmagic'': more details in the [[#extra_packages_list|Extra packages list section]] |
- | * ''conda activate cdatm19_py2''\\ ''pip install ferretmagic'' | + | |
+ | === Packages with no dependency problems and were added (or updated) later === | ||
- | The following packages have no dependency problems and were installed (or updated) //later// | ||
* [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | ||
- | * ''conda install -n cdatm19_py2 -c conda-forge cmor'' | + | * ''conda install -n cdatm19_py3 -c conda-forge cmor'' |
* Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' '' | * Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' '' | ||
* <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]] | * <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]] | ||
- | |||
- | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | ||
- | * ''conda install -n cdatm19_py2 -c conda-forge -c vacumm vacumm'' | ||
- | * <wrap hi>Warning!</wrap> [[https://github.com/VACUMM/vacumm/issues/15|standard install of vacumm does not work for Python3]] | ||
- | |||
- | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
- | * ''conda install -n cdatm19_py2 -c stefraynaud -c conda-forge spanlib'' | ||
- | * Test: ''python -c 'from spanlib.analyzer import Analyzer' '' | ||
* [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | ||
- | * ''conda install -n cdatm19_py2 -c conda-forge cartopy'' | + | * ''conda install -n cdatm19_py3 -c conda-forge cartopy'' |
* [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs | * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs | ||
- | * ''conda install -n cdatm19_py2 -c conda-forge joblib'' | + | * ''conda install -n cdatm19_py3 -c conda-forge joblib'' |
* [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | ||
* [[other:uvcdat:cdat_conda:climaf_install|Installation notes]] | * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]] | ||
- | |||
- | * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python | ||
- | * ''conda install -n cdatm19_py2 -c conda-forge palettable'' | ||
==== TODO ==== | ==== TODO ==== | ||
//Add here packages that would be useful and have not been installed yet, or have some problems that prevent their installation// | //Add here packages that would be useful and have not been installed yet, or have some problems that prevent their installation// | ||
- | |||
- | * [[http://scitools.org.uk/iris/index.html|iris]]: A Python library for Meteorology and Climatology | ||
- | * ''conda install -n cdatxxx -c conda-forge iris'' | ||
* [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | * [[http://wrf-python.readthedocs.io/en/latest/|wrf-python]]: A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model | ||
Line 262: | Line 315: | ||
===== Extra packages list ===== | ===== Extra packages list ===== | ||
- | * [[https://python-pillow.org/|pillow]]: the friendly PIL (//Python Imaging Library//) fork | + | FIXME: need to clean the list below, and move the packages to the ordered list further down |
- | * [[http://pandas.pydata.org/|pandas]]: Python Data Analysis Library | + | |
- | * [[http://statsmodels.sourceforge.net/|statsmodels]]: a Python module that allows users to explore data, estimate statistical models, and perform statistical tests | + | |
- | * [[https://seaborn.pydata.org/|seaborn]]: statistical data visualization | + | |
- | * [[http://scikit-image.org/|scikit-image]]: image processing in Python | + | |
- | * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox | + | |
- | * [[https://pypi.python.org/pypi/gsw/|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater | + | |
- | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | + | |
* [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library | * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library | ||
- | * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in Python | ||
- | * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret | ||
- | * [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: high resolution data for ''basemap'' | ||
* [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests | * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests | ||
* [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files | * [[https://xlsxwriter.readthedocs.io/|XlsxWriter]]: a Python module for creating Excel XLSX files | ||
Line 279: | Line 323: | ||
* [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API | * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API | ||
* [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | ||
- | * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | ||
* [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) | * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) | ||
- | * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | ||
- | * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python | ||
- | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography | ||
- | * [[http://www.gdal.org/|OSGeo/GDAL]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | ||
- | * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] | ||
- | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
* [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | ||
* [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | ||
+ | |||
+ | <WRAP center round tip 60%> | ||
+ | The extra packages below are (more or less) listed in package name alphabetical order. These are the extra packages that we explicitly installed, but there are many more packages installed (e.g. {{ :other:uvcdat:cdat_conda:cdatm19_nompi_py3_21-03-06.txt |}})! | ||
+ | |||
+ | Initialize the environment and type ''conda list'' if you need an up-to-date list, including the packages' version and where they came from (conda-forge for most packages | ||
+ | </WRAP> | ||
+ | |||
+ | |||
+ | * [[https://matplotlib.org/basemap/|basemap]]: a library for plotting 2D data on maps in Python | ||
+ | * [[missing|basemap-data]] and [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: (high resolution) data for ''basemap'' | ||
+ | * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | ||
+ | * see also ''iris'' | ||
+ | * [[https://github.com/Try2Code/cdo-bindings|python-cdo]]: Python scripting interface of [[https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo#Documentation|cdo]] | ||
+ | * ''python-cdo'' will install the ''cdo'' package (providing the ''cdo ''executable) as a dependency | ||
+ | * see also [[https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo%7Brbpy%7D|Using CDO from python or ruby]] | ||
+ | * [[https://cds.climate.copernicus.eu/api-how-to|cdsapi]]: The Climate Data Store (CDS) Application Program Interface (API) is a service providing programmatic access to CDS data | ||
+ | * CDS = Copernicus [[https://cds.climate.copernicus.eu/|Climate Data Store]] | ||
+ | * Example: [[https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=form|ERA5 hourly data on pressure levels from 1959 to present]] | ||
+ | * [[https://github.com/ecmwf/cfgrib|cfgrib]]: Python interface to map GRIB files to the Unidata's Common Data Model v4 following the CF Conventions | ||
+ | * see also ''eccodes'' and ''pygrib'' | ||
+ | * [[https://unidata.github.io/cftime/|cftime]]: Python library for decoding time units and variable values in a netCDF file conforming to the Climate and Forecasting (CF) netCDF conventions | ||
+ | * Used with ''xarray'' | ||
+ | * [[https://cf-xarray.readthedocs.io/|cf_xarray]]: provides an accessor (''DataArray.cf'' or ''Dataset.cf'') that allows you to interpret //Climate and Forecast// metadata convention attributes present on ''xarray'' objects | ||
+ | * Used with ''xarray'' | ||
+ | * [[https://clustergram.readthedocs.io/|clustergram]]: visualization and diagnostics for cluster analysis | ||
+ | * [[https://www.fabiocrameri.ch/colourmaps/|cmcrameri]]: Crameri's Scientific colour maps ''[color]'' | ||
+ | * the colormaps are also available in [[https://jiffyclub.github.io/palettable/scientific/|palettable.scientific]] | ||
+ | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography ''[color]'' | ||
+ | * [[https://dash.plotly.com/|dash]] and [[https://github.com/plotly/jupyter-dash|jupyter-dash]]: the original low-code framework for rapidly building data apps in Python, R, Julia, and F# | ||
+ | * see also ''plotly'' | ||
+ | * [[https://dynamictimewarping.github.io/|python-dtw]]: implementation of Dynamic Time Warping-type (DTW) | ||
+ | * Note: older installations provided [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | ||
+ | * [[https://confluence.ecmwf.int/display/ECC/|ecCodes]]: ecCodes is a package developed by ECMWF which provides an application programming interface and a set of tools for decoding and encoding GRIB and BUFR messages | ||
+ | * need to install ''eccodes'' and ''python-eccodes'' ([[https://github.com/ecmwf/eccodes-python/issues/56|details]]) | ||
+ | * see also ''cfgrib'' and ''pygrib'' | ||
+ | * [[https://ajdawson.github.io/eofs/|eofs]]: a Python package for EOF analysis of spatial-temporal data | ||
+ | * [[https://docs.esmvaltool.org/en/latest/|ESMValTool]]: Earth System Model Evaluation Tool | ||
+ | * not installed yet (March 2022) because it required download of 302 Mb extra packages and a downgrade of gdal | ||
+ | * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]] and ''ferret_datasets'': Ferret encapsulated in Python | ||
+ | * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret | ||
+ | * Install with: ''pip install ferretmagic'' | ||
+ | * OSGeo/[[http://www.gdal.org/|gdal]]: Geospatial Data Abstraction Library. GDAL is a translator library for raster and vector geospatial data formats | ||
+ | * [[https://pcjericks.github.io/py-gdalogr-cookbook/|Python GDAL/OGR Cookbook]] | ||
+ | * [[https://github.com/TEOS-10/GSW-python|gsw]]: Python implementation of the Thermodynamic Equation Of Seawater | ||
+ | * see also //seawater// | ||
+ | * [[https://noaa-orr-erd.github.io/gridded/index.html|gridded]]: a single API for accessing / working with gridded model results on multiple grid types | ||
+ | * Supports the [[http://cfconventions.org/|CF]], [[https://ugrid-conventions.github.io/ugrid-conventions/|UGRID]] and [[http://sgrid.github.io/sgrid/|SGRID]] conventions | ||
+ | * [[https://icclim.readthedocs.io/|icclim]]: icclim (Index Calculation for CLIMate) is a Python library to compute climate indices | ||
+ | * [[https://intake.readthedocs.io/|intake]]: a lightweight package for finding, investigating, loading and disseminating data | ||
+ | * [[https://intake-esm.readthedocs.io/|intake-esm]]: data cataloging utility built on top of intake, pandas, and xarray | ||
+ | * [[https://ipyleaflet.readthedocs.io/en/latest/|ipyleaflet]]: interactive maps in the Jupyter notebook | ||
+ | * [[https://ipywidgets.readthedocs.io/|ipywidgets]]: ipywidgets, also known as jupyter-widgets or simply widgets, are interactive HTML widgets for Jupyter notebooks and the IPython kernel | ||
+ | * [[https://scitools-iris.readthedocs.io/en/stable/|iris]]: a powerful, format-agnostic, community-driven Python package for analysing and visualising Earth science data | ||
+ | * see also ''cartopy'' | ||
+ | * install sample data with ''conda install -c conda-forge iris-sample-data'' | ||
* [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs | * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs | ||
- | * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python | + | * [[https://jupyterlab.readthedocs.io/|jupyterlab]]: the next-generation web-based user interface for Project Jupyter |
+ | * [[https://github.com/plotly/Kaleido|python-kaleido]]: a cross-platform library for generating static images (e.g. png, svg, pdf, etc.) for web-based visualization libraries | ||
+ | * [[https://mpltern.readthedocs.io/|mpltern]]: a Python plotting library based on Matplotlib specifically designed for ternary plots | ||
+ | * [[https://nc-time-axis.readthedocs.io/en/stable/|nc-time-axis]]: a package that enables making plots in matplotlib with axes made up of ''cftime.datetime'' dates with any calendar type | ||
+ | * [[https://gitlab.com/remikz/nccmp|nccmp]]: compare two NetCDF files bitwise, semantically or with a user defined tolerance (absolute or relative percentage) | ||
+ | * can probably also be done with [[https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-580002.1.3|cdo -v diffn file1.nc file2.nc]] | ||
+ | * [[https://opencv.org/|OpenCV]]: OpenCV (Open Source Computer Vision Library) is an open-source library that includes several hundreds of computer vision algorithms. See also [[https://www.geeksforgeeks.org/opencv-python-tutorial/|OpenCV Python Tutorial]] and **scikit-image** | ||
+ | * [[https://openpyxl.readthedocs.io/|openpyxl]]: a Python library to read/write Excel 2010 ''xlsx''/''xlsm'' files | ||
+ | * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python ''[color]'' | ||
+ | * [[http://pandas.pydata.org/|pandas]]: Python Data Analysis Library | ||
+ | * [[https://geopandas.org/|geopandas]]: an open source project to make working with geospatial data in python easier | ||
+ | * [[https://peakutils.readthedocs.io/|PeakUtils]]: utilities related to the detection of peaks on 1D data | ||
+ | * [[https://python-pillow.org/|pillow]]: the friendly PIL (//Python Imaging Library//) fork | ||
+ | * [[https://plotly.com/python/|plotly]]: Plotly's Python graphing library (sometimes referred to as //plotly.py//) makes interactive, publication-quality graphs | ||
+ | * see also ''python-kaleido'' and ''dash'' | ||
+ | * [[https://www.fatiando.org/pooch/|pooch]]: a friend to fetch your data files (makes it easy to download a file, without messing with ''requests'' and ''urllib'') | ||
+ | * [[https://proplot.readthedocs.io/en/latest/|proplot]]: a lightweight **matplotlib wrapper** for making beautiful, publication-quality graphics | ||
+ | * [[https://psyplot.github.io/|psyplot]]: Interactive Data Visualization from Python and GUIs | ||
+ | * [[https://github.com/jswhit/pygrib|pygrib]]: high-level interface to the ECWMF ECCODES C library for reading GRIB files | ||
+ | * see also ''eccodes'' and ''cfgrib'' | ||
+ | * [[https://pyleoclim-util.readthedocs.io/|pyleoclim]]: a Python package designed for the analysis of paleoclimate data | ||
+ | * **Wait** till it can be installed with ''conda'' ([[https://github.com/LinkedEarth/Pyleoclim_util/discussions/205|Why I'm not installing Pyleoclim yet]]) | ||
+ | * [[https://requests.readthedocs.io/|requests]]: is an elegant and simple HTTP library for Python, built for human beings | ||
+ | * See also ''pooch'' | ||
+ | * [[https://rpy2.github.io/|rpy2]]: an interface to R running embedded in a Python process | ||
+ | * [[http://scikit-image.org/|scikit-image]]: a collection of algorithms for image processing in Python | ||
+ | * [[https://scikit-learn.org/|scikit-learn]]: an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection and evaluation, and many other utilities. | ||
+ | * [[https://seaborn.pydata.org/|seaborn]]: statistical data visualization | ||
+ | * [[http://pythonhosted.org/seawater/|seawater]]: Python re-write of the CSIRO seawater toolbox | ||
+ | * see also ''gsw'' | ||
+ | * [[https://www.statsmodels.org/|statsmodels]]: a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration. | ||
+ | * [[https://github.com/SatAgro/suntime|suntime]]: simple sunset and sunrise time calculation python library | ||
+ | * **Warning**: not available in conda, use ''pip install suntime'' | ||
+ | * [[https://www.tensorflow.org/|tensorflow-mkl]]: an end-to-end open source machine learning platform | ||
+ | * [[https://anaconda.org/anaconda/tensorflow-mkl|tensorflow-mkl]] will install the **CPU**-based (**not GPU**) version | ||
+ | * [[https://github.com/tqdm|tqdm]]: make your loops show a smart progress meter | ||
+ | * [[https://uxarray.readthedocs.io/|uxarray]]: provide xarray styled functionality for unstructured grid datasets following [[https://ugrid-conventions.github.io/ugrid-conventions/|UGRID Conventions]] | ||
+ | * [[https://docs.xarray.dev/en/stable/|xarray]]: Xarray makes working with labelled multi-dimensional arrays in Python simple, efficient, and fun! | ||
+ | * [[https://xcdat.readthedocs.io/|xcdat]]: xarray extended with Climate Data Analysis Tools | ||
+ | * [[https://xclim.readthedocs.io/|xclim]]: an operational Python library for climate services, providing numerous climate-related indicator tools with an extensible framework for constructing custom climate indicators, statistical downscaling and bias adjustment of climate model simulations, as well as climate model ensemble analysis tools. | ||
+ | * [[https://xgrads.readthedocs.io/|xgrads]]: parse and read binary dataset described by a ''.ctl'' file commonly used by [[http://cola.gmu.edu/grads/|GrADS]] or [[http://www.opengrads.org/|openGrADS]] | ||
+ | * Install with: ''pip install xgrads'' | ||
+ | * [[https://xlrd.readthedocs.io/|xlrd]]: a library for reading data and formatting information from Excel files in the historical .xls format | ||
+ | * [[https://xoa.readthedocs.io/en/latest/|xoa]]: xarray-based ocean analysis library | ||
+ | * ''xoa'' is the successor of [[http://www.ifremer.fr/vacumm/|vacumm]] (vacumm does **not** support Python3) | ||
==== Removed packages ==== | ==== Removed packages ==== | ||
Line 296: | Line 432: | ||
* NO removed packages! | * NO removed packages! | ||
+ | ==== Packages NOT supported in Python3 ==== | ||
+ | |||
+ | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
+ | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | ||
+ | * Check the ''xoa'' replacement package | ||
===== Environments summary ===== | ===== Environments summary ===== | ||
Line 303: | Line 444: | ||
^ Environment\\ name ^ Server ^ Summary ^ | ^ Environment\\ name ^ Server ^ Summary ^ | ||
- | | cdat-8.2.1_py2 | LSCE\\ ciclad | CDAT 8.2.1 & Python 2.7 | | + | | cdat-8.2.1_py3 | LSCE\\ ciclad | CDAT 8.2.1 & Python 2.7 | |
| cdat-8.2.1_py3 | LSCE | CDAT 8.2.1 & Python 3.6 | | | cdat-8.2.1_py3 | LSCE | CDAT 8.2.1 & Python 3.6 | | ||
- | | cdatm19_py2\\ or //cdatm_py2// | LSCE\\ ciclad | CDAT 8.2.1 & P 2.7 //JYP version// | | + | | cdatm19_py3\\ or //cdatm_py3// | LSCE\\ ciclad | CDAT 8.2.1 & P 2.7 //JYP version// | |
| cdatm19_py3\\ or //cdatm_py3// | LSCE | CDAT 8.2.1 & P 3.7 //JYP version// | | | cdatm19_py3\\ or //cdatm_py3// | LSCE | CDAT 8.2.1 & P 3.7 //JYP version// | | ||
- | | cdat-nightly_py2 | LSCE\\ ciclad | CDAT nightly & Python 2.7 | | ||
- | | cdatm-nightly_py2 | LSCE\\ ciclad | CDAT nightly & P 2.7 //JYP version// | | ||
/* standard page footer */ | /* standard page footer */ |