This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:uvcdat:cdat_conda:cdat_8_1 [2019/03/07 15:14] jypeter |
other:uvcdat:cdat_conda:cdat_8_1 [2019/03/18 10:52] jypeter [Environments summary] |
||
---|---|---|---|
Line 4: | Line 4: | ||
<WRAP center round tip 60%> | <WRAP center round tip 60%> | ||
- | Follow the instructions about the [[other:python:starting#conda-based_versions_of_uv-cdat|conda-based versions of UV-CDAT initialization]] for actually using an installed version of ''8.1'' | + | You have questions about Python? |
+ | |||
+ | Read [[other:python:jyp_steps|JYP's recommended steps for learning python]]! | ||
</WRAP> | </WRAP> | ||
+ | ===== Using JYP version ===== | ||
+ | |||
+ | If you mostly want to use the CDAT 8.1 installed by JYP, just use the following steps and skip the next sections: | ||
+ | * Initialize conda: | ||
+ | |||
+ | ^ Server ^ tcsh ^ bash ^ | ||
+ | | LSCE | ''source ~jypeter/.conda3_jyp.csh'' | ''source ~jypeter/.conda3_jyp.sh'' | | ||
+ | | ciclad | ''source ~jypmce/.conda3_jyp.csh'' | ''source ~jypmce/.conda3_jyp.sh'' | | ||
+ | |||
+ | * Activate the default environment: ''conda activate cdatm_py2'' | ||
+ | * Check the [[#environments_summary|Available environments]] for more options | ||
===== What's New? ===== | ===== What's New? ===== | ||
Line 16: | Line 29: | ||
We assume that Miniconda3 is already installed. Otherwise, follow the the [[other:uvcdat:cdat_conda:cdat_8_0_py2#installing_miniconda3|Installing Miniconda]] steps (and the //Post-Miniconda3 installation// section) we followed when installing CDAT 8.0 | We assume that Miniconda3 is already installed. Otherwise, follow the the [[other:uvcdat:cdat_conda:cdat_8_0_py2#installing_miniconda3|Installing Miniconda]] steps (and the //Post-Miniconda3 installation// section) we followed when installing CDAT 8.0 | ||
- | ==== Installing CDAT 8.0 ==== | + | ==== Updating conda ==== |
+ | |||
+ | The conda package itself can be updated (if need be) with\\ ''conda update -n base -c defaults conda'' | ||
+ | ==== Installing CDAT 8.1 ==== | ||
We first check that we have indeed access to a conda installation, and we assume that we have a write-access to the conda hierarchy, and a few Gb of disk space available | We first check that we have indeed access to a conda installation, and we assume that we have a write-access to the conda hierarchy, and a few Gb of disk space available | ||
Line 34: | Line 50: | ||
=== Python 2.7 version === | === Python 2.7 version === | ||
- | <code>$ conda create -n cdat-8.1_py2 -c cdat/label/v81 -c conda-forge -c cdat python=2.7 cdat | + | <code>$ conda create -n cdat-8.1_py2 -c cdat/label/v81 -c conda-forge python=2.7 cdat |
# Generate the list of installed packages | # Generate the list of installed packages | ||
$ conda list -n cdat-8.1_py2 > /home/scratch01/jypeter/cdat-8.1_py2_list_190307.txt</code> | $ conda list -n cdat-8.1_py2 > /home/scratch01/jypeter/cdat-8.1_py2_list_190307.txt</code> | ||
Line 42: | Line 58: | ||
=== Python 3.6 version === | === Python 3.6 version === | ||
- | <code>$ conda create -n cdat-8.1_py3 -c cdat/label/v81 -c conda-forge -c cdat python=3.6 cdat</code> | + | <code>$ conda create -n cdat-8.1_py3 -c cdat/label/v81 -c conda-forge python=3.6 cdat</code> |
==== Installing CDAT nightly ==== | ==== Installing CDAT nightly ==== | ||
Line 54: | Line 70: | ||
=== Python 2.7 nightly version === | === Python 2.7 nightly version === | ||
- | <code>$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat python=2.7 cdat</code> | + | <code>$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge python=2.7 cdat</code> |
- | <wrap hi>Not tested!</wrap> Can this be updated with ''conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat --all'' ??? | + | <wrap hi>Not tested!</wrap> Can this be updated with ''conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge --all'' ??? |
=== Python 3.6 nightly version === | === Python 3.6 nightly version === | ||
Line 63: | Line 79: | ||
Not installed yet!</note> | Not installed yet!</note> | ||
- | <code>$ conda create -n cdat-nightly_py3 -c cdat/label/nightly -c conda-forge -c cdat python=3.6 cdat</code> | + | <code>$ conda create -n cdat-nightly_py3 -c cdat/label/nightly -c conda-forge python=3.6 cdat</code> |
===== Cloning cdat to add specific packages for LSCE ===== | ===== Cloning cdat to add specific packages for LSCE ===== | ||
Line 74: | Line 90: | ||
- | Note: using hard links, cloning a full environment //only// adds an extra 582M of disk space | + | Note: using hard links, //cloning// a full environment uses less disk space than making a real copy |
- | <code>$ conda create -n cdatm18_py2 --clone cdat-8.1_py2</code> | + | <code>$ conda create -n cdatm18_py2 --clone cdat-8.1_py2 |
+ | $ du -sh cdat-8.1_py2 cdatm18_py2 | ||
+ | 1.9G cdat-8.1_py2 | ||
+ | 448M cdatm18_py2 | ||
+ | </code> | ||
==== cdat nightly case ==== | ==== cdat nightly case ==== | ||
Line 93: | Line 113: | ||
===== Customizing UV-CDAT for LSCE ===== | ===== Customizing UV-CDAT for LSCE ===== | ||
+ | ==== Testing vcs ==== | ||
+ | Start python and paste the following lines | ||
+ | |||
+ | <code>import numpy as np, vcs | ||
+ | id100 = np.identity(100) | ||
+ | x = vcs.init() | ||
+ | x.plot(id100) | ||
+ | </code> | ||
+ | |||
+ | If you get the kind of errors described in [[https://github.com/CDAT/vcs/issues/393|x.plot crashes with OpenGL2 error]], the drivers on your server are probably not compatible with an interactive use of vcs... The workaround is to **install mesalib and work in //headless mode//** (e.g. without a display)... | ||
+ | |||
+ | Install //mesalib// with ''conda install -c cdat/label/v81 -c conda-forge mesalib vtk-cdat'' and try pasting again the few code lines above in python. If things work as expected, no canvas will be opened, but you'll be able to save the canvas to a file with | ||
+ | |||
+ | <code>x.png('test_mesalib') | ||
+ | x.pdf('test_mesalib')</code> | ||
+ | |||
+ | Note: using //mesalib// for headless mode work makes it possible to work without an X server, and with no DISPLAY variable defined | ||
==== Downloading cdms2/vcs test data ==== | ==== Downloading cdms2/vcs test data ==== | ||
Line 100: | Line 137: | ||
<code>$ conda activate cdatm18_py2 | <code>$ conda activate cdatm18_py2 | ||
- | (cdatm17_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data' | + | (cdatm18_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)' |
[...] | [...] | ||
- | Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data | + | Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data |
- | (cdatm17_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data | + | (cdatm18_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data |
- | 174M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data | + | 174M /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data |
</code> | </code> | ||
Line 116: | Line 153: | ||
* We then install individual extra packages with ''conda install'' or ''pip install'' | * We then install individual extra packages with ''conda install'' or ''pip install'' | ||
- | <code># Keep a trace of what will be installed | + | <code># You can use the following to keep a trace of what will be installed |
- | $ conda install --dry-run -n cdatm17_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm > /home/scratch01/jypeter/lsce-extra_01_install_190304.txt | + | #$ conda install --dry-run -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm > somewhere/extra_packages.txt |
# Install... | # Install... | ||
- | $ conda install -n cdatm17_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm | + | $ conda install -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm |
[...] | [...] | ||
- | # Install extra packages in the nightly version (if you need 'nightly') | + | # Use a similar install line if you want to install the same packages in the python3 version</code> |
- | $ conda install -n cdatm-nightly_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm | + | |
- | + | ||
- | # Check the disk space after installation | + | |
- | $ du -sh /home/share/unix_files/cdat/miniconda3 | + | |
- | 7.5G /home/share/unix_files/cdat/miniconda3 | + | |
- | + | ||
- | $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* | + | |
- | 2.5G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | + | |
- | 1.2G /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | + | |
- | 938M /home/share/unix_files/cdat/miniconda3/envs/cdatm-nightly_py2 | + | |
- | 450M /home/share/unix_files/cdat/miniconda3/envs/cdat-nightly_py2</code> | + | |
List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_190304.txt |}} | List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_190304.txt |}} | ||
Line 140: | Line 166: | ||
Packages installed with //pip// | Packages installed with //pip// | ||
* [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API | * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API | ||
- | * ''pip install dreqPy'' | + | * ''conda activate cdatm18_py2''\\ ''pip install dreqPy'' |
* Update with: ''pip install %%--%%upgrade dreqPy'' | * Update with: ''pip install %%--%%upgrade dreqPy'' | ||
* Get version number with:\\ <code>$ drq -v | * Get version number with:\\ <code>$ drq -v | ||
Line 146: | Line 172: | ||
The following packages have no dependency problems and were installed (or updated) //later// | The following packages have no dependency problems and were installed (or updated) //later// | ||
+ | |||
+ | * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files | ||
+ | * ''conda install -n cdatm18_py2 -c conda-forge cmor'' | ||
+ | * Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' '' | ||
+ | * <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]] | ||
* [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations | ||
- | * ''conda install -n cdatm17_py2 -c conda-forge -c vacumm vacumm'' | + | * ''conda install -n cdatm18_py2 -c conda-forge -c vacumm vacumm'' |
+ | * <wrap hi>Warning!</wrap> [[https://github.com/VACUMM/vacumm/issues/15|standard install of vacumm does not work for Python3]] | ||
* [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
- | * ''conda install -n cdatm17_py2 -c stefraynaud -c conda-forge spanlib'' | + | * ''conda install -n cdatm18_py2 -c stefraynaud -c conda-forge spanlib'' |
* Test: ''python -c 'from spanlib.analyzer import Analyzer' '' | * Test: ''python -c 'from spanlib.analyzer import Analyzer' '' | ||
+ | |||
+ | * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses | ||
+ | * ''conda install -n cdatm18_py2 -c conda-forge cartopy'' | ||
+ | |||
+ | * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs | ||
+ | * ''conda install -n cdatm18_py2 -c conda-forge joblib'' | ||
+ | |||
+ | * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | ||
+ | * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]] | ||
==== TODO ==== | ==== TODO ==== | ||
Line 177: | Line 218: | ||
Some packages change more often than others, and can be easily updated the following way: | Some packages change more often than others, and can be easily updated the following way: | ||
- | |||
- | * [[https://cmor.llnl.gov/|CMOR]] | ||
- | * Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor'' | ||
- | * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' '' | ||
* [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]] | * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]] | ||
Line 186: | Line 223: | ||
* Get version number with: ''drq -v'' | * Get version number with: ''drq -v'' | ||
- | ==== Installing CLIMAF ==== | ||
- | |||
- | The [[https://climaf.readthedocs.io/|CLIMAF]] installation is a special case, because we may want to use it with the Python installed for CDAT, but it is not installed using conda | ||
- | |||
- | We follow the steps for the [[https://climaf.readthedocs.io/en/latest/installing.html#installing-climaf-if-you-are-not-at-cnrm-on-ciclad-or-climserv|installation of a non-standard server]] | ||
- | |||
- | * Go to the appropriate installation directory (we choose the directory where we have also installed miniconda3): | ||
- | * LSCE: ''cd /home/share/unix_files/cdat'' | ||
- | * ciclad: ''/data/jypmce/cdat/'' | ||
- | * ''git clone https://github.com/senesis/climaf climaf'' | ||
- | * Add CLIMAF to the PYTHONPATH environment variable | ||
- | * LSCE & tcsh: ''setenv PYTHONPATH ${PYTHONPATH}:/home/share/unix_files/cdat/climaf'' | ||
- | * LSCE & bash: ''export PYTHONPATH=${PYTHONPATH}:/home/share/unix_files/cdat/climaf'' | ||
- | * ciclad: same as above, using ''/data/jypmce/cdat/climaf'' | ||
- | * Test the installation | ||
- | * make sure the [[https://climaf.readthedocs.io/en/latest/requirements.html#requirements|Requirements]] are available: | ||
- | * LSCE: ''module load netcdf/4 nco cdo ncl ncview'' | ||
- | * ciclad: ''module load netcdf4 nco cdo ncl/6.3.0'' | ||
- | * optional: start an X server | ||
- | * <code>cd climaf/testing | ||
- | ./test_install.sh</code> | ||
- | * Once the tests are OK, add the required //module load// and environment variables definitions to the ''.basrc'' and ''.cshrc'' files. <wrap hi>Note: maybe these lines should go to the ''.profile'' and ''.login'' files...(check [[https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html|Bash Startup Files]])</wrap> | ||
- | * LSCE: add to your .cshrc | ||
- | * <code># Prerequisites for using CLIMAF | ||
- | module load netcdf/4 nco cdo ncl ncview | ||
- | |||
- | # CLIMAF environment variables | ||
- | setenv CLIMAF /home/share/unix_files/cdat/climaf | ||
- | setenv PYTHONPATH "${PYTHONPATH}:${CLIMAF}" | ||
- | #setenv PYTHONPATH $CLIMAF | ||
- | setenv PATH "${PATH}:${CLIMAF}/bin" | ||
- | setenv CLIMAF_CACHE "/home/scratch01/${USER}/climaf_cache" | ||
- | setenv TMPDIR $CLIMAF_CACHE | ||
- | setenv CLIMAF_LOG_DIR $CLIMAF_CACHE | ||
- | |||
- | #conda activate cdatm-nightly_py2</code> | ||
- | * ciclad: add to your .bashrc | ||
- | * <code># Prerequisites for using CLIMAF when not using the | ||
- | # default 'module load climaf' | ||
- | module load netcdf4 nco cdo ncl/6.3.0 | ||
- | |||
- | # CLIMAF environment variables | ||
- | export CLIMAF=/data/jypmce/cdat/climaf | ||
- | #export PYTHONPATH=$PYTHONPATH:$CLIMAF | ||
- | export PYTHONPATH=$CLIMAF | ||
- | export PATH=$PATH:$CLIMAF/bin | ||
- | export CLIMAF_CACHE=/data/jypmce/climaf_cache | ||
- | export TMPDIR=$CLIMAF_CACHE | ||
- | export CLIMAF_LOG_DIR=$CLIMAF_CACHE | ||
- | |||
- | #conda activate cdatm-nightly_py2 | ||
- | </code> | ||
- | * Open a new shell and <wrap em>run the tests again</wrap>. If there is an error that was not there, check what you have added to the shell init files | ||
- | * [[other:uvcdat:cdat_conda:climaf_test|Test CLIMAF on ciclad]] | ||
==== Cleaning up things ==== | ==== Cleaning up things ==== | ||
Line 270: | Line 253: | ||
* [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module | ||
* [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) | * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane) | ||
- | * [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python | + | * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses |
* [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python | * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python | ||
* [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography | * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography | ||
Line 277: | Line 260: | ||
* [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library | ||
* [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python | ||
+ | * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework | ||
+ | * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs | ||
==== Removed packages ==== | ==== Removed packages ==== | ||
Line 284: | Line 269: | ||
===== Environments summary ===== | ===== Environments summary ===== | ||
- | After following the steps above, we get the following environments. Use the ''conda info %%--%%envs'' or the ''conda env list'' command to get the up-to-date list of available environments | + | After following the steps above, we get the following environments. Use the ''conda env list'' command (same result as ''conda info %%--%%envs'') to get the up-to-date list of available environments. |
+ | |||
+ | Use ''conda list -n env_name'' to get a detailed list of which packages/modules are installed, or check the [[other:uvcdat:conda_notes#installation_history|conda Installation history]] section to get more details | ||
- | ^ Environment\\ name ^ Server ^ ''conda list'' ^ | + | ^ Environment\\ name ^ Server ^ Summary ^ |
- | | cdat-8.1_py2 | LSCE\\ ciclad | | | + | | cdat-8.1_py2 | LSCE\\ ciclad | CDAT 8.1 & Python 2.7 | |
- | | cdat-8.1_py3 | LSCE | | | + | | cdat-8.1_py3 | LSCE | CDAT 8.1 & Python 3.6 | |
- | | cdatm18_py2 | LSCE\\ ciclad | | | + | | cdatm18_py2\\ or //cdatm_py2// | LSCE\\ ciclad | CDAT 8.1 & P 2.7 //JYP version// | |
- | | cdat-nightly_py2 | LSCE\\ ciclad | | | + | | cdatm18_py3\\ or //cdatm_py3// | LSCE | CDAT 8.1 & P 3.7 //JYP version// | |
- | | cdatm-nightly_py2 | LSCE\\ ciclad | | | + | | cdat-nightly_py2 | LSCE\\ ciclad | CDAT nightly & Python 2.7 | |
+ | | cdatm-nightly_py2 | LSCE\\ ciclad | CDAT nightly & P 2.7 //JYP version// | | ||
/* standard page footer */ | /* standard page footer */ |