This is an old revision of the document!
If you have questions about Python, read:
If you mostly want to use the CDAT 8.2.1 installed by JYP, just use the following steps and then read the What should I do now? section below
Server | tcsh | bash |
---|---|---|
LSCE | source ~jypeter/.conda3_21-02_jyp.csh | source ~jypeter/.conda3_21-02_jyp.sh |
ciclad | source ~jypmce/.conda3_jyp.csh n/a | source ~jypmce/.conda3_jyp.sh n/a |
conda activate env_name
conda activate cdatm_py3
which python
(or the wp
alias) and make sure you get something like[…]/miniconda3/envs/env_name/bin/python
You are ready if you have typed the conda activate
command specified in the previous section!
source
line to your shell configuration file so that you don't have to remember it and type it each time you open a new terminalactivate
command in a terminal only when you want to use this specific version of python, rather than having the activate
in your shell configuration file. This may have some side effects in some casesNote: this particular CDAT installation at LSCE (and on other machines/servers) provides many extra (non-CDAT) packages. You can jump directly to the Extra package list at the bottom of this page to see what is available
We assume that Miniconda3 is already installed. Otherwise, follow the the Installing Miniconda steps (and the Post-Miniconda3 installation section) we followed when installing CDAT 8.0
The conda package itself can be updated (if need be) with
conda update -n base -c defaults conda
We first check that we have indeed access to a (mini)conda installation (the base environment we get if we have correctly installed miniconda3), and we assume that we have a write-access to the conda disk hierarchy, and a few Gb of disk space available
> conda env list | grep base base * /home/share/unix_files/cdat/miniconda3_21-02 > df -h /home/share/ Filesystem Size Used Avail Use% Mounted on prolix3:/share 917G 245G 626G 29% /home/share > du -sh /home/share/unix_files/cdat/miniconda3_21-02 577M /home/share/unix_files/cdat/miniconda3_21-02
We can also use conda env list
and remove some obsolete versions with conda remove -n obsolete_name –all
to get some space
Python 2 is deprecated in CDAT, starting with CDAT 8.2.1. If you still need to use python 2.7, please use CDAT 8.1
It's possible that, when using the tcsh shell and trying to use the conda create -n cdat-8.2.1_py3 lots_of_parameters_with_stars
line, you will get the following error message:
conda: No match
This may be due to CSH install/search fail with '*' variable in an argument. In that case, just type bash
and run the installation command in a bash shell
The installation line below is based on information from the CDAT wiki page. See also Some questions and notes about 8.2.1 installation
$ conda create -n cdat-8.2.1_py3 -c conda-forge -c cdat/label/v8.2.1 cdat "libnetcdf=*=mpi_openmpi_*" "mesalib=18.3.1" "python=3"
You can skip this section if you are installing CDAT just for one user (yourself) on a Linux machine (or a Windows 10 machine with WSL)
You will directly make changes and install packages in your main python environment
This section is about the creation of the cdatm19 environment
Notes about actually using the cdatm19 conda-based python
Notes:
$ conda create -n cdatm19_py3 --clone cdat-8.2.1_py3 $ cd /home/share/unix_files/cdat/miniconda3_21-02/envs $ du -sh cdat-8.2.1_py3 cdatm19_py3 2.5G cdat-8.2.1_py3 538M cdatm19_py3
You can skip this section if you are installing CDAT just for one user (yourself) on a Linux machine (or a Windows 10 machine with WSL)
You will directly make changes and install packages in your main python environment
This step could probably be listed at the end, especially in a multi-user environment!
If there is already a cdatm_py3
link, make sure that the new CDAT version is stable and working correctly before updating the cdatm_py3
link
We create a cdatm_py3 symbolic link in the envs
directory, that has a stable name but can be moved to point to the latest default (and hopefully stable) CDAT. In that case, most users can just activate this cdatm_py3 version and always get the latest stable version
$ cd /home/share/unix_files/cdat/miniconda3_21-02/envs $ ln -s cdatm19_py3 cdatm_py3
Start python and paste the following lines
import numpy as np, vcs id100 = np.identity(100) x = vcs.init() x.plot(id100)
If you get the kind of errors described in x.plot crashes with OpenGL2 error, the drivers on your server are probably not compatible with an interactive use of vcs… The workaround is to install mesalib and work in headless mode (e.g. without a display)…
Install mesalib with conda install -c cdat/label/v81 -c conda-forge mesalib vtk-cdat
and try pasting again the few code lines above in python. If things work as expected, no canvas will be opened, but you'll be able to save the canvas to a file with
x.png('test_mesalib') x.pdf('test_mesalib')
Note: using mesalib for headless mode work makes it possible to work without an X server, and with no DISPLAY variable defined
You should download the test data (174M of data…) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any…)
$ conda activate cdatm19_py2 (cdatm19_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)' [...] Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data (cdatm19_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data 174M /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data
After cloning, we are ready to install some extra packages that may be requested by LSCE users
conda install
or pip install
# You can use the following to keep a trace of what will be installed #$ conda install --dry-run -n cdatm19_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm > somewhere/extra_packages.txt # Install... $ conda install -n cdatm19_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret ferret_datasets basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm [...] # Use a similar install line if you want to install the same packages in the python3 version
List of installed packages: lsce-extra_01_install_190304.txt
Packages installed with pip
conda activate cdatm19_py2
pip install dreqPy
pip install --upgrade dreqPy
$ drq -v dreqPy version 01.00.29 [Version 01.00.29]
conda activate cdatm19_py2
pip install ferretmagic
The following packages have no dependency problems and were installed (or updated) later
conda install -n cdatm19_py2 -c conda-forge cmor
python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )'
conda install -n cdatm19_py2 -c conda-forge -c vacumm vacumm
conda install -n cdatm19_py2 -c stefraynaud -c conda-forge spanlib
python -c 'from spanlib.analyzer import Analyzer'
conda install -n cdatm19_py2 -c conda-forge cartopy
conda install -n cdatm19_py2 -c conda-forge joblib
conda install -n cdatm19_py2 -c conda-forge palettable
Add here packages that would be useful and have not been installed yet, or have some problems that prevent their installation
conda install -n cdatxxx -c conda-forge iris
conda install -n cdatm15 -c conda-forge wrf-python
top
)conda install -n cdatm15 -c conda-forge glances
Some packages change more often than others, and can be easily updated the following way:
pip install --upgrade dreqPy
drq -v
Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg pcmdi-metrics in 2.8.0 and cdp in 2.10)
We check if some of the installed files are missing read access for the group or other, and we manually change the permissions
>find /home/share/unix_files/cdat/miniconda3/envs \! -perm /g+r,o+r -ls # Everything OK!
basemap
dreqPy
After following the steps above, we get the following environments. Use the conda env list
command (same result as conda info --envs
) to get the up-to-date list of available environments.
Use conda list -n env_name
to get a detailed list of which packages/modules are installed, or check the conda Installation history section to get more details
Environment name | Server | Summary |
---|---|---|
cdat-8.2.1_py2 | LSCE ciclad | CDAT 8.2.1 & Python 2.7 |
cdat-8.2.1_py3 | LSCE | CDAT 8.2.1 & Python 3.6 |
cdatm19_py2 or cdatm_py2 | LSCE ciclad | CDAT 8.2.1 & P 2.7 JYP version |
cdatm19_py3 or cdatm_py3 | LSCE | CDAT 8.2.1 & P 3.7 JYP version |
cdat-nightly_py2 | LSCE ciclad | CDAT nightly & Python 2.7 |
cdatm-nightly_py2 | LSCE ciclad | CDAT nightly & P 2.7 JYP version |
[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]