This is an old revision of the document!
Follow the instructions about the conda-based versions of UV-CDAT initialization for actually using an installed version of 8.0
We have installed a new version of Miniconda3 (steps below adapted from the installing miniconda instructions)
cd /home/scratch01/jypeter
cd /mnt/h/Scratch/jypeter/
(assuming that there is a H:\Scratch\jypeter
directory)wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
(67 Mo, 15 Feb 2019)bash Miniconda3-latest-Linux-x86_64.sh
/home/share/unix_files/cdat/miniconda3
/data/jypmce/cdat/miniconda3
/home/
does not work (e.g. /mnt/h/CDAT/miniconda3
,assuming there is a H:\CDAT\
directory, does not work)/home/jypeter/miniconda3
miniconda3
directory size is 302M>du -sh miniconda3 302M miniconda3 >cd miniconda3 /home/share/unix_files/cdat/miniconda3 >du -sh * 6.3M bin 2.0M compiler_compat 524K conda-meta 4.0K envs 28K etc 5.2M include 146M lib 8.0K LICENSE.txt 142M pkgs 928K share 92K ssl 12K x86_64-conda_cos6-linux-gnu
.bashrc
(note: the installer automatically creates a backup of .bashrc
). This will some extra lines at the end of .bashrc
used by the old-style conda initialization (we will later switch to the new-style). The added lines should look like:# added by Miniconda3 installer
export PATH=“/home/share/unix_files/cdat/miniconda3/bin:$PATH”
# added by Miniconda4.xxx installer
[some really fancy bash shell commands]
rm Miniconda3-latest-Linux-x86_64.sh
bash
> bash $ which python /home/share/unix_files/cdat/miniconda3/bin/python $ which conda /home/share/unix_files/cdat/miniconda3/bin/conda
$ conda update --all $ du -sh /home/share/unix_files/cdat/miniconda3 515M /home/share/unix_files/cdat/miniconda3 $ conda clean --all $ du -sh /home/share/unix_files/cdat/miniconda3 270M /home/share/unix_files/cdat/miniconda3 $ conda list # packages in environment at /home/share/unix_files/cdat/miniconda3: # # Name Version Build Channel [...] python 3.7.1 h0371630_3 readline 7.0 h7b6447c_5 requests 2.20.0 py37_0 [...]
The idea is to remove the miniconda3 initialization lines that were automatically added at the end of .bashrc
and put them (and other useful commands) in a special initialization file, that can be sourced only when we actually want to use conda and CDAT (in order to avoid potentiel side effects)
Note: this is the new conda activate some_version
style
See the ~jypeter/.conda3_jyp.sh
file below, and how to use it, in a bash shell. In a tcsh shell, see the ~jypeter/.conda3_jyp.csh
further down. In both shell cases, if you are installing your own version of python, you need to use your own location of the initialization files in the source
lines, and you can use another file name than conda3_jyp
$ which python /usr/bin/python $ cat ~jypeter/.conda3_jyp.sh # Conda initialization by JYP, NEW style # # Use this for working with conda and CDAT centrally managed by JYP # # Execute this file in a BASH shell with # source path/this_file # Then get the list of available python distributions with # conda env list # Then activate a specific distribution with # conda activate version_name # # More details in: # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes # # Jean-Yves Peterschmitt - LSCE - 11/2018 source /home/share/unix_files/cdat/miniconda3/etc/profile.d/conda.sh # Use the alias below to easily determine where your python # interpreter is located alias wp="which python" # Where are ALL the python interpreters in the search path alias wpa="which -a python" # The end $ source ~jypeter/.conda3_jyp.sh $ conda activate (base) $ which python /home/share/unix_files/cdat/miniconda3/bin/python
tcsh shell usage example
>which python /usr/bin/python >cat ~jypeter/.conda3_jyp.csh # Conda initialization by JYP, NEW style # # Use this for working with conda and CDAT centrally managed by JYP # # Execute this file in a TCSH shell with # source path/this_file # Then get the list of available python distributions with # conda env list # Then activate a specific distribution with # conda activate version_name # # More details in: # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes # # Jean-Yves Peterschmitt - LSCE - 11/2018 source /home/share/unix_files/cdat/miniconda3/etc/profile.d/conda.csh # Use the alias below to easily determine where your python # interpreter is located alias wp "which python" # The end >source ~jypeter/.conda3_jyp.csh >conda activate (base) >which python /home/share/unix_files/cdat/miniconda3/bin/python
You probably don't want to type the source line each time you need to use your conda based python, so you can add a source ~jypeter/.conda3_jyp.sh
line in your ~/.bashrc
file, and source ~jypeter/.conda3_jyp.csh
line in your ~/.cshrc
file. Then, when you need a specific python environment, just type conda activate name_of_the_specific_environment
The old style notes below are just for reference sake and you can skip them
Note: this is the old source activate some_version
style
See the ~jypeter/.conda3old_jyp.sh
file below, and how to use it, in a bash shell
bash-4.2$ which python /usr/bin/python $ cat .conda3old_jyp.sh # Conda initialization by JYP, OLD style # # Use this for working with conda and CDAT centrally managed by JYP # # Execute this file in a BASH shell with # source path/this_file # Then get the list of available python distributions with # conda env list # Then activate a specific distribution with # source activate version_name # # More details in: # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes # # Jean-Yves Peterschmitt - LSCE - 11/2018 export PATH="/home/share/unix_files/cdat/miniconda3/bin:$PATH" # Use the alias below to easily determine where your python # interpreter is located alias wp="which python" # Where are ALL the python interpreters in the search path alias wpa="which -a python" # The end $ source ~jypeter/.conda3old_jyp.sh $ wp /home/share/unix_files/cdat/miniconda3/bin/python
$ conda create -n cdat-8.0_py2 -c cdat/label/v80 -c conda-forge -c cdat python=2.7 cdat # Generate the list of installed packages $ conda list -n cdat-8.0_py2 > /home/scratch01/jypeter/cdat-8.0_py2_list_181108.txt
List of installed packages: cdat-8.0_py2_list_181108.txt
Disk space after 8.0 installation
$ du -sh /home/share/unix_files/cdat/miniconda3 4.3G /home/share/unix_files/cdat/miniconda3 $ du -sh /home/share/unix_files/cdat/miniconda3/* 15M /home/share/unix_files/cdat/miniconda3/bin 2.4M /home/share/unix_files/cdat/miniconda3/compiler_compat 4.0K /home/share/unix_files/cdat/miniconda3/conda-bld 3.7M /home/share/unix_files/cdat/miniconda3/conda-meta 2.8G /home/share/unix_files/cdat/miniconda3/envs 28K /home/share/unix_files/cdat/miniconda3/etc 4.9M /home/share/unix_files/cdat/miniconda3/include 156M /home/share/unix_files/cdat/miniconda3/lib 8.0K /home/share/unix_files/cdat/miniconda3/LICENSE.txt 1.4G /home/share/unix_files/cdat/miniconda3/pkgs 1.3M /home/share/unix_files/cdat/miniconda3/share 24K /home/share/unix_files/cdat/miniconda3/ssl 12K /home/share/unix_files/cdat/miniconda3/x86_64-conda_cos6-linux-gnu $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2
$ conda create -n cdat-8.0_py3 -c cdat/label/v80 -c conda-forge -c cdat python=3.6 cdat
Notes:
$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat python=2.7 cdat
Not tested! Can this be updated with conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat –all
???
$ conda create -n cdat-nightly_py3 -c cdat/label/nightly -c conda-forge -c cdat python=3.6 cdat
This section is about the creation of the cdatm17 environment
Notes about actually using the cdatm17 conda-based python
Note: using hard links, cloning a full environment only adds an extra 582M of disk space
$ du -sh /home/share/unix_files/cdat/miniconda3/envs/* 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 $ conda create -n cdatm17_py2 --clone cdat-8.0_py2 Source: /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 Destination: /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 Packages: 226 Files: 3 [...] $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 582M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2
conda create -n cdatm-nightly_py2 –clone cdat-nightly_py2
We create a cdatm symbolic link in the envs
directory, that has a stable name but can be moved to point to the latest default CDAT. In that case, most users can just activate this cdatm version and always get the latest stable version
$ cd /home/share/unix_files/cdat/miniconda3/envs $ conda env list base * /home/share/unix_files/cdat/miniconda3 cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 $ ln -s cdatm17_py2 cdatm $ conda env list base * /home/share/unix_files/cdat/miniconda3 cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 cdatm /home/share/unix_files/cdat/miniconda3/envs/cdatm cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 $ source activate cdatm (cdatm) bash-4.2$ conda env list base /home/share/unix_files/cdat/miniconda3 cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 cdatm * /home/share/unix_files/cdat/miniconda3/envs/cdatm cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2
You should download the test data (174M of data…) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any…)
$ source activate cdatm17_py2 (cdatm17_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data' [...] Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data (cdatm17_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data 174M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data
After cloning, we are ready to install some extra packages that may be requested by LSCE users
# Keep a trace of what will be installed bash-4.1$ conda install --dry-run -n cdatm15 -c conda-forge -c uvcdat -c pcmdi pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy rpy2 > /home/scratch01/jypeter/lsce-extra_01_install_170808.txt # Install... bash-4.1$ conda install -n cdatm15 -c conda-forge -c uvcdat -c pcmdi pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires pcmdi_metrics xlsxwriter cmor shapely cartopy rpy2 [...] # Check the disk space after installation bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 11G /home/share/unix_files/cdat/miniconda2 bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* 0 /home/share/unix_files/cdat/miniconda2/envs/cdatm 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 3.7G /home/share/unix_files/cdat/miniconda2/envs/cdatm15 574M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 # Check the disk space again after cleaning bash-4.1$ conda clean --all bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 9.9G /home/share/unix_files/cdat/miniconda2
List of installed packages: lsce-extra_01_install_170808.txt
Disk space used after adding the extra packages, in in the new directory hierarchy (cdat/miniconda2b
, on the obelix servers)
bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b 6.4G /home/share/unix_files/cdat/miniconda2b bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/* 0 /home/share/unix_files/cdat/miniconda2b/envs/cdatm 3.8G /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 630M /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 # Check the disk space again after cleaning bash-4.2$ conda clean --all bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b 5.4G /home/share/unix_files/cdat/miniconda2b
Packages installed with pip
pip install dreqPy
pip install --upgrade dreqPy
$ drq -v dreqPy version 01.00.06 [Version 01.00.06]
pip install dtw
The following packages have no dependency problems and were installed (or updated) later
bash-4.1$ conda list | grep -i esm esmf 7.0.0 6 conda-forge esmpy 7.0.0 py27_1 conda-forge bash-4.1$ conda install -n cdatm15 -c nesii/channel/dev-esmf -c conda-forge esmpy=7.1.0.dev32 [...] bash-4.1$ conda list | grep -i esm esmf 7.1.0.dev32 1 nesii/channel/dev-esmf esmpy 7.1.0.dev32 py27_1 nesii/channel/dev-esmf
conda install -n cdatm15 -c conda-forge cmocean
conda install -n cdatm15 -c conda-forge -c scitools iris
conda install -c conda-forge gdal
conda install -n cdatm15 -c stefraynaud -c conda-forge spanlib
python -c 'from spanlib.analyzer import Analyzer'
conda install -n cdatm15 -c conda-forge wrf-python
top
)conda install -n cdatm15 -c conda-forge glances
conda install -n cdatm15 -c conda-forge windspharm
Add here packages that would be useful but have some problems that prevent their installation
conda install -n cdatm15 -c conda-forge -c vacumm vacumm
conda install -n cdatm15 -c conda-forge fiona geopandas
pip install pysplit
Some packages change more often than others, and can be easily updated the following way:
conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor
python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH'
pip install --upgrade dreqPy
drq -v
Some packages may have files that can only be read by the person who installed CDAT and the LSCE extensions (eg pcmdi-metrics in 2.8.0 and cdp in 2.10)
We check if some of the installed files are missing read access for the group or other, and we manually change the permissions
(cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls 50333969 4 -rw------- 2 jypeter lsce 234 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/PKG-INFO 50333971 4 -rw------- 2 jypeter lsce 291 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/SOURCES.txt 50333962 4 -rw------- 2 jypeter lsce 4 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/top_level.txt 50333961 4 -rw------- 2 jypeter lsce 1 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/dependency_links.txt 50333960 4 -rw------- 2 jypeter lsce 1 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp-1.0.3-py2.7.egg-info/not-zip-safe 50333987 8 -rw------- 2 jypeter lsce 5149 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.pyc 50333986 8 -rw------- 2 jypeter lsce 4208 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.py 50333966 4 -rw------- 2 jypeter lsce 150 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.pyc 50333968 4 -rw------- 2 jypeter lsce 201 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.py 50333988 8 -rw------- 2 jypeter lsce 5365 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_metric.pyc 50333982 4 -rw------- 2 jypeter lsce 1821 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.py 50333981 4 -rw------- 2 jypeter lsce 1773 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.py 50333973 4 -rw------- 2 jypeter lsce 670 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.py 50333980 4 -rw------- 2 jypeter lsce 1642 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_output.pyc 50333979 4 -rw------- 2 jypeter lsce 1474 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.pyc 50333967 4 -rw------- 2 jypeter lsce 187 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.py 50333989 0 -rw------- 2 jypeter lsce 0 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/__init__.py 50333985 4 -rw------- 2 jypeter lsce 3843 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_viewer.py 50333984 4 -rw------- 2 jypeter lsce 2436 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parameter.pyc 50333970 4 -rw------- 2 jypeter lsce 288 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.py 50333974 4 -rw------- 2 jypeter lsce 692 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_tool.pyc 50333978 4 -rw------- 2 jypeter lsce 878 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_io.pyc 50333976 4 -rw------- 2 jypeter lsce 734 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/_cache.py 50333983 4 -rw------- 2 jypeter lsce 2269 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_parser.pyc 50333975 4 -rw------- 2 jypeter lsce 724 Apr 4 20:35 cdatm15/lib/python2.7/site-packages/cdp/cdp_provenance.pyc (cdatm15) bash-4.2$ chmod -R a+r cdatm15/lib/python2.7/site-packages/cdp* (cdatm15) bash-4.2$ find cdatm15 \! -perm /g+r,o+r -ls
basemap
dreqPy
top
)
After following the steps above, we get the following environments. Use the conda info --envs
or the conda env list
command to get the up-to-date list of available environments
Environment name | Server | conda list |
---|---|---|
uvcdat-2.10 | LSCE | package list |
cdatm15 | LSCE | package list |
[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]