This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:uvcdat:cdat_conda:cdat_8_0_py2 [2018/11/08 09:47] jypeter |
other:uvcdat:cdat_conda:cdat_8_0_py2 [2018/11/08 17:01] jypeter |
||
---|---|---|---|
Line 16: | Line 16: | ||
===== Installation with Miniconda3 ===== | ===== Installation with Miniconda3 ===== | ||
- | We have installed a new version of Miniconda3, based on the old [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions | + | ==== Installing Miniconda3 ==== |
+ | |||
+ | We have installed a new version of Miniconda3, by following the existing [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions | ||
* Follow the [[https://conda.io/docs/install/quick.html|Quick install]] steps to install Miniconda | * Follow the [[https://conda.io/docs/install/quick.html|Quick install]] steps to install Miniconda | ||
Line 66: | Line 68: | ||
requests 2.20.0 py37_0 | requests 2.20.0 py37_0 | ||
[...]</code> | [...]</code> | ||
- | * Remove the miniconda3 initialization lines automatically added at the end of ''.bashrc'' and put them (and other useful commands) in a special initialization file, that can be sourced only when we actually want to use conda and CDAT | + | * Remove the miniconda3 initialization lines that were automatically added at the end of ''.bashrc'' and put them (and other useful commands) in a special initialization file (see ''~jypeter/.conda3_jyp.sh'' below), that can be sourced only when we actually want to use conda and CDAT\\ <code>bash-4.2$ which python |
+ | /usr/bin/python | ||
+ | $ cat ~jypeter/.conda3_jyp.sh | ||
+ | # Conda initialization by JYP | ||
+ | # | ||
+ | # Use this for working with conda and CDAT centrally managed by JYP | ||
+ | # | ||
+ | # More details in: | ||
+ | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_v ersions_of_uv-cdat | ||
+ | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes | ||
+ | # | ||
+ | # Jean-Yves Peterschmitt - LSCE - 11/2018 | ||
- | Disk space used before 2.10 installation (after running ''conda clean --all'') | + | export PATH="/home/share/unix_files/cdat/miniconda3/bin:$PATH" |
- | <code> > du -sh /home/share/unix_files/cdat/miniconda2 | + | # Use the alias below to easily determine where your python |
- | 6.5G /home/share/unix_files/cdat/miniconda2 | + | # interpreter is located |
+ | alias wp="which python" | ||
- | > du -sh /home/share/unix_files/cdat/miniconda2/* | + | # Where are ALL the python interpreters in the search path |
- | 4.4M /home/share/unix_files/cdat/miniconda2/bin | + | alias wpa="which -a python" |
- | 4.0K /home/share/unix_files/cdat/miniconda2/conda-bld | + | |
- | 272K /home/share/unix_files/cdat/miniconda2/conda-meta | + | |
- | 6.4G /home/share/unix_files/cdat/miniconda2/envs | + | |
- | 20K /home/share/unix_files/cdat/miniconda2/etc | + | |
- | 3.8M /home/share/unix_files/cdat/miniconda2/include | + | |
- | 99M /home/share/unix_files/cdat/miniconda2/lib | + | |
- | 4.0K /home/share/unix_files/cdat/miniconda2/LICENSE.txt | + | |
- | 76K /home/share/unix_files/cdat/miniconda2/pkgs | + | |
- | 376K /home/share/unix_files/cdat/miniconda2/share | + | |
- | 364K /home/share/unix_files/cdat/miniconda2/ssl | + | |
- | >du -sh /home/share/unix_files/cdat/miniconda2/envs/* | + | # The end |
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | + | $ source ~jypeter/.conda3_jyp.sh |
- | 1.7G /home/share/unix_files/cdat/miniconda2/envs/keras | + | $ wp |
- | 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0</code> | + | /home/share/unix_files/cdat/miniconda3/bin/python |
+ | </code> | ||
- | Actual installation | + | ==== Installing CDAT 8.0 ==== |
- | + | ||
- | <code>bash-4.1$ conda create -n uvcdat-2.10 -c conda-forge -c uvcdat uvcdat=2.10 | + | |
+ | <code>conda create -n cdat-8.0_py2 -c cdat/label/v80 -c conda-forge -c cdat python=2.7 cdat | ||
# Generate the list of installed packages | # Generate the list of installed packages | ||
- | conda list -n uvcdat-2.10 > /home/scratch01/jypeter/uvcdat-2.10_list_170721.txt</code> | + | conda list -n cdat-8.0_py2 > /home/scratch01/jypeter/cdat-8.0_py2_list_181108.txt</code> |
- | List of installed packages: {{:other:uvcdat:cdat_conda:uvcdat-2.10_list_170721.txt}} | + | List of installed packages: {{:other:uvcdat:cdat_conda:cdat-8.0_py2_list_181108.txt}} |
- | Disk space after 2.10 installation | + | Disk space after 8.0 installation |
- | <code>bash-4.1$ conda env list | + | <code>$ du -sh /home/share/unix_files/cdat/miniconda3 |
- | cdatm14 /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | + | 4.3G /home/share/unix_files/cdat/miniconda3 |
- | keras /home/share/unix_files/cdat/miniconda2/envs/keras | + | $ du -sh /home/share/unix_files/cdat/miniconda3/* |
- | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | + | 15M /home/share/unix_files/cdat/miniconda3/bin |
- | uvcdat-2.8.0 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 | + | 2.4M /home/share/unix_files/cdat/miniconda3/compiler_compat |
- | root * /home/share/unix_files/cdat/miniconda2 | + | 4.0K /home/share/unix_files/cdat/miniconda3/conda-bld |
- | + | 3.7M /home/share/unix_files/cdat/miniconda3/conda-meta | |
- | > du -sh /home/share/unix_files/cdat/miniconda2 | + | 2.8G /home/share/unix_files/cdat/miniconda3/envs |
- | 11G /home/share/unix_files/cdat/miniconda2 | + | 28K /home/share/unix_files/cdat/miniconda3/etc |
- | + | 4.9M /home/share/unix_files/cdat/miniconda3/include | |
- | > du -sh /home/share/unix_files/cdat/miniconda2/* | + | 156M /home/share/unix_files/cdat/miniconda3/lib |
- | 4.4M /home/share/unix_files/cdat/miniconda2/bin | + | 8.0K /home/share/unix_files/cdat/miniconda3/LICENSE.txt |
- | 4.0K /home/share/unix_files/cdat/miniconda2/conda-bld | + | 1.4G /home/share/unix_files/cdat/miniconda3/pkgs |
- | 272K /home/share/unix_files/cdat/miniconda2/conda-meta | + | 1.3M /home/share/unix_files/cdat/miniconda3/share |
- | 9.2G /home/share/unix_files/cdat/miniconda2/envs | + | 24K /home/share/unix_files/cdat/miniconda3/ssl |
- | 20K /home/share/unix_files/cdat/miniconda2/etc | + | 12K /home/share/unix_files/cdat/miniconda3/x86_64-conda_cos6-linux-gnu |
- | 3.8M /home/share/unix_files/cdat/miniconda2/include | + | $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* |
- | 99M /home/share/unix_files/cdat/miniconda2/lib | + | 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2</code> |
- | 4.0K /home/share/unix_files/cdat/miniconda2/LICENSE.txt | + | |
- | 1.4G /home/share/unix_files/cdat/miniconda2/pkgs | + | |
- | 376K /home/share/unix_files/cdat/miniconda2/share | + | |
- | 364K /home/share/unix_files/cdat/miniconda2/ssl | + | |
- | + | ||
- | > du -sh /home/share/unix_files/cdat/miniconda2/envs/* | + | |
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | + | |
- | 1.7G /home/share/unix_files/cdat/miniconda2/envs/keras | + | |
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | + | |
- | 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 | + | |
- | </code> | + | |
- | + | ||
- | Disk space after removing the keras environment that we don't need any more | + | |
- | + | ||
- | <code>jypeter@asterix2 - ...cdat - 67 >du -sh /home/share/unix_files/cdat/miniconda2 | + | |
- | 9.0G /home/share/unix_files/cdat/miniconda2 | + | |
- | jypeter@asterix2 - ...cdat - 68 >du -sh /home/share/unix_files/cdat/miniconda2/* | + | |
- | 4.4M /home/share/unix_files/cdat/miniconda2/bin | + | |
- | 4.0K /home/share/unix_files/cdat/miniconda2/conda-bld | + | |
- | 272K /home/share/unix_files/cdat/miniconda2/conda-meta | + | |
- | 7.5G /home/share/unix_files/cdat/miniconda2/envs | + | |
- | 20K /home/share/unix_files/cdat/miniconda2/etc | + | |
- | 3.8M /home/share/unix_files/cdat/miniconda2/include | + | |
- | 99M /home/share/unix_files/cdat/miniconda2/lib | + | |
- | 4.0K /home/share/unix_files/cdat/miniconda2/LICENSE.txt | + | |
- | 1.4G /home/share/unix_files/cdat/miniconda2/pkgs | + | |
- | 376K /home/share/unix_files/cdat/miniconda2/share | + | |
- | 364K /home/share/unix_files/cdat/miniconda2/ssl | + | |
- | jypeter@asterix2 - ...cdat - 69 >du -sh /home/share/unix_files/cdat/miniconda2/envs/* | + | |
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | + | |
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | + | |
- | 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0</code> | + | |
- | + | ||
- | Disk space used in <wrap em>a brand new directory hierarchy</wrap> (''cdat/miniconda2b'', on the obelix servers) | + | |
- | + | ||
- | <code>bash-4.2$ conda env list | + | |
- | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | + | |
- | root * /home/share/unix_files/cdat/miniconda2b | + | |
- | + | ||
- | bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b | + | |
- | 4.3G /home/share/unix_files/cdat/miniconda2b | + | |
- | + | ||
- | bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/* | + | |
- | 4.4M /home/share/unix_files/cdat/miniconda2b/bin | + | |
- | 4.0K /home/share/unix_files/cdat/miniconda2b/conda-bld | + | |
- | 280K /home/share/unix_files/cdat/miniconda2b/conda-meta | + | |
- | 2.8G /home/share/unix_files/cdat/miniconda2b/envs | + | |
- | 20K /home/share/unix_files/cdat/miniconda2b/etc | + | |
- | 3.8M /home/share/unix_files/cdat/miniconda2b/include | + | |
- | 100M /home/share/unix_files/cdat/miniconda2b/lib | + | |
- | 8.0K /home/share/unix_files/cdat/miniconda2b/LICENSE.txt | + | |
- | 1.4G /home/share/unix_files/cdat/miniconda2b/pkgs | + | |
- | 16K /home/share/unix_files/cdat/miniconda2b/share | + | |
- | 376K /home/share/unix_files/cdat/miniconda2b/ssl | + | |
- | </code> | + | |
- | ===== Cloning uvcdat to add specific packages for LSCE ===== | + | ===== Cloning cdat to add specific packages for LSCE ===== |
<WRAP center round tip 60%> | <WRAP center round tip 60%> | ||
- | This section is about the creation of the **cdatm15** environment | + | This section is about the creation of the **cdatm17** environment |
- | Notes about [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat|actually using the cdatm15 conda-based python]] | + | Notes about [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat|actually using the cdatm17 conda-based python]] |
</WRAP> | </WRAP> | ||
- | Note: using hard links, cloning a full environment //only// adds an extra 500M of disk space | + | Note: using hard links, cloning a full environment //only// adds an extra 582M of disk space |
- | <code>bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 | + | <code>$ du -sh /home/share/unix_files/cdat/miniconda3/envs/* |
- | 9.0G /home/share/unix_files/cdat/miniconda2 | + | 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 |
+ | $ conda create -n cdatm17_py2 --clone cdat-8.0_py2 | ||
+ | Source: /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | ||
+ | Destination: /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | ||
+ | Packages: 226 | ||
+ | Files: 3 | ||
- | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* | ||
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | ||
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | ||
- | 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 | ||
- | |||
- | bash-4.1$ conda create -n cdatm15 --clone uvcdat-2.10 | ||
- | Source: /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | ||
- | Destination: /home/share/unix_files/cdat/miniconda2/envs/cdatm15 | ||
[...] | [...] | ||
- | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 | + | $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* |
- | 9.5G /home/share/unix_files/cdat/miniconda2 | + | 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 |
+ | 582M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | ||
- | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* | ||
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | ||
- | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm15 | ||
- | 573M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | ||
- | 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 | ||
- | </code> | ||
- | |||
- | Disk space used after cloning, in <wrap em>in the new directory hierarchy</wrap> (''cdat/miniconda2b'', on the obelix servers) | ||
- | |||
- | <code>bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b | ||
- | 4.8G /home/share/unix_files/cdat/miniconda2b | ||
- | |||
- | bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/* | ||
- | 4.4M /home/share/unix_files/cdat/miniconda2b/bin | ||
- | 4.0K /home/share/unix_files/cdat/miniconda2b/conda-bld | ||
- | 280K /home/share/unix_files/cdat/miniconda2b/conda-meta | ||
- | 3.4G /home/share/unix_files/cdat/miniconda2b/envs | ||
- | 20K /home/share/unix_files/cdat/miniconda2b/etc | ||
- | 3.8M /home/share/unix_files/cdat/miniconda2b/include | ||
- | 100M /home/share/unix_files/cdat/miniconda2b/lib | ||
- | 8.0K /home/share/unix_files/cdat/miniconda2b/LICENSE.txt | ||
- | 1.4G /home/share/unix_files/cdat/miniconda2b/pkgs | ||
- | 16K /home/share/unix_files/cdat/miniconda2b/share | ||
- | 376K /home/share/unix_files/cdat/miniconda2b/ssl | ||
- | |||
- | bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/* | ||
- | 2.8G /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | ||
- | 574M /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | ||
</code> | </code> | ||
Line 238: | Line 157: | ||
We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version | We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version | ||
- | <code>bash-4.2$ cd /home/share/unix_files/cdat/miniconda2b/envs | + | <code>$ cd /home/share/unix_files/cdat/miniconda3/envs |
- | + | $ conda env list | |
- | bash-4.2$ conda env list | + | base * /home/share/unix_files/cdat/miniconda3 |
- | cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | + | cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 |
- | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | + | cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 |
- | root * /home/share/unix_files/cdat/miniconda2b | + | $ ln -s cdatm17_py2 cdatm |
- | + | $ conda env list | |
- | bash-4.2$ ln -s cdatm15 cdatm | + | base * /home/share/unix_files/cdat/miniconda3 |
- | + | cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | |
- | bash-4.2$ conda env list | + | cdatm /home/share/unix_files/cdat/miniconda3/envs/cdatm |
- | cdatm /home/share/unix_files/cdat/miniconda2b/envs/cdatm | + | cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 |
- | cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | + | $ source activate cdatm |
- | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | + | |
- | root * /home/share/unix_files/cdat/miniconda2b | + | |
- | + | ||
- | bash-4.2$ source activate cdatm | + | |
(cdatm) bash-4.2$ conda env list | (cdatm) bash-4.2$ conda env list | ||
- | cdatm * /home/share/unix_files/cdat/miniconda2b/envs/cdatm | + | base /home/share/unix_files/cdat/miniconda3 |
- | cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | + | cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 |
- | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | + | cdatm * /home/share/unix_files/cdat/miniconda3/envs/cdatm |
- | root /home/share/unix_files/cdat/miniconda2b | + | cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2</code> |
- | </code> | + | |
===== Customizing UV-CDAT for LSCE ===== | ===== Customizing UV-CDAT for LSCE ===== | ||
Line 267: | Line 180: | ||
==== Downloading cdms2/vcs test data ==== | ==== Downloading cdms2/vcs test data ==== | ||
- | You should download the test data and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...) | + | You should download the test data (174M of data...) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...) |
- | <code>bash-4.1$ source activate cdatm15 | + | <code>$ source activate cdatm17_py2 |
- | (cdatm15) bash-4.1$ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data' | + | (cdatm17_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data' |
[...] | [...] | ||
- | Finished downloading sample data to /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data | + | Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data |
- | (cdatm15) bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data | + | (cdatm17_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data |
- | 174M /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data</code> | + | 174M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data |
+ | </code> | ||