other:uvcdat:cdat_conda:cdat_8_0_py2
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
other:uvcdat:cdat_conda:cdat_8_0_py2 [2018/11/08 15:30] jypeter |
other:uvcdat:cdat_conda:cdat_8_0_py2 [2018/11/13 17:21] jypeter Added details for activating conda |
||
|---|---|---|---|
| Line 18: | Line 18: | ||
| ==== Installing Miniconda3 ==== | ==== Installing Miniconda3 ==== | ||
| - | We have installed a new version of Miniconda3, based on the old [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions | + | We have installed a new version of Miniconda3, by following the existing [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions |
| * Follow the [[https://conda.io/docs/install/quick.html|Quick install]] steps to install Miniconda | * Follow the [[https://conda.io/docs/install/quick.html|Quick install]] steps to install Miniconda | ||
| Line 68: | Line 68: | ||
| requests 2.20.0 py37_0 | requests 2.20.0 py37_0 | ||
| [...]</code> | [...]</code> | ||
| - | * Remove the miniconda3 initialization lines that were automatically added at the end of ''.bashrc'' and put them (and other useful commands) in a special initialization file (see ''~jypeter/.conda3_jyp.sh'' below), that can be sourced only when we actually want to use conda and CDAT\\ <code>bash-4.2$ which python | + | |
| + | ==== Post-Miniconda3 installation ==== | ||
| + | |||
| + | The idea is to remove the miniconda3 initialization lines that were automatically added at the end of ''.bashrc'' and put them (and other useful commands) in a special initialization file, that can be sourced only when we actually want to use conda and CDAT | ||
| + | |||
| + | === New style initialization === | ||
| + | |||
| + | Note: this is the **new** ''conda activate some_version'' style | ||
| + | |||
| + | See the ''~jypeter/.conda3_jyp.sh'' file below, and how to use it, **in a bash shell**. In a **tcsh shell**, see the ''~jypeter/.conda3_jyp.csh'' further down | ||
| + | |||
| + | <code>$ which python | ||
| /usr/bin/python | /usr/bin/python | ||
| + | |||
| $ cat ~jypeter/.conda3_jyp.sh | $ cat ~jypeter/.conda3_jyp.sh | ||
| - | # Conda initialization by JYP | + | # Conda initialization by JYP, NEW style |
| # | # | ||
| # Use this for working with conda and CDAT centrally managed by JYP | # Use this for working with conda and CDAT centrally managed by JYP | ||
| + | # | ||
| + | # Execute this file in a BASH shell with | ||
| + | # source path/this_file | ||
| + | # Then get the list of available python distributions with | ||
| + | # conda env list | ||
| + | # Then activate a specific distribution with | ||
| + | # conda activate version_name | ||
| # | # | ||
| # More details in: | # More details in: | ||
| - | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_v ersions_of_uv-cdat | + | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat |
| # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes | ||
| # | # | ||
| # Jean-Yves Peterschmitt - LSCE - 11/2018 | # Jean-Yves Peterschmitt - LSCE - 11/2018 | ||
| - | export PATH="/home/share/unix_files/cdat/miniconda3/bin:$PATH" | + | . /home/share/unix_files/cdat/miniconda3/etc/profile.d/conda.sh |
| # Use the alias below to easily determine where your python | # Use the alias below to easily determine where your python | ||
| Line 91: | Line 110: | ||
| # The end | # The end | ||
| + | |||
| $ source ~jypeter/.conda3_jyp.sh | $ source ~jypeter/.conda3_jyp.sh | ||
| - | $ wp | + | |
| - | /home/share/unix_files/cdat/miniconda3/bin/python | + | $ conda activate cdatm |
| + | (cdatm) $ which python | ||
| + | /home/share/unix_files/cdat/miniconda3/envs/cdatm/bin/python | ||
| </code> | </code> | ||
| - | ==== Installing CDAT 8.0 ==== | + | **tcsh shell** usage example |
| + | <code> >which python | ||
| + | /usr/bin/python | ||
| - | <code>bash-4.1$ conda create -n uvcdat-2.10 -c conda-forge -c uvcdat uvcdat=2.10 | + | >cat ~jypeter/.conda3_jyp.csh |
| - | conda create -n cdat-8.0 -c cdat/label/v80 -c conda-forge -c cdat python=2.7 cdat | + | # Conda initialization by JYP, NEW style |
| - | # Generate the list of installed packages | + | # |
| - | conda list -n uvcdat-2.10 > /home/scratch01/jypeter/uvcdat-2.10_list_170721.txt</code> | + | # Use this for working with conda and CDAT centrally managed by JYP |
| + | # | ||
| + | # Execute this file in a TCSH shell with | ||
| + | # source path/this_file | ||
| + | # Then get the list of available python distributions with | ||
| + | # conda env list | ||
| + | # Then activate a specific distribution with | ||
| + | # conda activate version_name | ||
| + | # | ||
| + | # More details in: | ||
| + | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat | ||
| + | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes | ||
| + | # | ||
| + | # Jean-Yves Peterschmitt - LSCE - 11/2018 | ||
| - | List of installed packages: {{:other:uvcdat:cdat_conda:uvcdat-2.10_list_170721.txt}} | + | source /home/share/unix_files/cdat/miniconda3/etc/profile.d/conda.csh |
| - | Disk space after 8.0 installation | + | # Use the alias below to easily determine where your python |
| + | # interpreter is located | ||
| + | alias wp "which python" | ||
| - | <code>bash-4.1$ conda env list | + | # Where are ALL the python interpreters in the search path |
| - | cdatm14 /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | + | alias wpa "\which -a python" |
| - | keras /home/share/unix_files/cdat/miniconda2/envs/keras | + | |
| - | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | + | |
| - | uvcdat-2.8.0 /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 | + | |
| - | root * /home/share/unix_files/cdat/miniconda2 | + | |
| - | > du -sh /home/share/unix_files/cdat/miniconda2 | + | # The end |
| - | 11G /home/share/unix_files/cdat/miniconda2 | + | |
| - | > du -sh /home/share/unix_files/cdat/miniconda2/* | + | >source ~jypeter/.conda3_jyp.csh |
| - | 4.4M /home/share/unix_files/cdat/miniconda2/bin | + | |
| - | 4.0K /home/share/unix_files/cdat/miniconda2/conda-bld | + | |
| - | 272K /home/share/unix_files/cdat/miniconda2/conda-meta | + | |
| - | 9.2G /home/share/unix_files/cdat/miniconda2/envs | + | |
| - | 20K /home/share/unix_files/cdat/miniconda2/etc | + | |
| - | 3.8M /home/share/unix_files/cdat/miniconda2/include | + | |
| - | 99M /home/share/unix_files/cdat/miniconda2/lib | + | |
| - | 4.0K /home/share/unix_files/cdat/miniconda2/LICENSE.txt | + | |
| - | 1.4G /home/share/unix_files/cdat/miniconda2/pkgs | + | |
| - | 376K /home/share/unix_files/cdat/miniconda2/share | + | |
| - | 364K /home/share/unix_files/cdat/miniconda2/ssl | + | |
| - | > du -sh /home/share/unix_files/cdat/miniconda2/envs/* | + | >conda activate cdatm |
| - | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | + | |
| - | 1.7G /home/share/unix_files/cdat/miniconda2/envs/keras | + | |
| - | 2.8G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | + | |
| - | 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 | + | |
| - | </code> | + | |
| + | (cdatm) >which python | ||
| + | /home/share/unix_files/cdat/miniconda3/envs/cdatm/bin/python</code> | ||
| - | ===== Cloning cdat to add specific packages for LSCE ===== | + | === Old style initialization === |
| - | <WRAP center round tip 60%> | + | Note: this is the **old** ''source activate some_version'' style |
| - | This section is about the creation of the **cdatm17** environment | + | |
| - | Notes about [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat|actually using the cdatm15 conda-based python]] | + | See the ''~jypeter/.conda3old_jyp.sh'' file below, and how to use it, **in a bash shell** |
| - | </WRAP> | + | |
| + | <code>bash-4.2$ which python | ||
| + | /usr/bin/python | ||
| - | Note: using hard links, cloning a full environment //only// adds an extra 500M of disk space | + | $ cat .conda3old_jyp.sh |
| + | # Conda initialization by JYP, OLD style | ||
| + | # | ||
| + | # Use this for working with conda and CDAT centrally managed by JYP | ||
| + | # | ||
| + | # Execute this file in a BASH shell with | ||
| + | # source path/this_file | ||
| + | # Then get the list of available python distributions with | ||
| + | # conda env list | ||
| + | # Then activate a specific distribution with | ||
| + | # source activate version_name | ||
| + | # | ||
| + | # More details in: | ||
| + | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat | ||
| + | # https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes | ||
| + | # | ||
| + | # Jean-Yves Peterschmitt - LSCE - 11/2018 | ||
| - | <code>bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda3 | + | export PATH="/home/share/unix_files/cdat/miniconda3/bin:$PATH" |
| - | 9.0G /home/share/unix_files/cdat/miniconda3 | + | |
| - | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda3/envs/* | + | # Use the alias below to easily determine where your python |
| + | # interpreter is located | ||
| + | alias wp="which python" | ||
| - | bash-4.1$ conda create -n cdatm16 --clone uvcdat-2.10 | + | # Where are ALL the python interpreters in the search path |
| - | Source: /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | + | alias wpa="which -a python" |
| - | Destination: /home/share/unix_files/cdat/miniconda2/envs/cdatm15 | + | |
| - | [...] | + | |
| - | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2 | + | # The end |
| - | 9.5G /home/share/unix_files/cdat/miniconda2 | + | |
| - | bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/* | + | $ source ~jypeter/.conda3old_jyp.sh |
| - | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm14 | + | |
| - | 2.8G /home/share/unix_files/cdat/miniconda2/envs/cdatm15 | + | $ wp |
| - | 573M /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 | + | /home/share/unix_files/cdat/miniconda3/bin/python |
| - | 2.0G /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.8.0 | + | |
| </code> | </code> | ||
| - | Disk space used after cloning, in <wrap em>in the new directory hierarchy</wrap> (''cdat/miniconda2b'', on the obelix servers) | + | ==== Installing CDAT 8.0 ==== |
| - | <code>bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b | + | <code>$ conda create -n cdat-8.0_py2 -c cdat/label/v80 -c conda-forge -c cdat python=2.7 cdat |
| - | 4.8G /home/share/unix_files/cdat/miniconda2b | + | # Generate the list of installed packages |
| + | $ conda list -n cdat-8.0_py2 > /home/scratch01/jypeter/cdat-8.0_py2_list_181108.txt</code> | ||
| - | bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/* | + | List of installed packages: {{:other:uvcdat:cdat_conda:cdat-8.0_py2_list_181108.txt}} |
| - | 4.4M /home/share/unix_files/cdat/miniconda2b/bin | + | |
| - | 4.0K /home/share/unix_files/cdat/miniconda2b/conda-bld | + | |
| - | 280K /home/share/unix_files/cdat/miniconda2b/conda-meta | + | |
| - | 3.4G /home/share/unix_files/cdat/miniconda2b/envs | + | |
| - | 20K /home/share/unix_files/cdat/miniconda2b/etc | + | |
| - | 3.8M /home/share/unix_files/cdat/miniconda2b/include | + | |
| - | 100M /home/share/unix_files/cdat/miniconda2b/lib | + | |
| - | 8.0K /home/share/unix_files/cdat/miniconda2b/LICENSE.txt | + | |
| - | 1.4G /home/share/unix_files/cdat/miniconda2b/pkgs | + | |
| - | 16K /home/share/unix_files/cdat/miniconda2b/share | + | |
| - | 376K /home/share/unix_files/cdat/miniconda2b/ssl | + | |
| - | bash-4.2$ du -sh /home/share/unix_files/cdat/miniconda2b/envs/* | + | Disk space after 8.0 installation |
| - | 2.8G /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | + | |
| - | 574M /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | + | |
| - | </code> | + | |
| - | ===== Getting ready for a moving default CDAT ===== | + | <code>$ du -sh /home/share/unix_files/cdat/miniconda3 |
| + | 4.3G /home/share/unix_files/cdat/miniconda3 | ||
| + | $ du -sh /home/share/unix_files/cdat/miniconda3/* | ||
| + | 15M /home/share/unix_files/cdat/miniconda3/bin | ||
| + | 2.4M /home/share/unix_files/cdat/miniconda3/compiler_compat | ||
| + | 4.0K /home/share/unix_files/cdat/miniconda3/conda-bld | ||
| + | 3.7M /home/share/unix_files/cdat/miniconda3/conda-meta | ||
| + | 2.8G /home/share/unix_files/cdat/miniconda3/envs | ||
| + | 28K /home/share/unix_files/cdat/miniconda3/etc | ||
| + | 4.9M /home/share/unix_files/cdat/miniconda3/include | ||
| + | 156M /home/share/unix_files/cdat/miniconda3/lib | ||
| + | 8.0K /home/share/unix_files/cdat/miniconda3/LICENSE.txt | ||
| + | 1.4G /home/share/unix_files/cdat/miniconda3/pkgs | ||
| + | 1.3M /home/share/unix_files/cdat/miniconda3/share | ||
| + | 24K /home/share/unix_files/cdat/miniconda3/ssl | ||
| + | 12K /home/share/unix_files/cdat/miniconda3/x86_64-conda_cos6-linux-gnu | ||
| + | $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* | ||
| + | 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2</code> | ||
| - | We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version | ||
| - | <code>bash-4.2$ cd /home/share/unix_files/cdat/miniconda2b/envs | + | ===== Cloning cdat to add specific packages for LSCE ===== |
| - | bash-4.2$ conda env list | + | <WRAP center round tip 60%> |
| - | cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | + | This section is about the creation of the **cdatm17** environment |
| - | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | + | |
| - | root * /home/share/unix_files/cdat/miniconda2b | + | |
| - | bash-4.2$ ln -s cdatm15 cdatm | + | Notes about [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat|actually using the cdatm17 conda-based python]] |
| + | </WRAP> | ||
| - | bash-4.2$ conda env list | ||
| - | cdatm /home/share/unix_files/cdat/miniconda2b/envs/cdatm | ||
| - | cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | ||
| - | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | ||
| - | root * /home/share/unix_files/cdat/miniconda2b | ||
| - | bash-4.2$ source activate cdatm | + | Note: using hard links, cloning a full environment //only// adds an extra 582M of disk space |
| + | |||
| + | <code>$ du -sh /home/share/unix_files/cdat/miniconda3/envs/* | ||
| + | 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | ||
| + | $ conda create -n cdatm17_py2 --clone cdat-8.0_py2 | ||
| + | Source: /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | ||
| + | Destination: /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | ||
| + | Packages: 226 | ||
| + | Files: 3 | ||
| + | |||
| + | [...] | ||
| + | |||
| + | $ du -sh /home/share/unix_files/cdat/miniconda3/envs/* | ||
| + | 2.8G /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | ||
| + | 582M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | ||
| - | (cdatm) bash-4.2$ conda env list | ||
| - | cdatm * /home/share/unix_files/cdat/miniconda2b/envs/cdatm | ||
| - | cdatm15 /home/share/unix_files/cdat/miniconda2b/envs/cdatm15 | ||
| - | uvcdat-2.10 /home/share/unix_files/cdat/miniconda2b/envs/uvcdat-2.10 | ||
| - | root /home/share/unix_files/cdat/miniconda2b | ||
| </code> | </code> | ||
| + | |||
| + | ===== Getting ready for a moving default CDAT ===== | ||
| + | |||
| + | We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version | ||
| + | |||
| + | <code>$ cd /home/share/unix_files/cdat/miniconda3/envs | ||
| + | $ conda env list | ||
| + | base * /home/share/unix_files/cdat/miniconda3 | ||
| + | cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | ||
| + | cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | ||
| + | $ ln -s cdatm17_py2 cdatm | ||
| + | $ conda env list | ||
| + | base * /home/share/unix_files/cdat/miniconda3 | ||
| + | cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | ||
| + | cdatm /home/share/unix_files/cdat/miniconda3/envs/cdatm | ||
| + | cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2 | ||
| + | $ source activate cdatm | ||
| + | (cdatm) bash-4.2$ conda env list | ||
| + | base /home/share/unix_files/cdat/miniconda3 | ||
| + | cdat-8.0_py2 /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2 | ||
| + | cdatm * /home/share/unix_files/cdat/miniconda3/envs/cdatm | ||
| + | cdatm17_py2 /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2</code> | ||
| ===== Customizing UV-CDAT for LSCE ===== | ===== Customizing UV-CDAT for LSCE ===== | ||
| Line 226: | Line 288: | ||
| ==== Downloading cdms2/vcs test data ==== | ==== Downloading cdms2/vcs test data ==== | ||
| - | You should download the test data and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...) | + | You should download the test data (174M of data...) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...) |
| - | <code>bash-4.1$ source activate cdatm15 | + | <code>$ source activate cdatm17_py2 |
| - | (cdatm15) bash-4.1$ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data' | + | (cdatm17_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data' |
| [...] | [...] | ||
| - | Finished downloading sample data to /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data | + | Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data |
| - | (cdatm15) bash-4.1$ du -sh /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data | + | (cdatm17_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data |
| - | 174M /home/share/unix_files/cdat/miniconda2/envs/cdatm15/share/uvcdat/sample_data</code> | + | 174M /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data |
| + | </code> | ||
other/uvcdat/cdat_conda/cdat_8_0_py2.txt · Last modified: 2021/02/27 13:55 by jypeter
