Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:cdat_8_1 [2019/03/07 10:39]
jypeter created
+++ other:uvcdat:cdat_conda:cdat_8_1 [2021/03/26 09:10] (current)
jypeter [You want to replicate this installation on your own server or desktop/laptop]
@@ Line 4: / Line 4: @@
 <WRAP center round tip 60%>
-Follow the instructions about the [[other:python:starting#conda-based_versions_of_uv-cdat|conda-based versions of UV-CDAT initialization]] for actually using an installed version of ''8.1''
+If you have questions about Python, read:
+  * [[other:python:starting|Working with Python]]
+  * [[other:python:jyp_steps|JYP's recommended steps for learning python]]!
 </WRAP>
-===== What's New? =====
+===== Using JYP version =====
-  * [[https://github.com/cdat/cdat/releases/tag/v8.1|8.1 announcement]] and features summary ([[https://github.com/UV-CDAT/uvcdat/releases|all versions]])
+If you mostly want to use the CDAT 8.1 installed by JYP, just use the following steps and then read the //What should I do now?// section below
-  * Full [[https://uvcdat.llnl.gov/changelog.html#8.1|Change log]]
-===== Installation with Miniconda3 =====
+  * Initialize //conda// with:
-==== Installing Miniconda3 ====
+^  Server  ^  tcsh  ^  bash  ^
+|  LSCE  | ''source ~jypeter/.conda3_jyp.csh'' | ''source ~jypeter/.conda3_jyp.sh'' |
+|  ciclad  | ''source ~jypmce/.conda3_jyp.csh'' | ''source ~jypmce/.conda3_jyp.sh'' |
-We have installed a new version of Miniconda3 (steps below adapted from the [[other:uvcdat:conda_notes#installing_miniconda|installing miniconda]] instructions)
+  * Choose one of the [[#environments_summary|installed environments]] and activate it with: ''conda activate env_name''
+    * python 2.7.x: ''conda activate cdatm_py2''
+    * python 3.x: ''conda activate cdatm_py3'' (not available on ciclad, but could be installed)
-  * Get the [[https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh|Python 3.7 Linux 64-bit (bash installer)]] (unless there is a [[https://conda.io/miniconda.html|more recent version?]])
+  * Type ''which python'' (or the ''wp'' alias) and make sure you get something like\\ ''[...]/miniconda3/envs/env_name/bin/python''
-    * Find some temporary space on Linux or Windows
-      * Linux:''cd /home/scratch01/jypeter''
-      * WSL: ''cd /mnt/h/Scratch/jypeter/'' (assuming that there is a ''H:\Scratch\jypeter'' directory)
-    * ''wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh'' (67 Mo, 15 Feb 2019)
-  * Execute the installer
-    * ''bash Miniconda3-latest-Linux-x86_64.sh''
-      * Accept the license
-      * Specify an explicit installation path **outside of the home directory**, with enough disk space (more than 3G if you are going to install UV-CDAT and some extra packages), preferably on a disk that is not backed up:
-        * Linux LSCE: ''/home/share/unix_files/cdat/miniconda3''
-        * Linux ciclad: ''/data/jypmce/cdat/miniconda3''
-        * WSL: installing to a directory that is not in ''/home/'' <wrap hi>does not work</wrap> (e.g. ''/mnt/h/CDAT/miniconda3'',assuming there is a ''H:\CDAT\'' directory, does not work)\\ Need to accept the installation in the default location: ''/home/jypeter/miniconda3''
-        * Resulting ''miniconda3'' directory size is 302M\\ <code> >du -sh miniconda3
-M    miniconda3
- >cd miniconda3
-/home/share/unix_files/cdat/miniconda3
- >du -sh *
-.3M    bin
-.0M    compiler_compat
-K    conda-meta
-.0K    envs
-K     etc
-.2M    include
-M    lib
-.0K    LICENSE.txt
-M    pkgs
-K    share
-K     ssl
-K     x86_64-conda_cos6-linux-gnu</code>
-      * Accept the automatic update of the //PATH// variable in ''.bashrc'' (note: the installer automatically creates a backup of ''.bashrc''). This will some extra lines at the end of ''.bashrc'' used by the //old-style conda initialization// (we will later switch to the //new-style//). The added lines should look like:
-        * ''# added by Miniconda3 installer''\\ ''export PATH="/home/share/unix_files/cdat/miniconda3/bin:$PATH"''
-        * ''# added by Miniconda4.xxx installer''\\ ''[some really fancy bash shell commands]''
-  * Remove the installer: ''rm Miniconda3-latest-Linux-x86_64.sh''
-  * Open a new terminal (if you use bash) or open a new terminal and type ''bash''
-    * check if the python from the conda distribution has been correctly installed\\ <code> > bash
-$ which python
-/home/share/unix_files/cdat/miniconda3/bin/python
-$ which conda
-/home/share/unix_files/cdat/miniconda3/bin/conda</code>
-    * Update and clean the new installation\\ <code>$ conda update --all
-$ du -sh /home/share/unix_files/cdat/miniconda3
-M    /home/share/unix_files/cdat/miniconda3
-$ conda clean --all
-$ du -sh /home/share/unix_files/cdat/miniconda3
-M    /home/share/unix_files/cdat/miniconda3
-$ conda list
-# packages in environment at /home/share/unix_files/cdat/miniconda3:
-#
-# Name                    Version                   Build  Channel
-[...]
-python                    3.7.1                h0371630_3
-readline                  7.0                  h7b6447c_5
-requests                  2.20.0                   py37_0
-[...]</code>
-==== Post-Miniconda3 installation ====
-The idea is to remove the miniconda3 initialization lines that were automatically added at the end of ''.bashrc'' and put them (and other useful commands) in a special initialization file, that can be sourced only when we actually want to use conda and CDAT (in order to avoid potentiel side effects)
+===== What should I do now? =====
-=== New style initialization ===
+==== You just want to use the installed version ====
-Note: this is the **new** ''conda activate some_version'' style
+You are ready if you have typed the ''conda activate'' command specified in the previous section!
+  * You may want to add the ''source'' line to your shell configuration file so that you don't have to remember it and type it each time you open a new terminal
+    * Note: it's better to type the ''activate'' command in a terminal only when you want to use this specific version of python, rather than having the ''activate'' in your shell configuration file. This may have some side effects in some cases
+  * Have a look at the [[other:uvcdat:cdat_conda:cdat_8_1#extra_packages_list|Extra package list]] at the bottom of this page to see what is available in this distribution
+  * If you have not done it yet, read (or at least have a look at the list of sections):
+    * [[other:python:starting|Working with Python]]
+    * [[other:python:jyp_steps|JYP's recommended steps for learning python]]
-See the ''~jypeter/.conda3_jyp.sh'' file below, and how to use it, **in a bash shell**. In a **tcsh shell**, see the ''~jypeter/.conda3_jyp.csh'' further down. In both shell cases, if you are installing your own version of python, you need to use your own location of the initialization files in the ''source'' lines, and you can use another file name than ''conda3_jyp''
+==== You want to replicate this installation on your own server or desktop/laptop ====
-<code>$ which python
+Read carefully the following sections of this page! 8-)
-/usr/bin/python
+===== What's New in CDAT 8.1? =====
-$ cat ~jypeter/.conda3_jyp.sh
+  * [[https://github.com/cdat/cdat/releases/tag/v8.1|8.1 announcement]] and features summary ([[https://github.com/UV-CDAT/uvcdat/releases|all versions]])
-# Conda initialization by JYP, NEW style
+  * Full [[https://uvcdat.llnl.gov/changelog.html#8.1|Change log]]
-#
-# Use this for working with conda and CDAT centrally managed by JYP
-#
-# Execute this file in a BASH shell with
-#     source path/this_file
-# Then get the list of available python distributions with
-#     conda env list
-# Then activate a specific distribution with
-#     conda activate version_name
-#
-# More details in:
-#   https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat
-#   https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes
-#
-# Jean-Yves Peterschmitt - LSCE - 11/2018
-source /home/share/unix_files/cdat/miniconda3/etc/profile.d/conda.sh
+Note: this particular CDAT installation at LSCE (and on other machines/servers) provides **many extra (non-CDAT) packages**. You can jump directly to the [[other:uvcdat:cdat_conda:cdat_8_1#extra_packages_list|Extra package list]] at the bottom of this page to see what is available
-# Use the alias below to easily determine where your python
+===== Installation with Miniconda3 =====
-# interpreter is located
-alias wp="which python"
-# Where are ALL the python interpreters in the search path
+We assume that Miniconda3 is already installed. Otherwise, follow the the [[other:uvcdat:cdat_conda:cdat_8_0_py2#installing_miniconda3|Installing Miniconda]] steps (and the //Post-Miniconda3 installation// section) we followed when installing CDAT 8.0
-alias wpa="which -a python"
-# The end
+==== Updating conda ====
-$ source ~jypeter/.conda3_jyp.sh
+The conda package itself can be updated (if need be) with\\ ''conda update -n base -c defaults conda''
+==== Installing CDAT 8.1 ====
-$ conda activate
+We first check that we have indeed access to a conda installation, and we assume that we have a write-access to the conda hierarchy, and a few Gb of disk space available
-(base) $ which python
+<code> >conda activate
-/home/share/unix_files/cdat/miniconda3/bin/python
+(base) >conda env list | grep base
+base                  *  /home/share/unix_files/cdat/miniconda3
-</code>
+(base) >df -h  /home/share/
+Filesystem      Size  Used Avail Use% Mounted on
+prolix3:/share  917G  171G  700G  20% /home/share
-**tcsh shell** usage example
+(base) >du -sh /home/share/unix_files/cdat/miniconda3
+.8G    /home/share/unix_files/cdat/miniconda3</code>
-<code> >which python
+We can also use ''conda env list'' and remove some obsolete versions with ''conda remove -n obsolete_name --all'' to get some space
-/usr/bin/python
- >cat ~jypeter/.conda3_jyp.csh
-# Conda initialization by JYP, NEW style
-#
-# Use this for working with conda and CDAT centrally managed by JYP
-#
-# Execute this file in a TCSH shell with
-#     source path/this_file
-# Then get the list of available python distributions with
-#     conda env list
-# Then activate a specific distribution with
-#     conda activate version_name
-#
-# More details in:
-#   https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat
-#   https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes
-#
-# Jean-Yves Peterschmitt - LSCE - 11/2018
-source /home/share/unix_files/cdat/miniconda3/etc/profile.d/conda.csh
-# Use the alias below to easily determine where your python
-# interpreter is located
-alias wp "which python"
-# The end
- >source  ~jypeter/.conda3_jyp.csh
- >conda activate
-(base) >which python
-/home/share/unix_files/cdat/miniconda3/bin/python</code>
-You probably don't want to type the //source// line each time you need to use your conda based python, so you can add a ''source  ~jypeter/.conda3_jyp.sh'' line in your ''~/.bashrc'' file, and ''source  ~jypeter/.conda3_jyp.csh'' line in your ''~/.cshrc'' file. Then, when you need a specific python environment, just type ''conda activate name_of_the_specific_environment''
-=== Old style initialization ===
-<WRAP center round tip 60%>
-The //old style// notes below are just for reference sake and you can skip them
-</WRAP>
-Note: this is the **old** ''source activate some_version'' style
-See the ''~jypeter/.conda3old_jyp.sh'' file below, and how to use it, **in a bash shell**
-<code>bash-4.2$ which python
-/usr/bin/python
-$ cat .conda3old_jyp.sh
-# Conda initialization by JYP, OLD style
-#
-# Use this for working with conda and CDAT centrally managed by JYP
-#
-# Execute this file in a BASH shell with
-#     source path/this_file
-# Then get the list of available python distributions with
-#     conda env list
-# Then activate a specific distribution with
-#     source activate version_name
-#
-# More details in:
-#   https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat
-#   https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes
-#
-# Jean-Yves Peterschmitt - LSCE - 11/2018
-export PATH="/home/share/unix_files/cdat/miniconda3/bin:$PATH"
-# Use the alias below to easily determine where your python
-# interpreter is located
-alias wp="which python"
-# Where are ALL the python interpreters in the search path
-alias wpa="which -a python"
-# The end
-$ source ~jypeter/.conda3old_jyp.sh
-$ wp
-/home/share/unix_files/cdat/miniconda3/bin/python
-</code>
-==== Installing CDAT 8.0 ====
 === Python 2.7 version ===
-<code>$ conda create -n cdat-8.0_py2 -c cdat/label/v80 -c conda-forge -c cdat python=2.7 cdat
+<code>$ conda create -n cdat-8.1_py2 -c cdat/label/v81 -c conda-forge python=2.7 cdat
 # Generate the list of installed packages
-$ conda list -n cdat-8.0_py2 > /home/scratch01/jypeter/cdat-8.0_py2_list_181108.txt</code>
+$ conda list -n cdat-8.1_py2 > /home/scratch01/jypeter/cdat-8.1_py2_list_190307.txt</code>
-List of installed packages: {{:other:uvcdat:cdat_conda:cdat-8.0_py2_list_181108.txt}}
+List of installed packages: {{:other:uvcdat:cdat_conda:cdat-8.1_py2_list_190307.txt}}
-Disk space after 8.0 installation
-<code>$ du -sh /home/share/unix_files/cdat/miniconda3
-.3G    /home/share/unix_files/cdat/miniconda3
-$ du -sh /home/share/unix_files/cdat/miniconda3/*
-M     /home/share/unix_files/cdat/miniconda3/bin
-.4M    /home/share/unix_files/cdat/miniconda3/compiler_compat
-.0K    /home/share/unix_files/cdat/miniconda3/conda-bld
-.7M    /home/share/unix_files/cdat/miniconda3/conda-meta
-.8G    /home/share/unix_files/cdat/miniconda3/envs
-K     /home/share/unix_files/cdat/miniconda3/etc
-.9M    /home/share/unix_files/cdat/miniconda3/include
-M    /home/share/unix_files/cdat/miniconda3/lib
-.0K    /home/share/unix_files/cdat/miniconda3/LICENSE.txt
-.4G    /home/share/unix_files/cdat/miniconda3/pkgs
-.3M    /home/share/unix_files/cdat/miniconda3/share
-K     /home/share/unix_files/cdat/miniconda3/ssl
-K     /home/share/unix_files/cdat/miniconda3/x86_64-conda_cos6-linux-gnu
-$ du -sh /home/share/unix_files/cdat/miniconda3/envs/*
-.8G    /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2</code>
 === Python 3.6 version ===
-<code>$ conda create -n cdat-8.0_py3 -c cdat/label/v80 -c conda-forge -c cdat python=3.6 cdat</code>
+<code>$ conda create -n cdat-8.1_py3 -c cdat/label/v81 -c conda-forge python=3.6 cdat</code>
 ==== Installing CDAT nightly ====
@@ Line 251: / Line 87: @@
 Notes:
-  * This page is about CDAT 8.0, but we have added a short nightly section here as a convenient shortcut. This should probably be moved to a stand-alone nightly page later
+  * This page is about CDAT 8.1, but we have added a short nightly section here as a convenient shortcut. This should probably be moved to a stand-alone nightly page later
+  * The nightly version is **not automatically updated**, and may well be out-of-date
   * [[https://github.com/CDAT/cdat/wiki/Obtain-CDAT-nightly-packages|Official nightly page]]
-=== Python 2.7 version ===
+=== Python 2.7 nightly version ===
-<code>$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat python=2.7 cdat</code>
+<code>$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge python=2.7 cdat</code>
-<wrap hi>Not tested!</wrap> Can this be updated with ''conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge -c cdat --all'' ???
+<wrap hi>Not tested!</wrap> Can this be updated with ''conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge --all'' ???
-=== Python 3.6 version ===
+=== Python 3.6 nightly version ===
+<note important>
+Not installed yet!</note>
-<code>$ conda create -n cdat-nightly_py3 -c cdat/label/nightly -c conda-forge -c cdat python=3.6 cdat</code>
+<code>$ conda create -n cdat-nightly_py3 -c cdat/label/nightly -c conda-forge python=3.6 cdat</code>
 ===== Cloning cdat to add specific packages for LSCE =====
 <WRAP center round tip 60%>
-This section is about the creation of the **cdatm17** environment
+This section is about the creation of the **cdatm18** environment
-Notes about [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat|actually using the cdatm17 conda-based python]]
+Notes about [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#conda-based_versions_of_uv-cdat|actually using the cdatm18 conda-based python]]
 </WRAP>
-Note: using hard links, cloning a full environment //only// adds an extra 582M of disk space
+Note: using hard links, //cloning// a full environment uses less disk space than making a real copy
-<code>$ conda create -n cdatm17_py2 --clone cdat-8.0_py2
-Source:      /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2
-Destination: /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2
-Packages: 226
-Files: 3
-[...]
-$ du -sh /home/share/unix_files/cdat/miniconda3/envs/*
-.5G    /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2
-M    /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2
-M    /home/share/unix_files/cdat/miniconda3/envs/cdat-nightly_py2
+<code>$ conda create -n cdatm18_py2 --clone cdat-8.1_py2
+$ du -sh cdat-8.1_py2 cdatm18_py2
+.9G    cdat-8.1_py2
+M    cdatm18_py2
 </code>
@@ Line 301: / Line 132: @@
 <code>$ cd /home/share/unix_files/cdat/miniconda3/envs
-$ conda env list
+$ ln -s cdatm18_py2 cdatm_py2</code>
-base                  *  /home/share/unix_files/cdat/miniconda3
-cdat-8.0_py2             /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2
-cdatm17_py2              /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2
-$ ln -s cdatm17_py2 cdatm
-$ conda env list
-base                  *  /home/share/unix_files/cdat/miniconda3
-cdat-8.0_py2             /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2
-cdatm                    /home/share/unix_files/cdat/miniconda3/envs/cdatm
-cdatm17_py2              /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2
-$ source activate cdatm
-(cdatm) bash-4.2$ conda env list
-base                     /home/share/unix_files/cdat/miniconda3
-cdat-8.0_py2             /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2
-cdatm                 *  /home/share/unix_files/cdat/miniconda3/envs/cdatm
-cdatm17_py2              /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2</code>
 ===== Customizing UV-CDAT for LSCE =====
+==== Testing vcs ====
+Start python and paste the following lines
+<code>import numpy as np, vcs
+id100 = np.identity(100)
+x = vcs.init()
+x.plot(id100)
+</code>
+If you get the kind of errors described in [[https://github.com/CDAT/vcs/issues/393|x.plot crashes with OpenGL2 error]], the drivers on your server are probably not compatible with an interactive use of vcs... The workaround is to **install mesalib and work in //headless mode//** (e.g. without a display)...
+Install //mesalib// with ''conda install -c cdat/label/v81 -c conda-forge mesalib vtk-cdat'' and try pasting again the few code lines above in python. If things work as expected, no canvas will be opened, but you'll be able to save the canvas to a file with
+<code>x.png('test_mesalib')
+x.pdf('test_mesalib')</code>
+Note: using //mesalib// for headless mode work makes it possible to work without an X server, and with no DISPLAY variable defined
 ==== Downloading cdms2/vcs test data ====
 You should download the test data (174M of data...) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...)
-<code>$ source activate cdatm17_py2
+<code>$ conda activate cdatm18_py2
-(cdatm17_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print "\nFinished downloading sample data to", vcs.sample_data'
+(cdatm18_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)'
 [...]
-Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data
+Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
-(cdatm17_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data
+(cdatm18_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
-M    /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2/share/uvcdat/sample_data
+M    /home/share/unix_files/cdat/miniconda3/envs/cdatm18_py2/share/cdat/sample_data
 </code>
@@ Line 343: / Line 176: @@
   * We then install individual extra packages with ''conda install'' or ''pip install''
-<code># Keep a trace of what will be installed
+<code># You can use the following to keep a trace of what will be installed
-$ conda install --dry-run -n cdatm17_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm  > /home/scratch01/jypeter/lsce-extra_01_install_190304.txt
+#$ conda install --dry-run -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm  > somewhere/extra_packages.txt
 # Install...
-$ conda install -n cdatm17_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
+$ conda install -n cdatm18_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret ferret_datasets basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
 [...]
-# Install extra packages in the nightly version (if you need 'nightly')
+# Use a similar install line if you want to install the same packages in the python3 version</code>
-$ conda install -n cdatm-nightly_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
-# Check the disk space after installation
-$ du -sh /home/share/unix_files/cdat/miniconda3
-.5G    /home/share/unix_files/cdat/miniconda3
-$ du -sh /home/share/unix_files/cdat/miniconda3/envs/*
-.5G    /home/share/unix_files/cdat/miniconda3/envs/cdat-8.0_py2
-.2G    /home/share/unix_files/cdat/miniconda3/envs/cdatm17_py2
-M    /home/share/unix_files/cdat/miniconda3/envs/cdatm-nightly_py2
-M    /home/share/unix_files/cdat/miniconda3/envs/cdat-nightly_py2</code>
 List of installed packages: {{ :other:uvcdat:cdat_conda:lsce-extra_01_install_190304.txt |}}
@@ Line 367: / Line 189: @@
 Packages installed with //pip//
   * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API
-    * ''pip install dreqPy''
+    * ''conda activate cdatm18_py2''\\ ''pip install dreqPy''
     * Update with: ''pip install %%--%%upgrade dreqPy''
       * Get version number with:\\ <code>$ drq -v
 dreqPy version 01.00.29 [Version 01.00.29]</code>
+  * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret
+    * ''conda activate cdatm18_py2''\\ ''pip install ferretmagic''
 The following packages have no dependency problems and were installed (or updated) //later//
+  * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files
+    * ''conda install -n cdatm18_py2 -c conda-forge cmor''
+    * Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' ''
+    * <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]]
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
-    * ''conda install -n cdatm17_py2 -c conda-forge -c vacumm vacumm''
+    * ''conda install -n cdatm18_py2 -c conda-forge -c vacumm vacumm''
+    * <wrap hi>Warning!</wrap> [[https://github.com/VACUMM/vacumm/issues/15|standard install of vacumm does not work for Python3]]
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
-    * ''conda install -n cdatm17_py2 -c stefraynaud -c conda-forge spanlib''
+    * ''conda install -n cdatm18_py2 -c stefraynaud -c conda-forge spanlib''
     * Test: ''python -c 'from spanlib.analyzer import Analyzer' ''
+  * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses
+    * ''conda install -n cdatm18_py2 -c conda-forge cartopy''
+  * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs
+    * ''conda install -n cdatm18_py2 -c conda-forge joblib''
+  * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework
+    * [[other:uvcdat:cdat_conda:climaf_install|Installation notes]]
+  * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python
+    * ''conda install -n cdatm18_py2 -c conda-forge palettable''
 ==== TODO ====
@@ Line 404: / Line 247: @@
 Some packages change more often than others, and can be easily updated the following way:
-  * [[https://cmor.llnl.gov/|CMOR]]
-    * Update with: ''conda update -n cdatm15 -c conda-forge -c pcmdi -c uvcdat cmor''
-      * Get version number with: ''python -c 'from cmor import *; print CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH' ''
   * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]
@@ Line 413: / Line 252: @@
       * Get version number with: ''drq -v''
-==== Installing CLIMAF ====
-The [[https://climaf.readthedocs.io/|CLIMAF]] installation is a special case, because we may want to use it with the Python installed for CDAT, but it is not installed using conda
-We follow the steps for the [[https://climaf.readthedocs.io/en/latest/installing.html#installing-climaf-if-you-are-not-at-cnrm-on-ciclad-or-climserv|installation of a non-standard server]]
-  * Go to the appropriate installation directory (we choose the directory where we have also installed miniconda3):
-    * LSCE: ''cd /home/share/unix_files/cdat''
-    * ciclad: ''/data/jypmce/cdat/''
-  * ''git clone https://github.com/senesis/climaf climaf''
-  * Add CLIMAF to the PYTHONPATH environment variable
-    * LSCE & tcsh: ''setenv PYTHONPATH ${PYTHONPATH}:/home/share/unix_files/cdat/climaf''
-    * LSCE & bash: ''export PYTHONPATH=${PYTHONPATH}:/home/share/unix_files/cdat/climaf''
-    * ciclad: same as above, using ''/data/jypmce/cdat/climaf''
-  * Test the installation
-    * make sure the [[https://climaf.readthedocs.io/en/latest/requirements.html#requirements|Requirements]] are available:
-      * LSCE: ''module load netcdf/4 nco cdo ncl ncview''
-      * ciclad: ''module load netcdf4 nco cdo ncl/6.3.0''
-      * optional: start an X server
-      * <code>cd climaf/testing
-./test_install.sh</code>
-  * Once the tests are OK, add the required //module load// and environment variables definitions to the ''.basrc'' and ''.cshrc'' files. <wrap hi>Note: maybe these lines should go to the ''.profile'' and ''.login'' files...(check [[https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html|Bash Startup Files]])</wrap>
-    * LSCE: add to your .cshrc
-      * <code># Prerequisites for using CLIMAF
-module load netcdf/4 nco cdo ncl ncview
-# CLIMAF environment variables
-setenv CLIMAF /home/share/unix_files/cdat/climaf
-setenv PYTHONPATH "${PYTHONPATH}:${CLIMAF}"
-#setenv PYTHONPATH $CLIMAF
-setenv PATH "${PATH}:${CLIMAF}/bin"
-setenv CLIMAF_CACHE "/home/scratch01/${USER}/climaf_cache"
-setenv TMPDIR $CLIMAF_CACHE
-setenv CLIMAF_LOG_DIR $CLIMAF_CACHE
-#conda activate cdatm-nightly_py2</code>
-    * ciclad: add to your .bashrc
-      * <code># Prerequisites for using CLIMAF when not using the
-# default 'module load climaf'
-module load netcdf4 nco cdo ncl/6.3.0
-# CLIMAF environment variables
-export CLIMAF=/data/jypmce/cdat/climaf
-#export PYTHONPATH=$PYTHONPATH:$CLIMAF
-export PYTHONPATH=$CLIMAF
-export PATH=$PATH:$CLIMAF/bin
-export CLIMAF_CACHE=/data/jypmce/climaf_cache
-export TMPDIR=$CLIMAF_CACHE
-export CLIMAF_LOG_DIR=$CLIMAF_CACHE
-#conda activate cdatm-nightly_py2
-</code>
-  * Open a new shell and <wrap em>run the tests again</wrap>. If there is an error that was not there, check what you have added to the shell init files
-  * [[other:uvcdat:cdat_conda:climaf_test|Test CLIMAF on ciclad]]
 ==== Cleaning up things ====
@@ Line 488: / Line 273: @@
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
   * [[http://unidata.github.io/netcdf4-python/|netcdf4]]: a Python interface to the netCDF C library
-  * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in
+  * [[http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret|pyferret]]: Ferret encapsulated in Python
+  * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret
   * [[https://github.com/conda-forge/basemap-data-hires-feedstock|basemap-data-hires]]: high resolution data for ''basemap''
   * [[https://github.com/PCMDI/pcmdi_metrics|PCMDI metrics package]] (PMP): objectively compare results from climate models with observations using well-established statistical tests
@@ Line 497: / Line 283: @@
   * [[https://github.com/pierre-rouanet/dtw|dtw]]: DTW (Dynamic Time Warping) python module
   * [[http://toblerity.org/shapely/project.html|shapely]]: a Python wrapper for GEOS for algebraic manipulation of geometry (manipulation and analysis of geometric objects in the Cartesian plane)
-  * [[http://scitools.org.uk/cartopy/|cartopy]]: a library providing cartographic tools for python
+  * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses
   * [[https://rpy2.readthedocs.io/en/version_2.8.x/|rpy2]]: providing simple and robust access to R from within Python
   * [[http://matplotlib.org/cmocean/|cmocean]]: beautiful colormaps for oceanography
@@ Line 504: / Line 290: @@
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
   * [[https://ajdawson.github.io/windspharm/latest/|windspharm]]: spherical harmonic wind analysis in Python
+  * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework
+  * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs
+  * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python
 ==== Removed packages ====
@@ Line 511: / Line 300: @@
 ===== Environments summary =====
-After following the steps above, we get the following environments. Use the ''conda info %%--%%envs'' or the ''conda env list'' command to get the up-to-date list of available environments
+After following the steps above, we get the following environments. Use the ''conda env list'' command (same result as ''conda info %%--%%envs'') to get the up-to-date list of available environments.
+Use ''conda list -n env_name'' to get a detailed list of which packages/modules are installed, or check the [[other:uvcdat:conda_notes#installation_history|conda Installation history]] section to get more details
-^  Environment\\ name  ^  Server  ^  ''conda list''  ^
+^  Environment\\ name  ^  Server  ^   Summary  ^
-| cdat-8.0_py2 |  LSCE\\ ciclad  | |
+| cdat-8.1_py2 |  LSCE\\ ciclad  | CDAT 8.1 & Python 2.7 |
-| cdatm17_py2 |  LSCE\\ ciclad  | |
+| cdat-8.1_py3 |  LSCE  | CDAT 8.1 & Python 3.6 |
-| cdat-nightly_py2 |  LSCE\\ ciclad  | |
+| cdatm18_py2\\ or //cdatm_py2// |  LSCE\\ ciclad  | CDAT 8.1 & P 2.7 //JYP version// |
-| cdatm-nightly_py2 |  LSCE\\ ciclad  | |
+| cdatm18_py3\\ or //cdatm_py3// |  LSCE  | CDAT 8.1 & P 3.7 //JYP version// |
+| cdat-nightly_py2 |  LSCE\\ ciclad  | CDAT nightly & Python 2.7 |
+| cdatm-nightly_py2 |  LSCE\\ ciclad  | CDAT nightly & P 2.7 //JYP version// |
 /* standard page footer */

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools