Differences

This shows you the differences between two versions of the page.

--- other:uvcdat:cdat_conda:cdat_8_2_1 [2021/02/27 12:34]
jypeter Replaces all 8.1 with 8.2.1, and 18 with 19
+++ other:uvcdat:cdat_conda:cdat_8_2_1 [2021/03/03 15:39]
jypeter [Packages that have no dependency problems]
@@ Line 16: / Line 16: @@
 ^  Server  ^  tcsh  ^  bash  ^
-|  LSCE  | ''source ~jypeter/.conda3_jyp.csh'' | ''source ~jypeter/.conda3_jyp.sh'' |
+|  LSCE  | ''source ~jypeter/.conda3_21-02_jyp.csh'' | ''source ~jypeter/.conda3_21-02_jyp.sh'' |
-|  ciclad  | ''source ~jypmce/.conda3_jyp.csh'' | ''source ~jypmce/.conda3_jyp.sh'' |
+|  ciclad  | <del>''source ~jypmce/.conda3_jyp.csh''</del>\\ n/a | <del>''source ~jypmce/.conda3_jyp.sh''</del>\\ n/a |
   * Choose one of the [[#environments_summary|installed environments]] and activate it with: ''conda activate env_name''
-    * python 2.7.x: ''conda activate cdatm_py2''
+    * python 2.7.x: Deprecated! If you still need to use python 2.7, please use [[other:uvcdat:cdat_conda:cdat_8_1|CDAT 8.1]]
-    * python 3.x: ''conda activate cdatm_py3'' (not available on ciclad, but could be installed)
+    * python 3.x: ''conda activate cdatm_py3''
   * Type ''which python'' (or the ''wp'' alias) and make sure you get something like\\ ''[...]/miniconda3/envs/env_name/bin/python''
@@ Line 57: / Line 57: @@
 ==== Installing CDAT 8.2.1 ====
-We first check that we have indeed access to a conda installation, and we assume that we have a write-access to the conda hierarchy, and a few Gb of disk space available
+=== Prerequisites ===
-<code> >conda activate
-(base) >conda env list | grep base
-base                  *  /home/share/unix_files/cdat/miniconda3
-(base) >df -h  /home/share/
+We first check that we have indeed access to a //(mini)conda// installation (the //base// environment we get if we have correctly installed miniconda3), and we assume that we have a write-access to the conda disk hierarchy, and a few Gb of disk space available
+<code> > conda env list | grep base
+base                  *  /home/share/unix_files/cdat/miniconda3_21-02
+ > df -h  /home/share/
 Filesystem      Size  Used Avail Use% Mounted on
-prolix3:/share  917G  171G  700G  20% /home/share
+prolix3:/share  917G  245G  626G  29% /home/share
-(base) >du -sh /home/share/unix_files/cdat/miniconda3
+ > du -sh /home/share/unix_files/cdat/miniconda3_21-02
-.8G    /home/share/unix_files/cdat/miniconda3</code>
+M    /home/share/unix_files/cdat/miniconda3_21-02</code>
-We can also use ''conda env list'' and remove some obsolete versions with ''conda remove -n obsolete_name --all'' to get some space
+We can also use ''conda env list'' and remove some obsolete versions with ''conda remove -n obsolete_environment %%--%%all'', in order to free up some space
 === Python 2.7 version ===
-<code>$ conda create -n cdat-8.2.1_py2 -c cdat/label/v81 -c conda-forge python=2.7 cdat
+<wrap hi>Python 2.x is deprecated in CDAT, starting with CDAT 8.2.1</wrap>.
-# Generate the list of installed packages
-$ conda list -n cdat-8.2.1_py2 > /home/scratch01/jypeter/cdat-8.2.1_py2_list_190307.txt</code>
-List of installed packages: {{:other:uvcdat:cdat_conda:cdat-8.2.1_py2_list_190307.txt}}
+If you still need to use python 2.7, please use [[other:uvcdat:cdat_conda:cdat_8_1|CDAT 8.1]]
-=== Python 3.6 version ===
+=== Python 3.8 version ===
-<code>$ conda create -n cdat-8.2.1_py3 -c cdat/label/v81 -c conda-forge python=3.6 cdat</code>
+<WRAP center round important 60%>
+It's possible that, when using the //tcsh// shell and trying to use the ''conda create -n cdat-8.2.1_py3 lots_of_parameters_with_wildcards'' line, you will get the following error message:
-==== Installing CDAT nightly ====
+**conda: No match**
-Notes:
+Apparently the //tcsh// version of conda does not like wildcards too much ([[https://gitmemory.com/issue/conda/conda/9771/608209654|CSH install/search fail with '*' variable in an argument]])...
-  * This page is about CDAT 8.2.1, but we have added a short nightly section here as a convenient shortcut. This should probably be moved to a stand-alone nightly page later
+In that case, just type ''bash'' and run the installation command in a //bash// shell
-  * The nightly version is **not automatically updated**, and may well be out-of-date
+</WRAP>
-  * [[https://github.com/CDAT/cdat/wiki/Obtain-CDAT-nightly-packages|Official nightly page]]
-=== Python 2.7 nightly version ===
+The installation line below is based on information from the [[https://github.com/CDAT/cdat/wiki/install#for-linux-python|CDAT wiki page]]. See also [[https://github.com/CDAT/cdat/issues/2265|Some questions and notes about 8.2.1 installation]]
-<code>$ conda create -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge python=2.7 cdat</code>
-<wrap hi>Not tested!</wrap> Can this be updated with ''conda update -n cdat-nightly_py2 -c cdat/label/nightly -c conda-forge --all'' ???
+<code>$ conda create -n cdat-8.2.1_py3 -c conda-forge -c cdat/label/v8.2.1 cdat "libnetcdf=*=mpi_openmpi_*" "mesalib=18.3.1" "python=3"</code>
-=== Python 3.6 nightly version ===
+Resulting list of installed packages
+  * ''conda list -n cdat-8.2.1_py3 > cdat_8.2.1_installed_packages.txt''
+  * {{ :other:uvcdat:cdat_conda:cdat_8.2.1_installed_packages.txt |}}
-<note important>
+[[https://github.com/CDAT/cdat/issues/2265|Some extra installation notes]]
-Not installed yet!</note>
+===== Cloning the base CDAT environment before adding specific packages for LSCE =====
-<code>$ conda create -n cdat-nightly_py3 -c cdat/label/nightly -c conda-forge python=3.6 cdat</code>
+<WRAP center round alert 60%>
+You can skip this section if you are **installing CDAT just for one user (yourself)** on a Linux machine (or a Windows 10 machine with WSL)
+You will directly make changes and install packages in **your** main python environment
+</WRAP>
-===== Cloning cdat to add specific packages for LSCE =====
 <WRAP center round tip 60%>
@@ Line 112: / Line 117: @@
 </WRAP>
+Notes:
+  * Why //cloning//? The initial CDAT environment is strictly the one created at PCMDI and certified by PCMDI. Rather than making changes directly in there, we keep it as it is, clone it, and make changes in the cloned environment
+  * Carefully working on different python environments (possibly cloned) is safer in a multi-user environment (i.e. you can have people use a specific environment, and make tests in another environment)
+  * Cloning a full environment uses Linux hard links and  requires less disk space than making a real copy
-Note: using hard links, //cloning// a full environment uses less disk space than making a real copy
+<code>$ conda create -n cdatm19_py3 --clone cdat-8.2.1_py3
+$ cd /home/share/unix_files/cdat/miniconda3_21-02/envs
+$ du -sh cdat-8.2.1_py3 cdatm19_py3
+.5G    cdat-8.2.1_py3
+M    cdatm19_py3
-<code>$ conda create -n cdatm19_py2 --clone cdat-8.2.1_py2
-$ du -sh cdat-8.2.1_py2 cdatm19_py2
-.9G    cdat-8.2.1_py2
-M    cdatm19_py2
 </code>
-==== cdat nightly case ====
-''conda create -n cdatm-nightly_py2 %%--%%clone cdat-nightly_py2''
-===== Getting ready for a moving default CDAT =====
+===== Getting ready for a moving default CDAT environment =====
-<note warning>This step should probably be listed at the **end**, especially in a multi-user environment! If there is already a ''cdatm'' link, make sure that the new version is stable and working correctly before updating the ''cdatm'' link</note>
+<WRAP center round alert 60%>
+You can skip this section if you are **installing CDAT just for one user (yourself)** on a Linux machine (or a Windows 10 machine with WSL)
-We create a **//cdatm// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default CDAT. In that case, most users can just activate this //cdatm// version and always get the latest stable version
+You will directly make changes and install packages in **your** main python environment
+</WRAP>
-<code>$ cd /home/share/unix_files/cdat/miniconda3/envs
+<WRAP center round tip 60%>
-$ ln -s cdatm19_py2 cdatm_py2</code>
+This step could probably be listed at the **end**, especially in a multi-user environment!
-===== Customizing UV-CDAT for LSCE =====
+If there is already a ''cdatm_py3'' link (pointing to an older environment), make sure that the new CDAT environment is stable and working correctly before updating the ''cdatm_py3'' link
+</WRAP>
-==== Testing vcs ====
+We create a **//cdatm_py3// symbolic link** in the ''envs'' directory, that has a //stable name// but can be moved to point to the latest default (and hopefully stable) CDAT environment. In that case, most users can just activate this //cdatm_py3// environment and always get the latest stable version.
-Start python and paste the following lines
+<code>$ cd /home/share/unix_files/cdat/miniconda3_21-02/envs
+$ ln -s cdatm19_py3 cdatm_py3
-<code>import numpy as np, vcs
+conda env list
-id100 = np.identity(100)
+# conda environments:
-x = vcs.init()
+#
-x.plot(id100)
+base                  *  /home/share/unix_files/cdat/miniconda3_21-02
+cdat-8.2.1_py3           /home/share/unix_files/cdat/miniconda3_21-02/envs/cdat-8.2.1_py3
+cdatm19_py3              /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_py3
+cdatm_py3                /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3
+$ ls -l /home/share/unix_files/cdat/miniconda3_21-02/envs/
+drwxr-xr-x [...] cdat-8.2.1_py3/
+drwxr-xr-x [...] cdatm19_py3/
+lrwxrwxrwx [...] cdatm_py3 -> cdatm19_py3/
 </code>
-If you get the kind of errors described in [[https://github.com/CDAT/vcs/issues/393|x.plot crashes with OpenGL2 error]], the drivers on your server are probably not compatible with an interactive use of vcs... The workaround is to **install mesalib and work in //headless mode//** (e.g. without a display)...
+===== Customizing UV-CDAT for LSCE =====
-Install //mesalib// with ''conda install -c cdat/label/v81 -c conda-forge mesalib vtk-cdat'' and try pasting again the few code lines above in python. If things work as expected, no canvas will be opened, but you'll be able to save the canvas to a file with
+==== A quick test of cdms2 and vcs ====
-<code>x.png('test_mesalib')
+You can use the quick test shown in [[https://github.com/CDAT/cdat/wiki/install#to-test-your-cdat-installation|To Test Your CDAT Installation]], and check if you get the expected png file
-x.pdf('test_mesalib')</code>
-Note: using //mesalib// for headless mode work makes it possible to work without an X server, and with no DISPLAY variable defined
+You can also check the [[other:uvcdat:cdat_conda:cdat_8_1#testing_vcs|Testing vcs]] section of the LSCE installation of CDAT 8.1
 ==== Downloading cdms2/vcs test data ====
-You should download the test data (174M of data...) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...)
+You should download the test data (174M of data...) and use it in the example scripts that you want to distribute, and scripts you write for reporting the errors you find (if any...). The downloaded data files will be available in: ''vcs.sample_data''
-<code>$ conda activate cdatm19_py2
+<code>$ conda activate cdatm19_py3
-(cdatm19_py2) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)'
+(cdatm19_py3) $ python -c 'import vcs; vcs.download_sample_data_files(); print("\nFinished downloading sample data to " + vcs.sample_data)'
 [...]
-Finished downloading sample data to /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data
+Finished downloading sample data to /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_py3/share/cdat/sample_data
-(cdatm19_py2) $ du -sh /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data
+(cdatm19_py3) $ du -sh /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_py3/share/cdat/sample_data
-M    /home/share/unix_files/cdat/miniconda3/envs/cdatm19_py2/share/cdat/sample_data
+M    /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_py3/share/cdat/sample_data
-</code>
+$ python
+>>> import os, vcs
+>>> vcs.sample_data
+'/home/jypeter/miniconda3_21-02/envs/cdat-8.2.1_py3/share/cdat/sample_data'
+>>> os.listdir(vcs.sample_data)
+['BlueMarble.ppm', 'clt.nc', 'geo.1deg.ctl', 'geo.1deg.gmp', 'geo.1deg.grb', ...]</code>
 ==== Packages that have no dependency problems ====
-After cloning, we are ready to install some extra packages that may be requested by LSCE users
+After [[#cloning_the_base_cdat_environment_before_adding_specific_packages_for_lsce|cloning]], we are ready to install some extra packages that may be requested by LSCE users
   * We first try to install together as many packages as possible that don't require other channels than //conda-forge//, and that don't request a downgrade of what is already installed
@@ Line 177: / Line 202: @@
 <code># You can use the following to keep a trace of what will be installed
-#$ conda install --dry-run -n cdatm19_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm  > somewhere/extra_packages.txt
+#$ conda install --dry-run -n cdatm19_py3 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm  > somewhere/extra_packages.txt
 # Install...
-$ conda install -n cdatm19_py2 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret ferret_datasets basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
+$ conda install -n cdatm19_py3 -c conda-forge pillow pandas statsmodels seaborn scikit-image seawater gsw netcdf4 pyferret ferret_datasets basemap-data-hires xlsxwriter cmocean rpy2 gdal windspharm
 [...]
@@ Line 189: / Line 214: @@
 Packages installed with //pip//
   * [[https://earthsystemcog.org/projects/wip/CMIP6DataRequest|dreqPy]]: CMIP6 Data Request Python API
-    * ''conda activate cdatm19_py2''\\ ''pip install dreqPy''
+    * ''conda activate cdatm19_py3''\\ ''pip install dreqPy''
     * Update with: ''pip install %%--%%upgrade dreqPy''
       * Get version number with:\\ <code>$ drq -v
@@ Line 195: / Line 220: @@
   * [[https://github.com/PBrockmann/ipython_ferretmagic|ipython_ferretmagic]]: IPython notebook extension for ferret
-    * ''conda activate cdatm19_py2''\\ ''pip install ferretmagic''
+    * ''conda activate cdatm19_py3''\\ ''pip install ferretmagic''
 The following packages have no dependency problems and were installed (or updated) //later//
   * [[https://cmor.llnl.gov/|CMOR]]: CMOR (//Climate Model Output Rewriter//) is used to produce CF-compliant netCDF files
-    * ''conda install -n cdatm19_py2 -c conda-forge cmor''
+    * ''conda install -n cdatm19_py3 -c conda-forge cmor''
     * Get version number with: ''python -c 'from cmor import *; print( (CMOR_VERSION_MAJOR, CMOR_VERSION_MINOR, CMOR_VERSION_PATCH) )' ''
     * <wrap hi>Warning!</wrap> [[https://github.com/PCMDI/cmor/issues/449|CMOR currently requires Python 2.7]]
   * [[http://www.ifremer.fr/vacumm/|vacumm]]: Validation, Analysis, Comparison - Utilities written in Python to validate and analyze Multi-Model outputs, and compare them to observations
-    * ''conda install -n cdatm19_py2 -c conda-forge -c vacumm vacumm''
+    * ''conda install -n cdatm19_py3 -c conda-forge -c vacumm vacumm''
     * <wrap hi>Warning!</wrap> [[https://github.com/VACUMM/vacumm/issues/15|standard install of vacumm does not work for Python3]]
   * [[https://github.com/stefraynaud/spanlib|spanlib]]: Spectral Analysis Library
-    * ''conda install -n cdatm19_py2 -c stefraynaud -c conda-forge spanlib''
+    * ''conda install -n cdatm19_py3 -c stefraynaud -c conda-forge spanlib''
     * Test: ''python -c 'from spanlib.analyzer import Analyzer' ''
   * [[http://scitools.org.uk/cartopy/|cartopy]]: a Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses
-    * ''conda install -n cdatm19_py2 -c conda-forge cartopy''
+    * ''conda install -n cdatm19_py3 -c conda-forge cartopy''
   * [[https://joblib.readthedocs.io/en/latest/|joblib]]: running Python functions as pipeline jobs
-    * ''conda install -n cdatm19_py2 -c conda-forge joblib''
+    * ''conda install -n cdatm19_py3 -c conda-forge joblib''
   * [[https://climaf.readthedocs.io/en/latest/|CliMAF]]: a Climate Model Assessment Framework
@@ Line 222: / Line 247: @@
   * [[https://jiffyclub.github.io/palettable/|Palettable]]: Color palettes for Python
-    * ''conda install -n cdatm19_py2 -c conda-forge palettable''
+    * ''conda install -n cdatm19_py3 -c conda-forge palettable''
 ==== TODO ====
@@ Line 305: / Line 330: @@
 ^  Environment\\ name  ^  Server  ^   Summary  ^
-| cdat-8.2.1_py2 |  LSCE\\ ciclad  | CDAT 8.2.1 & Python 2.7 |
+| cdat-8.2.1_py3 |  LSCE\\ ciclad  | CDAT 8.2.1 & Python 2.7 |
 | cdat-8.2.1_py3 |  LSCE  | CDAT 8.2.1 & Python 3.6 |
-| cdatm19_py2\\ or //cdatm_py2// |  LSCE\\ ciclad  | CDAT 8.2.1 & P 2.7 //JYP version// |
+| cdatm19_py3\\ or //cdatm_py3// |  LSCE\\ ciclad  | CDAT 8.2.1 & P 2.7 //JYP version// |
 | cdatm19_py3\\ or //cdatm_py3// |  LSCE  | CDAT 8.2.1 & P 3.7 //JYP version// |
-| cdat-nightly_py2 |  LSCE\\ ciclad  | CDAT nightly & Python 2.7 |
-| cdatm-nightly_py2 |  LSCE\\ ciclad  | CDAT nightly & P 2.7 //JYP version// |
 /* standard page footer */

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools