Differences

This shows you the differences between two versions of the page.

--- other:python:jyp_steps [2016/07/21 09:18]
jypeter [cdms2] Added a link to UV-CDAT in the "Working with Python" page
+++ other:python:jyp_steps [2020/02/04 08:33]
jypeter [Cartopy + Iris]
@@ Line 13: / Line 13: @@
 You can start using python by reading the {{:other:python:python_intro_ipsl_oct2013_v2.pdf|Bien démarrer avec python}} tutorial that was used during a 2013 IPSL python class:
   * this tutorial is in French (my apologies for the lack of translation, but it should be easy to understand)
+    * If you have too much trouble understanding this French Tutorial, you can read the first 6 chapters of the **Tutorial** in [[#the_official_python_documentation|the official Python documentation]] and chapters 1.2.1 to 1.2.5 in the [[#scipy_lecture_notes|Scipy Lecture Notes]]. Once you have read these, you can try to read the French tutorial again
   * it's an introduction to python (and programming) for the climate scientist: after reading this tutorial, you should be able to do most of the things you usually do in a shell script
     * python types, tests, loops, reading a text file
     * the tutorial is very detailed about string handling, because strings offer an easy way to practice working with indices (indexing and slicing), before indexing numpy arrays. And our usual pre/post-processing scripts often need to do a lot of string handling in order to generate the file/variable/experiment names
   * after reading this tutorial, you should practice with the following:
-    * [[https://files.lsce.ipsl.fr/public.php?service=files&t=9731fdad4521ac5fa6e84b392d3a2e44|Basic python training test (ipython notebook version)]]
+    * [[https://sharebox.lsce.ipsl.fr/index.php/s/S3EO8cLrhVDeQWA|Basic python training test (ipython notebook version)]]
     * {{:other:python:tp_intro_python_oct2013_no_solutions.pdf|Basic python training test (pdf version)}}
     * {{:other:python:tp_intro_python_oct2013_full.pdf|Basic python training test (pdf version, with answers)}}
@@ Line 57: / Line 58: @@
     - [[https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html|Numpy for Matlab users]]
     - [[http://mathesaurus.sourceforge.net/matlab-numpy.html|NumPy for MATLAB users]] (nice, but does not seem to be maintained any more)
-  - read the [[https://docs.scipy.org/doc/numpy-dev/user/quickstart.html|Quickstart tutorial]]
+  - read the really nice [[https://docs.scipy.org/doc/numpy/user/quickstart.html|numpy Quickstart tutorial]]
   - have a quick look at the full documentation to know where things are
     - Numpy User Guide
@@ Line 119: / Line 120: @@
        [20, 21, -1, -1, 24, 25, 26, 27, 28, 29]])
 </code></note>
+==== Extra numpy information ====
+  * More information about array indexing:
+    * Examples:
+      * {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path'
+    * [[https://docs.scipy.org/doc/numpy/user/basics.indexing.html|Indexing]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...)
+    * [[https://docs.scipy.org/doc/numpy/user/quickstart.html#fancy-indexing-and-index-tricks|Fancy indexing]] and [[https://docs.scipy.org/doc/numpy/user/quickstart.html#the-ix-function|the ix_() function]]
+    * [[https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html|Indexing (in the numpy reference manual)]]
+    * [[https://docs.scipy.org/doc/numpy/reference/routines.indexing.html#routines-indexing|Indexing routines]]
+  * More information about arrays:
+    * [[https://docs.scipy.org/doc/numpy/reference/routines.array-creation.html#routines-array-creation|Array creation routines]]
+    * [[https://docs.scipy.org/doc/numpy/reference/routines.array-manipulation.html|Array manipulation routines]]
+    * [[https://docs.scipy.org/doc/numpy/reference/maskedarray.html|Masked arrays]]
+      * [[https://docs.scipy.org/doc/numpy/reference/routines.ma.html|Masked array operations]]
+  * [[https://docs.scipy.org/doc/numpy/user/misc.html#ieee-754-floating-point-special-values|Dealing with special numerical values]] (//Nan//, //inf//)
+    * If you know that your data has missing values, it is cleaner and safer to handle them with [[https://docs.scipy.org/doc/numpy/reference/maskedarray.html|masked arrays]]!
+    * [[https://docs.scipy.org/doc/numpy/user/misc.html#how-numpy-handles-numerical-exceptions|Handling numerical exceptions]]
+    * [[https://docs.scipy.org/doc/numpy/reference/routines.err.html|Floating point error handling]]
 ===== cdms2 and netCDF4 =====
-There is a good chance that your input array data will come from a file in the [[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]] format. Depending on which [[other:python:starting#some_python_distributions|python distribution]] you are using, you can use the //cdms2// or or //netCDF4// modules to read the data.
+There is a good chance that your input array data will come from a file in the [[other:newppl:starting#netcdf_and_file_formats|NetCDF format]].
-Note: the NetCDF file format is self-documented, and the metadata of climate date files often follows the [[http://cfconventions.org/|CF (Climate and Forecast) Metadata Conventions]]
+Depending on which [[other:python:starting#some_python_distributions|python distribution]] you are using, you can use the //cdms2// or or //netCDF4// modules to read the data.
 ==== cdms2 ====
-Summary: cdms2 can read/write netCDF files (and read //grads// dat+ctl files) and provides a higher level interface than netCDF4. Unfortunately, cdms2 is only available in the [[other:python:starting#uv-cdat|UV-CDAT distribution]], and distributions where somebody has installed some version of //cdat-lite//. When you can use cdms2, you also have access to //cdtime//, that is very useful for handling time axis data.
+Summary: cdms2 can read/write netCDF files (and read //grads// dat+ctl files) and provides a higher level interface than netCDF4. cdms2 is available in the [[other:python:starting#uv-cdat|UV-CDAT distribution]], and can theoretically be installed independently of UV-CDAT (e.g. it will be installed when you install [[https://cmor.llnl.gov/mydoc_cmor3_conda/|CMOR in conda)]]. When you can use cdms2, you also have access to //cdtime//, that is very useful for handling time axis data.
 How to get started:
@@ Line 134: / Line 154: @@
     - the tutorial is in French (soooorry!)
     - you have to replace //cdms// with **cdms2**, and //MV// with **MV2** (sooorry about that, the tutorial was written when CDAT was based on //Numeric// instead of //numpy// to handle array data)
-  - read the [[http://uv-cdat.llnl.gov/documentation/cdms/cdms.html|official cdms documentation]]
+  - read the [[http://cdms.readthedocs.io/en/docstanya/index.html|official cdms documentation]] (link may change)
-  - ask questions and get answers on the [[http://uvcdat.askbot.com/questions/|UV-CDAT askbot]]
 ==== netCDF4 ====
-Summary: netCDF4 can read/write netCDF files and is available in most python distributions
+Summary: //netCDF4 can read/write netCDF files and is available in most python distributions//
 Where: [[http://unidata.github.io/netcdf4-python/]]
+===== CDAT-related resources =====
+Some links, in case they can't be found easily on the [[https://uv-cdat.llnl.gov|UV-CDAT]] web site...
+  * [[https://uv-cdat.llnl.gov/tutorials.html|Tutorials in ipython notebooks]]
+  * [[http://cdat-vcs.readthedocs.io/en/latest/|VCS: Visualization Control System]]
+    * [[https://github.com/CDAT/vcs/issues/238|Colormaps in vcs examples]]
+  * [[https://github.com/CDAT/cdat-site/blob/master/eztemplate.md|EzTemplate Documentation]]
 ===== Matplotlib =====
+<note important>
+The full content of this //matplotlib// section has been moved to\\ [[other:python:matplotlib_by_jyp|Working with matplotlib (JYP version)]]\\ after becoming too big to manage here</note>
 Summary: there are lots of python libraries that you can use for plotting, but Matplotlib has become a //de facto// standard
@@ Line 151: / Line 181: @@
 Where: [[http://matplotlib.org|Matplotlib web site]]
-The documentation is good, but not always easy to use. A good way to start with matplotlib is to:
+Help on //stack overflow//: [[https://stackoverflow.com/questions/tagged/matplotlib|matplotlib help]]
-  - Look at the [[http://matplotlib.org/gallery.html|matplotlib gallery]] to get an idea of all you can do with matplotlib. Later, when you need to plot something, come back to the gallery to find some examples that are close to what you need and click on them to get the sources
-  - Use the free hints provided by JY!
+===== Graphics related resources =====
-    - a Matplotlib //Figure// is a graphical window in which you make your plots...
-    - a Matplotlib //Axis// is a plot inside a Figure... [[http://matplotlib.org/faq/usage_faq.html#parts-of-a-figure|More details]]
+  * [[http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003833|Ten Simple Rules for Better Figures]]
-    - some examples are more //pythonic// (ie object oriented) than others, some example mix different styles of coding, all this can be confusing. Try to [[http://matplotlib.org/faq/usage_faq.html#coding-styles|use an object oriented way of doing things]]!
+  * [[https://www.machinelearningplus.com/plots/top-50-matplotlib-visualizations-the-master-plots-python/|Top 50 matplotlib Visualizations]]
-    - sometimes the results of the python/matplolib commands are displayed directly, sometimes not. It depends if you are in [[http://matplotlib.org/faq/usage_faq.html#what-is-interactive-mode|interactive or non-interactive]] mode
+  * [[http://seaborn.pydata.org/|Seaborn]] is a library for making attractive and informative statistical graphics in Python, built on top of matplotlib
-    - the documentation may mention [[http://matplotlib.org/faq/usage_faq.html#what-is-a-backend|backends]]. What?? Basically, you use python commands to create a plot, and the backend is the //thing// that will render your plot on the screen or in a file (png, pdf, etc...)
+    * See also: [[https://www.datacamp.com/community/tutorials/seaborn-python-tutorial|
-  - Read the [[http://www.labri.fr/perso/nrougier/teaching/matplotlib/|Matplotlib tutorial by Nicolas Rougier]]
+Python Seaborn Tutorial For Beginners]]
-  - Download the [[http://matplotlib.org/contents.html|pdf version of the manual]]. **Do not print** the 2800+ pages of the manual! Read the beginner's guide (Chapter //FIVE// of //Part II//) and have a super quick look at the table of contents of the whole document.
+  * Communicating/displaying/plotting your data (possibly for people not of your field):
+    * [[https://uxknowledgebase.com/introduction-to-designing-data-visualizations-part-1-31c056556133|Introduction to Designing Data Visualizations — Part 1]]
+    * [[https://uxknowledgebase.com/tables-other-charts-data-visualization-part-2-cfc582e4712c|Tables & Other Charts — Data Visualization Part 2]]
+    * [[https://uxknowledgebase.com/tables-other-charts-data-visualization-part-3-5bfab15ce525|Tables & Other Charts — Data Visualization Part 3]]
+  * Working with colors
+    * [[https://matplotlib.org/users/colormaps.html|Choosing colormaps]]
+    * [[https://matplotlib.org/cmocean/|cmocean: Beautiful colormaps for oceanography]]
+    * [[https://jiffyclub.github.io/palettable/|Palettable: Color palettes for Python]]
+    * [[http://colorbrewer2.org|ColorBrewer 2.0]] is a tool that can help you understand, and experiment with //sequential//, //diverging// and //qualitative// colormaps
 ===== Basemap =====
-<note warning>It seems that basemap is going to be slowly phased out, in favor of [[#cartopy]]\\ More information in this [[https://github.com/matplotlib/basemap/issues/267|basemap github issue]]
+<note warning>Basemap is going to be slowly phased out, in favor of [[#cartopy]]\\ More information in this:
+  * [[https://github.com/SciTools/cartopy/issues/920|cartopy github issue]]
+  * [[https://github.com/matplotlib/basemap/issues/267|basemap github issue]]
 </note>
-Summary: Basemap is an extension of Matplotlib that you can use for plotting maps, using different projections
+Summary: //Basemap is an extension of Matplotlib that you can use for plotting maps, using different projections//
 Where: [[http://matplotlib.org/basemap/|Basemap web site]]
+Help on //stack overflow//: [[https://stackoverflow.com/questions/tagged/matplotlib-basemap|basemap help]]
 How to use basemap?
@@ Line 178: / Line 221: @@
     - look at the [[http://matplotlib.org/basemap/api/basemap_api.html#module-mpl_toolkits.basemap|detailed official documentation]]
-===== Cartopy =====
+===== Cartopy + Iris =====
-Summary: //Cartopy makes use of the powerful PROJ.4, numpy and shapely libraries and has a simple and intuitive drawing interface to matplotlib for creating publication quality maps//
+Summary: //Cartopy is a Python package for advanced map generation with a simple matplotlib interface// and //Iris is a Python package for analysing and visualising meteorological and oceanographic data sets//
+Where: [[http://scitools.org.uk/cartopy/docs/latest/|Cartopy]] and [[http://scitools.org.uk/iris/index.html|Iris]] web sites
+Examples:
+  * [[other:python:maps_by_jyp|Examples provided by JYP]]
+  * [[http://scitools.org.uk/cartopy/docs/latest/gallery.html|Gallery on the Cartopy web site]]
+  * [[http://scitools.org.uk/iris/docs/latest/gallery.html|Gallery on the Iris web site]]
+  * [[http://scitools.org.uk/iris/docs/latest/examples/index.html|Examples on the Iris web site]]
+Help on //stack overflow//: [[https://stackoverflow.com/questions/tagged/cartopy|cartopy help]]
+===== Maps and projections resources =====
+==== About projections ====
+  * [[https://egsc.usgs.gov/isb//pubs/MapProjections/projections.html|Map projections from USGS poster]]
+  * [[https://pubs.usgs.gov/pp/1395/report.pdf|Map projections - A working manual (USGS)]]
+==== Libraries ====
+  * Projections in vcs
+  * [[http://matplotlib.org/basemap/users/mapsetup.html|Projections in basemap]]
+  * [[https://scitools.org.uk/cartopy/docs/latest/crs/projections.html|Projections in cartopy]]
+===== 3D resources =====
+  * [[https://ipyvolume.readthedocs.io/en/latest/|Ipyvolume]]
+  * [[https://zulko.wordpress.com/2012/09/29/animate-your-3d-plots-with-pythons-matplotlib/|Animate your 3D plots with Python’s Matplotlib]]
+  * [[https://stackoverflow.com/questions/26796997/how-to-get-vertical-z-axis-in-3d-surface-plot-of-matplotlib|How to get vertical Z axis in 3D surface plot of Matplotlib?]]
+=====  Data file formats =====
+We list here some resources about non-NetCDF data formats that can be useful
+==== The shelve package ====
+The [[https://docs.python.org/3/library/shelve.html|built-in shelve package]], can be easily used for storing data (python objects like lists, dictionaries, numpy arrays that are not too big, ...) on disk and retrieving them later
+Use case:
+  - Use a script do to the heavy data pre-processing and store the (intermediate) results in a file using ''shelve'', or update the results
+  - Use another script for plotting the results stored with ''shelve''. This way you don't have to wait for the pre-processing step to finish each time you want to improve your plot(s)
+Warning:
+  * read the [[https://docs.python.org/3/library/shelve.html|documentation]] and the example carefully (it's quite small)
+    * if you get the impression that the data is not saved correctly, re-read the parts about updating correctly the content of the shelve file
+    * you should be able to store most python objects in a shelve file, but it is safer to make tests
+  * do not forget to close the output file
+  * if you are dealing with big arrays and want to avoid performance issues, you should use netCDF files for storing the intermediate results
+==== json files ====
+More and more applications use //json files// as configuration files or as a mean to use text files to exchange data (through serialization/deserialization ).
+//json// files look basically like a **list of (nested) python dictionaries** that would have been dumped to a text file
+  * [[https://docs.python.org/2/library/json.html|json module]] documentation
+  * [[https://realpython.com/python-json/|Working With JSON Data in Python]] tutorial
+  * example script: ''/home/users/jypeter/CDAT/Progs/Devel/beaugendre/nc2json.py''
+  * A compact (not easy to read...) //json// file can be pretty-printed with\\ ''cat file.json | python -m json.tool | less''
+==== LiPD files ====
+Resources for //Linked PaleoData//:
+  * [[http://linked.earth/projects/lipd/|LiPD]]
+  * [[https://doi.org/10.5194/cp-12-1093-2016|Technical note: The Linked Paleo Data framework –
+a common tongue for paleoclimatology]] @ GMD
+  * [[https://github.com/nickmckay/LiPD-utilities|LiPD-utilities]] @ github
+==== BagIt files ====
+//BagIt//, a set of hierarchical file layout conventions for storage and transfer of arbitrary digital content.
+  * [[https://tools.ietf.org/html/draft-kunze-bagit-16|The BagIt File Packaging Format]]
+  * [[https://github.com/LibraryOfCongress/bagger|Bagger]] (BagIt GUI)
+  * [[https://github.com/LibraryOfCongress/bagit-python|bagit-python]]
+===== Pandas =====
+Summary: //pandas is a library providing high-performance, easy-to-use data structures and data analysis tools//
+Where: [[http://pandas.pydata.org|Pandas web site]]
+JYP's comment: pandas is supposed to be quite good for loading, processing and plotting time series, without writing custom code. It is **very convenient for processing tables in xlsx files** (or csv, etc...). You should at least have a quick look at:
+  * Some //Cheat Sheets// (in the following order):
+    - Basics: [[http://datacamp-community-prod.s3.amazonaws.com/dbed353d-2757-4617-8206-8767ab379ab3|Pandas basics]] (associated with the [[https://www.datacamp.com/community/blog/python-pandas-cheat-sheet|Pandas Cheat Sheet for Data Science in Python]] pandas introduction page)
+    - Intermediate: [[https://github.com/pandas-dev/pandas/tree/master/doc/cheatsheet|github Pandas doc page]]
+    - Advanced: the cheat sheet on the [[https://www.enthought.com/services/training/pandas-mastery-workshop/|Enthought workshops advertising page]]
+  * Some tutorials:
+    * [[https://www.datacamp.com/community/blog/python-pandas-cheat-sheet|Pandas Cheat Sheet for Data Science in Python]] pandas introduction page
+    * The [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines Pandas, [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]]
-Where: [[http://scitools.org.uk/cartopy/docs/latest/|Cartopy web site]]
 ===== Scipy Lecture Notes =====
@@ Line 189: / Line 321: @@
 Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]]
-This is a really nice document that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. You will learn more things about python, numpy and matplotlib, debugging and optimizing scripts, and also learn about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc...
+This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. You will learn more things about python, numpy and matplotlib, debugging and optimizing scripts, and also learn about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc...
-===== Quick Reference =====
+===== Quick Reference and cheat sheets =====
   * The nice and convenient Python 2.7 Quick Reference: [[http://rgruet.free.fr/PQR27/PQR2.7_printing_a4.pdf|pdf]] - [[http://rgruet.free.fr/PQR27/PQR2.7.html|html]]
+    * A possibly more [[http://iysik.com/PQR2.7/PQR2.7.html|up-date-version]]
+  * Python 3 [[https://perso.limsi.fr/pointal/python:abrege|Quick reference]] and [[https://perso.limsi.fr/pointal/python:memento|Cheat sheet]]
+  * [[https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/pdf_bw/|Jupyter Notebook Keyboard Shortcuts]]
+===== Misc tutorials =====
+  * [[https://pyformat.info/|PyFormat]]: //With this site we try to show you the most common use-cases covered by the old and new style string formatting API with practical examples//
 ===== Some good coding tips =====
@@ Line 222: / Line 362: @@
 Depending on the distribution, the editor and the programming environment you use, you may have access to a graphical version of the debugger. UV-CDAT users can use ''pydebug my_script.py''
+===== Using a Python IDE =====
+**IDE** = //Integrated Development Environment//
+There are lots of ways to use Python and develop scripts, from using a lightweight approach (your favorite text editor with builtin python syntax highlighting, e.g. **emacs** and ''python -i myscript.py'') to a full-fledged IDE. You'll find below some IDE related links
+  * [[https://www.datacamp.com/community/tutorials/data-science-python-ide|Top 5 Python IDEs For Data Science]]
+  * [[http://noeticforce.com/best-python-ide-for-programmers-windows-and-mac|Python IDE: The10 Best IDEs for Python Programmers]]
+  * [[https://www.techbeamers.com/best-python-ide-python-programming/|Get the Best Python IDE]]
+  * [[https://wiki.python.org/moin/IntegratedDevelopmentEnvironments]]
+==== Spyder ====
+  * [[https://github.com/spyder-ide/spyder|Home page]]
+  * [[http://pythonhosted.org/spyder/|Documentation]]
 ===== Improving the performance of your code =====

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools