other:python:jyp_steps
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
other:python:jyp_steps [2016/01/20 10:58] jypeter |
other:python:jyp_steps [2016/01/22 11:40] jypeter added cdms2/netcdf4 |
||
|---|---|---|---|
| Line 11: | Line 11: | ||
| ==== Python 2.7 ==== | ==== Python 2.7 ==== | ||
| - | [[https://docs.python.org/2.7/|html]] - [[https://docs.python.org/2.7/download.html|pdf]] | + | [[https://docs.python.org/2.7/|html]] - [[https://docs.python.org/2.7/download.html|pdf (in a zip file)]] |
| ==== Python 3 ==== | ==== Python 3 ==== | ||
| - | [[https://docs.python.org/3/|html]] - [[https://docs.python.org/3/download.html|pdf]] | + | [[https://docs.python.org/3/|html]] - [[https://docs.python.org/3/download.html|pdf (in a zip file)]] |
| + | |||
| + | |||
| + | ===== Numpy and Scipy ===== | ||
| + | |||
| + | Summary: Python provides //ordered// objects (e.g. lists, strings, basic arrays, ...) and some math operators, but you can't do real heavy computation with these. **Numpy** makes it possible to work with multi-dimensional data arrays, and using array syntax and masks (instead of explicit nested loops and tests) and the apropriate numpy functions will allow you to get performance similar to what you would get with a compiled program! **Scipy** adds more scientific functions | ||
| + | |||
| + | How to get started? | ||
| + | - always remember that indices start at ''0'' and that the last element of an array is at index ''-1''! Learn about indexing and slicing by manipulating a string (try '''This document by JY is awesome!'[::-1]'') | ||
| + | - read the [[https://docs.scipy.org/doc/numpy-dev/user/quickstart.html|Quickstart tutorial]] | ||
| + | - have a quick look at the full documentation to know where things are | ||
| + | - Numpy User Guide: | ||
| + | - Numpy Reference Guide | ||
| + | - Scipy Reference Guide | ||
| + | |||
| + | ===== cdms2 and netcdf4 ===== | ||
| + | |||
| + | There is a good chance that your input array data will come from a file in the [[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]] format. Depending on which [[other:python:starting#some_python_distributions|python distribution]] you are using, you can use the //cdms2// or or //netCDF4// modules to read the data. | ||
| + | |||
| + | Note: the NetCDF file format is self-documented, and the metadata of climate date files often follows the [[http://cfconventions.org/|CF (Climate and Forecast) Metadata Conventions]] | ||
| + | |||
| + | ==== cdms2 ==== | ||
| + | |||
| + | Summary: cdms2 can read/write netCDF files (and read //grads// dat+ctl files) and provides a higher level interface than netCDF4. Unfortunately, cdms2 is only available in the UV-CDAT distribution, and distributions where somebody has installed some version of //cdat-lite//. When you can use cdms2, you also have access to //cdtime//, that is very useful for handling time axis data. | ||
| + | |||
| + | How to get started: | ||
| + | - read [[http://www.lsce.ipsl.fr/Phocea/file.php?class=page&file=5/pythonCDAT_jyp_2sur2_070306.pdf|JYP's cdms tutorial]], starting at page 54 | ||
| + | - the tutorial is in French (soooorry!) | ||
| + | - you have to replace //cdms// with **cdms2**, and //MV// with **MV2** (sooorry about that, the tutorial was written when CDAT was based on Numeric instead of numpy to handle array data) | ||
| + | - read the [[http://uv-cdat.llnl.gov/documentation/cdms/cdms.html|official cdms documentation]] | ||
| + | - ask questions and get answers on the [[http://uvcdat.askbot.com/questions/|UV-CDAT askbot]] | ||
| + | |||
| + | |||
| + | ==== netCDF4 ==== | ||
| + | |||
| + | Summary: netCDF4 can read/write netCDF files and is available in most python distributions | ||
| + | |||
| + | Where: [[http://unidata.github.io/netcdf4-python/]] | ||
| + | |||
| + | |||
| + | ===== Matplotlib ===== | ||
| + | |||
| + | Summary: there are lots of python libraries that you can use for plotting, but Matplotlib has become a //de facto// standard | ||
| + | |||
| + | Where: [[http://matplotlib.org|Matplotlib web site]] | ||
| + | |||
| + | The documentation is good, but not always easy to use. A good way to start with matplotlib is to: | ||
| + | - Look at the [[http://matplotlib.org/gallery.html|matplotlib gallery]] to get an idea of all you can do with matplotlib. Later, when you need to plot something, come back to the gallery to find some examples that are close to what you need and click on them to get the sources | ||
| + | - Use the free hints provided by JY! | ||
| + | - a Matplotlib //Figure// is a graphical window in which you make your plots... | ||
| + | - a Matplotlib //Axis// is a plot inside a Figure... [[http://matplotlib.org/faq/usage_faq.html#parts-of-a-figure|More details]] | ||
| + | - some examples are more //pythonic// (ie object oriented) than others, some example mix different styles of coding, all this can be confusing. Try to [[http://matplotlib.org/faq/usage_faq.html#coding-styles|use an object oriented way of doing things]]! | ||
| + | - sometimes the results of the python/matplolib commands are shown directly, sometimes not. It depends if you are in [[http://matplotlib.org/faq/usage_faq.html#what-is-interactive-mode|interactive or non-interactive]] mode | ||
| + | - the documentation may mention [[http://matplotlib.org/faq/usage_faq.html#what-is-a-backend|backends]]. What?? Basically, you use python commands to create a plot, and the backend is the //thing// that will render your plot on the screen or in a file (png, pdf, etc...) | ||
| + | - Read the [[http://www.labri.fr/perso/nrougier/teaching/matplotlib/|Matplotlib tutorial by Nicolas Rougier]] | ||
| + | - Download the [[http://matplotlib.org/contents.html|pdf version of the manual]]. **Do not print** the 2800+ pages of the manual! Read the beginner's guide (Chapter //FIVE// of //Part II//) and have a super quick look at the table of contents of the whole document. | ||
| + | |||
| + | ===== Basemap ===== | ||
| + | |||
| + | Summary: Basemap is an extension of Matplotlib that you can use for plotting maps, using different projections | ||
| + | |||
| + | Where: [[http://matplotlib.org/basemap/|Basemap web site]] | ||
| + | |||
| + | How to use basemap? | ||
| + | - look at the [[http://matplotlib.org/basemap/users/examples.html|examples]] | ||
| + | - check the [[http://matplotlib.org/basemap/users/mapsetup.html|different projections]] | ||
| + | - look at the [[http://matplotlib.org/basemap/api/basemap_api.html#module-mpl_toolkits.basemap|detailed documentation]] | ||
| ===== Scipy Lecture Notes ===== | ===== Scipy Lecture Notes ===== | ||
| Line 27: | Line 94: | ||
| ===== Quick Reference ===== | ===== Quick Reference ===== | ||
| - | * The nice Python 2.7 Quick Reference: [[http://rgruet.free.fr/PQR27/PQR2.7_printing_a4.pdf|pdf]] - [[http://rgruet.free.fr/PQR27/PQR2.7.html|html]] | + | * The nice and convenient Python 2.7 Quick Reference: [[http://rgruet.free.fr/PQR27/PQR2.7_printing_a4.pdf|pdf]] - [[http://rgruet.free.fr/PQR27/PQR2.7.html|html]] |
| ===== Some good coding tips ===== | ===== Some good coding tips ===== | ||
| Line 41: | Line 108: | ||
| * **make sure that your script is not using too much memory** (the amount depends on the computer you are using)! Your script should be scalable (e.g. keeps on working even when your data gets bigger), so it's a good idea to load only the data you need in memory (e.g. not all the time steps), and learn how to load chunks of data | * **make sure that your script is not using too much memory** (the amount depends on the computer you are using)! Your script should be scalable (e.g. keeps on working even when your data gets bigger), so it's a good idea to load only the data you need in memory (e.g. not all the time steps), and learn how to load chunks of data | ||
| - | * **make sure that you are using array/vector syntax and masks**, instead of using explicit loops and tests. The numpy documentation is big, because there are lots of optimized functions to help you! | + | * **make sure that you are using array/vector syntax and masks**, instead of using explicit loops and tests. The numpy documentation is big, because there are lots of optimized functions to help you! If you are stuck, ask JY or somebody else who is used to numpy. |
| If your script is still not fast enough, there is a lot you can do to improve it, without resorting to parallelization (that may introduce extra bugs rather that extra performance). See the sections below | If your script is still not fast enough, there is a lot you can do to improve it, without resorting to parallelization (that may introduce extra bugs rather that extra performance). See the sections below | ||
other/python/jyp_steps.txt · Last modified: 2024/03/07 10:15 by jypeter
