PMIP3 wiki

User Tools

Site Tools

You are here: index » other » python » jyp_steps
other:python:jyp_steps

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Next revisionBoth sides next revision
other:python:jyp_steps [2016/01/19 17:51] – jypeterother:python:jyp_steps [2017/03/08 10:48] – Rewrote the netcdf part to add a link to the tuto for new people jypeter
Line 1: Line 1:
-====== Following JYP's steps ======+====== JYP's recommended steps for learning python ====== 
 + 
 +<note tip>If you don't know which python distribution to use and how to start the python interpreter, you should first read the [[starting|Working with Python]] page</note>
  
 As can be expected, there is **a lot** of online python documentation available, and it's easy to get lost. You can always use google to find an answer to your problem, and you will probably end up looking at lots of answers on [[http://stackoverflow.com/questions/tagged/python|Stack Overflow]] or a similar site. But it's always better to know where you can find some good documentation... and to spend some time to read the documentation As can be expected, there is **a lot** of online python documentation available, and it's easy to get lost. You can always use google to find an answer to your problem, and you will probably end up looking at lots of answers on [[http://stackoverflow.com/questions/tagged/python|Stack Overflow]] or a similar site. But it's always better to know where you can find some good documentation... and to spend some time to read the documentation
  
-This page tries to list some //python for the scientist// related resources, in a suggested reading order. You should not print everything, but it's a good idea to download all the //pdf// files in the same place, so that you can easily open and search the document+This page tries to list some //python for the scientist// related resources, in a suggested reading order. **Do not print anything** (or at least not everything), but it's a good idea to download all the //pdf// files in the same place, so that you can easily open and search the documents 
 + 
 +===== JYP's introduction to python ===== 
 + 
 +==== Part 1 ==== 
 + 
 +You can start using python by reading the {{:other:python:python_intro_ipsl_oct2013_v2.pdf|Bien démarrer avec python}} tutorial that was used during a 2013 IPSL python class: 
 +  * this tutorial is in French (my apologies for the lack of translation, but it should be easy to understand) 
 +    * If you have too much trouble understanding this French Tutorial, you can read the first 6 chapters of the **Tutorial** in [[#the_official_python_documentation|the official Python documentation]] and chapters 1.2.1 to 1.2.5 in the [[#scipy_lecture_notes|Scipy Lecture Notes]]. Once you have read these, you can try to read the French tutorial again 
 +  * it's an introduction to python (and programming) for the climate scientist: after reading this tutorial, you should be able to do most of the things you usually do in a shell script 
 +    * python types, tests, loops, reading a text file 
 +    * the tutorial is very detailed about string handling, because strings offer an easy way to practice working with indices (indexing and slicing), before indexing numpy arrays. And our usual pre/post-processing scripts often need to do a lot of string handling in order to generate the file/variable/experiment names 
 +  * after reading this tutorial, you should practice with the following: 
 +    * [[https://files.lsce.ipsl.fr/public.php?service=files&t=9731fdad4521ac5fa6e84b392d3a2e44|Basic python training test (ipython notebook version)]] 
 +    * {{:other:python:tp_intro_python_oct2013_no_solutions.pdf|Basic python training test (pdf version)}} 
 +    * {{:other:python:tp_intro_python_oct2013_full.pdf|Basic python training test (pdf version, with answers)}} 
 + 
 +==== Part 2 ==== 
 + 
 +Once you have done your first steps, you should read [[http://www.lsce.ipsl.fr/Phocea/file.php?class=page&file=5/pythonCDAT_jyp_1sur2_070227.pdf|Plus loin avec Python]] (start at page 39, the previous pages are an old version of what was covered in //Part 1// above) 
 +  * this tutorial is in French (sorry again) 
 +  * after reading this tutorial, you will be able to do more than you can do in a shell script, in an easier way 
 +    * advanced string formatting 
 +    * creating functions and using modules 
 +    * working with file paths and handling files without calling external Linux programs\\ (e.g. using ''os.remove(file_name)'' instead of ''rm $file_name'') 
 +    * using command-line options for scripts, or using configuration files 
 +    * calling external programs 
 + 
 +===== The official python documentation ===== 
 + 
 +You do not need to read all the python documentation at this step, but it is really well made and you should at least have a look at it. The **Tutorial** is very good, and you should have a look at the table of content of the **Python Standard Library**. There is a lot in the default library that can make your life easier 
 + 
 +==== Python 2.7 ==== 
 + 
 +[[https://docs.python.org/2.7/|html]] - [[https://docs.python.org/2.7/download.html|pdf (in a zip file)]] 
 + 
 +==== Python 3 ==== 
 + 
 +[[https://docs.python.org/3/|html]] - [[https://docs.python.org/3/download.html|pdf (in a zip file)]] 
 + 
 + 
 +===== Numpy and Scipy ===== 
 + 
 +Summary: Python provides //ordered// objects (e.g. lists, strings, basic arrays, ...) and some math operators, but you can't do real heavy computation with these. **Numpy** makes it possible to work with multi-dimensional data arrays, and using array syntax and masks (instead of explicit nested loops and tests) and the apropriate numpy functions will allow you to get performance similar to what you would get with a compiled program! **Scipy** adds more scientific functions 
 + 
 +Where: [[http://docs.scipy.org/doc/|html and pdf documentation]] 
 + 
 +==== Getting started ==== 
 + 
 +  - always remember that indices start at ''0'' and that the last element of an array is at index ''-1''!\\ First learn about //indexing// and //slicing// by manipulating strings, as shown in [[#part1|Part 1]] above (try '''This document by JY is awesome!'[::-1]'' and '''This document by JY is awesome!'[slice(None, None, -1)]'') 8-) 
 +  - if you are a Matlab user (but the references are interesting for others as well), you can read the following: 
 +    - [[https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html|Numpy for Matlab users]] 
 +    - [[http://mathesaurus.sourceforge.net/matlab-numpy.html|NumPy for MATLAB users]] (nice, but does not seem to be maintained any more) 
 +  - read the [[https://docs.scipy.org/doc/numpy-dev/user/quickstart.html|Quickstart tutorial]] 
 +  - have a quick look at the full documentation to know where things are 
 +    - Numpy User Guide 
 +    - Numpy Reference Guide 
 +    - Scipy Reference Guide 
 + 
 +==== Beware of the array view side effects ==== 
 + 
 +<note warning>When you take a slice of an array, you get a **//View//** : an array that has a new shape but that still shares its data with the first array. 
 + 
 +That is not a problem when you only read the values, but **if you change the values of the //View//, you change the values of the first array** (and vice-versa)! If that is not what want, do not forget to **make a copy** of the data before working on it! 
 + 
 +//Views// are a good thing most of the time, so only make a copy of your data when needed, because otherwise copying a big array will just be a waste of CPU and computer memory. Anyway, it is always better to understand what you are doing... :-P 
 + 
 +Check the example below and the [[https://docs.scipy.org/doc/numpy-dev/user/quickstart.html#copies-and-views|copies and views]] part of the quickstart tutorial. 
 + 
 +<code python> 
 +>>> import numpy as np 
 +>>> a = np.arange(30).reshape((3,10)) 
 +>>> a 
 +array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9], 
 +       [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], 
 +       [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]]) 
 + 
 +>>> b = a[1, :] 
 +>>> b 
 +array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) 
 + 
 +>>> b[3:7] = 0 
 +>>> b 
 +array([10, 11, 12,  0,  0,  0,  0, 17, 18, 19]) 
 + 
 +>>> a 
 +array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9], 
 +       [10, 11, 12,  0,  0,  0,  0, 17, 18, 19], 
 +       [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]]) 
 +        
 +>>> a[:, 2:4] = -1 
 +>>> a 
 +array([[ 0,  1, -1, -1,  4,  5,  6,  7,  8,  9], 
 +       [10, 11, -1, -1,  0,  0,  0, 17, 18, 19], 
 +       [20, 21, -1, -1, 24, 25, 26, 27, 28, 29]]) 
 +        
 +>>> b 
 +array([10, 11, -1, -1,  0,  0,  0, 17, 18, 19]) 
 + 
 +>>> c = a[1, :].copy() 
 +>>> c 
 +array([10, 11, -1, -1,  0,  0,  0, 17, 18, 19]) 
 + 
 +>>> c[:] = 9 
 +>>> c 
 +array([9, 9, 9, 9, 9, 9, 9, 9, 9, 9]) 
 + 
 +>>> b 
 +array([10, 11, -1, -1,  0,  0,  0, 17, 18, 19]) 
 + 
 +>>> a 
 +array([[ 0,  1, -1, -1,  4,  5,  6,  7,  8,  9], 
 +       [10, 11, -1, -1,  0,  0,  0, 17, 18, 19], 
 +       [20, 21, -1, -1, 24, 25, 26, 27, 28, 29]]) 
 +</code></note> 
 + 
 +===== cdms2 and netCDF4 ===== 
 + 
 +There is a good chance that your input array data will come from a file in the [[other:newppl:starting#netcdf_and_file_formats|NetCDF format]]. 
 + 
 +Depending on which [[other:python:starting#some_python_distributions|python distribution]] you are using, you can use the //cdms2// or or //netCDF4// modules to read the data. 
 + 
 +==== cdms2 ==== 
 + 
 +Summary: cdms2 can read/write netCDF files (and read //grads// dat+ctl files) and provides a higher level interface than netCDF4. Unfortunately, cdms2 is only available in the [[other:python:starting#uv-cdat|UV-CDAT distribution]], and distributions where somebody has installed some version of //cdat-lite//. When you can use cdms2, you also have access to //cdtime//, that is very useful for handling time axis data. 
 + 
 +How to get started: 
 +  - read [[http://www.lsce.ipsl.fr/Phocea/file.php?class=page&file=5/pythonCDAT_jyp_2sur2_070306.pdf|JYP's cdms tutorial]], starting at page 54 
 +    - the tutorial is in French (soooorry!) 
 +    - you have to replace //cdms// with **cdms2**, and //MV// with **MV2** (sooorry about that, the tutorial was written when CDAT was based on //Numeric// instead of //numpy// to handle array data) 
 +  - read the [[http://uv-cdat.llnl.gov/documentation/cdms/cdms.html|official cdms documentation]] 
 +  - ask questions and get answers on the [[http://uvcdat.askbot.com/questions/|UV-CDAT askbot]] 
 + 
 + 
 +==== netCDF4 ==== 
 + 
 +Summary: netCDF4 can read/write netCDF files and is available in most python distributions 
 + 
 +Where: [[http://unidata.github.io/netcdf4-python/]] 
 + 
 + 
 +===== Matplotlib ===== 
 + 
 +Summary: there are lots of python libraries that you can use for plotting, but Matplotlib has become a //de facto// standard 
 + 
 +Where: [[http://matplotlib.org|Matplotlib web site]] 
 + 
 +The documentation is good, but not always easy to use. A good way to start with matplotlib is to: 
 +  - Look at the [[http://matplotlib.org/gallery.html|matplotlib gallery]] to get an idea of all you can do with matplotlib. Later, when you need to plot something, come back to the gallery to find some examples that are close to what you need and click on them to get the sources 
 +  - Use the free hints provided by JY! 
 +    - a Matplotlib //Figure// is a graphical window in which you make your plots...  
 +    - a Matplotlib //Axis// is a plot inside a Figure... [[http://matplotlib.org/faq/usage_faq.html#parts-of-a-figure|More details]] 
 +    - some examples are more //pythonic// (ie object oriented) than others, some example mix different styles of coding, all this can be confusing. Try to [[http://matplotlib.org/faq/usage_faq.html#coding-styles|use an object oriented way of doing things]]! 
 +    - sometimes the results of the python/matplolib commands are displayed directly, sometimes not. It depends if you are in [[http://matplotlib.org/faq/usage_faq.html#what-is-interactive-mode|interactive or non-interactive]] mode 
 +    - the documentation may mention [[http://matplotlib.org/faq/usage_faq.html#what-is-a-backend|backends]]. What?? Basically, you use python commands to create a plot, and the backend is the //thing// that will render your plot on the screen or in a file (png, pdf, etc...) 
 +  - Read the [[http://www.labri.fr/perso/nrougier/teaching/matplotlib/|Matplotlib tutorial by Nicolas Rougier]] 
 +  - Download the [[http://matplotlib.org/contents.html|pdf version of the manual]]. **Do not print** the 2800+ pages of the manual! Read the beginner's guide (Chapter //FIVE// of //Part II//) and have a super quick look at the table of contents of the whole document. 
 + 
 +===== Basemap ===== 
 + 
 +<note warning>It seems that basemap is going to be slowly phased out, in favor of [[#cartopy]]\\ More information in this [[https://github.com/matplotlib/basemap/issues/267|basemap github issue]] 
 +</note> 
 + 
 +Summary: Basemap is an extension of Matplotlib that you can use for plotting maps, using different projections 
 + 
 +Where: [[http://matplotlib.org/basemap/|Basemap web site]] 
 + 
 +How to use basemap? 
 +  - look at the [[http://matplotlib.org/basemap/users/examples.html|examples]] 
 +  - check the [[http://matplotlib.org/basemap/users/mapsetup.html|different projections]] 
 +  - read some documentation! 
 +    - the **really nice** [[http://basemaptutorial.readthedocs.io/en/latest/index.html|basemap tutorial]] seems much better than the official documentation below 
 +    - look at the [[http://matplotlib.org/basemap/api/basemap_api.html#module-mpl_toolkits.basemap|detailed official documentation]] 
 + 
 +===== Cartopy ===== 
 + 
 +Summary: //Cartopy makes use of the powerful PROJ.4, numpy and shapely libraries and has a simple and intuitive drawing interface to matplotlib for creating publication quality maps// 
 + 
 +Where: [[http://scitools.org.uk/cartopy/docs/latest/|Cartopy web site]] 
 + 
 +===== Pandas ===== 
 + 
 +Summary: //pandas is a library providing high-performance, easy-to-use data structures and data analysis tools// 
 + 
 +Where: [[http://pandas.pydata.org|Pandas web site]] 
 + 
 +JYP's comment: pandas is supposed to be quite good for loading, processing and plotting time series, without writing custom code. You should at least have a quick look at: 
 +  * The [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines Pandas, [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] 
 +  * the cheat sheet on the [[https://www.enthought.com/services/training/pandas-mastery-workshop/|Enthought workshops advertising page]] 
 +  * the cheat sheet on the [[https://github.com/pandas-dev/pandas/tree/master/doc/cheatsheet|github Pandas doc page]]
  
 ===== Scipy Lecture Notes ===== ===== Scipy Lecture Notes =====
Line 11: Line 202:
 Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]] Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]]
  
-This is a really nice document that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials+This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. You will learn more things about python, numpy and matplotlib, debugging and optimizing scripts, and also learn about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc...
  
 ===== Quick Reference ===== ===== Quick Reference =====
  
-  * The nice Python 2.7 Quick Reference: [[http://rgruet.free.fr/PQR27/PQR2.7_printing_a4.pdf|pdf]] - [[http://rgruet.free.fr/PQR27/PQR2.7.html|html]]+  * The nice and convenient Python 2.7 Quick Reference: [[http://rgruet.free.fr/PQR27/PQR2.7_printing_a4.pdf|pdf]] - [[http://rgruet.free.fr/PQR27/PQR2.7.html|html]]
  
 ===== Some good coding tips ===== ===== Some good coding tips =====
Line 22: Line 213:
  
   * [[http://blog.codinghorror.com/a-pragmatic-quick-reference/|A Pragmatic Quick Reference]]   * [[http://blog.codinghorror.com/a-pragmatic-quick-reference/|A Pragmatic Quick Reference]]
 +
 +===== Debugging your code =====
 +
 +There is only so much you can do with staring at your code in your favorite text editor, and adding ''print'' lines in your code (or using [[https://docs.python.org/2/howto/logging.html#logging-basic-tutorial|logging]] instead of ''print''). The next step is to **use the python debugger**!
 +
 +==== Debugging in text mode ====
 +
 +  - Start the script with: ''python -m pdb my_script.py''
 +  - Type ''run'' (or **r**) to go to the first line of the script
 +  - Type ''continue'' (or **c**) to execute the script to the end, or till the first breakpoint or error is reached
 +  - Use ''where'' (or **w**) to check the call stack that led to the current stop. Use ''up'' and ''down'' to navigate through the call stack and examine the values of the functions' parameters
 +  - Type ''break NNN'' to stop at line NNN
 +  - Use ''type(var)'' and ''print var'' to check the type and values of variables. You can also change the variables' values on the fly!
 +  - Type ''run'' (or **r**) to restart the script
 +  - Use ''next'' and ''step'' to execute some parts of the script line by line. If a code line calls a function:
 +    * ''next'' (or **n**) will execute a function and stop on the next line
 +    * ''step'' (or **s**) will stop at the first line **inside the function**  
 +  - Check the [[https://docs.python.org/2/library/pdb.html#debugger-commands|debugger commands]] for details, or type ''help'' in the debugger for using the built-in help
 +
 +==== Using pydebug ====
 +
 +Depending on the distribution, the editor and the programming environment you use, you may have access to a graphical version of the debugger. UV-CDAT users can use ''pydebug my_script.py''
  
 ===== Improving the performance of your code ===== ===== Improving the performance of your code =====
Line 29: Line 242:
   * **make sure that your script is not using too much memory** (the amount depends on the computer you are using)! Your script should be scalable (e.g. keeps on working even when your data gets bigger), so it's a good idea to load only the data you need in memory (e.g. not all the time steps), and learn how to load chunks of data   * **make sure that your script is not using too much memory** (the amount depends on the computer you are using)! Your script should be scalable (e.g. keeps on working even when your data gets bigger), so it's a good idea to load only the data you need in memory (e.g. not all the time steps), and learn how to load chunks of data
  
-  * **make sure that you are using array/vector syntax and masks**, instead of using explicit loops and tests. The numpy documentation is big, because there are lots of optimized functions to help you!+  * **make sure that you are using array/vector syntax and masks**, instead of using explicit loops and tests. The [[#numpy_and_scipy|numpy documentation]] is big, because there are lots of optimized functions to help you! If you are stuck, ask JY or somebody else who is used to numpy.
  
 If your script is still not fast enough, there is a lot you can do to improve it, without resorting to parallelization (that may introduce extra bugs rather that extra performance). See the sections below If your script is still not fast enough, there is a lot you can do to improve it, without resorting to parallelization (that may introduce extra bugs rather that extra performance). See the sections below
 +
 +Hint: before optimizing your script, you should spent some time //profiling// it, in order to only spend time improving the slow parts of your script
 +
 +==== Useful packages ====
 +
 +  * [[https://github.com/pydata/numexpr|Numexpr]]: //Numexpr is a **fast numerical expression evaluator for NumPy**. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python.//
 +  * [[http://www.pytables.org/|PyTables]]: //PyTables is a package for managing hierarchical datasets and designed to efficiently and **easily cope with extremely large amounts of data**//
  
 ==== Tutorials by Ian Osvald ==== ==== Tutorials by Ian Osvald ====
Line 41: Line 261:
  
 The official [[https://docs.python.org/2.7/howto/pyporting.html|Porting Python 2 Code to Python 3]] page gives the required information to make the transition from python 2 to python 3. It is still safe to use Python 2.7, so there is no rush to change to Python 3. The official [[https://docs.python.org/2.7/howto/pyporting.html|Porting Python 2 Code to Python 3]] page gives the required information to make the transition from python 2 to python 3. It is still safe to use Python 2.7, so there is no rush to change to Python 3.
 +
 +===== What now? =====
 +
 +You can do a lot more with python! But if you have read at least a part of this page, you should be able to find and use the modules you need. Make sure you do not reinvent the wheel! Use existing packages when possible, and make sure to report bugs or errors in the documentations when you find some
  
 /* standard page footer */ /* standard page footer */
other/python/jyp_steps.txt · Last modified: 2025/02/26 11:40 by jypeter
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki