This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:python:jyp_steps [2019/05/23 14:54] jypeter [Matplotlib] Added ref to subplot_adjust |
other:python:jyp_steps [2020/10/08 13:37] jypeter [Python 2.7 vs Python 3] Added more information |
||
---|---|---|---|
Line 18: | Line 18: | ||
* the tutorial is very detailed about string handling, because strings offer an easy way to practice working with indices (indexing and slicing), before indexing numpy arrays. And our usual pre/post-processing scripts often need to do a lot of string handling in order to generate the file/variable/experiment names | * the tutorial is very detailed about string handling, because strings offer an easy way to practice working with indices (indexing and slicing), before indexing numpy arrays. And our usual pre/post-processing scripts often need to do a lot of string handling in order to generate the file/variable/experiment names | ||
* after reading this tutorial, you should practice with the following: | * after reading this tutorial, you should practice with the following: | ||
- | * [[https://files.lsce.ipsl.fr/public.php?service=files&t=9731fdad4521ac5fa6e84b392d3a2e44|Basic python training test (ipython notebook version)]] | + | * [[https://sharebox.lsce.ipsl.fr/index.php/s/S3EO8cLrhVDeQWA|Basic python training test (ipython notebook version)]] |
* {{:other:python:tp_intro_python_oct2013_no_solutions.pdf|Basic python training test (pdf version)}} | * {{:other:python:tp_intro_python_oct2013_no_solutions.pdf|Basic python training test (pdf version)}} | ||
* {{:other:python:tp_intro_python_oct2013_full.pdf|Basic python training test (pdf version, with answers)}} | * {{:other:python:tp_intro_python_oct2013_full.pdf|Basic python training test (pdf version, with answers)}} | ||
Line 173: | Line 173: | ||
===== Matplotlib ===== | ===== Matplotlib ===== | ||
+ | |||
+ | <note important> | ||
+ | The full content of this //matplotlib// section has been moved to\\ [[other:python:matplotlib_by_jyp|Working with matplotlib (JYP version)]]\\ after becoming too big to manage here | ||
+ | |||
+ | \\ Note: [[other:python:maps_by_jyp|Plotting maps with matplotlib+cartopy]] (examples provided by JYP) | ||
+ | </note> | ||
Summary: there are lots of python libraries that you can use for plotting, but Matplotlib has become a //de facto// standard | Summary: there are lots of python libraries that you can use for plotting, but Matplotlib has become a //de facto// standard | ||
Line 179: | Line 185: | ||
Help on //stack overflow//: [[https://stackoverflow.com/questions/tagged/matplotlib|matplotlib help]] | Help on //stack overflow//: [[https://stackoverflow.com/questions/tagged/matplotlib|matplotlib help]] | ||
- | |||
- | The documentation is good, but not always easy to use. <wrap hi>A good way to start with matplotlib</wrap> is to: | ||
- | - Look at the [[http://matplotlib.org/gallery.html|matplotlib gallery]] to get an idea of all you can do with matplotlib. Later, when you need to plot something, come back to the gallery to find some examples that are close to what you need and click on them to get the sources | ||
- | - Use the free hints provided by JY! | ||
- | - a Matplotlib //Figure// is a graphical window in which you make your plots... | ||
- | - a Matplotlib //Axis// is a plot inside a Figure... [[http://matplotlib.org/faq/usage_faq.html#parts-of-a-figure|More details]] | ||
- | - some resources for having multiple plots on the same figure | ||
- | * [[https://matplotlib.org/gallery/recipes/create_subplots.html#sphx-glr-gallery-recipes-create-subplots-py|Easily creating subplots]] | ||
- | * [[https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplots_adjust.html|subplots_adjust]] can be used to change the boundaries | ||
- | * [[https://matplotlib.org/gallery/index.html#subplots-axes-and-figures|Subplots, axes and figures]] gallery | ||
- | * [[https://matplotlib.org/tutorials/intermediate/gridspec.html#sphx-glr-tutorials-intermediate-gridspec-py|Customizing Figure Layouts Using GridSpec and Other Functions]], [[https://matplotlib.org/tutorials/intermediate/constrainedlayout_guide.html|constrained layout]] and [[https://matplotlib.org/tutorials/intermediate/tight_layout_guide.html|tight layout]] | ||
- | * [[http://matplotlib.org/faq/usage_faq.html#parts-of-a-figure|parts of a figure]] | ||
- | - some examples are more //pythonic// (ie object oriented) than others, some example mix different styles of coding, all this can be confusing. Try to [[http://matplotlib.org/faq/usage_faq.html#coding-styles|use an object oriented way of doing things]]! | ||
- | - it may be hard to (remember how to) work with colors. Some examples from the [[http://matplotlib.org/gallery.html|Gallery]] can help you! | ||
- | * [[https://matplotlib.org/examples/pylab_examples/leftventricle_bulleye.html|leftventricle_bulleye.py]]: associating different types of colormaps to a plot and colorbar | ||
- | * [[https://matplotlib.org/examples/api/colorbar_only.html|colorbar_only.py]]: the different types of colorbars (or plotting only a colorbar) | ||
- | * [[https://matplotlib.org/examples/color/colormaps_reference.html|colormaps_reference.py]]: pre-defined colormaps | ||
- | * [[https://matplotlib.org/examples/color/named_colors.html|named_colors.py]]: named colors | ||
- | * More details about the colors below, in the [[#graphics_related_resources|Resources section]] | ||
- | - sometimes the results of the python/matplolib commands are displayed immediately, sometimes not. It depends if you are in [[http://matplotlib.org/faq/usage_faq.html#what-is-interactive-mode|interactive or non-interactive]] mode | ||
- | - if your matplotlib is executed in a batch script, it will generate an error when trying to create (''show()'') a plot, because matplotlib expects to be able to display the figure on a screen by default. | ||
- | * Check how you can [[https://matplotlib.org/faq/howto_faq.html?highlight=web#generate-images-without-having-a-window-appear|generate images offline]] | ||
- | - the documentation may mention [[http://matplotlib.org/faq/usage_faq.html#what-is-a-backend|backends]]. What?? Basically, you use python commands to create a plot, and the backend is the //thing// that will render your plot on the screen or in a file (png, pdf, etc...) | ||
- | - if you don't see a part of what you have plotted, maybe it's hidden behind other elements! Use the [[https://matplotlib.org/examples/pylab_examples/zorder_demo.html|zorder parameter]] to explicitly specify the plotting order/layers | ||
- | - Read the [[http://www.labri.fr/perso/nrougier/teaching/matplotlib/|Matplotlib tutorial by Nicolas Rougier]] | ||
- | - Download the [[http://matplotlib.org/contents.html|pdf version of the manual]]. **Do not print** the 2800+ pages of the manual! Read the beginner's guide (Chapter //FIVE// of //Part II//) and have a super quick look at the table of contents of the whole document. | ||
- | |||
- | ==== Useful reference pages ==== | ||
- | |||
- | * [[https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html|plot(...)]]: Plot y versus x as lines and/or markers | ||
- | * [[https://matplotlib.org/api/_as_gen/matplotlib.pyplot.scatter.html|scatter(...)]]: A scatter plot of y vs x with varying marker size and/or color | ||
- | * The ''plot'' function will be faster for scatterplots where markers don't vary in size or color | ||
- | * [[https://matplotlib.org/api/_as_gen/matplotlib.lines.Line2D.html|line]] parameters | ||
- | * ''linestyle'': ''solid'', ''None'', [[https://matplotlib.org/api/_as_gen/matplotlib.lines.Line2D.html#matplotlib.lines.Line2D.set_linestyle|other]] ([[https://matplotlib.org/examples/lines_bars_and_markers/line_styles_reference.html|default styles example]], [[https://matplotlib.org/examples/lines_bars_and_markers/linestyles.html|custom styles example]]) | ||
- | * [[https://matplotlib.org/api/markers_api.html|marker types]] | ||
- | * ''fillstyle'': ''full'', ''None'', [[https://matplotlib.org/gallery/lines_bars_and_markers/marker_fillstyle_reference.html|other]] | ||
- | * Other attributes: ''markersize'', ''markerfacecolor'' (and ''markerfacecoloralt'' for dual color markers), ''markeredgecolor'', ''markeredgewidth'' | ||
- | * [[https://matplotlib.org/api/colors_api.html|colors]] | ||
- | * [[https://matplotlib.org/gallery/color/color_demo.html|color demo]] | ||
- | * [[https://matplotlib.org/examples/color/named_colors.html|named colors]] | ||
- | * [[https://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.legend|legend(...)]] ([[https://matplotlib.org/examples/pylab_examples/legend_demo3.html|legend demo]]) | ||
- | * The legend will //show// the lines (or other objects) that were associated with a //label// with the ''label='' keyword when creating/updating a plot | ||
- | * If there are some elements of a plot that you do not want to associate with a legend (e.g. there are several lines with the same color and markers, but you want to plot the legend only once), do not specify a ''label='' keyword for these elements, or add a ''_'' at the front of the label strings | ||
- | * The legend is positioned somewhere (that can be specified) **inside** the plot. In order to place a legend **outside** the plot, use the ''bbox_to_anchor'' parameter | ||
- | * the parameters of ''bbox_to_anchor'' are in normalized coordinates of the current (sub)plot: | ||
- | * ''(0, 0)'' is the lower left corner of the plot, and ''(1, 1)'' the upper right corner | ||
- | * ''legend(... bbox_to_anchor=(1.05, 1.), loc='upper left', ...)'' will put the upper left corner of the legend slightly right (''(1.05, 1.)'') of the upper right corner (''(1, 1)'') of the plot | ||
- | * if the legend is outside of the plot, you have to **explicitly provide enough space for the legend on the page** | ||
- | * e.g. with [[https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplots_adjust.html|subplots_adjust]], ''plt.subplots_adjust(right=0.75)'' will make all the plots use 75% on the left of the page, and leave 25% on the right for the legend | ||
- | |||
- | ==== Misc Matplotlib tricks ==== | ||
- | |||
- | * Specifying the background color of a plot (e.g. when plotting a masked variable and you don't want the masked areas to be white) | ||
- | * ''# make the background dark gray (call this before the contourf)''\\ ''plt.gca().patch.set_color('.25')''\\ ''plt.contourf(d)''\\ ''plt.show()'' | ||
- | * [[https://stackoverflow.com/questions/9797520/masking-part-of-a-contourf-plot-in-matplotlib|trick source]] | ||
===== Graphics related resources ===== | ===== Graphics related resources ===== | ||
Line 242: | Line 193: | ||
* See also: [[https://www.datacamp.com/community/tutorials/seaborn-python-tutorial| | * See also: [[https://www.datacamp.com/community/tutorials/seaborn-python-tutorial| | ||
Python Seaborn Tutorial For Beginners]] | Python Seaborn Tutorial For Beginners]] | ||
+ | * Communicating/displaying/plotting your data (possibly for people not of your field): | ||
+ | * [[https://uxknowledgebase.com/introduction-to-designing-data-visualizations-part-1-31c056556133|Introduction to Designing Data Visualizations — Part 1]] | ||
+ | * [[https://uxknowledgebase.com/tables-other-charts-data-visualization-part-2-cfc582e4712c|Tables & Other Charts — Data Visualization Part 2]] | ||
+ | * [[https://uxknowledgebase.com/tables-other-charts-data-visualization-part-3-5bfab15ce525|Tables & Other Charts — Data Visualization Part 3]] | ||
* Working with colors | * Working with colors | ||
+ | * **Do not use the outdated //rainbow// colormap!** | ||
+ | * [[http://geog.uoregon.edu/datagraphics/EOS/Light-and-Bartlein.pdf|The End of the Rainbow? Color Schemes for Improved Data Graphics]] (Light and Bartlein, EOS 2004, including replies and comments) | ||
+ | * [[http://colorspace.r-forge.r-project.org/articles/endrainbow.html|Somewhere over the Rainbow]] | ||
* [[https://matplotlib.org/users/colormaps.html|Choosing colormaps]] | * [[https://matplotlib.org/users/colormaps.html|Choosing colormaps]] | ||
- | * [[https://matplotlib.org/cmocean/|Beautiful colormaps for oceanography: cmocean]] | + | * [[https://matplotlib.org/cmocean/|cmocean: Beautiful colormaps for oceanography]] |
+ | * [[https://jiffyclub.github.io/palettable/|Palettable: Color palettes for Python]] | ||
* [[http://colorbrewer2.org|ColorBrewer 2.0]] is a tool that can help you understand, and experiment with //sequential//, //diverging// and //qualitative// colormaps | * [[http://colorbrewer2.org|ColorBrewer 2.0]] is a tool that can help you understand, and experiment with //sequential//, //diverging// and //qualitative// colormaps | ||
+ | * The [[http://hclwizard.org/|hclwizard]] provides tools for manipulating and assessing colors and palettes based on the underlying ''colorspace'' software | ||
Line 275: | Line 235: | ||
Examples: | Examples: | ||
+ | * [[other:python:maps_by_jyp|Examples provided by JYP]] | ||
* [[http://scitools.org.uk/cartopy/docs/latest/gallery.html|Gallery on the Cartopy web site]] | * [[http://scitools.org.uk/cartopy/docs/latest/gallery.html|Gallery on the Cartopy web site]] | ||
* [[http://scitools.org.uk/iris/docs/latest/gallery.html|Gallery on the Iris web site]] | * [[http://scitools.org.uk/iris/docs/latest/gallery.html|Gallery on the Iris web site]] | ||
Line 305: | Line 266: | ||
We list here some resources about non-NetCDF data formats that can be useful | We list here some resources about non-NetCDF data formats that can be useful | ||
+ | ==== The shelve package ==== | ||
+ | |||
+ | The [[https://docs.python.org/3/library/shelve.html|built-in shelve package]], can be easily used for storing data (python objects like lists, dictionaries, numpy arrays that are not too big, ...) on disk and retrieving them later | ||
+ | |||
+ | Use case: | ||
+ | - Use a script do to the heavy data pre-processing and store the (intermediate) results in a file using ''shelve'', or update the results | ||
+ | - Use another script for plotting the results stored with ''shelve''. This way you don't have to wait for the pre-processing step to finish each time you want to improve your plot(s) | ||
+ | |||
+ | Warning: | ||
+ | * read the [[https://docs.python.org/3/library/shelve.html|documentation]] and the example carefully (it's quite small) | ||
+ | * if you get the impression that the data is not saved correctly, re-read the parts about updating correctly the content of the shelve file | ||
+ | * you should be able to store most python objects in a shelve file, but it is safer to make tests | ||
+ | * do not forget to close the output file | ||
+ | * if you are dealing with big arrays and want to avoid performance issues, you should use netCDF files for storing the intermediate results | ||
==== json files ==== | ==== json files ==== | ||
Line 337: | Line 312: | ||
Where: [[http://pandas.pydata.org|Pandas web site]] | Where: [[http://pandas.pydata.org|Pandas web site]] | ||
- | JYP's comment: pandas is supposed to be quite good for loading, processing and plotting time series, without writing custom code. You should at least have a quick look at: | + | JYP's comment: pandas is supposed to be quite good for loading, processing and plotting time series, without writing custom code. It is **very convenient for processing tables in xlsx files** (or csv, etc...). You should at least have a quick look at: |
- | * The [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines Pandas, [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | + | |
- | * the cheat sheet on the [[https://www.enthought.com/services/training/pandas-mastery-workshop/|Enthought workshops advertising page]] | + | * Some //Cheat Sheets// (in the following order): |
- | * the cheat sheet on the [[https://github.com/pandas-dev/pandas/tree/master/doc/cheatsheet|github Pandas doc page]] | + | - Basics: [[http://datacamp-community-prod.s3.amazonaws.com/dbed353d-2757-4617-8206-8767ab379ab3|Pandas basics]] (associated with the [[https://www.datacamp.com/community/blog/python-pandas-cheat-sheet|Pandas Cheat Sheet for Data Science in Python]] pandas introduction page) |
+ | - Intermediate: [[https://github.com/pandas-dev/pandas/tree/master/doc/cheatsheet|github Pandas doc page]] | ||
+ | - Advanced: the cheat sheet on the [[https://www.enthought.com/services/training/pandas-mastery-workshop/|Enthought workshops advertising page]] | ||
+ | * Some tutorials: | ||
+ | * [[https://www.datacamp.com/community/blog/python-pandas-cheat-sheet|Pandas Cheat Sheet for Data Science in Python]] pandas introduction page | ||
+ | * The [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines Pandas, [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
===== Scipy Lecture Notes ===== | ===== Scipy Lecture Notes ===== | ||
Line 348: | Line 328: | ||
Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]] | Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]] | ||
- | This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. You will learn more things about python, numpy and matplotlib, debugging and optimizing scripts, and also learn about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc... | + | This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. |
+ | |||
+ | This document will teach you even more things about python, numpy and matplotlib, debugging and optimizing scripts, and about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc... | ||
+ | * Example: the [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines [[other:python:jyp_steps#pandas|Pandas]], [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
===== Quick Reference and cheat sheets ===== | ===== Quick Reference and cheat sheets ===== | ||
Line 389: | Line 372: | ||
Depending on the distribution, the editor and the programming environment you use, you may have access to a graphical version of the debugger. UV-CDAT users can use ''pydebug my_script.py'' | Depending on the distribution, the editor and the programming environment you use, you may have access to a graphical version of the debugger. UV-CDAT users can use ''pydebug my_script.py'' | ||
+ | |||
+ | ===== jupyter and notebook stuff ===== | ||
+ | |||
+ | FIXME Misc notes, resources and links to organize later | ||
+ | |||
+ | * [[https://beta.jupyterbook.org/|jupyter {book}]]: Jupyter Book is an open source project for building beautiful, publication-quality books and documents from computational material. | ||
===== Using a Python IDE ===== | ===== Using a Python IDE ===== | ||
Line 431: | Line 420: | ||
===== Python 2.7 vs Python 3 ===== | ===== Python 2.7 vs Python 3 ===== | ||
- | The official [[https://docs.python.org/2.7/howto/pyporting.html|Porting Python 2 Code to Python 3]] page gives the required information to make the transition from python 2 to python 3. It is still safe to use Python 2.7, so there is no rush to change to Python 3. | + | It is still safe to use Python 2.7, but **you should consider upgrading to Python 3**, unless some key modules you need are not compatible (yet) with Python 3 |
+ | |||
+ | You should start writing code that will, when possible, work both in Python 2 and Python 3 | ||
+ | |||
+ | Some interesting reading: | ||
+ | |||
+ | * [[https://docs.python.org/3/whatsnew/3.0.html|What’s New In Python 3.0]].\\ Examples: | ||
+ | * ''print'' is now a function. Use ''print('Hello')'' | ||
+ | * You cannot test a difference with ''<>'' any longer! Use ''!='' | ||
+ | |||
+ | * The official [[https://docs.python.org/2.7/howto/pyporting.html|Porting Python 2 Code to Python 3]] page gives the required information to make the transition from python 2 to python 3. | ||
===== What now? ===== | ===== What now? ===== |