other:python:jyp_steps
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
other:python:jyp_steps [2020/02/04 08:33] jypeter [Cartopy + Iris] |
other:python:jyp_steps [2021/09/22 13:21] jypeter [cdms2] uv-cdat => cdat |
||
|---|---|---|---|
| Line 55: | Line 55: | ||
| - always remember that indices start at ''0'' and that the last element of an array is at index ''-1''!\\ First learn about //indexing// and //slicing// by manipulating strings, as shown in [[#part1|Part 1]] above (try '''This document by JY is awesome!'[::-1]'' and '''This document by JY is awesome!'[slice(None, None, -1)]'') 8-) | - always remember that indices start at ''0'' and that the last element of an array is at index ''-1''!\\ First learn about //indexing// and //slicing// by manipulating strings, as shown in [[#part1|Part 1]] above (try '''This document by JY is awesome!'[::-1]'' and '''This document by JY is awesome!'[slice(None, None, -1)]'') 8-) | ||
| - | - if you are a Matlab user (but the references are interesting for others as well), you can read the following: | + | - if you are a **Matlab user** (but the references are interesting for others as well), you can read the following: |
| + | - [[https://www.enthought.com/wp-content/uploads/2019/08/Enthought-MATLAB-to-Python-White-Paper-1.pdf|Migrating from MATLAB to Python]] on the [[https://www.enthought.com/software-development/|Enthought Software Development page]] | ||
| - [[https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html|Numpy for Matlab users]] | - [[https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html|Numpy for Matlab users]] | ||
| - [[http://mathesaurus.sourceforge.net/matlab-numpy.html|NumPy for MATLAB users]] (nice, but does not seem to be maintained any more) | - [[http://mathesaurus.sourceforge.net/matlab-numpy.html|NumPy for MATLAB users]] (nice, but does not seem to be maintained any more) | ||
| Line 123: | Line 124: | ||
| ==== Extra numpy information ==== | ==== Extra numpy information ==== | ||
| - | * More information about array indexing: | + | * More information about **array indexing**:\\ <wrap em>Always check what you are doing on a simple test case, when you use advanced/fancy indexing!</wrap> |
| * Examples: | * Examples: | ||
| * {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path' | * {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path' | ||
| - | * [[https://docs.scipy.org/doc/numpy/user/basics.indexing.html|Indexing]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...) | + | * [[https://numpy.org/doc/stable/user/basics.indexing.html|Array indexing basics (user guide)]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...) |
| - | * [[https://docs.scipy.org/doc/numpy/user/quickstart.html#fancy-indexing-and-index-tricks|Fancy indexing]] and [[https://docs.scipy.org/doc/numpy/user/quickstart.html#the-ix-function|the ix_() function]] | + | * [[https://numpy.org/doc/stable/reference/arrays.indexing.html|Array indexing (reference manual)]] |
| - | * [[https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html|Indexing (in the numpy reference manual)]] | + | * [[https://numpy.org/doc/stable/user/quickstart.html#advanced-indexing-and-index-tricks|Advanced indexing and index tricks]] and [[https://numpy.org/doc/stable/user/quickstart.html#the-ix-function|the ix_() function]] |
| - | * [[https://docs.scipy.org/doc/numpy/reference/routines.indexing.html#routines-indexing|Indexing routines]] | + | * [[https://numpy.org/doc/stable/reference/routines.indexing.html#routines-indexing|Indexing routines]] |
| * More information about arrays: | * More information about arrays: | ||
| - | * [[https://docs.scipy.org/doc/numpy/reference/routines.array-creation.html#routines-array-creation|Array creation routines]] | + | * [[https://numpy.org/doc/stable/reference/routines.array-creation.html|Array creation routines]] |
| - | * [[https://docs.scipy.org/doc/numpy/reference/routines.array-manipulation.html|Array manipulation routines]] | + | * [[https://numpy.org/doc/stable/reference/routines.array-manipulation.html|Array manipulation routines]] |
| - | * [[https://docs.scipy.org/doc/numpy/reference/maskedarray.html|Masked arrays]] | + | * [[https://numpy.org/doc/stable/reference/maskedarray.html|Masked arrays]] |
| - | * [[https://docs.scipy.org/doc/numpy/reference/routines.ma.html|Masked array operations]] | + | * [[https://numpy.org/doc/stable/reference/routines.ma.html|Masked array operations]] |
| - | * [[https://docs.scipy.org/doc/numpy/user/misc.html#ieee-754-floating-point-special-values|Dealing with special numerical values]] (//Nan//, //inf//) | + | * [[https://numpy.org/doc/stable/user/misc.html#ieee-754-floating-point-special-values|Dealing with special numerical values]] (//Nan//, //inf//) |
| - | * If you know that your data has missing values, it is cleaner and safer to handle them with [[https://docs.scipy.org/doc/numpy/reference/maskedarray.html|masked arrays]]! | + | * If you know that your data has missing values, it is cleaner and safer to handle them with [[https://numpy.org/doc/stable/reference/maskedarray.html|masked arrays]]! |
| - | * [[https://docs.scipy.org/doc/numpy/user/misc.html#how-numpy-handles-numerical-exceptions|Handling numerical exceptions]] | + | * [[https://numpy.org/doc/stable/user/misc.html#how-numpy-handles-numerical-exceptions|Handling numerical exceptions]] |
| - | * [[https://docs.scipy.org/doc/numpy/reference/routines.err.html|Floating point error handling]] | + | * [[https://numpy.org/doc/stable/reference/routines.err.html|Floating point error handling]] |
| ===== cdms2 and netCDF4 ===== | ===== cdms2 and netCDF4 ===== | ||
| Line 148: | Line 149: | ||
| ==== cdms2 ==== | ==== cdms2 ==== | ||
| - | Summary: cdms2 can read/write netCDF files (and read //grads// dat+ctl files) and provides a higher level interface than netCDF4. cdms2 is available in the [[other:python:starting#uv-cdat|UV-CDAT distribution]], and can theoretically be installed independently of UV-CDAT (e.g. it will be installed when you install [[https://cmor.llnl.gov/mydoc_cmor3_conda/|CMOR in conda)]]. When you can use cdms2, you also have access to //cdtime//, that is very useful for handling time axis data. | + | Summary: cdms2 can read/write netCDF files (and read //grads// dat+ctl files) and provides a higher level interface than netCDF4. cdms2 is available in the [[other:python:starting#cdat|CDAT distribution]], and can theoretically be installed independently of CDAT (e.g. it will be installed when you install [[https://cmor.llnl.gov/mydoc_cmor3_conda/|CMOR in conda)]]. When you can use cdms2, you also have access to //cdtime//, that is very useful for handling time axis data. |
| How to get started: | How to get started: | ||
| Line 165: | Line 166: | ||
| ===== CDAT-related resources ===== | ===== CDAT-related resources ===== | ||
| - | Some links, in case they can't be found easily on the [[https://uv-cdat.llnl.gov|UV-CDAT]] web site... | + | Some links, in case they can't be found easily on the [[https://cdat.llnl.gov|CDAT]] web site... |
| - | * [[https://uv-cdat.llnl.gov/tutorials.html|Tutorials in ipython notebooks]] | + | * [[https://cdat.llnl.gov/tutorials.html|Tutorials in ipython notebooks]] |
| * [[http://cdat-vcs.readthedocs.io/en/latest/|VCS: Visualization Control System]] | * [[http://cdat-vcs.readthedocs.io/en/latest/|VCS: Visualization Control System]] | ||
| * [[https://github.com/CDAT/vcs/issues/238|Colormaps in vcs examples]] | * [[https://github.com/CDAT/vcs/issues/238|Colormaps in vcs examples]] | ||
| Line 175: | Line 176: | ||
| <note important> | <note important> | ||
| - | The full content of this //matplotlib// section has been moved to\\ [[other:python:matplotlib_by_jyp|Working with matplotlib (JYP version)]]\\ after becoming too big to manage here</note> | + | The full content of this //matplotlib// section has been moved to\\ [[other:python:matplotlib_by_jyp|Working with matplotlib (JYP version)]]\\ after becoming too big to manage here |
| + | |||
| + | \\ Note: [[other:python:maps_by_jyp|Plotting maps with matplotlib+cartopy]] (examples provided by JYP) | ||
| + | </note> | ||
| Summary: there are lots of python libraries that you can use for plotting, but Matplotlib has become a //de facto// standard | Summary: there are lots of python libraries that you can use for plotting, but Matplotlib has become a //de facto// standard | ||
| Line 194: | Line 198: | ||
| * [[https://uxknowledgebase.com/tables-other-charts-data-visualization-part-2-cfc582e4712c|Tables & Other Charts — Data Visualization Part 2]] | * [[https://uxknowledgebase.com/tables-other-charts-data-visualization-part-2-cfc582e4712c|Tables & Other Charts — Data Visualization Part 2]] | ||
| * [[https://uxknowledgebase.com/tables-other-charts-data-visualization-part-3-5bfab15ce525|Tables & Other Charts — Data Visualization Part 3]] | * [[https://uxknowledgebase.com/tables-other-charts-data-visualization-part-3-5bfab15ce525|Tables & Other Charts — Data Visualization Part 3]] | ||
| - | * Working with colors | + | * **IPCC**-related //stuff//... |
| + | * [[https://www.ipcc.ch/site/assets/uploads/2019/04/IPCC-visual-style-guide.pdf|IPCC Visual Style Guide for Authors]] | ||
| + | * [[https://wg1.ipcc.ch/sites/default/files/documents/ipcc_visual-identity_guidelines.pdf|A new assessment cycle,A new visual identity]] | ||
| + | * [[https://link.springer.com/article/10.1007/s10584-019-02537-z|Communication of IPCC visuals: IPCC authors’ views and assessments of visual complexity]] | ||
| + | * [[https://www.carbonbrief.org/guest-post-the-perils-of-counter-intuitive-design-in-ipcc-graphics|The perils of counter-intuitive design in IPCC graphics]] | ||
| + | * Working with **colors** | ||
| + | * Choosing specific colors: use [[https://www.w3schools.com/colors/colors_names.asp|HTML color names]], the [[https://www.w3schools.com/colors/colors_picker.asp|HTML color picker]], etc... | ||
| + | * **Do not use the outdated //rainbow// and //jet// colormaps!** | ||
| + | * [[https://pjbartlein.github.io/datagraphics/index.html|The End of the Rainbow? Color Schemes for Improved Data Graphics]] (Light and Bartlein, EOS 2004, including replies and comments) | ||
| + | * [[http://colorspace.r-forge.r-project.org/articles/endrainbow.html|Somewhere over the Rainbow]] | ||
| + | * [[https://www.nature.com/articles/s41467-020-19160-7|The misuse of colour in science communication]] | ||
| * [[https://matplotlib.org/users/colormaps.html|Choosing colormaps]] | * [[https://matplotlib.org/users/colormaps.html|Choosing colormaps]] | ||
| * [[https://matplotlib.org/cmocean/|cmocean: Beautiful colormaps for oceanography]] | * [[https://matplotlib.org/cmocean/|cmocean: Beautiful colormaps for oceanography]] | ||
| * [[https://jiffyclub.github.io/palettable/|Palettable: Color palettes for Python]] | * [[https://jiffyclub.github.io/palettable/|Palettable: Color palettes for Python]] | ||
| * [[http://colorbrewer2.org|ColorBrewer 2.0]] is a tool that can help you understand, and experiment with //sequential//, //diverging// and //qualitative// colormaps | * [[http://colorbrewer2.org|ColorBrewer 2.0]] is a tool that can help you understand, and experiment with //sequential//, //diverging// and //qualitative// colormaps | ||
| + | * The [[http://hclwizard.org/|hclwizard]] provides tools for manipulating and assessing colors and palettes based on the underlying ''colorspace'' software | ||
| + | * NCL (NCAR Command Language) [[https://www.ncl.ucar.edu/Document/Graphics/color_table_gallery.shtml|Color table Gallery]] | ||
| + | * JYP's favorite title: [[https://www.researchgate.net/publication/220943662_The_Which_Blair_Project_A_Quick_Visual_Method_for_Evaluating_Perceptual_Color_Maps|The "Which Blair Project": A Quick Visual Method for Evaluating Perceptual Color Maps]] | ||
| ===== Basemap ===== | ===== Basemap ===== | ||
| - | <note warning>Basemap is going to be slowly phased out, in favor of [[#cartopy]]\\ More information in this: | + | <note warning>Basemap is going to be slowly phased out, in favor of [[#cartopy_iris|cartopy]]\\ More information in this: |
| * [[https://github.com/SciTools/cartopy/issues/920|cartopy github issue]] | * [[https://github.com/SciTools/cartopy/issues/920|cartopy github issue]] | ||
| * [[https://github.com/matplotlib/basemap/issues/267|basemap github issue]] | * [[https://github.com/matplotlib/basemap/issues/267|basemap github issue]] | ||
| Line 223: | Line 240: | ||
| ===== Cartopy + Iris ===== | ===== Cartopy + Iris ===== | ||
| - | Summary: //Cartopy is a Python package for advanced map generation with a simple matplotlib interface// and //Iris is a Python package for analysing and visualising meteorological and oceanographic data sets// | + | Summary: |
| + | * **Cartopy** is //a matplolib-based Python package designed for geospatial data processing in order to produce maps and other geospatial data analyses// | ||
| + | * **Iris** is //a powerful, format-agnostic, community-driven Python package for analysing and visualising Earth science data.// | ||
| - | Where: [[http://scitools.org.uk/cartopy/docs/latest/|Cartopy]] and [[http://scitools.org.uk/iris/index.html|Iris]] web sites | + | Where: [[http://scitools.org.uk/cartopy/docs/latest/|Cartopy]] and [[https://scitools-iris.readthedocs.io/en/stable/|Iris]] web sites |
| Examples: | Examples: | ||
| * [[other:python:maps_by_jyp|Examples provided by JYP]] | * [[other:python:maps_by_jyp|Examples provided by JYP]] | ||
| - | * [[http://scitools.org.uk/cartopy/docs/latest/gallery.html|Gallery on the Cartopy web site]] | + | * Official gallery pages: [[https://scitools.org.uk/cartopy/docs/latest/gallery/index.html|Cartopy]] - [[https://scitools-iris.readthedocs.io/en/stable/generated/gallery/|Iris]] |
| - | * [[http://scitools.org.uk/iris/docs/latest/gallery.html|Gallery on the Iris web site]] | + | |
| - | * [[http://scitools.org.uk/iris/docs/latest/examples/index.html|Examples on the Iris web site]] | + | |
| - | Help on //stack overflow//: [[https://stackoverflow.com/questions/tagged/cartopy|cartopy help]] | + | Help on //stack overflow//: [[https://stackoverflow.com/questions/tagged/cartopy|Cartopy help]] - [[https://stackoverflow.com/questions/tagged/python-iris|Iris help]] |
| ===== Maps and projections resources ===== | ===== Maps and projections resources ===== | ||
| Line 279: | Line 296: | ||
| //json// files look basically like a **list of (nested) python dictionaries** that would have been dumped to a text file | //json// files look basically like a **list of (nested) python dictionaries** that would have been dumped to a text file | ||
| - | * [[https://docs.python.org/2/library/json.html|json module]] documentation | + | * [[https://docs.python.org/3/library/json.html|json module]] documentation |
| * [[https://realpython.com/python-json/|Working With JSON Data in Python]] tutorial | * [[https://realpython.com/python-json/|Working With JSON Data in Python]] tutorial | ||
| * example script: ''/home/users/jypeter/CDAT/Progs/Devel/beaugendre/nc2json.py'' | * example script: ''/home/users/jypeter/CDAT/Progs/Devel/beaugendre/nc2json.py'' | ||
| Line 321: | Line 338: | ||
| Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]] | Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]] | ||
| - | This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. You will learn more things about python, numpy and matplotlib, debugging and optimizing scripts, and also learn about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc... | + | This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. |
| + | |||
| + | This document will teach you even more things about python, numpy and matplotlib, debugging and optimizing scripts, and about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc... | ||
| + | * Example: the [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines [[other:python:jyp_steps#pandas|Pandas]], [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
| ===== Quick Reference and cheat sheets ===== | ===== Quick Reference and cheat sheets ===== | ||
| Line 331: | Line 351: | ||
| * [[https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/pdf_bw/|Jupyter Notebook Keyboard Shortcuts]] | * [[https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/pdf_bw/|Jupyter Notebook Keyboard Shortcuts]] | ||
| + | |||
| + | ===== Miscellaneous Python stuff ===== | ||
| + | |||
| + | Check the page about [[other:python:misc_by_jyp|useful python stuff that has not been sorted yet]] | ||
| ===== Misc tutorials ===== | ===== Misc tutorials ===== | ||
| Line 362: | Line 386: | ||
| Depending on the distribution, the editor and the programming environment you use, you may have access to a graphical version of the debugger. UV-CDAT users can use ''pydebug my_script.py'' | Depending on the distribution, the editor and the programming environment you use, you may have access to a graphical version of the debugger. UV-CDAT users can use ''pydebug my_script.py'' | ||
| + | |||
| + | ===== jupyter and notebook stuff ===== | ||
| + | |||
| + | FIXME Misc notes, resources and links to organize later | ||
| + | |||
| + | * [[https://beta.jupyterbook.org/|jupyter {book}]]: Jupyter Book is an open source project for building beautiful, publication-quality books and documents from computational material. | ||
| ===== Using a Python IDE ===== | ===== Using a Python IDE ===== | ||
| Line 404: | Line 434: | ||
| ===== Python 2.7 vs Python 3 ===== | ===== Python 2.7 vs Python 3 ===== | ||
| - | The official [[https://docs.python.org/2.7/howto/pyporting.html|Porting Python 2 Code to Python 3]] page gives the required information to make the transition from python 2 to python 3. It is still safe to use Python 2.7, so there is no rush to change to Python 3. | + | It is still safe to use Python 2.7, but **you should consider upgrading to Python 3**, unless some key modules you need are not compatible (yet) with Python 3 |
| + | |||
| + | You should start writing code that will, when possible, work both in Python 2 and Python 3 | ||
| + | |||
| + | Some interesting reading: | ||
| + | |||
| + | * [[https://docs.python.org/3/whatsnew/3.0.html|What’s New In Python 3.0]].\\ Examples: | ||
| + | * ''print'' is now a function. Use ''print('Hello')'' | ||
| + | * You cannot test a difference with ''<>'' any longer! Use ''!='' | ||
| + | |||
| + | * The official [[https://docs.python.org/2.7/howto/pyporting.html|Porting Python 2 Code to Python 3]] page gives the required information to make the transition from python 2 to python 3. | ||
| ===== What now? ===== | ===== What now? ===== | ||
other/python/jyp_steps.txt · Last modified: 2024/03/07 10:15 by jypeter
