other:python:jyp_steps
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
other:python:jyp_steps [2021/09/22 13:42] jypeter Added xarray |
other:python:jyp_steps [2023/12/13 14:49] jypeter Moved and renamed Scipy Lecture Notes |
||
|---|---|---|---|
| Line 44: | Line 44: | ||
| [[https://docs.python.org/3/|html]] - [[https://docs.python.org/3/download.html|pdf (in a zip file)]] | [[https://docs.python.org/3/|html]] - [[https://docs.python.org/3/download.html|pdf (in a zip file)]] | ||
| + | |||
| + | |||
| + | ===== Scientific Python Lectures ===== | ||
| + | |||
| + | Summary: //One document to learn numerics, science, and data with Python// | ||
| + | |||
| + | Note: this used to be called //Scipy Lecture Notes// | ||
| + | |||
| + | Where: [[https://lectures.scientific-python.org/_downloads/ScientificPythonLectures-simple.pdf|pdf]] - [[https://lectures.scientific-python.org/|html]] | ||
| + | |||
| + | This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. | ||
| + | |||
| + | This document will teach you lots of things about python, numpy and matplotlib, debugging and optimizing scripts, and about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc... | ||
| + | * Example: the [[https://lectures.scientific-python.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines [[other:python:jyp_steps#pandas|Pandas]], [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
| Line 64: | Line 78: | ||
| - Numpy Reference Guide | - Numpy Reference Guide | ||
| - Scipy Reference Guide | - Scipy Reference Guide | ||
| + | - read [[https://github.com/rougier/numpy-100/blob/master/100_Numpy_exercises.ipynb|100 numpy exercises]] | ||
| ==== Beware of the array view side effects ==== | ==== Beware of the array view side effects ==== | ||
| Line 123: | Line 138: | ||
| ==== Extra numpy information ==== | ==== Extra numpy information ==== | ||
| + | |||
| + | <WRAP center round tip 60%> | ||
| + | You can also check the [[other:python:misc_by_jyp#numpy_related_stuff|numpy section]] of the //Useful python stuff// page | ||
| + | </WRAP> | ||
| + | |||
| * More information about **array indexing**:\\ <wrap em>Always check what you are doing on a simple test case, when you use advanced/fancy indexing!</wrap> | * More information about **array indexing**:\\ <wrap em>Always check what you are doing on a simple test case, when you use advanced/fancy indexing!</wrap> | ||
| Line 128: | Line 148: | ||
| * {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path' | * {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path' | ||
| * [[https://numpy.org/doc/stable/user/basics.indexing.html|Array indexing basics (user guide)]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...) | * [[https://numpy.org/doc/stable/user/basics.indexing.html|Array indexing basics (user guide)]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...) | ||
| - | * [[https://numpy.org/doc/stable/reference/arrays.indexing.html|Array indexing (reference manual)]] | + | * [[https://numpy.org/doc/stable/reference/arrays.indexing.html|Indexing routines (reference manual)]] |
| * [[https://numpy.org/doc/stable/user/quickstart.html#advanced-indexing-and-index-tricks|Advanced indexing and index tricks]] and [[https://numpy.org/doc/stable/user/quickstart.html#the-ix-function|the ix_() function]] | * [[https://numpy.org/doc/stable/user/quickstart.html#advanced-indexing-and-index-tricks|Advanced indexing and index tricks]] and [[https://numpy.org/doc/stable/user/quickstart.html#the-ix-function|the ix_() function]] | ||
| - | * [[https://numpy.org/doc/stable/reference/routines.indexing.html#routines-indexing|Indexing routines]] | ||
| * More information about arrays: | * More information about arrays: | ||
| * [[https://numpy.org/doc/stable/reference/routines.array-creation.html|Array creation routines]] | * [[https://numpy.org/doc/stable/reference/routines.array-creation.html|Array creation routines]] | ||
| * [[https://numpy.org/doc/stable/reference/routines.array-manipulation.html|Array manipulation routines]] | * [[https://numpy.org/doc/stable/reference/routines.array-manipulation.html|Array manipulation routines]] | ||
| + | * [[https://numpy.org/doc/stable/reference/routines.sort.html|Sorting, searching, and counting routines]] | ||
| * [[https://numpy.org/doc/stable/reference/maskedarray.html|Masked arrays]] | * [[https://numpy.org/doc/stable/reference/maskedarray.html|Masked arrays]] | ||
| * [[https://numpy.org/doc/stable/reference/routines.ma.html|Masked array operations]] | * [[https://numpy.org/doc/stable/reference/routines.ma.html|Masked array operations]] | ||
| * [[https://numpy.org/doc/stable/user/misc.html#ieee-754-floating-point-special-values|Dealing with special numerical values]] (//Nan//, //inf//) | * [[https://numpy.org/doc/stable/user/misc.html#ieee-754-floating-point-special-values|Dealing with special numerical values]] (//Nan//, //inf//) | ||
| * If you know that your data has missing values, it is cleaner and safer to handle them with [[https://numpy.org/doc/stable/reference/maskedarray.html|masked arrays]]! | * If you know that your data has missing values, it is cleaner and safer to handle them with [[https://numpy.org/doc/stable/reference/maskedarray.html|masked arrays]]! | ||
| + | * If you know that some of your data //may// have masked values, play safe by explicitly using ''np.ma.some_function()'' rather than just ''np.some_function()'' | ||
| + | * More details in the [[https://github.com/numpy/numpy/issues/18675|Why/when does np.something remove the mask of a np.ma array ?]] discussion | ||
| * [[https://numpy.org/doc/stable/user/misc.html#how-numpy-handles-numerical-exceptions|Handling numerical exceptions]] | * [[https://numpy.org/doc/stable/user/misc.html#how-numpy-handles-numerical-exceptions|Handling numerical exceptions]] | ||
| * [[https://numpy.org/doc/stable/reference/routines.err.html|Floating point error handling]] | * [[https://numpy.org/doc/stable/reference/routines.err.html|Floating point error handling]] | ||
| - | ===== NetCDF files: using cdms2, xarray and netCDF4 ===== | + | ===== Using NetCDF files with Python ===== |
| + | |||
| + | <note tip>People using CMIPn and model data on the IPSL servers can easily search and process NetCDF files using: | ||
| + | * the [[https://climaf.readthedocs.io/|Climate Model Assessment Framework (CliMAF)]] environment | ||
| + | * and the [[https://github.com/jservonnat/C-ESM-EP/wiki|CliMAF Earth System Evaluation Platform (C-ESM-EP)]] | ||
| + | </note> | ||
| - | There is a good chance that your input array data will come from a file in the [[other:newppl:starting#netcdf_and_file_formats|NetCDF format]]. | + | * There is a good chance that your input array data will be stored in a [[other:newppl:starting#netcdf_and_related_conventions|NetCDF]] file. |
| - | Depending on which [[other:python:starting#some_python_distributions|python distribution]] you are using, you can use the //cdms2//, //xarray// or //netCDF4// modules to read the data. | + | * There may be different ways of dealing with NetCDF files, depending on which [[other:python:starting#some_python_distributions|python distribution]] you have access to |
| ==== cdms2 ==== | ==== cdms2 ==== | ||
| Line 160: | Line 187: | ||
| Summary: [[http://xarray.pydata.org/en/stable/|xarray]] is an open source project and Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun! [...] It is particularly tailored to working with netCDF files | Summary: [[http://xarray.pydata.org/en/stable/|xarray]] is an open source project and Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun! [...] It is particularly tailored to working with netCDF files | ||
| + | |||
| + | === Some xarray related resources === | ||
| + | |||
| + | Note: more packages (than listed below) may be listed in the [[other:uvcdat:cdat_conda:cdat_8_2_1#extra_packages_list|Extra packages list]] | ||
| + | |||
| + | * [[https://xcdat.readthedocs.io/|xcdat]]: xarray extended with Climate Data Analysis Tools | ||
| + | |||
| + | * [[https://xoa.readthedocs.io/en/latest/|xoa]]: xarray-based ocean analysis library | ||
| + | |||
| + | * [[https://uxarray.readthedocs.io/|uxarray]]: provide xarray styled functionality for unstructured grid datasets following [[https://ugrid-conventions.github.io/ugrid-conventions/|UGRID Conventions]] | ||
| + | |||
| Line 336: | Line 374: | ||
| * The [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines Pandas, [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | * The [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines Pandas, [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
| - | ===== Scipy Lecture Notes ===== | ||
| - | Summary: //One document to learn numerics, science, and data with Python// | + | ===== statsmodels ===== |
| - | Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]] | + | [[https://www.statsmodels.org/|statsmodels ]] is a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration. |
| - | This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. | + | ===== scikit-learn ===== |
| + | |||
| + | [[http://scikit-learn.org/|scikit-learn]] is an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection and evaluation, and many other utilities. | ||
| + | |||
| + | ===== scikit-image ===== | ||
| - | This document will teach you even more things about python, numpy and matplotlib, debugging and optimizing scripts, and about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc... | + | [[https://scikit-image.org/|scikit-image]] is a collection of algorithms for image processing in Python |
| - | * Example: the [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines [[other:python:jyp_steps#pandas|Pandas]], [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | + | |
| ===== Quick Reference and cheat sheets ===== | ===== Quick Reference and cheat sheets ===== | ||
other/python/jyp_steps.txt · Last modified: 2024/03/07 10:15 by jypeter
