This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:python:jyp_steps [2023/02/01 16:49] jypeter [xarray] Added some xarray resources |
other:python:jyp_steps [2023/12/14 15:18] jypeter [xarray] Updated |
||
---|---|---|---|
Line 13: | Line 13: | ||
You can start using python by reading the {{:other:python:python_intro_ipsl_oct2013_v2.pdf|Bien démarrer avec python}} tutorial that was used during a 2013 IPSL python class: | You can start using python by reading the {{:other:python:python_intro_ipsl_oct2013_v2.pdf|Bien démarrer avec python}} tutorial that was used during a 2013 IPSL python class: | ||
* this tutorial is in French (my apologies for the lack of translation, but it should be easy to understand) | * this tutorial is in French (my apologies for the lack of translation, but it should be easy to understand) | ||
- | * If you have too much trouble understanding this French Tutorial, you can read the first 6 chapters of the **Tutorial** in [[#the_official_python_documentation|the official Python documentation]] and chapters 1.2.1 to 1.2.5 in the [[#scipy_lecture_notes|Scipy Lecture Notes]]. Once you have read these, you can try to read the French tutorial again | + | * If you have too much trouble understanding this French Tutorial, you can read the first 6 chapters of the **Tutorial** in [[#the_official_python_documentation|the official Python documentation]] and chapters 1.2.1 to 1.2.5 in the [[#scientific_python_lectures|Scientific Python Lectures]]. Once you have read these, you can try to read the French tutorial again |
* it's an introduction to python (and programming) for the climate scientist: after reading this tutorial, you should be able to do most of the things you usually do in a shell script | * it's an introduction to python (and programming) for the climate scientist: after reading this tutorial, you should be able to do most of the things you usually do in a shell script | ||
* python types, tests, loops, reading a text file | * python types, tests, loops, reading a text file | ||
Line 44: | Line 44: | ||
[[https://docs.python.org/3/|html]] - [[https://docs.python.org/3/download.html|pdf (in a zip file)]] | [[https://docs.python.org/3/|html]] - [[https://docs.python.org/3/download.html|pdf (in a zip file)]] | ||
+ | |||
+ | |||
+ | ===== Scientific Python Lectures ===== | ||
+ | |||
+ | Summary: //One document to learn numerics, science, and data with Python// | ||
+ | |||
+ | Note: this used to be called //Scipy Lecture Notes// | ||
+ | |||
+ | Where: [[https://lectures.scientific-python.org/_downloads/ScientificPythonLectures-simple.pdf|pdf]] - [[https://lectures.scientific-python.org/|html]] | ||
+ | |||
+ | This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. | ||
+ | |||
+ | This document will teach you lots of things about python, numpy and matplotlib, debugging and optimizing scripts, and about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc... | ||
+ | * Example: the [[https://lectures.scientific-python.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines [[other:python:jyp_steps#pandas|Pandas]], [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
Line 64: | Line 78: | ||
- Numpy Reference Guide | - Numpy Reference Guide | ||
- Scipy Reference Guide | - Scipy Reference Guide | ||
+ | - read [[https://github.com/rougier/numpy-100/blob/master/100_Numpy_exercises.ipynb|100 numpy exercises]] | ||
==== Beware of the array view side effects ==== | ==== Beware of the array view side effects ==== | ||
Line 133: | Line 148: | ||
* {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path' | * {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path' | ||
* [[https://numpy.org/doc/stable/user/basics.indexing.html|Array indexing basics (user guide)]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...) | * [[https://numpy.org/doc/stable/user/basics.indexing.html|Array indexing basics (user guide)]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...) | ||
- | * [[https://numpy.org/doc/stable/reference/arrays.indexing.html|Array indexing (reference manual)]] | + | * [[https://numpy.org/doc/stable/reference/arrays.indexing.html|Indexing routines (reference manual)]] |
* [[https://numpy.org/doc/stable/user/quickstart.html#advanced-indexing-and-index-tricks|Advanced indexing and index tricks]] and [[https://numpy.org/doc/stable/user/quickstart.html#the-ix-function|the ix_() function]] | * [[https://numpy.org/doc/stable/user/quickstart.html#advanced-indexing-and-index-tricks|Advanced indexing and index tricks]] and [[https://numpy.org/doc/stable/user/quickstart.html#the-ix-function|the ix_() function]] | ||
- | * [[https://numpy.org/doc/stable/reference/routines.indexing.html#routines-indexing|Indexing routines]] | ||
* More information about arrays: | * More information about arrays: | ||
* [[https://numpy.org/doc/stable/reference/routines.array-creation.html|Array creation routines]] | * [[https://numpy.org/doc/stable/reference/routines.array-creation.html|Array creation routines]] | ||
Line 172: | Line 186: | ||
==== xarray ==== | ==== xarray ==== | ||
- | Summary: [[http://xarray.pydata.org/en/stable/|xarray]] is an open source project and Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun! [...] It is particularly tailored to working with netCDF files | + | Summary: [[https://docs.xarray.dev/|xarray]] makes working with labelled multi-dimensional arrays in Python simple, efficient, and fun! [...] It is particularly tailored to working with netCDF files |
=== Some xarray related resources === | === Some xarray related resources === | ||
- | Note: more packages (than listed below) may be listed in the [[other:uvcdat:cdat_conda:cdat_8_2_1#extra_packages_list|Extra packages list]] | + | Note: more packages (than listed below) may be listed in the [[other:uvcdat:cdat_conda:cdat_8_2_1#extra_packages_list|Extra packages list]] page |
+ | |||
+ | * [[https://docs.xarray.dev/en/stable/generated/xarray.tutorial.load_dataset.html|xarray test datasets]] | ||
* [[https://xcdat.readthedocs.io/|xcdat]]: xarray extended with Climate Data Analysis Tools | * [[https://xcdat.readthedocs.io/|xcdat]]: xarray extended with Climate Data Analysis Tools | ||
Line 294: | Line 310: | ||
- | ===== 3D resources ===== | + | ===== 3D plots resources ===== |
* [[https://ipyvolume.readthedocs.io/en/latest/|Ipyvolume]] | * [[https://ipyvolume.readthedocs.io/en/latest/|Ipyvolume]] | ||
* [[https://zulko.wordpress.com/2012/09/29/animate-your-3d-plots-with-pythons-matplotlib/|Animate your 3D plots with Python’s Matplotlib]] | * [[https://zulko.wordpress.com/2012/09/29/animate-your-3d-plots-with-pythons-matplotlib/|Animate your 3D plots with Python’s Matplotlib]] | ||
* [[https://stackoverflow.com/questions/26796997/how-to-get-vertical-z-axis-in-3d-surface-plot-of-matplotlib|How to get vertical Z axis in 3D surface plot of Matplotlib?]] | * [[https://stackoverflow.com/questions/26796997/how-to-get-vertical-z-axis-in-3d-surface-plot-of-matplotlib|How to get vertical Z axis in 3D surface plot of Matplotlib?]] | ||
+ | |||
+ | ===== Data analysis ===== | ||
+ | |||
+ | ==== Easy to use datasets ==== | ||
+ | |||
+ | If you need standard datasets for testing, example, demos, ... | ||
+ | |||
+ | * [[https://scikit-learn.org/stable/datasets.html|Toy, real-world and generated datasets]] from [[#scikit-learn]] | ||
+ | * Example: [[https://lectures.scientific-python.org/packages/scikit-learn/index.html#a-simple-example-the-iris-dataset|using the 'iris' dataset]] | ||
+ | |||
+ | * [[https://scikit-image.org/docs/stable/api/skimage.data.html|Test images and datasets]] from [[#scikit-image]] | ||
+ | * Example: [[https://lectures.scientific-python.org/packages/scikit-image/index.html#data-types|Using the 'camera' dataset]] | ||
+ | ==== Pandas ==== | ||
+ | |||
+ | Summary: //pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool// | ||
+ | |||
+ | Where: [[http://pandas.pydata.org|Pandas web site]] | ||
+ | |||
+ | JYP's comment: pandas is supposed to be quite good for loading, processing and plotting time series, without writing custom code. It is **very convenient for processing tables in xlsx files** (or csv, etc...). You should at least have a quick look at: | ||
+ | |||
+ | * Some //Cheat Sheets//: | ||
+ | - Basics: [[https://github.com/fralfaro/DS-Cheat-Sheets/blob/main/docs/files/pandas_cs.pdf|Pandas Basics Cheat Sheet]] (associated with the [[https://www.datacamp.com/cheat-sheet/pandas-cheat-sheet-for-data-science-in-python#python-for-data-science-cheat-sheet:-pandas-basics-useth|Pandas basics]] //datacamp// introduction page) | ||
+ | - Intermediate: [[https://github.com/pandas-dev/pandas/blob/main/doc/cheatsheet/Pandas_Cheat_Sheet.pdf|Data Wrangling with pandas Cheat Sheet]] | ||
+ | * Some tutorials: | ||
+ | * [[http://pandas.pydata.org/docs/user_guide/10min.html|10 minutes to pandas]] | ||
+ | * The [[https://lectures.scientific-python.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines Pandas, [[#statsmodels|statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
+ | * More [[http://pandas.pydata.org/docs/getting_started/tutorials.html|Community tutorials]]... | ||
+ | |||
+ | |||
+ | ==== statsmodels ==== | ||
+ | |||
+ | [[https://www.statsmodels.org/|statsmodels]] is a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration. | ||
+ | |||
+ | Note: check the example in the [[https://lectures.scientific-python.org/packages/statistics/index.html|Statistics in Python]] tutorial | ||
+ | |||
+ | |||
+ | ==== scikit-learn ==== | ||
+ | |||
+ | [[http://scikit-learn.org/|scikit-learn]] is a Python library for machine learning, and is one of the most widely used tools for supervised and unsupervised machine learning. Scikit–learn provides an easy-to-use, consistent interface to a large collection of machine learning models, as well as tools for model evaluation and data preparation | ||
+ | |||
+ | Note: check the example in [[https://lectures.scientific-python.org/packages/scikit-learn/index.html|scikit-learn: machine learning in Python]] | ||
+ | ==== scikit-image ==== | ||
+ | |||
+ | [[https://scikit-image.org/|scikit-image]] is a collection of algorithms for image processing in Python | ||
+ | |||
+ | Note: check the example in [[https://lectures.scientific-python.org/packages/scikit-image/index.html|scikit-image: image processing]] | ||
+ | |||
===== Data file formats ===== | ===== Data file formats ===== | ||
Line 344: | Line 407: | ||
* [[https://github.com/LibraryOfCongress/bagger|Bagger]] (BagIt GUI) | * [[https://github.com/LibraryOfCongress/bagger|Bagger]] (BagIt GUI) | ||
* [[https://github.com/LibraryOfCongress/bagit-python|bagit-python]] | * [[https://github.com/LibraryOfCongress/bagit-python|bagit-python]] | ||
- | ===== Pandas ===== | ||
- | |||
- | Summary: //pandas is a library providing high-performance, easy-to-use data structures and data analysis tools// | ||
- | |||
- | Where: [[http://pandas.pydata.org|Pandas web site]] | ||
- | |||
- | JYP's comment: pandas is supposed to be quite good for loading, processing and plotting time series, without writing custom code. It is **very convenient for processing tables in xlsx files** (or csv, etc...). You should at least have a quick look at: | ||
- | |||
- | * Some //Cheat Sheets// (in the following order): | ||
- | - Basics: [[http://datacamp-community-prod.s3.amazonaws.com/dbed353d-2757-4617-8206-8767ab379ab3|Pandas basics]] (associated with the [[https://www.datacamp.com/community/blog/python-pandas-cheat-sheet|Pandas Cheat Sheet for Data Science in Python]] pandas introduction page) | ||
- | - Intermediate: [[https://github.com/pandas-dev/pandas/tree/master/doc/cheatsheet|github Pandas doc page]] | ||
- | - Advanced: the cheat sheet on the [[https://www.enthought.com/services/training/pandas-mastery-workshop/|Enthought workshops advertising page]] | ||
- | * Some tutorials: | ||
- | * [[https://www.datacamp.com/community/blog/python-pandas-cheat-sheet|Pandas Cheat Sheet for Data Science in Python]] pandas introduction page | ||
- | * The [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines Pandas, [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
- | |||
- | ===== Scipy Lecture Notes ===== | ||
- | |||
- | Summary: //One document to learn numerics, science, and data with Python// | ||
- | |||
- | Where: [[http://www.scipy-lectures.org/_downloads/ScipyLectures-simple.pdf|pdf]] - [[http://www.scipy-lectures.org/|html]] | ||
- | |||
- | This is **a really nice and useful document** that is regularly updated and used for the [[https://www.euroscipy.org/|EuroScipy]] tutorials. | ||
- | |||
- | This document will teach you even more things about python, numpy and matplotlib, debugging and optimizing scripts, and about using python for statistics, image processing, machine learning, washing dishes (this is just to check if you have read this page), etc... | ||
- | * Example: the [[http://www.scipy-lectures.org/packages/statistics/index.html|Statistics in Python]] tutorial that combines [[other:python:jyp_steps#pandas|Pandas]], [[http://statsmodels.sourceforge.net/|Statsmodels]] and [[http://seaborn.pydata.org/|Seaborn]] | ||
- | |||
- | ===== statsmodels ===== | ||
- | |||
- | [[https://www.statsmodels.org/|statsmodels ]] is a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration. | ||
- | |||
- | ===== scikit-learn ===== | ||
- | |||
- | [[http://scikit-learn.org/|scikit-learn]] is an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection and evaluation, and many other utilities. | ||
- | |||
- | ===== scikit-image ===== | ||
- | |||
- | [[https://scikit-image.org/|scikit-image]] is a collection of algorithms for image processing in Python | ||
===== Quick Reference and cheat sheets ===== | ===== Quick Reference and cheat sheets ===== |