This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
other:python:jyp_steps [2023/12/15 10:25] jypeter [YData Profiling] Added D-Tale |
other:python:jyp_steps [2023/12/15 14:45] jypeter [Data file formats] Added link to the NetCDF section |
||
---|---|---|---|
Line 317: | Line 317: | ||
===== Data analysis ===== | ===== Data analysis ===== | ||
+ | |||
+ | ==== EDA (Exploratory Data Analysis) ? ==== | ||
+ | |||
+ | <note tip> | ||
+ | The //EDA concept// seems to apply to **time series** (and tabular data), which is not exactly the case of full climate model output data</note> | ||
+ | |||
+ | * [[https://www.geeksforgeeks.org/what-is-exploratory-data-analysis/|What is Exploratory Data Analysis ?]] | ||
+ | * //The method of studying and exploring record sets to apprehend their predominant traits, discover patterns, locate outliers, and identify relationships between variables. EDA is normally carried out as a preliminary step before undertaking extra formal statistical analyses or modeling.// | ||
+ | |||
+ | * [[https://medium.com/codex/automate-the-exploratory-data-analysis-eda-to-understand-the-data-faster-not-better-2ed6ff230eed|Automate the exploratory data analysis (EDA) to understand the data faster and easier]]: a nice comparison of some Python libraries listed below ([[#ydata_profiling|YData Profiling]], [[#d-tale|D-Tale]], [[#sweetviz|sweetviz]], [[#autoviz|AutoViz]]) | ||
+ | |||
+ | * [[https://www.geeksforgeeks.org/exploratory-data-analysis-in-python/|EDA in Python]] | ||
+ | |||
==== Easy to use datasets ==== | ==== Easy to use datasets ==== | ||
Line 337: | Line 350: | ||
* [[https://github.com/xCDAT/xcdat/issues/277|xCDAT test data GH discussion]] | * [[https://github.com/xCDAT/xcdat/issues/277|xCDAT test data GH discussion]] | ||
+ | |||
+ | |||
==== Pandas ==== | ==== Pandas ==== | ||
Line 359: | Line 374: | ||
Note: check the example in the [[https://lectures.scientific-python.org/packages/statistics/index.html|Statistics in Python]] tutorial | Note: check the example in the [[https://lectures.scientific-python.org/packages/statistics/index.html|Statistics in Python]] tutorial | ||
+ | |||
+ | |||
+ | ==== scikit-learn ==== | ||
+ | |||
+ | [[http://scikit-learn.org/|scikit-learn]] is a Python library for machine learning, and is one of the most widely used tools for supervised and unsupervised machine learning. Scikit–learn provides an easy-to-use, consistent interface to a large collection of machine learning models, as well as tools for model evaluation and data preparation | ||
+ | |||
+ | Note: check the example in [[https://lectures.scientific-python.org/packages/scikit-learn/index.html|scikit-learn: machine learning in Python]] | ||
+ | |||
+ | |||
+ | ==== scikit-image ==== | ||
+ | |||
+ | [[https://scikit-image.org/|scikit-image]] is a collection of algorithms for image processing in Python | ||
+ | |||
+ | Note: check the example in [[https://lectures.scientific-python.org/packages/scikit-image/index.html|scikit-image: image processing]] | ||
Line 369: | Line 398: | ||
[[https://github.com/man-group/dtale|D-Tale]] brings you an easy way to view & analyze Pandas data structures. It integrates seamlessly with ipython notebooks & python/ipython terminals. | [[https://github.com/man-group/dtale|D-Tale]] brings you an easy way to view & analyze Pandas data structures. It integrates seamlessly with ipython notebooks & python/ipython terminals. | ||
- | ==== scikit-learn ==== | ||
- | [[http://scikit-learn.org/|scikit-learn]] is a Python library for machine learning, and is one of the most widely used tools for supervised and unsupervised machine learning. Scikit–learn provides an easy-to-use, consistent interface to a large collection of machine learning models, as well as tools for model evaluation and data preparation | ||
- | Note: check the example in [[https://lectures.scientific-python.org/packages/scikit-learn/index.html|scikit-learn: machine learning in Python]] | + | ==== Sweetviz ==== |
- | ==== scikit-image ==== | + | |
- | [[https://scikit-image.org/|scikit-image]] is a collection of algorithms for image processing in Python | + | [[https://github.com/fbdesignpro/sweetviz|Sweetviz]] is pandas based Python library that generates beautiful, high-density visualizations to kickstart EDA (Exploratory Data Analysis) with just two lines of code. |
- | Note: check the example in [[https://lectures.scientific-python.org/packages/scikit-image/index.html|scikit-image: image processing]] | + | |
+ | ==== AutoViz ==== | ||
+ | |||
+ | [[https://github.com/AutoViML/AutoViz|AutoViz]]: the One-Line Automatic Data Visualization Library. Automatically Visualize any dataset, any size with a single line of code | ||
===== Data file formats ===== | ===== Data file formats ===== | ||
- | We list here some resources about non-NetCDF data formats that can be useful | + | * We list below some resources about **non-NetCDF data formats** that can be useful |
+ | |||
+ | * Check the [[#using_netcdf_files_with_python|Using NetCDF files with Python]] section otherwise | ||
==== The shelve package ==== | ==== The shelve package ==== |