| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| other:python:jyp_steps [2023/12/15 16:40] – [Using NetCDF files with Python] Rewrote the beginning of the section jypeter | other:python:jyp_steps [2025/02/26 11:40] (current) – [Extra numpy information] Added more links for indexing jypeter |
|---|
| ==== Part 2 ==== | ==== Part 2 ==== |
| |
| Once you have done your first steps, you should read [[http://www.lsce.ipsl.fr/Phocea/file.php?class=page&file=5/pythonCDAT_jyp_1sur2_070227.pdf|Plus loin avec Python]] (start at page 39, the previous pages are an old version of what was covered in //Part 1// above) | Once you have done your first steps, you should read {{:other:python:pythoncdat_jyp_1sur2_070227.pdf|Plus loin avec Python}} (start at page 39, the previous pages are an old version of what was covered in //Part 1// above) |
| * this tutorial is in French (sorry again) | * this tutorial is in French (sorry again) |
| * after reading this tutorial, you will be able to do more than you can do in a shell script, in an easier way | * after reading this tutorial, you will be able to do more than you can do in a shell script, in an easier way |
| * calling external programs | * calling external programs |
| |
| | You can also look at the [[other:python:misc_by_jyp|Useful python stuff]] page |
| ===== The official python documentation ===== | ===== The official python documentation ===== |
| |
| * {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path' | * {{ :other:python:indirect_indexing_2.py.txt |}}: Take a vertical slice in a 3D zyx array, along a varying y 'path' |
| * [[https://numpy.org/doc/stable/user/basics.indexing.html|Array indexing basics (user guide)]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...) | * [[https://numpy.org/doc/stable/user/basics.indexing.html|Array indexing basics (user guide)]] (//index arrays//, //boolean index arrays//, //np.newaxis//, //Ellipsis//, //variable numbers of indices//, ...) |
| | * [[https://numpy.org/doc/stable/user/basics.indexing.html#dealing-with-variable-numbers-of-indices-within-programs|Dealing with variable numbers of indices within programs]] |
| | * [[https://numpy.org/doc/stable/user/basics.indexing.html#field-access|Field access]] |
| | * [[https://numpy.org/doc/stable/user/basics.rec.html#indexing-and-assignment-to-structured-arrays|Indexing and assignment to structured arrays]] |
| * [[https://numpy.org/doc/stable/reference/arrays.indexing.html|Indexing routines (reference manual)]] | * [[https://numpy.org/doc/stable/reference/arrays.indexing.html|Indexing routines (reference manual)]] |
| * [[https://numpy.org/doc/stable/user/quickstart.html#advanced-indexing-and-index-tricks|Advanced indexing and index tricks]] and [[https://numpy.org/doc/stable/user/quickstart.html#the-ix-function|the ix_() function]] | * [[https://numpy.org/doc/stable/user/quickstart.html#advanced-indexing-and-index-tricks|Advanced indexing and index tricks]] and [[https://numpy.org/doc/stable/user/quickstart.html#the-ix-function|the ix_() function]] |
| |
| ===== Using NetCDF files with Python ===== | ===== Using NetCDF files with Python ===== |
| |
| <note tip>People using CMIPn and model data on the IPSL servers can easily search and process NetCDF files using: | |
| * the [[https://climaf.readthedocs.io/|Climate Model Assessment Framework (CliMAF)]] environment | |
| * and the [[https://github.com/jservonnat/C-ESM-EP/wiki|CliMAF Earth System Evaluation Platform (C-ESM-EP)]] | |
| </note> | |
| |
| |
| |
| * There may be different ways of dealing with NetCDF files, depending on which [[other:python:starting#some_python_distributions|python distribution]] you have access to | * There may be different ways of dealing with NetCDF files, depending on which [[other:python:starting#some_python_distributions|python distribution]] you have access to |
| | |
| | |
| | ==== CliMAF and C-ESM-EP ==== |
| | |
| | People using **//CMIPn// and model data on the IPSL servers** can easily search and process NetCDF files using: |
| | |
| | * the [[https://climaf.readthedocs.io/|Climate Model Assessment Framework (CliMAF)]] environment |
| | |
| | * and the [[https://github.com/jservonnat/C-ESM-EP/wiki|CliMAF Earth System Evaluation Platform (C-ESM-EP)]] |
| | |
| | |
| ==== xarray ==== | ==== xarray ==== |
| |
| Summary: [[https://docs.xarray.dev/|xarray]] makes working with labelled multi-dimensional arrays in Python simple, efficient, and fun! [...] It is particularly tailored to working with netCDF files | [[https://docs.xarray.dev/|xarray]] makes working with labelled multi-dimensional arrays in Python simple, efficient, and fun! [...] It is particularly tailored to working with netCDF files |
| |
| === Some xarray related resources === | === Some xarray related resources === |
| |
| * [[https://uxarray.readthedocs.io/|uxarray]]: provide xarray styled functionality for unstructured grid datasets following [[https://ugrid-conventions.github.io/ugrid-conventions/|UGRID Conventions]] | * [[https://uxarray.readthedocs.io/|uxarray]]: provide xarray styled functionality for unstructured grid datasets following [[https://ugrid-conventions.github.io/ugrid-conventions/|UGRID Conventions]] |
| | |
| | |
| | ==== netCDF4 ==== |
| | |
| | [[http://unidata.github.io/netcdf4-python/|netCDF4]] is a Python interface to the netCDF C library |
| |
| |
| ==== cdms2 ==== | ==== cdms2 ==== |
| |
| <note important>''cdms2'' is unfortunately not maintained anymore and is slowly being **phased out in favor of a combination of [[#xarray|xarray]] and [[https://xcdat.readthedocs.io/|xCDAT]]**</note> | <note important> |
| | * ''cdms2'' is unfortunately not maintained anymore and has been slowly **phased out in favor of a combination of [[#xarray|xarray]] and [[https://xcdat.readthedocs.io/|xCDAT]]** |
| |
| Summary: cdms2 can read/write netCDF files (and read //grads// dat+ctl files) and provides a higher level interface than netCDF4. cdms2 is available in the [[other:python:starting#cdat|CDAT distribution]], and can theoretically be installed independently of CDAT (e.g. it will be installed when you install [[https://cmor.llnl.gov/mydoc_cmor3_conda/|CMOR in conda)]]. When you can use cdms2, you also have access to //cdtime//, that is very useful for handling time axis data. | * ''cdms2'' is [[https://github.com/CDAT/cdms/issues/449|not compatible with numpy after numpy 1.23.5]] :-( |
| | </note> |
| | |
| | [[https://cdms.readthedocs.io/en/docstanya/|cdms2]] can read/write netCDF files (and read //grads// dat+ctl files) and provides a higher level interface than netCDF4. ''cdms2'' is available in the [[other:python:starting#cdat|CDAT distribution]], and can theoretically be installed independently of CDAT (e.g. it will be installed when you install [[https://cmor.llnl.gov/mydoc_cmor3_conda/|CMOR in conda)]]. When you can use cdms2, you also have access to //cdtime//, that is very useful for handling time axis data. |
| |
| How to get started: | How to get started: |
| - read [[http://www.lsce.ipsl.fr/Phocea/file.php?class=page&file=5/pythonCDAT_jyp_2sur2_070306.pdf|JYP's cdms tutorial]], starting at page 54 | - read {{:other:python:pythoncdat_jyp_2sur2_070306.pdf|JYP's cdms tutorial}}, starting at page 54 |
| - the tutorial is in French (soooorry!) | - the tutorial is in French (soooorry!) |
| - you have to replace //cdms// with **cdms2**, and //MV// with **MV2** (sooorry about that, the tutorial was written when CDAT was based on //Numeric// instead of //numpy// to handle array data) | - you have to replace //cdms// with **cdms2**, and //MV// with **MV2** (sooorry about that, the tutorial was written when CDAT was based on //Numeric// instead of //numpy// to handle array data) |
| - read the [[http://cdms.readthedocs.io/en/docstanya/index.html|official cdms documentation]] (link may change) | - read the [[http://cdms.readthedocs.io/en/docstanya/index.html|official cdms documentation]] (link may change) |
| |
| |
| ==== netCDF4 ==== | |
| |
| Summary: //netCDF4 can read/write netCDF files and is available in most python distributions// | |
| |
| Where: [[http://unidata.github.io/netcdf4-python/]] | |
| |
| ===== CDAT-related resources ===== | |
| |
| Some links, in case they can't be found easily on the [[https://cdat.llnl.gov|CDAT]] web site... | |
| |
| * [[https://cdat.llnl.gov/tutorials.html|Tutorials in ipython notebooks]] | |
| * [[http://cdat-vcs.readthedocs.io/en/latest/|VCS: Visualization Control System]] | |
| * [[https://github.com/CDAT/vcs/issues/238|Colormaps in vcs examples]] | |
| * [[https://github.com/CDAT/cdat-site/blob/master/eztemplate.md|EzTemplate Documentation]] | |
| |
| ===== Matplotlib ===== | ===== Matplotlib ===== |
| * [[https://github.com/LibraryOfCongress/bagger|Bagger]] (BagIt GUI) | * [[https://github.com/LibraryOfCongress/bagger|Bagger]] (BagIt GUI) |
| * [[https://github.com/LibraryOfCongress/bagit-python|bagit-python]] | * [[https://github.com/LibraryOfCongress/bagit-python|bagit-python]] |
| | |
| | ==== Protocol Buffers ==== |
| | |
| | //Protocol Buffers are (Google's) language-neutral, platform-neutral extensible mechanisms for serializing structured data// |
| | |
| | * https://protobuf.dev/ |
| | * [[https://protobuf.dev/getting-started/pythontutorial/|Protocol Buffer Basics: Python]] |
| | * ''mamba install protobuf'' |
| |
| ===== Quick Reference and cheat sheets ===== | ===== Quick Reference and cheat sheets ===== |
| * [[https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/pdf_bw/|Jupyter Notebook Keyboard Shortcuts]] | * [[https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/pdf_bw/|Jupyter Notebook Keyboard Shortcuts]] |
| |
| ===== Miscellaneous Python stuff ===== | ===== Miscellaneous Python stuff and tutorials ===== |
| |
| Check the page about [[other:python:misc_by_jyp|useful python stuff that has not been sorted yet]] | Check the page about [[other:python:misc_by_jyp|useful python stuff that has not been sorted yet]] |
| |
| ===== Misc tutorials ===== | |
| |
| * [[https://pyformat.info/|PyFormat]]: //With this site we try to show you the most common use-cases covered by the old and new style string formatting API with practical examples// | |
| ===== Some good coding tips ===== | ===== Some good coding tips ===== |
| |
| |
| You can do a lot more with python! But if you have read at least a part of this page, you should be able to find and use the modules you need. Make sure you do not reinvent the wheel! Use existing packages when possible, and make sure to report bugs or errors in the documentations when you find some | You can do a lot more with python! But if you have read at least a part of this page, you should be able to find and use the modules you need. Make sure you do not reinvent the wheel! Use existing packages when possible, and make sure to report bugs or errors in the documentations when you find some |
| | |
| | |
| | ===== Out-of-date stuff ===== |
| | |
| | |
| | ==== CDAT-related resources ==== |
| | |
| | Some links, in case they can't be found easily on the [[https://cdat.llnl.gov|CDAT]] web site... |
| | |
| | * [[https://cdat.llnl.gov/tutorials.html|Tutorials in ipython notebooks]] |
| | * [[http://cdat-vcs.readthedocs.io/en/latest/|VCS: Visualization Control System]] |
| | * [[https://github.com/CDAT/vcs/issues/238|Colormaps in vcs examples]] |
| | * [[https://github.com/CDAT/cdat-site/blob/master/eztemplate.md|EzTemplate Documentation]] |
| | |
| |
| /* standard page footer */ | /* standard page footer */ |
