other:python:misc_by_jyp
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
other:python:misc_by_jyp [2023/09/27 13:53] jypeter Added a section with a link to "100 numpy exercises" |
other:python:misc_by_jyp [2024/04/19 12:02] (current) jypeter [Data representation] Corrected link to an old JYP tutorial |
||
|---|---|---|---|
| Line 5: | Line 5: | ||
| </WRAP> | </WRAP> | ||
| + | ===== Extra tutorials ===== | ||
| + | Only **when you have already read all the content of this page several times**, and you are looking for new ideas | ||
| + | |||
| + | * [[https://medium.com/@yaduvanshineelam09/ultimate-python-cheat-sheet-practical-python-for-everyday-tasks-8a33abc0892f|Ultimate Python Cheat Sheet: Practical Python For Everyday Tasks]] | ||
| ===== Reading/setting environments variables ===== | ===== Reading/setting environments variables ===== | ||
| Line 65: | Line 69: | ||
| ==== Working with paths and filenames ==== | ==== Working with paths and filenames ==== | ||
| - | If you are in a hurry, you can just use string functions to work with path and file names. But you will need some specific functions to check if a file exists, and similar operations. All these are available in 2 libraries that have similar functions. Both of these libraries can deal with Unix-type paths on Linux computers, and Windows-type paths on Windows computers | + | If you are in a hurry, you can just use string functions to work with paths and file names. |
| - | * [[https://docs.python.org/3/library/os.path.html|os.path]] //Common pathname manipulations// | + | |
| + | You will need some specific objects and functions to check if a file exists, and similar operations. Check the libraries listed below, that can automatically deal with Unix-type paths on Linux and MacOS computers, and Windows-type paths on Windows computers | ||
| + | |||
| + | * [[https://docs.python.org/3/library/os.path.html|os.path]]: //common pathname manipulations// | ||
| * Available since... a long time! Use this if you want to avoid backward compatibility problems | * Available since... a long time! Use this if you want to avoid backward compatibility problems | ||
| * Some functions are directly in [[https://docs.python.org/3/library/os.html|os]] //Miscellaneous operating system interfaces//\\ e.g. [[https://docs.python.org/3/library/os.html#os.remove|os.remove]] and [[https://docs.python.org/3/library/os.html#os.rmdir|os.rmdir]] | * Some functions are directly in [[https://docs.python.org/3/library/os.html|os]] //Miscellaneous operating system interfaces//\\ e.g. [[https://docs.python.org/3/library/os.html#os.remove|os.remove]] and [[https://docs.python.org/3/library/os.html#os.rmdir|os.rmdir]] | ||
| - | * [[https://docs.python.org/3/library/pathlib.html|pathlib]] //Object-oriented filesystem paths// | + | * [[https://docs.python.org/3/library/pathlib.html|pathlib]]: a **more recent** //object-oriented// way to deal with //filesystem paths// |
| * Available since Python version 3.4 | * Available since Python version 3.4 | ||
| * [[https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module|Matching pathlib, and os or os.path functions]] | * [[https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module|Matching pathlib, and os or os.path functions]] | ||
| - | * [[https://docs.python.org/3/library/shutil.html|High-level file operations]] | + | * [[https://docs.python.org/3/library/shutil.html|shutil]]: High-level file operations, e.g copy/move a file or directory tree |
| - | === Example: getting the full path of the Python used === | + | === Example: getting the full path of the Python executable used === |
| Note: the actual python may be different from the default python! | Note: the actual python may be different from the default python! | ||
| Line 83: | Line 90: | ||
| /usr/bin/python | /usr/bin/python | ||
| - | $ /modfs/modtools/miniconda3//envs/analyse_3.6_test/bin/python | + | $ /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python |
| >>> import sys, shutil | >>> import sys, shutil | ||
| >>> shutil.which('python') | >>> shutil.which('python') | ||
| '/usr/bin/python' | '/usr/bin/python' | ||
| >>> sys.executable | >>> sys.executable | ||
| - | '/modfs/modtools/miniconda3//envs/analyse_3.6_test/bin/python'</code> | + | '/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python'</code> |
| Line 104: | Line 111: | ||
| </code> | </code> | ||
| + | |||
| + | === Example: system independent paths with pathlib === | ||
| + | |||
| + | Note: the following example was generated on a Linux server and uses a <wrap em>/</wrap> character as a path separator | ||
| + | |||
| + | <code>>>> my_home = Path.home() | ||
| + | >>> my_home | ||
| + | PosixPath('/home/users/my_login') | ||
| + | >>> my_conf = my_home / '.config' / 'evince' | ||
| + | >>> my_conf | ||
| + | PosixPath('/home/users/my_login/.config/evince') | ||
| + | >>> my_conf.is_dir() | ||
| + | True | ||
| + | >>> my_conf.is_file() | ||
| + | False | ||
| + | >>> list(my_conf.glob('*')) | ||
| + | [PosixPath('/home/users/my_login/.config/evince/evince_toolbar.xml'), PosixPath(' /home/users/my_login/.config/evince/accels')] | ||
| + | >>> [ ff.name for ff in my_conf.glob('*') ] | ||
| + | ['evince_toolbar.xml', 'accels'] | ||
| + | </code> | ||
| === Example: getting the size(s) of all the files in a directory === | === Example: getting the size(s) of all the files in a directory === | ||
| Line 290: | Line 317: | ||
| ['c', 'd', 'b', 'a']</code> | ['c', 'd', 'b', 'a']</code> | ||
| + | |||
| + | ===== Efficient looping with numpy, map, itertools and list comprehension ===== | ||
| + | |||
| + | <wrap hi>Big, nested, explicit ''for'' loops should be avoided at all cost</wrap>, in order to reduce a script execution time! | ||
| + | |||
| + | * **''numpy'' arrays** should be used when dealing with //numerical data// | ||
| + | * **Masked arrays** can be used to deal with //special cases// and remove tests from loops | ||
| + | |||
| + | * The built-in [[https://docs.python.org/3/library/functions.html?highlight=map#map|map]] function (and similar functions like [[https://docs.python.org/3/library/functions.html?highlight=zip#zip|zip]], [[https://docs.python.org/3/library/functions.html?highlight=filter#filter|filter]], ...) can be used to efficiently apply a function (possibly a //simple// [[https://docs.python.org/3/tutorial/controlflow.html#lambda-expressions|lambda]] function) to all the elements of a list | ||
| + | * <code>>>> my_ints = [1, 2, 3] | ||
| + | |||
| + | >>> map(str, my_ints) | ||
| + | ['1', '2', '3'] | ||
| + | |||
| + | >>> map(lambda ii: str(10*ii + 5), my_ints) | ||
| + | ['15', '25', '35']</code> | ||
| + | |||
| + | * The [[https://docs.python.org/3/library/itertools.html|itertools]] module defines many more fancy iterators that can be used for efficient looping | ||
| + | * Example: replacing nested loops with [[https://docs.python.org/3/library/itertools.html#itertools.product|product]] | ||
| + | * <code>>>> it.product('AB', '01') | ||
| + | <itertools.product object at 0x2b35a7b5f100> | ||
| + | |||
| + | >>> list(it.product('AB', '01')) | ||
| + | [('A', '0'), ('A', '1'), ('B', '0'), ('B', '1')] | ||
| + | |||
| + | >>> for c1, c2 in it.product('AB', '01'): | ||
| + | ... print(c1 + c2) | ||
| + | ... | ||
| + | A0 | ||
| + | A1 | ||
| + | B0 | ||
| + | B1 | ||
| + | |||
| + | >>> for c1, c2 in it.product(['A', 'B'], ['0', '1']): | ||
| + | ... print(c1 + c2) | ||
| + | ... | ||
| + | A0 | ||
| + | A1 | ||
| + | B0 | ||
| + | B1 | ||
| + | |||
| + | >>> for c1, c2, c3 in it.product('AB', '01', '$!'): | ||
| + | ... print(c1 + c2 + c3, end=', ') | ||
| + | ... | ||
| + | A0$, A0!, A1$, A1!, B0$, B0!, B1$, B1!,</code> | ||
| + | |||
| + | * The [[https://docs.python.org/3/tutorial/datastructures.html?highlight=comprehension#list-comprehensions|list comprehension]] (aka //implicit loops//) can also be used to generate lists from lists | ||
| + | * Example: converting a list of integers to a list of strings\\ Note: in that case, you should rather use the ''map'' function detailed above | ||
| + | * <code>>>> my_ints = [1, 2, 3] | ||
| + | |||
| + | >>> [ str(ii) for ii in my_ints ] | ||
| + | ['1', '2', '3']</code> | ||
| ===== numpy related stuff ===== | ===== numpy related stuff ===== | ||
| Line 469: | Line 548: | ||
| A few notes for a future section or page about about //data representation// (bits and bytes) on disk and in memory, vs //data format// | A few notes for a future section or page about about //data representation// (bits and bytes) on disk and in memory, vs //data format// | ||
| - | FIXME Add parts (pages 28 to 37) of this [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:jyp_steps#part_2|old tutorial]] to this section | + | FIXME Add parts (pages 28 to 37) of this [[http://www.lsce.ipsl.fr/Phocea/file.php?class=page&file=5/pythonCDAT_jyp_2sur2_070306.pdf|old tutorial]] to this section |
| ==== Base notions ==== | ==== Base notions ==== | ||
other/python/misc_by_jyp.1695822811.txt.gz · Last modified: 2023/09/27 13:53 by jypeter
