Differences

This shows you the differences between two versions of the page.

--- other:python:misc_by_jyp [2023/09/27 13:53]
jypeter Added a section with a link to "100 numpy exercises"
+++ other:python:misc_by_jyp [2023/12/08 15:36]
jypeter Added the efficient looping section
@@ Line 65: / Line 65: @@
 ==== Working with paths and filenames ====
-If you are in a hurry, you can just use string functions to work with path and file names. But you will need some specific functions to check if a file exists, and similar operations. All these are available in 2 libraries that have similar functions. Both of these libraries can deal with Unix-type paths on Linux computers, and Windows-type paths on Windows computers
+If you are in a hurry, you can just use string functions to work with paths and file names.
-  * [[https://docs.python.org/3/library/os.path.html|os.path]] //Common pathname manipulations//
+You will need some specific objects and functions to check if a file exists, and similar operations. Check the libraries listed below, that can automatically deal with Unix-type paths on Linux and MacOS computers, and Windows-type paths on Windows computers
+  * [[https://docs.python.org/3/library/os.path.html|os.path]]: //common pathname manipulations//
     * Available since... a long time! Use this if you want to avoid backward compatibility problems
     * Some functions are directly in [[https://docs.python.org/3/library/os.html|os]] //Miscellaneous operating system interfaces//\\ e.g. [[https://docs.python.org/3/library/os.html#os.remove|os.remove]] and [[https://docs.python.org/3/library/os.html#os.rmdir|os.rmdir]]
-  * [[https://docs.python.org/3/library/pathlib.html|pathlib]] //Object-oriented filesystem paths//
+  * [[https://docs.python.org/3/library/pathlib.html|pathlib]]: a **more recent** //object-oriented// way to deal with //filesystem paths//
     * Available since Python version 3.4
     * [[https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module|Matching pathlib, and os or os.path functions]]
-  * [[https://docs.python.org/3/library/shutil.html|High-level file operations]]
+  * [[https://docs.python.org/3/library/shutil.html|shutil]]: High-level file operations, e.g copy/move a file or directory tree
-=== Example: getting the full path of the Python used ===
+=== Example: getting the full path of the Python executable used ===
 Note: the actual python may be different from the default python!
@@ Line 83: / Line 86: @@
 /usr/bin/python
-$ /modfs/modtools/miniconda3//envs/analyse_3.6_test/bin/python
+$ /home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python
 >>> import sys, shutil
 >>> shutil.which('python')
 '/usr/bin/python'
 >>> sys.executable
-'/modfs/modtools/miniconda3//envs/analyse_3.6_test/bin/python'</code>
+'/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm_py3/bin/python'</code>
@@ Line 104: / Line 107: @@
 </code>
+=== Example: system independent paths with pathlib ===
+Note: the following example was generated on a Linux server and uses a <wrap em>/</wrap> character as a path separator
+<code>>>> my_home = Path.home()
+>>> my_home
+PosixPath('/home/users/my_login')
+>>> my_conf = my_home / '.config' / 'evince'
+>>> my_conf
+PosixPath('/home/users/my_login/.config/evince')
+>>> my_conf.is_dir()
+True
+>>> my_conf.is_file()
+False
+>>> list(my_conf.glob('*'))
+[PosixPath('/home/users/my_login/.config/evince/evince_toolbar.xml'), PosixPath(' /home/users/my_login/.config/evince/accels')]
+>>> [ ff.name for ff in my_conf.glob('*') ]
+['evince_toolbar.xml', 'accels']
+</code>
 === Example: getting the size(s) of all the files in a directory ===
@@ Line 290: / Line 313: @@
 ['c', 'd', 'b', 'a']</code>
+===== Efficient looping with numpy, map and itertools =====
+<wrap hi>Big, nested, explicit loops should be avoided at all cost</wrap>, in order to reduce a script execution time!
+  * **''numpy'' arrays** should be used when dealing with //numerical data//
+    * **Masked arrays** can be used to deal with //special cases// and remove tests from loops
+  * The built-in [[https://docs.python.org/3/library/functions.html?highlight=map#map|map]] function (and similar functions like [[https://docs.python.org/3/library/functions.html?highlight=zip#zip|zip]], [[https://docs.python.org/3/library/functions.html?highlight=filter#filter|filter]], ...) can be used to efficiently apply a function (possibly a //simple// [[https://docs.python.org/3/tutorial/controlflow.html#lambda-expressions|lambda]] function) to all the elements of a list
+    * <code>>>> my_ints = [1, 2, 3]
+>>> map(str, my_ints)
+['1', '2', '3']
+>>> map(lambda ii: str(10*ii + 5), my_ints)
+['15', '25', '35']</code>
+  * The [[https://docs.python.org/3/library/itertools.html|itertools]] module defines many more fancy iterators that can be used for efficient looping
+    * Example: replacing nested loops with [[https://docs.python.org/3/library/itertools.html#itertools.product|product]]
+      * <code>>>> it.product('AB', '01')
+<itertools.product object at 0x2b35a7b5f100>
+>>> list(it.product('AB', '01'))
+[('A', '0'), ('A', '1'), ('B', '0'), ('B', '1')]
+>>> for c1, c2 in it.product('AB', '01'):
+...   print(c1 + c2)
+...
+A0
+A1
+B0
+B1
+>>> for c1, c2 in it.product(['A', 'B'], ['0', '1']):
+...   print(c1 + c2)
+...
+A0
+A1
+B0
+B1
+>>> for c1, c2, c3 in it.product('AB', '01', '$!'):
+...   print(c1 + c2 + c3, end=', ')
+...
+A0$, A0!, A1$, A1!, B0$, B0!, B1$, B1!,</code>
 ===== numpy related stuff =====

PMIP3 wiki

User Tools

Site Tools

Differences

Page Tools