User Tools

Site Tools


other:python:misc_by_jyp

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
Next revision Both sides next revision
other:python:misc_by_jyp [2021/06/30 13:17]
jypeter created
other:python:misc_by_jyp [2021/09/17 14:05]
jypeter [Printing a readable version of long lists or dictionaries] Added long list example
Line 2: Line 2:
  
 <WRAP center round tip 60%> <WRAP center round tip 60%>
-You will find on this page some useful, but unsorted, python tips and tricks that can't fit in a section of the main [[other:​python:​jyp_steps|JYP'​s recommended steps for learning python]] page+You will find on this page some **useful, but unsorted, python tips and tricks** that can't fit in a section of the main [[other:​python:​jyp_steps|JYP'​s recommended steps for learning python]] page
 </​WRAP>​ </​WRAP>​
  
- ​* ​Reading/​setting environments variables\\ <​code>>>>​ os.environ['​TMPDIR'​]+==== Reading/​setting environments variables ​==== 
 + 
 + 
 +<​code>>>>​ os.environ['​TMPDIR'​]
 '/​data/​jypmce/​climafcache'​ '/​data/​jypmce/​climafcache'​
 >>>​ os.environ.get('​SCRATCHDIR',​ '/​data/​jypmce/​some_scratch_stuff'​) >>>​ os.environ.get('​SCRATCHDIR',​ '/​data/​jypmce/​some_scratch_stuff'​)
Line 14: Line 17:
 </​code>​ </​code>​
  
- ​* ​Generating (aka //raising//) an errorThis will stop the script, unless it is called in a function, and the code calling the function explicitely catches and deals with errors+==== Generating (aka raising) an error ==== 
 + 
 +This will stop the script, unless it is called in a function, and the code calling the function explicitely catches and deals with errors
     * <​code>​raise RuntimeError('​\n\nOMG! An error! :​-(\nAborting script...'​)</​code>​     * <​code>​raise RuntimeError('​\n\nOMG! An error! :​-(\nAborting script...'​)</​code>​
     * [[https://​docs.python.org/​3/​tutorial/​errors.html|Errors and Exceptions tutorial]]     * [[https://​docs.python.org/​3/​tutorial/​errors.html|Errors and Exceptions tutorial]]
     * [[https://​docs.python.org/​3/​library/​exceptions.html|Built-in Exceptions reference]]     * [[https://​docs.python.org/​3/​library/​exceptions.html|Built-in Exceptions reference]]
  
- ​* ​Stopping a script\\ <​code>​sys.exit('​Some optional message about why we are stopping'​)</​code>​+ 
 +==== Stopping a script ​==== 
 + 
 +A user can use ''​CTRL-C''​ or ''​kill''​ to stop a script, or ''​CTRL-Z''​ to suspend it temporarily (use ''​fg''​ to resume a suspended script). The code below can be used by the script itself to interrupt its execution, instead of raising an error 
 + 
 +<​code>​sys.exit('​Some optional message about why we are stopping'​)</​code>​ 
 + 
 + 
 +==== Checking if a file/​directory is writable by the current user ==== 
 + 
 +<​code>>>>​ os.access('/',​ os.W_OK) 
 +False 
 +>>>​ os.access('/​home/​jypmce/​.bashrc',​ os.W_OK) 
 +True</​code>​ 
 + 
 +==== Working with paths and filenames ==== 
 + 
 +If you are in a hurry, you can just use string functions to work with path and file names. But you will need some specific functions to check if a file exists, and similar operations. All these are available in 2 libraries that have similar functions. Both of these libraries can deal with Unix-type paths on Linux computers, and Windows-type paths on Windows computers 
 + 
 +  * [[https://​docs.python.org/​3/​library/​os.path.html|os.path]] //Common pathname manipulations//​ 
 +    * Available since... a long time! Use this if you want to avoid backward compatibility problems 
 +    * Some functions are directly in [[https://​docs.python.org/​3/​library/​os.html|os]] //​Miscellaneous operating system interfaces//​\\ e.g. [[https://​docs.python.org/​3/​library/​os.html#​os.remove|os.remove]] and [[https://​docs.python.org/​3/​library/​os.html#​os.rmdir|os.rmdir]] 
 +  * [[https://​docs.python.org/​3/​library/​pathlib.html|pathlib]] //​Object-oriented filesystem paths// 
 +    * Available since Python version 3.4 
 +    * [[https://​docs.python.org/​3/​library/​pathlib.html#​correspondence-to-tools-in-the-os-module|Matching pathlib, and os or os.path functions]] 
 +  * [[https://​docs.python.org/​3/​library/​shutil.html|High-level file operations]] 
 + 
 + 
 +=== Example: getting the full path of the Python used === 
 + 
 +Note: the actual python may be different from the default python! 
 + 
 +<​code>​$ which python 
 +/​usr/​bin/​python 
 + 
 +$ /​modfs/​modtools/​miniconda3//​envs/​analyse_3.6_test/​bin/​python 
 +>>>​ import sys, shutil 
 +>>>​ shutil.which('​python'​) 
 +'/​usr/​bin/​python'​ 
 +>>>​ sys.executable 
 +'/​modfs/​modtools/​miniconda3//​envs/​analyse_3.6_test/​bin/​python'</​code>​ 
 + 
 + 
 +=== Example: getting the full path of a script === 
 + 
 +<​code>>>>​ import os 
 +>>>​ os.getcwd() 
 +'/​home/​jypmce/​PMIP4'​ 
 +>>>​ os.path.exists('​./​argv_test.py'​) 
 +True 
 +>>>​ os.path.abspath('​./​argv_test.py'​) 
 +'/​home/​jypmce/​PMIP4/​argv_test.py'​ 
 +>>>​ os.path.exists('/​home/​jypmce/​PMIP4/​argv_test.py'​) 
 +True 
 +</​code>​ 
 + 
 + 
 +=== Example: getting the size(s) of all the files in a directory === 
 + 
 +<​code>​$ cd /​data/​jypmce/​TestDir 
 +$ ls -l 
 +total 72 
 +-rw-r--r-- 1 jypmce ipsl 18147 Jun 25  2012 get_TS_cmip5.py 
 +-rw-r--r-- 1 jypmce ipsl 16152 Jun 21  2012 get_TS_cmip5.py~ 
 +-rw-r--r-- 1 jypmce ipsl 13954 Jul  3  2012 get_TS_cmip5_regular.py 
 +-rw-r--r-- 1 jypmce ipsl 16539 Jun 22  2012 get_TS_cmip5_regular.py~</​code>​ 
 + 
 +<​code>>>>​ os.chdir('/​data/​jypmce/​TestDir'​) 
 +>>>​ print(os.getcwd()) 
 +/​data/​jypmce/​TestDir 
 +>>>​ files_list = os.listdir() 
 +>>>​ files_list 
 +['​get_TS_cmip5.py~',​ '​get_TS_cmip5_regular.py',​ '​get_TS_cmip5_regular.py~',​ '​get_TS_cmip5.py'​] 
 +>>>​ files_sizes = list(map(os.path.getsize,​ files_list)) 
 +>>>​ files_sizes 
 +[16152, 13954, 16539, 18147] 
 +>>>​ sum(files_sizes) 
 +64792</​code>​ 
 + 
 +==== Generating file names ==== 
 + 
 +=== Name depending on the current date/time === 
 + 
 +<​code>>>>​ import time 
 +>>>​ plot_version = time.strftime('​%Y%m%d_%H%M'​) 
 +>>>​ f_name = '​test_%s.nc'​ % (plot_version,​) 
 +>>>​ f_name 
 +'​test_20210827_1334.nc'​ 
 +</​code>​ 
 + 
 +=== Temporary file === 
 + 
 +<​code>>>>​ import tempfile, os 
 +>>>​ f_tmp = tempfile.NamedTemporaryFile(mode='​w',​ suffix='​.nc',​ delete=False) 
 +>>>​ f_tmp 
 +<​tempfile._TemporaryFileWrapper object at 0x2b5614743820>​ 
 +>>>​ f_tmp.name 
 +'/​tmp/​tmpi6uk9hre.nc'​ 
 +>>>​ f_tmp.close() 
 +>>>​ os.remove(f_tmp.name)</​code>​ 
 +==== Using command-line arguments ==== 
 + 
 +=== The extremely easy but non-flexible way: sys.argv === 
 + 
 +The name of a script, the number of arguments (including the name of the script), and the arguments (as strings) can be accessed through the ''​sys.argv''​ strings'​ list 
 + 
 +Simple ''​argv_test.py''​ test script: 
 +<​code>#​!/​usr/​bin/​env python 
 +import sys 
 +nb_args = len(sys.argv) 
 +print('​Number of script arguments (including script name) =', nb_args) 
 +for idx, val in enumerate(sys.argv):​ 
 +    print(idx, val)</​code>​ 
 + 
 +<​code>​$ python argv_test.py 
 +Number of script arguments (including script name) = 1 
 +0 argv_test.py 
 + 
 +$ python argv_test.py tas tas_tes.nc 
 +Number of script arguments (including script name) = 3 
 +0 argv_test.py 
 +1 tas 
 +2 tas_tes.nc</​code>​ 
 + 
 +=== The C-style way: getopt === 
 + 
 +Use [[https://​docs.python.org/​3/​library/​getopt.html|getopt]] (//C-style parser for command line options//​) 
 + 
 +=== The deprecated Python way: optparse === 
 + 
 +[[https://​docs.python.org/​3/​library/​optparse.html|optparse]] (//parser for command line options//) is **deprecated since Python version 3.2**! You should now use argparse (check [[https://​docs.python.org/​3/​library/​argparse.html#​upgrading-optparse-code|Upgrading optparse code]] for converting from ''​optparse''​ to ''​argparse''​) 
 + 
 +=== The current Python way: argparse === 
 + 
 +[[https://​docs.python.org/​3/​library/​argparse.html|argparse]] (//parser for command-line options, arguments and sub-commands//​) is available since Python version 3.2 
 + 
 +==== Using ordered dictionaries ==== 
 + 
 +**Dictionary order is guaranteed to be insertion order**! Note that the [[https://​docs.python.org/​3/​library/​stdtypes.html#​dict|usual Python dictionary]] also guarantees the order since version **3.6** 
 + 
 +Check the [[https://​docs.python.org/​3/​library/​collections.html#​collections.OrderedDict|OrderedDict class]] (''​from collections import OrderedDict''​) and the [[https://​realpython.com/​python-ordereddict/​|OrderedDict vs dict in Python: The Right Tool for the Job]] tutorial 
 + 
 +==== Printing a readable version of long lists or dictionaries ==== 
 + 
 +The [[https://​docs.python.org/​3/​library/​pprint.html|pprint]] module can be used for //pretty printing// objects (lists, dictionaries,​ ...). It will wrap long lines in a meaningful way 
 + 
 +<​code>>>>​ import pprint 
 + 
 +>>>​ test_dic = {'​AWI-ESM-1-1-LR_AWI':​{'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ '​CESM2_NCAR':​{'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ '​IPSL-CM6A-LR_IPSL':​{'​r1i1p1f1':​ {'​grid':​ '​gr'​},​ '​r1i1p1f2':​ {'​grid':​ '​gr'​},​ '​r1i1p1f3':​ {'​grid':​ '​gr'​},​ '​r1i1p1f4':​ {'​grid':​ '​gr'​}}} 
 + 
 +>>>​ print(test_dic) 
 +{'​AWI-ESM-1-1-LR_AWI':​ {'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ '​CESM2_NCAR':​ {'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ '​IPSL-CM6A-LR_IPSL':​ {'​r1i1p1f1':​ {'​grid':​ '​gr'​},​ '​r1i1p1f2':​ {'​grid':​ '​gr'​},​ '​r1i1p1f3':​ {'​grid':​ '​gr'​},​ '​r1i1p1f4':​ {'​grid':​ '​gr'​}}} 
 + 
 +>>>​ pprint.pprint(test_dic) 
 +{'​AWI-ESM-1-1-LR_AWI':​ {'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ 
 + '​CESM2_NCAR':​ {'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ 
 + '​IPSL-CM6A-LR_IPSL':​ {'​r1i1p1f1':​ {'​grid':​ '​gr'​},​ 
 +                       '​r1i1p1f2':​ {'​grid':​ '​gr'​},​ 
 +                       '​r1i1p1f3':​ {'​grid':​ '​gr'​},​ 
 +                       '​r1i1p1f4':​ {'​grid':​ '​gr'​}}} 
 +                        
 +>>>​ dir(test_dic) 
 +['​__class__',​ '​__contains__',​ '​__delattr__',​ [... lots of unreadable stuff removed...] '​setdefault',​ '​update',​ '​values'​] 
 + 
 +>>>​ pprint.pprint(dir(test_dic)) 
 +['​__class__',​ 
 + '​__contains__',​ 
 + 
 +[... lots of lines removed in this example ] 
 + 
 + '​setdefault',​ 
 + '​update',​ 
 + '​values'​] 
 + 
 +</​code>​
  
 /* /*
- * tip template\\ <​code>​Some code</​code>​+==== Tip template ​==== 
 + 
 +<​code>​Some code</​code>​
  */  */
  
other/python/misc_by_jyp.txt · Last modified: 2023/12/08 15:51 by jypeter