User Tools

Site Tools


other:python:misc_by_jyp

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
other:python:misc_by_jyp [2021/06/30 15:20]
jypeter
other:python:misc_by_jyp [2021/09/17 14:05] (current)
jypeter [Printing a readable version of long lists or dictionaries] Added long list example
Line 2: Line 2:
  
 <WRAP center round tip 60%> <WRAP center round tip 60%>
-You will find on this page some useful, but unsorted, python tips and tricks that can't fit in a section of the main [[other:​python:​jyp_steps|JYP'​s recommended steps for learning python]] page+You will find on this page some **useful, but unsorted, python tips and tricks** that can't fit in a section of the main [[other:​python:​jyp_steps|JYP'​s recommended steps for learning python]] page
 </​WRAP>​ </​WRAP>​
  
- ​* ​Reading/​setting environments variables\\ <​code>>>>​ os.environ['​TMPDIR'​]+==== Reading/​setting environments variables ​==== 
 + 
 + 
 +<​code>>>>​ os.environ['​TMPDIR'​]
 '/​data/​jypmce/​climafcache'​ '/​data/​jypmce/​climafcache'​
 >>>​ os.environ.get('​SCRATCHDIR',​ '/​data/​jypmce/​some_scratch_stuff'​) >>>​ os.environ.get('​SCRATCHDIR',​ '/​data/​jypmce/​some_scratch_stuff'​)
Line 14: Line 17:
 </​code>​ </​code>​
  
- ​* ​Generating (aka //raising//) an errorThis will stop the script, unless it is called in a function, and the code calling the function explicitely catches and deals with errors+==== Generating (aka raising) an error ==== 
 + 
 +This will stop the script, unless it is called in a function, and the code calling the function explicitely catches and deals with errors
     * <​code>​raise RuntimeError('​\n\nOMG! An error! :​-(\nAborting script...'​)</​code>​     * <​code>​raise RuntimeError('​\n\nOMG! An error! :​-(\nAborting script...'​)</​code>​
     * [[https://​docs.python.org/​3/​tutorial/​errors.html|Errors and Exceptions tutorial]]     * [[https://​docs.python.org/​3/​tutorial/​errors.html|Errors and Exceptions tutorial]]
     * [[https://​docs.python.org/​3/​library/​exceptions.html|Built-in Exceptions reference]]     * [[https://​docs.python.org/​3/​library/​exceptions.html|Built-in Exceptions reference]]
  
- * Stopping a script\\ <​code>​sys.exit('​Some optional message about why we are stopping'​)</​code>​ 
  
- ​* ​Checking if a file/​directory is writable by the current user\\ <​code>>>>​ os.access('/',​ os.W_OK)+==== Stopping a script ==== 
 + 
 +A user can use ''​CTRL-C''​ or ''​kill''​ to stop a script, or ''​CTRL-Z''​ to suspend it temporarily (use ''​fg''​ to resume a suspended script). The code below can be used by the script itself to interrupt its execution, instead of raising an error 
 + 
 +<​code>​sys.exit('​Some optional message about why we are stopping'​)</​code>​ 
 + 
 + 
 +==== Checking if a file/​directory is writable by the current user ==== 
 + 
 +<​code>>>>​ os.access('/',​ os.W_OK)
 False False
 >>>​ os.access('/​home/​jypmce/​.bashrc',​ os.W_OK) >>>​ os.access('/​home/​jypmce/​.bashrc',​ os.W_OK)
 True</​code>​ True</​code>​
 +
 +==== Working with paths and filenames ====
 +
 +If you are in a hurry, you can just use string functions to work with path and file names. But you will need some specific functions to check if a file exists, and similar operations. All these are available in 2 libraries that have similar functions. Both of these libraries can deal with Unix-type paths on Linux computers, and Windows-type paths on Windows computers
 +
 +  * [[https://​docs.python.org/​3/​library/​os.path.html|os.path]] //Common pathname manipulations//​
 +    * Available since... a long time! Use this if you want to avoid backward compatibility problems
 +    * Some functions are directly in [[https://​docs.python.org/​3/​library/​os.html|os]] //​Miscellaneous operating system interfaces//​\\ e.g. [[https://​docs.python.org/​3/​library/​os.html#​os.remove|os.remove]] and [[https://​docs.python.org/​3/​library/​os.html#​os.rmdir|os.rmdir]]
 +  * [[https://​docs.python.org/​3/​library/​pathlib.html|pathlib]] //​Object-oriented filesystem paths//
 +    * Available since Python version 3.4
 +    * [[https://​docs.python.org/​3/​library/​pathlib.html#​correspondence-to-tools-in-the-os-module|Matching pathlib, and os or os.path functions]]
 +  * [[https://​docs.python.org/​3/​library/​shutil.html|High-level file operations]]
 +
 +
 +=== Example: getting the full path of the Python used ===
 +
 +Note: the actual python may be different from the default python!
 +
 +<​code>​$ which python
 +/​usr/​bin/​python
 +
 +$ /​modfs/​modtools/​miniconda3//​envs/​analyse_3.6_test/​bin/​python
 +>>>​ import sys, shutil
 +>>>​ shutil.which('​python'​)
 +'/​usr/​bin/​python'​
 +>>>​ sys.executable
 +'/​modfs/​modtools/​miniconda3//​envs/​analyse_3.6_test/​bin/​python'</​code>​
 +
 +
 +=== Example: getting the full path of a script ===
 +
 +<​code>>>>​ import os
 +>>>​ os.getcwd()
 +'/​home/​jypmce/​PMIP4'​
 +>>>​ os.path.exists('​./​argv_test.py'​)
 +True
 +>>>​ os.path.abspath('​./​argv_test.py'​)
 +'/​home/​jypmce/​PMIP4/​argv_test.py'​
 +>>>​ os.path.exists('/​home/​jypmce/​PMIP4/​argv_test.py'​)
 +True
 +</​code>​
 +
 +
 +=== Example: getting the size(s) of all the files in a directory ===
 +
 +<​code>​$ cd /​data/​jypmce/​TestDir
 +$ ls -l
 +total 72
 +-rw-r--r-- 1 jypmce ipsl 18147 Jun 25  2012 get_TS_cmip5.py
 +-rw-r--r-- 1 jypmce ipsl 16152 Jun 21  2012 get_TS_cmip5.py~
 +-rw-r--r-- 1 jypmce ipsl 13954 Jul  3  2012 get_TS_cmip5_regular.py
 +-rw-r--r-- 1 jypmce ipsl 16539 Jun 22  2012 get_TS_cmip5_regular.py~</​code>​
 +
 +<​code>>>>​ os.chdir('/​data/​jypmce/​TestDir'​)
 +>>>​ print(os.getcwd())
 +/​data/​jypmce/​TestDir
 +>>>​ files_list = os.listdir()
 +>>>​ files_list
 +['​get_TS_cmip5.py~',​ '​get_TS_cmip5_regular.py',​ '​get_TS_cmip5_regular.py~',​ '​get_TS_cmip5.py'​]
 +>>>​ files_sizes = list(map(os.path.getsize,​ files_list))
 +>>>​ files_sizes
 +[16152, 13954, 16539, 18147]
 +>>>​ sum(files_sizes)
 +64792</​code>​
 +
 +==== Generating file names ====
 +
 +=== Name depending on the current date/time ===
 +
 +<​code>>>>​ import time
 +>>>​ plot_version = time.strftime('​%Y%m%d_%H%M'​)
 +>>>​ f_name = '​test_%s.nc'​ % (plot_version,​)
 +>>>​ f_name
 +'​test_20210827_1334.nc'​
 +</​code>​
 +
 +=== Temporary file ===
 +
 +<​code>>>>​ import tempfile, os
 +>>>​ f_tmp = tempfile.NamedTemporaryFile(mode='​w',​ suffix='​.nc',​ delete=False)
 +>>>​ f_tmp
 +<​tempfile._TemporaryFileWrapper object at 0x2b5614743820>​
 +>>>​ f_tmp.name
 +'/​tmp/​tmpi6uk9hre.nc'​
 +>>>​ f_tmp.close()
 +>>>​ os.remove(f_tmp.name)</​code>​
 +==== Using command-line arguments ====
 +
 +=== The extremely easy but non-flexible way: sys.argv ===
 +
 +The name of a script, the number of arguments (including the name of the script), and the arguments (as strings) can be accessed through the ''​sys.argv''​ strings'​ list
 +
 +Simple ''​argv_test.py''​ test script:
 +<​code>#​!/​usr/​bin/​env python
 +import sys
 +nb_args = len(sys.argv)
 +print('​Number of script arguments (including script name) =', nb_args)
 +for idx, val in enumerate(sys.argv):​
 +    print(idx, val)</​code>​
 +
 +<​code>​$ python argv_test.py
 +Number of script arguments (including script name) = 1
 +0 argv_test.py
 +
 +$ python argv_test.py tas tas_tes.nc
 +Number of script arguments (including script name) = 3
 +0 argv_test.py
 +1 tas
 +2 tas_tes.nc</​code>​
 +
 +=== The C-style way: getopt ===
 +
 +Use [[https://​docs.python.org/​3/​library/​getopt.html|getopt]] (//C-style parser for command line options//)
 +
 +=== The deprecated Python way: optparse ===
 +
 +[[https://​docs.python.org/​3/​library/​optparse.html|optparse]] (//parser for command line options//) is **deprecated since Python version 3.2**! You should now use argparse (check [[https://​docs.python.org/​3/​library/​argparse.html#​upgrading-optparse-code|Upgrading optparse code]] for converting from ''​optparse''​ to ''​argparse''​)
 +
 +=== The current Python way: argparse ===
 +
 +[[https://​docs.python.org/​3/​library/​argparse.html|argparse]] (//parser for command-line options, arguments and sub-commands//​) is available since Python version 3.2
 +
 +==== Using ordered dictionaries ====
 +
 +**Dictionary order is guaranteed to be insertion order**! Note that the [[https://​docs.python.org/​3/​library/​stdtypes.html#​dict|usual Python dictionary]] also guarantees the order since version **3.6**
 +
 +Check the [[https://​docs.python.org/​3/​library/​collections.html#​collections.OrderedDict|OrderedDict class]] (''​from collections import OrderedDict''​) and the [[https://​realpython.com/​python-ordereddict/​|OrderedDict vs dict in Python: The Right Tool for the Job]] tutorial
 +
 +==== Printing a readable version of long lists or dictionaries ====
 +
 +The [[https://​docs.python.org/​3/​library/​pprint.html|pprint]] module can be used for //pretty printing// objects (lists, dictionaries,​ ...). It will wrap long lines in a meaningful way
 +
 +<​code>>>>​ import pprint
 +
 +>>>​ test_dic = {'​AWI-ESM-1-1-LR_AWI':​{'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ '​CESM2_NCAR':​{'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ '​IPSL-CM6A-LR_IPSL':​{'​r1i1p1f1':​ {'​grid':​ '​gr'​},​ '​r1i1p1f2':​ {'​grid':​ '​gr'​},​ '​r1i1p1f3':​ {'​grid':​ '​gr'​},​ '​r1i1p1f4':​ {'​grid':​ '​gr'​}}}
 +
 +>>>​ print(test_dic)
 +{'​AWI-ESM-1-1-LR_AWI':​ {'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ '​CESM2_NCAR':​ {'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​ '​IPSL-CM6A-LR_IPSL':​ {'​r1i1p1f1':​ {'​grid':​ '​gr'​},​ '​r1i1p1f2':​ {'​grid':​ '​gr'​},​ '​r1i1p1f3':​ {'​grid':​ '​gr'​},​ '​r1i1p1f4':​ {'​grid':​ '​gr'​}}}
 +
 +>>>​ pprint.pprint(test_dic)
 +{'​AWI-ESM-1-1-LR_AWI':​ {'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​
 + '​CESM2_NCAR':​ {'​r1i1p1f1':​ {'​grid':​ '​gn'​}},​
 + '​IPSL-CM6A-LR_IPSL':​ {'​r1i1p1f1':​ {'​grid':​ '​gr'​},​
 +                       '​r1i1p1f2':​ {'​grid':​ '​gr'​},​
 +                       '​r1i1p1f3':​ {'​grid':​ '​gr'​},​
 +                       '​r1i1p1f4':​ {'​grid':​ '​gr'​}}}
 +                       
 +>>>​ dir(test_dic)
 +['​__class__',​ '​__contains__',​ '​__delattr__',​ [... lots of unreadable stuff removed...] '​setdefault',​ '​update',​ '​values'​]
 +
 +>>>​ pprint.pprint(dir(test_dic))
 +['​__class__',​
 + '​__contains__',​
 +
 +[... lots of lines removed in this example ]
 +
 + '​setdefault',​
 + '​update',​
 + '​values'​]
 +
 +</​code>​
  
 /* /*
- * tip template\\ <​code>​Some code</​code>​+==== Tip template ​==== 
 + 
 +<​code>​Some code</​code>​
  */  */
  
other/python/misc_by_jyp.1625066431.txt.gz · Last modified: 2021/06/30 15:20 by jypeter