User Tools

Site Tools


other:python:misc_by_jyp

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
other:python:misc_by_jyp [2022/02/21 14:47]
jypeter [numpy related stuff] Added np.unique example
other:python:misc_by_jyp [2022/05/23 16:01]
jypeter [numpy related stuff] Changed the VIEW update warnng example
Line 39: Line 39:
 True</​code>​ True</​code>​
  
 +==== Playing with strings ====
 +
 +=== Filenames, etc... ===
 +
 +Check [[other:​python:​misc_by_jyp#​working_with_paths_and_filenames|Working with paths and filenames]] and [[other:​python:​misc_by_jyp#​generating_file_names|Generating file names]]
 +
 +=== Splitting strings ===
 +
 +It's easy to split a string with multiple blank delimiters, or a specific delimiter, but it can be harder to deal with sub-strings
 +
 +<​code>>>>​ str_with_blanks = '​one ​   two\t3\t\tFOUR'​
 +>>>​ str_with_blanks.split()
 +['​one',​ '​two',​ '​3',​ '​FOUR'​]
 +
 +>>>​ str_with_simple_delimiters = '​1,​2,​3.14, ​ 4'
 +>>>​ str_with_simple_delimiters.split(','​)
 +['​1',​ '​2',​ '​3.14',​ ' ​ 4']
 +
 +>>>​ complex_string='​-o 1 --long "A string with accented chars: é è à ç"'​
 +>>>​ complex_string.split()
 +['​-o',​ '​1',​ '​--long',​ '"​A',​ '​string',​ '​with',​ '​accented',​ '​chars:',​ '​\xc3\xa9',​ '​\xc3\xa8',​ '​\xc3\xa0',​ '​\xc3\xa7"'​]
 +
 +>>>​ import shlex
 +>>>​ shlex.split(complex_string)
 +['​-o',​ '​1',​ '​--long',​ 'A string with accented chars: \xc3\xa9 \xc3\xa8 \xc3\xa0 \xc3\xa7'​]</​code>​
 ==== Working with paths and filenames ==== ==== Working with paths and filenames ====
  
Line 222: Line 247:
  
 ==== numpy related stuff ==== ==== numpy related stuff ====
 +
 +=== Dealing with a variable number of indices ===
 +
 +[[https://​numpy.org/​doc/​stable/​user/​basics.indexing.html#​dealing-with-variable-indices|Official reference]]
 +
 +<​code>>>>​ i10 = np.identity(10)
 +>>>​ i10
 +array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
 +       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
 +...
 +       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]])
 +>>>​ i10.shape
 +(10, 10)
 +
 +>>>​ i10[3:7, 4:6]
 +array([[0., 0.],
 +       [1., 0.],
 +       [0., 1.],
 +       [0., 0.]])
 +       
 +>>>​ s0 = slice(3, 7)
 +>>>​ s1 = slice(4, 6)
 +>>>​ i10[s0, s1]
 +array([[0., 0.],
 +       [1., 0.],
 +       [0., 1.],
 +       [0., 0.]])
 +       
 +>>>​ my_slices = (s0, s1)
 +>>>​ i10[my_slices]
 +array([[0., 0.],
 +       [1., 0.],
 +       [0., 1.],
 +       [0., 0.]])
 +       
 +>>>​ my_fancy_slices = (s0, Ellipsis)
 +>>>​ i10[my_fancy_slices]
 +array([[0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
 +       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
 +       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
 +       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.]])
 +>>>​ i10[my_fancy_slices].shape
 +(4, 10)
 +
 +>>>​ # WARNING! DANGERRRR! NEVER forget that a VIEW is NOT A COPY
 +>>>​ # and that you can change the content of the original array by mistake
 +>>>​ my_view = i10[my_slices]
 +>>>​ my_view[:, :] = -1
 +>>>​ my_view
 +array([[-1.,​ -1.],
 +       [-1., -1.],
 +       [-1., -1.],
 +       [-1., -1.]])
 +>>>​ i10
 +array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
 +       [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
 +       [ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
 +       [ 0.,  0.,  0.,  1., -1., -1.,  0.,  0.,  0.,  0.],
 +       [ 0.,  0.,  0.,  0., -1., -1.,  0.,  0.,  0.,  0.],
 +       [ 0.,  0.,  0.,  0., -1., -1.,  0.,  0.,  0.,  0.],
 +       [ 0.,  0.,  0.,  0., -1., -1.,  1.,  0.,  0.,  0.],
 +       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.],
 +       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.],
 +       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]])</​code>​
  
 === Finding and counting unique values === === Finding and counting unique values ===
Line 230: Line 319:
 >>>​ vals >>>​ vals
 array([1. , 2. , 1. , 2. , 2. , 1.5, 1. , 1.5, 2. , 1.5]) array([1. , 2. , 1. , 2. , 2. , 1.5, 1. , 1.5, 2. , 1.5])
 +
 >>>​ np.unique(vals) >>>​ np.unique(vals)
 array([1. , 1.5, 2. ]) array([1. , 1.5, 2. ])
->>>​ np.unique(vals,​ return_counts=True) +>>> ​unique_vals,​ nb_unique = np.unique(vals,​ return_counts=True) 
-(array([1. , 1.5, 2. ])array([3, 3, 4])+>>>​ unique_vals 
->>>​ np.sort(vals) # Sorted copy, in order to check the result+array([1. , 1.5, 2. ]) 
 +>>>​ nb_unique 
 +array([3, 3, 4]) 
 + 
 +>>> ​sorted_vals = np.sort(vals) # Sorted copy, in order to check the result 
 +>>>​ sorted_vals
 array([1. , 1. , 1. , 1.5, 1.5, 1.5, 2. , 2. , 2. , 2. ])</​code>​ array([1. , 1. , 1. , 1.5, 1.5, 1.5, 2. , 2. , 2. , 2. ])</​code>​
  
 +=== Applying a ufunc over all the elements of an array ===
 +
 +There are all sorts of //ufuncs// (Universal Functions), and we will just use below ''​add''​ from the [[https://​numpy.org/​doc/​stable/​reference/​ufuncs.html#​math-operations|math operations]],​ applied on the arrays defined in [[#​finding_and_counting_unique_values|Finding and counting unique values]]
 +
 +<​code>#​ Get the sum of all the elements of '​vals'​
 +>>>​ np.add.reduce(vals)
 +15.5
 +>>>​ np.add.reduce(sorted_vals)
 +15.5
 +>>>​ vals.sum() # The usual and easy way to do it
 +15.5
 +
 +# Compute the sum of the elements of '​nb_unique'​
 +# AND keep (accumulate) the intermediate results
 +>>>​ nb_unique
 +array([3, 3, 4])
 +>>>​ np.add.accumulate(nb_unique)
 +array([ 3,  6, 10])
 +
 +# The accumulated values can be used as indices to separate the different groups of sorted values!
 +>>>​ sorted_vals
 +array([1. , 1. , 1. , 1.5, 1.5, 1.5, 2. , 2. , 2. , 2. ])
 +>>>​ sorted_vals[0:​3]
 +array([1., 1., 1.])
 +>>>​ sorted_vals[3:​6]
 +array([1.5, 1.5, 1.5])
 +>>>​ sorted_vals[6:​10]
 +array([2., 2., 2., 2.])
 +
 +# Compute the sum of each equal-value group
 +>>>​ sorted_vals[0:​3].sum(),​ sorted_vals[3:​6].sum(),​ sorted_vals[6:​10].sum()
 +(3.0, 4.5, 8.0)</​code>​
 +
 +=== Applying a ufunc over specified sections of an array ===
 +
 +The [[https://​numpy.org/​doc/​stable/​reference/​generated/​numpy.ufunc.reduceat.html#​numpy.ufunc.reduceat|reduceat]] function can be used to avoid explicit python loops, and improve the speed (but not the readability...) of a script. The example below //​improves//​ what has been shown above
 +
 +<​code>#​ Define a list with the boundaries of the intervals we want to apply the '​add'​ function to
 +# We need to add the beginning index (0), AND remove the last index
 +# (reduceat will automatically go to the end of the input array
 +>>>​ nb_unique
 +array([3, 3, 4])
 +>>>​ slices_indices = [0] + list(np.add.accumulate(nb_unique))
 +>>>​ slices_indices.pop() # Remove last element
 +10
 +>>>​ slices_indices
 +[0, 3, 6]
 +
 +# Compute the sums over the selected intervals with just one call
 +>>>​ np.add.reduceat(np.sort(vals),​ slices_indices)
 +array([3. , 4.5, 8. ])</​code>​
  
 /* /*
other/python/misc_by_jyp.txt · Last modified: 2023/12/08 15:51 by jypeter