other:python:jyp_steps
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
other:python:jyp_steps [2016/01/22 17:07] jypeter More changes |
other:python:jyp_steps [2016/02/18 11:02] jypeter Started the debug section |
||
|---|---|---|---|
| Line 49: | Line 49: | ||
| Where: [[http://docs.scipy.org/doc/|html and pdf documentation]] | Where: [[http://docs.scipy.org/doc/|html and pdf documentation]] | ||
| - | How to get started? | + | ==== Getting started ==== |
| - always remember that indices start at ''0'' and that the last element of an array is at index ''-1''!\\ First learn about //indexing// and //slicing// by manipulating strings, as shown in [[#part1|Part 1]] above (try '''This document by JY is awesome!'[::-1]'' and '''This document by JY is awesome!'[slice(None, None, -1)]'') 8-) | - always remember that indices start at ''0'' and that the last element of an array is at index ''-1''!\\ First learn about //indexing// and //slicing// by manipulating strings, as shown in [[#part1|Part 1]] above (try '''This document by JY is awesome!'[::-1]'' and '''This document by JY is awesome!'[slice(None, None, -1)]'') 8-) | ||
| + | - if you are a Matlab user (but the references are interesting for others as well), you can read the following: | ||
| + | - [[https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html|Numpy for Matlab users]] | ||
| + | - [[http://mathesaurus.sourceforge.net/matlab-numpy.html|NumPy for MATLAB users]] (nice, but does not seem to be maintained any more) | ||
| - read the [[https://docs.scipy.org/doc/numpy-dev/user/quickstart.html|Quickstart tutorial]] | - read the [[https://docs.scipy.org/doc/numpy-dev/user/quickstart.html|Quickstart tutorial]] | ||
| - have a quick look at the full documentation to know where things are | - have a quick look at the full documentation to know where things are | ||
| Line 56: | Line 60: | ||
| - Numpy Reference Guide | - Numpy Reference Guide | ||
| - Scipy Reference Guide | - Scipy Reference Guide | ||
| + | |||
| + | ==== Beware of the array view side effects ==== | ||
| + | |||
| + | <note warning>When you take a slice of an array, you get a **//View//** : an array that has a new shape but that still shares its data with the first array. | ||
| + | |||
| + | That is not a problem when you only read the values, but **if you change the values of the //View//, you change the values of the first array** (and vice-versa)! If that is not what want, do not forget to **make a copy** of the data before working on it! | ||
| + | |||
| + | //Views// are a good thing most of the time, so only make a copy of your data when needed, because otherwise copying a big array will just be a waste of CPU and computer memory. Anyway, it is always better to understand what you are doing... :-P | ||
| + | |||
| + | Check the example below and the [[https://docs.scipy.org/doc/numpy-dev/user/quickstart.html#copies-and-views|copies and views]] part of the quickstart tutorial. | ||
| + | |||
| + | <code python> | ||
| + | >>> import numpy as np | ||
| + | >>> a = np.arange(30).reshape((3,10)) | ||
| + | >>> a | ||
| + | array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], | ||
| + | [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], | ||
| + | [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]]) | ||
| + | |||
| + | >>> b = a[1, :] | ||
| + | >>> b | ||
| + | array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) | ||
| + | |||
| + | >>> b[3:7] = 0 | ||
| + | >>> b | ||
| + | array([10, 11, 12, 0, 0, 0, 0, 17, 18, 19]) | ||
| + | |||
| + | >>> a | ||
| + | array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], | ||
| + | [10, 11, 12, 0, 0, 0, 0, 17, 18, 19], | ||
| + | [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]]) | ||
| + | |||
| + | >>> a[:, 2:4] = -1 | ||
| + | >>> a | ||
| + | array([[ 0, 1, -1, -1, 4, 5, 6, 7, 8, 9], | ||
| + | [10, 11, -1, -1, 0, 0, 0, 17, 18, 19], | ||
| + | [20, 21, -1, -1, 24, 25, 26, 27, 28, 29]]) | ||
| + | |||
| + | >>> b | ||
| + | array([10, 11, -1, -1, 0, 0, 0, 17, 18, 19]) | ||
| + | |||
| + | >>> c = a[1, :].copy() | ||
| + | >>> c | ||
| + | array([10, 11, -1, -1, 0, 0, 0, 17, 18, 19]) | ||
| + | |||
| + | >>> c[:] = 9 | ||
| + | >>> c | ||
| + | array([9, 9, 9, 9, 9, 9, 9, 9, 9, 9]) | ||
| + | |||
| + | >>> b | ||
| + | array([10, 11, -1, -1, 0, 0, 0, 17, 18, 19]) | ||
| + | |||
| + | >>> a | ||
| + | array([[ 0, 1, -1, -1, 4, 5, 6, 7, 8, 9], | ||
| + | [10, 11, -1, -1, 0, 0, 0, 17, 18, 19], | ||
| + | [20, 21, -1, -1, 24, 25, 26, 27, 28, 29]]) | ||
| + | </code></note> | ||
| ===== cdms2 and netCDF4 ===== | ===== cdms2 and netCDF4 ===== | ||
| Line 128: | Line 189: | ||
| * [[http://blog.codinghorror.com/a-pragmatic-quick-reference/|A Pragmatic Quick Reference]] | * [[http://blog.codinghorror.com/a-pragmatic-quick-reference/|A Pragmatic Quick Reference]] | ||
| + | |||
| + | ===== Debugging your code ===== | ||
| + | |||
| + | There is only so much you can do with staring at your code in your favorite text editor, and adding ''print'' lines in your code (or [[https://docs.python.org/2/howto/logging.html#logging-basic-tutorial|using logging]] instead of ''print''). The next step is to use the python debugger! | ||
| + | |||
| + | ==== Debugging in text mode ==== | ||
| + | |||
| + | ==== Using pydebug ==== | ||
| + | |||
| ===== Improving the performance of your code ===== | ===== Improving the performance of your code ===== | ||
| Line 135: | Line 205: | ||
| * **make sure that your script is not using too much memory** (the amount depends on the computer you are using)! Your script should be scalable (e.g. keeps on working even when your data gets bigger), so it's a good idea to load only the data you need in memory (e.g. not all the time steps), and learn how to load chunks of data | * **make sure that your script is not using too much memory** (the amount depends on the computer you are using)! Your script should be scalable (e.g. keeps on working even when your data gets bigger), so it's a good idea to load only the data you need in memory (e.g. not all the time steps), and learn how to load chunks of data | ||
| - | * **make sure that you are using array/vector syntax and masks**, instead of using explicit loops and tests. The numpy documentation is big, because there are lots of optimized functions to help you! If you are stuck, ask JY or somebody else who is used to numpy. | + | * **make sure that you are using array/vector syntax and masks**, instead of using explicit loops and tests. The [[#numpy_and_scipy|numpy documentation]] is big, because there are lots of optimized functions to help you! If you are stuck, ask JY or somebody else who is used to numpy. |
| If your script is still not fast enough, there is a lot you can do to improve it, without resorting to parallelization (that may introduce extra bugs rather that extra performance). See the sections below | If your script is still not fast enough, there is a lot you can do to improve it, without resorting to parallelization (that may introduce extra bugs rather that extra performance). See the sections below | ||
other/python/jyp_steps.txt · Last modified: 2024/03/07 10:15 by jypeter
