profiling and performance basics
Note
The newest version of the lecture notes includes a section on high(er) performance Python. Read that rather than this short post.
Original post
Python code can be very slow or very fast. For loops are slower than list comprehensions, which are much much slower than numpy calls or built in python functions. The latter two use optimized Fortran and C libraries. So the first rule of thumb is, whenever you find yourself writing a for loop or list comprehension, check if there is a built-in Python or numpy function that does the same thing.
The above rule always holds since built in functions will lead to simpler code (remember the importance of simplicity). However, there are often other optimizations that lead to slightly harder to read or more complicated code. To address this, first consider the following quote, credited to Donald Knuth, "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." Then solution is to profile your code first, to determine where the slow spots are, and then only re-write the parts that are slowing your code down.
To profile scientific code, you need a line-by-line readout of the time taken in different function calls in your code. This can be had by using the line_profiler in conjunction with the kernprof script. To install, simply type
pip install line_profiler
Then, at the top of a function that you want to profile (you can only profile functions), put @profile. For example:
@profile
def myfun(x):
y = 2 * x
return y
Then you need some way to call myfun from the command line. This could be for example a script run_myfun.py, which could be as simple as:
from mymodule import myfun
myfun(10)
Then, at the command line, type
kernprof.py -l run_myfun.py
If you are using anaconda and have installed line_profiler, then kernprof.py will be in your PATH, and the above line will work. The above line will produce the file run_myfun.py.lprof. This is the profiler output. You need to use the module line_profiler to read it. To do this, type
python -m line_profiler run_myfun.py.lprof | less
You should see a line-by-line breakdown of time taken to run your code.
blog comments powered by Disqus
