Benchmarking NumPy with Airspeed Velocity.
Airspeed Velocity manages building and Python virtualenvs by itself, unless told otherwise. Some of the benchmarking features in runtests.py also tell ASV to use the NumPy compiled by runtests.py. To run the benchmarks, you do not need to install a development version of NumPy to your current Python environment.
runtests.py
Run a benchmark against currently checked out NumPy version (don’t record the result):
python runtests.py --bench bench_core
Compare change in benchmark results to another version:
python runtests.py --bench-compare v1.6.2 bench_core
Run ASV commands (record results and generate HTML):
cd benchmarks asv run --skip-existing-commits --steps 10 ALL asv publish asv preview
More on how to use asv can be found in ASV documentation Command-line help is available as usual via asv --help and asv run --help.
asv
asv --help
asv run --help
See ASV documentation for basics on how to write benchmarks.
Some things to consider:
The benchmark suite should be importable with any NumPy version.
The benchmark parameters etc. should not depend on which NumPy version is installed.
Try to keep the runtime of the benchmark reasonable.
Prefer ASV’s time_ methods for benchmarking times rather than cooking up time measurements via time.clock, even if it requires some juggling when writing the benchmark.
time_
time.clock
Preparing arrays etc. should generally be put in the setup method rather than the time_ methods, to avoid counting preparation time together with the time of the benchmarked operation.
setup
Be mindful that large arrays created with np.empty or np.zeros might not be allocated in physical memory until the memory is accessed. If this is desired behaviour, make sure to comment it in your setup function. If you are benchmarking an algorithm, it is unlikely that a user will be executing said algorithm on a newly created empty/zero array. One can force pagefaults to occur in the setup phase either by calling np.ones or arr.fill(value) after creating the array,
np.empty
np.zeros
np.ones
arr.fill(value)