asv benchmark setup

[scottstanie]

(Dumping some of my initial notes about why I ended up picking this)

Airspeed Velocity (asv) seems to be the best existing way to run a few benchmarks and get progress over time. It's used by numpy, pandas, scikit-image, xarray, and a few others.

Basics: making these new configuration files, and running test benchmarks

1. I installed asv (which is added to the test requirements), ran asf quickstart to make a template asv.conf.json, then filled it out
2. To run locally, you can run asv machine or asv machine --yes to have it save info about the computer you are on
3. Runing benchmarks with asv run will install the environment(s) that are specified, and these can be reused for future runs.
4. You compile the results with asv publish, and view locally with asv preview

Naming the benchmark tests

The benchmarks almost look like normal pytest or UnitTest files, except all the names have time_ or mem_ as the function names... And that's because asv will use that to figure out what it's tracking

• time_<blah> will run a function (and call the result whatever you've named this function). and measure the runtime
• peakmem_<blah> records the peak memory usage

Using it in CI

• You can run against a specific git commit, or a tag, like

asv run v0.5.0^!

This PR is attempting to have a nice way to trigger this on Github actions only if you want it run on specific PRs (which I do), rather than every single change/push.
This section of the new Github Action:

on:
  pull_request:
    types: [labeled]

jobs:
  benchmark:
    if: ${{ github.event.label.name == 'run-benchmark' && github.event_name == 'pull_request' }}

means that when you label a PR with "run-benchmark" it should kick off the asv workflow. If you add more commits, you'll need to remove and re-add if you want it run again. You can also go to the "Actions" tab to run it.

Running on our big server

I started the test_run_bench.sh script for running on aurora, where I manually set the number of threads and use asvs --cpu-affinity 0-15 option to really limit the number of cores.

(The problem with only running this on the CI is that it'll be single CPU... we're also interested in how the main functions perform with access to multiple CPUs, since that's the more realistic setting. Open to discussing this part more.)

Examples repos' example usages

• TQDM has a longer GHA workflow using it on a scheduled cron
• scikit-image has a regular version, but also a cron version, and their docs have a lot of nice info about running/adding to their benchmarks
• Scipy has docs about it here and their graphs are up here
• scikit-learn has a separate repo setup
• Sarsen has a simple setup, but they've put semi-real data into git LFS... which is probably why it's in a separate repo
• napari has a complicated GHA setup

[scottstanie]

As a preview of what this gives us (this is from running it on aurora):

it seems like it might be possible to merge results from multiple machines too. I'm not too worried about that, since this alone was about 90% of what I wanted in the first place- you can tell from the bottom plot that with 20 SLCs, phase linking on CPU is taking ~4 seconds. For COMPASS-sized SLCs, that's ~18-20 minutes for the whole stack, which lines up quite well with the other rough time estimates we'd been giving.