Benchable

This is an IRC bot for benchmarking code at a given commit of Rakudo. It can be addressed by its full name ('benchable6') or its short name ('bench'). It will run the given code five times and return the minimum amount of time taken.

You can find some extra examples in tests: benchable.t

Usage examples

Basics

<MasterDuke> benchable6: HEAD .comb for ^10000
<benchable6> MasterDuke, starting to benchmark the 1 given commits
<benchable6> MasterDuke, ¦«HEAD»:0.3388

Multiple commits

<MasterDuke> bench: 2015.12,2016.02,2016.04,2016.06,2016.08.1 say 'hi'
<benchable6> MasterDuke, starting to benchmark the 5 given commits
<benchable6> MasterDuke, benchmarked the given commits, now zooming in on performance differences
<benchable6> MasterDuke, ¦«2015.12»:0.1158␤¦«2016.02»:0.1177␤¦«2016.04»:0.1170␤¦«2016.06»:0.1156␤¦«2016.08»:0.1206

<MasterDuke> bench: HEAD~100,HEAD my @a = rand xx 1000; my $m = @a.max
<benchable6> MasterDuke, starting to benchmark the 2 given commits
<benchable6> MasterDuke, benchmarked the given commits, now zooming in on performance differences
<benchable6> MasterDuke, ¦«HEAD~10»:0.2038␤¦«HEAD»:0.2083

Run something across all available releases

<MasterDuke> bench: releases say 'hi'
<benchable6> MasterDuke, starting to benchmark the 11 given commits
<benchable6> MasterDuke, benchmarked the given commits, now zooming in on performance differences
<benchable6> MasterDuke, https://gist.github.com/5a6f59d1e24a995a8b01cdf7348a084c

Ranges

You can try specifying a range. Keep in mind that it may time out if you try something that takes significant amount of time to run. It will only allow you run it across 300 commits (you probably don't want anything bigger than that).

<MasterDuke> bench: 2016.02~10..2016.02 my @a = (^10).combinations(3)
<benchable6> MasterDuke, starting to benchmark the 12 given commits
<benchable6> MasterDuke, ¦«280a487»:0.2239␤¦«f04a894»:0.2244␤¦«80a4998»:0.2293␤¦«2953d0a»:0.2227␤¦«4d5cb05»:0.2332␤¦«be7dd67»:0.2364␤¦«77581b1»:0.2317␤¦«7010f32»:0.2366␤¦«dae9f6a»:0.2368␤¦«5dcf96f»:0.2398␤¦«8fdaada»:0.2332␤¦«e401eeb»:0.2392

Zooming

If you run the code across all the available releases, or select more than one individual commit, once it has benchmarked the given commits it will then proceed to compare the times between the different commits. If it finds a 10% difference, it will then find a commit in the middle of the two it just compared and restart the comparison from that commit. It will keep doing this comparison until there are no more commits left to check or the time difference is less than 10%. Essentially it's bisecting by performance.

Graphing

If there are more than five commits after the initial benchmarking plus any zooming, the data will be graphed and the graph (in SVG) will be added to the gist.

Comparing code

Benchable also has a mode where it will compare multiple pieces of code at a single commit. The individual pieces of code are delimited by the string '|||'.

<MasterDuke> bench: compare HEAD .comb for ^1000 ||| .split('') for ^1000
<benchable6> MasterDuke, starting to benchmark the 1 given commits
<benchable6> MasterDuke, https://gist.github.com/8aa05f3fdb1b2b2a23296c5adaddc523