-
Notifications
You must be signed in to change notification settings - Fork 14
Bisectable
This bot is meant to help you find when something got broken. If you want to know if something has ever worked use Committable instead.
Defaults to old=2015.12 and new=HEAD. Note that there are builds available up to 2014.01, so you have to use old=2014.01 if you really want to go that far.
You can find some extra examples in tests: bisectable.t
⚠ Before trying to bisect, it is recommended that you use Committable (e.g. 6c: say rand to run on all v6.c-supporting releases or all: say rand to try even older commits). This will give you a better understanding of the situation and thus you'll be able to bisect exactly what you need (by supplying custom old= and new= values).
⚠ If you want to find which commit caused a change in performance, use Benchable. Bisectable does not guarantee consistent or meaningful timings.
<AlexDaniel> bisect: exit 1 if (^∞).grep({ last })[5] // 0 == 4 # RT 128181
<bisectable6> AlexDaniel, bisect log: https://gist.github.com/c3317662a9619214a587d1781fa495eb
<bisectable6> AlexDaniel, (2016-03-18) https://github.com/rakudo/rakudo/commit/6d120ca
In theory, this is what you are supposed to do. That is, you should exit 0 if this revision is OK and exit with any other code if it is not. Use exit 125 to skip a commit. However, over the weeks this bot was getting smarter and smarter, and nowadays it probably knows what you want better than you do. In other words, feel free to throw any piece of code into it and see how it goes. Inspect the log to make sure that you are getting what you were searching for.
Keep in mind that it is mostly a pure binary search, so do not expect it to figure out flappers or any other complex cases.
<AlexDaniel> bisect: say (^∞).grep({ last })[5] # same but without proper exit codes
<bisectable6> AlexDaniel, Exit code is 0 on both starting points (good=2015.12 bad=575dda1), bisecting by using the output
<bisectable6> AlexDaniel, bisect log: https://gist.github.com/f0e146242faf1ec01602eb6a66337e3a
<bisectable6> AlexDaniel, (2016-03-18) https://github.com/rakudo/rakudo/commit/6d120ca
<AlexDaniel> bisect: class A { has $.wut = [] }; my $a = A.new; $a.wut = [1,2,3]
<bisectable6> AlexDaniel, For the given starting points (good=2015.12 bad=575dda1), exit code on a ‘good’ revision is 1 (which is bad), bisecting with inverted logic
<bisectable6> AlexDaniel, bisect log: https://gist.github.com/831baa61429add18413a62ab22e974b0
<bisectable6> AlexDaniel, (2016-03-02) https://github.com/rakudo/rakudo/commit/fdd37a9
<AlexDaniel> bisect: say ‘hello world’; exit 42
<bisectable6> AlexDaniel, On both starting points (good=2015.12 bad=575dda1) the exit code is 42 and the output is identical as well
<bisectable6> AlexDaniel, Output on both points: hello world
Use good=… and bad=… in front of your query. You can use = or space as a delimiter, it does not matter.
<AlexDaniel> bisect: good=2016.02 bad 2016.03 say (^∞).grep({ last })[5]
<bisectable6> AlexDaniel, Exit code is 0 on both starting points (good=2016.02 bad=2016.03), bisecting by using the output
<bisectable6> AlexDaniel, bisect log: https://gist.github.com/87568a8cf839f8057968340b0ce5e4b8
<bisectable6> AlexDaniel, (2016-03-18) https://github.com/rakudo/rakudo/commit/6d120ca
Sometimes people confuse what is good and what is bad. If that happens, you are going to see this:
<AlexDaniel> bisect: good=2016.03 bad 2016.02 say (^∞).grep({ last })[5] # swapped good and bad revisions
<bisectable6> AlexDaniel, Exit code is 0 on both starting points (good=2016.03 bad=2016.02), bisecting by using the output
<bisectable6> AlexDaniel, bisect log: https://gist.github.com/bb672c28b8b9744f578b94259a2a7557
<bisectable6> AlexDaniel, bisect init failure
If you take a look at the log you will see this:
Some good revs are not ancestor of the bad rev.
git bisect cannot work properly in this case.
Maybe you mistake good and bad revs?
This error message comes from git.
Basically, good is always old and bad is always new.
You can use  characters as newlines. In other words, the behavior is identical to Committable#special-characters.
<AlexDaniel> bisect: # newline test  say ‘hello world’; exit 42
<bisectable6> AlexDaniel, On both starting points (good=2015.12 bad=575dda1) the exit code is 42 and the output is identical as well
<bisectable6> AlexDaniel, Output on both points: hello world
It also supports URLs, but note that you have to provide a direct link to the “raw” version:
<AlexDaniel> bisectable6: https://gist.githubusercontent.com/atweiden/9d1dfe825ade18a7db54d8e0733ca2e4/raw/16b7c39cfd12fef6eb74851c872097e0c655cff3/pkginfo.p6
<bisectable6> AlexDaniel, Successfully fetched the code from the provided URL.
<bisectable6> AlexDaniel, Exit code is 1 on both starting points (good=2015.12 bad=575dda1), bisecting by using the output
<bisectable6> AlexDaniel, bisect log: https://gist.github.com/81def212f00afc7dfcd51dde863ed23f
<bisectable6> AlexDaniel, (2016-02-18) https://github.com/rakudo/rakudo/commit/9983c2c
<AlexDaniel> bisect: bad=d3acb938 try { NaN.Rat == NaN; exit 0 }; exit 1
<bisectable6> AlexDaniel, bisect log: https://gist.github.com/53402fecc6601d9f92f611f17d74fe53
<bisectable6> AlexDaniel, (2016-05-02) https://github.com/rakudo/rakudo/commit/e2f1fa7
<AlexDaniel> bisect: for ‘q b c d’.words -> $a, $b { }; CATCH { exit 0 }; exit 1
<bisectable6> AlexDaniel, bisect log: https://gist.github.com/7ef27b210b3f86b06d1e0f703a91c708
<bisectable6> AlexDaniel, (2016-03-01) https://github.com/rakudo/rakudo/commit/1b6c901
<AlexDaniel> bisectable6: help
<bisectable6> AlexDaniel, Like this: bisectable6: good=2015.12 bad=HEAD exit 1 if (^∞).grep({ last })[5] // 0 == 4 # RT128181
<AlexDaniel> bisectable6: http://github.org/sntoheausnteoahuseoau
<bisectable6> AlexDaniel, It looks like a URL, but for some reason I cannot download it (HTTP status line is 404 Not Found).
<AlexDaniel> bisect: https://gist.github.com/atweiden/9d1dfe825ade18a7db54d8e0733ca2e4
<bisectable6> AlexDaniel, It looks like a URL, but mime type is ‘text/html; charset=utf-8’ while I was expecting ‘text/plain; charset=utf-8’. I can only understand raw links, sorry.
I don't get it, the exit code and output are all exactly identical for every step in the bisect log?
Yes, this is possible if the offending commit is the last one (i.e. HEAD).
It is quite fascinating to read the discussion about Bisectable when it was first introduced: https://irclog.perlgeek.de/perl6/2016-05-20#i_12514921
For example, Bisectable was much dumber at that time, and it seemed that it makes sense to require people to exit properly like if they were doing regular git bisect. Nowadays people rely on Bisectable to find out whether it's the output that is different or something else, and also on all of the checks for the behavior on starting points.