Modify the score calculation method of OpenCV #285
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem: The current score in PTS/OpenCV-1.3.0 is the wall time of all cases and it is not a fixed workload, such a score calculation is wrong.
The steps and methods introduced by OpenCV for performance testing are as follows:
https://github.com/opencv/opencv/wiki/HowToUsePerfTests#running-performance-tests-and-analyzing-the-results
• Description of the problem:
The score of PTS/OpenCV-1.3.0, which is the wall time, should be with the fixed work load. But the number of rounds each case runs is not a fixed value.
If the coefficient of variation(standard deviation/mean) of the previous rounds of the case that has been run before is greater than 3.0%, it will continue to run the next round, and if not, it will stop.
The number of rounds is determined according to stability: the coefficient of variation which is clearly indeterminate.
• Example of the problem:
Here is the operation log of a case:
[ RUN ] stitchDatasets_affine.affine/5, where GetParam() = ("newspaper", "akaze")
[ PERFSTAT ] (samples=11 mean=923.62 median=916.97 min=886.05 stddev=27.25 (3.0%))
[ OK ] stitchDatasets_affine.affine/5 (10193 ms)
10193 ms is the running time of the above case. The round number is 11 in the above case. 10193 ms is the computing time for the case running 11 times (923.62ms per round) plus the initialization time. In the same experimental setting, the round number could be 12, or 15, or some other value, then the total time may much greater than 10193 ms. The round number is determined by a random distribution, resulting in a different total code path each time, i.e., a different workload.
Solution:
The proposed score calculation still uses the tool functions provided by OpenCV, which is composed of the geomean of the mean time of all the running cases.