mirror/ git
1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-05-18 21:46:26 +02:00
Code

perf-lib: use a single filename for all measurement types 

The perf tests write files recording the results of tests.  These
results are later aggregated by 'aggregate.perl'.  If the tests are run
multiple times, those results are overwritten by the new results.  This
works just fine as long as there are only perf tests measuring the
times, whose results are stored in "$base".times files.

However 22bec79d1a ("t/perf: add infrastructure for measuring sizes",
2018-08-17) introduced a new type of test for measuring the size of
something.  The results of this are written to "$base".size files.

"$base" is essentially made up of the basename of the script plus the
test number.  So if test numbers shift because a new test was
introduced earlier in the script we might end up with both a ".times"
and a ".size" file for the same test.  In the aggregation script the
".times" file is preferred over the ".size" file, so some size tests
might end with performance numbers from a previous run of the test.

This is mainly relevant when writing perf tests that check both
performance and sizes, and can get quite confusing during
developement.

We could fix this by doing a more thorough job of cleaning out old
".times" and ".size" files before running each test. However, an even
easier solution is to just use the same filename for both types of
measurement, meaning we'll always overwrite the previous result. We
don't even need to change the file format to distinguish the two;
aggregate.perl already decides which is which based on a regex of the
content (this may become ambiguous if we add new types in the future,
but we could easily add a header field to the file at that point).

Based on an initial patch from Thomas Gummerer, who discovered the
problem and did all of the analysis (which I stole for the commit
message above):

  https://public-inbox.org/git/20191119185047.8550-1-t.gummerer@gmail.com/

Helped-by: Thomas Gummerer <t.gummerer@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This commit is contained in:
Jeff King 2019-11-25 09:09:25 -05:00 committed by Junio C Hamano
parent 5fa0f5238b
commit b8dcc45387
2 changed files with 5 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
t/perf/aggregate.perl
View File

@ -219,13 +219,7 @@ sub print_default_results {
for my $i (0..$#dirs) {
my $d = $dirs[$i];
my $base = "$resultsdir/$prefixes{$d}$t";
$times{$prefixes{$d}.$t} = [];
foreach my $type (qw(times size)) {
if (-e "$base.$type") {
$times{$prefixes{$d}.$t} = [get_times("$base.$type")];
last;
}
}
$times{$prefixes{$d}.$t} = [get_times("$base.result")];
my ($r,$u,$s) = @{$times{$prefixes{$d}.$t}};
my $w = length format_times($r,$u,$s,$firstr);
$colwidth[$i] = $w if $w > $colwidth[$i];
@ -267,7 +261,7 @@ sub print_sorted_results {
my ($prevr, $prevu, $prevs, $prevrev);
for my $i (0..$#dirs) {
my $d = $dirs[$i];
my ($r, $u, $s) = get_times("$resultsdir/$prefixes{$d}$t.times");
my ($r, $u, $s) = get_times("$resultsdir/$prefixes{$d}$t.result");
if ($i > 0 and defined $r and defined $prevr and $prevr > 0) {
my $percent = 100.0 * ($r - $prevr) / $prevr;
push @evolutions, { "percent" => $percent,
@ -327,7 +321,7 @@ sub print_codespeed_results {
my $commitid = $prefixes{$d};
$commitid =~ s/^build_//;
$commitid =~ s/\.$//;
my ($result_value, $u, $s) = get_times("$resultsdir/$prefixes{$d}$t.times");
my ($result_value, $u, $s) = get_times("$resultsdir/$prefixes{$d}$t.result");
my %vals = (
"commitid" => $commitid,

4
t/perf/perf-lib.sh
View File

@ -214,7 +214,7 @@ test_perf_ () {
else
test_ok_ "$1"
fi
"$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".times
"$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
}
test_perf () {
@ -223,7 +223,7 @@ test_perf () {
test_size_ () {
say >&3 "running: $2"
if test_eval_ "$2" 3>"$base".size; then
if test_eval_ "$2" 3>"$base".result; then
test_ok_ "$1"
else
test_failure_ "$@"