mirror/ git
1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-04-27 23:45:12 +02:00
Code
git/t/t4211-line-log.sh
SZEDER Gábor 3cb9d2b6f9 line-log: more responsive, incremental 'git log -L' 
The current line-level log implementation performs a preprocessing
step in prepare_revision_walk(), during which the line_log_filter()
function filters and rewrites history to keep only commits modifying
the given line range.  This preprocessing affects both responsiveness
and correctness:

  - Git doesn't produce any output during this preprocessing step.
    Checking whether a commit modified the given line range is
    somewhat expensive, so depending on the size of the given revision
    range this preprocessing can result in a significant delay before
    the first commit is shown.

  - Limiting the number of displayed commits (e.g. 'git log -3 -L...')
    doesn't limit the amount of work during preprocessing, because
    that limit is applied during history traversal.  Alas, by that
    point this expensive preprocessing step has already churned
    through the whole revision range to find all commits modifying the
    revision range, even though only a few of them need to be shown.

  - It rewrites parents, with no way to turn it off.  Without the user
    explicitly requesting parent rewriting any parent object ID shown
    should be that of the immediate parent, just like in case of a
    pathspec-limited history traversal without parent rewriting.

    However, after that preprocessing step rewrote history, the
    subsequent "regular" history traversal (i.e. get_revision() in a
    loop) only sees commits modifying the given line range.
    Consequently, it can only show the object ID of the last ancestor
    that modified the given line range (which might happen to be the
    immediate parent, but many-many times it isn't).

This patch addresses both the correctness and, at least for the common
case, the responsiveness issues by integrating line-level log
filtering into the regular revision walking machinery:

  - Make process_ranges_arbitrary_commit(), the static function in
    'line-log.c' deciding whether a commit modifies the given line
    range, public by removing the static keyword and adding the
    'line_log_' prefix, so it can be called from other parts of the
    revision walking machinery.

  - If the user didn't explicitly ask for parent rewriting (which, I
    believe, is the most common case):

    - Call this now-public function during regular history traversal,
      namely from get_commit_action() to ignore any commits not
      modifying the given line range.

      Note that while this check is relatively expensive, it must be
      performed before other, much cheaper conditions, because the
      tracked line range must be adjusted even when the commit will
      end up being ignored by other conditions.

    - Skip the line_log_filter() call, i.e. the expensive
      preprocessing step, in prepare_revision_walk(), because, thanks
      to the above points, the revision walking machinery is now able
      to filter out commits not modifying the given line range while
      traversing history.

      This way the regular history traversal sees the unmodified
      history, and is therefore able to print the object ids of the
      immediate parents of the listed commits.  The eliminated
      preprocessing step can greatly reduce the delay before the first
      commit is shown, see the numbers below.

  - However, if the user did explicitly ask for parent rewriting via
    '--parents' or a similar option, then stick with the current
    implementation for now, i.e. perform that expensive filtering and
    history rewriting in the preprocessing step just like we did
    before, leaving the initial delay as long as it was.

I tried to integrate line-level log filtering with parent rewriting
into the regular history traversal, but, unfortunately, several
subtleties resisted... :)  Maybe someday we'll figure out how to do
that, but until then at least the simple and common (i.e. without
parent rewriting) 'git log -L:func:file' commands can benefit from the
reduced delay.

This change makes the failing 'parent oids without parent rewriting'
test in 't4211-line-log.sh' succeed.

The reduced delay is most noticable when there's a commit modifying
the line range near the tip of a large-ish revision range:

  # no parent rewriting requested, no commit-graph present
  $ time git --no-pager log -L:read_alternate_refs:sha1-file.c -1 v2.23.0

  Before:

    real    0m9.570s
    user    0m9.494s
    sys     0m0.076s

  After:

    real    0m0.718s
    user    0m0.674s
    sys     0m0.044s

A significant part of the remaining delay is spent reading and parsing
commit objects in limit_list().  With the help of the commit-graph we
can eliminate most of that reading and parsing overhead, so here are
the timing results of the same command as above, but this time using
the commit-graph:

  Before:

    real    0m8.874s
    user    0m8.816s
    sys     0m0.057s

  After:

    real    0m0.107s
    user    0m0.091s
    sys     0m0.013s

The next patch will further reduce the remaining delay.

To be clear: this patch doesn't actually optimize the line-level log,
but merely moves most of the work from the preprocessing step to the
history traversal, so the commits modifying the line range can be
shown as soon as they are processed, and the traversal can be
terminated as soon as the given number of commits are shown.
Consequently, listing the full history of a line range, potentially
all the way to the root commit, will take the same time as before (but
at least the user might start reading the output earlier).
Furthermore, if the most recent commit modifying the line range is far
away from the starting revision, then that initial delay will still be
significant.

Additional testing by Derrick Stolee: In the Linux kernel repository,
the MAINTAINERS file was changed ~3,500 times across the ~915,000
commits. In addition to that edit frequency, the file itself is quite
large (~18,700 lines). This means that a significant portion of the
computation is taken up by computing the patch-diff of the file. This
patch improves the real time it takes to output the first result quite
a bit:

Command: git log -L 100,200:MAINTAINERS -n 1 >/dev/null
 Before: 3.88 s
  After: 0.71 s

If we drop the "-n 1" in the command, then there is no change in
end-to-end process time. This is because the command still needs to
walk the entire commit history, which negates the point of this
patch. This is expected.

As a note for future reference, the ~4.3 seconds in the old code
spends ~2.6 seconds computing the patch-diffs, and the rest of the
time is spent walking commits and computing diffs for which paths
changed at each commit. The changed-path Bloom filters could improve
the end-to-end computation time (i.e. no "-n 1" in the command).

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-11 09:33:56 -07:00

287 lines
8.1 KiB
Bash
Executable File
Raw Blame History

#!/bin/sh
test_description='test log -L'
. ./test-lib.sh
test_expect_success 'setup (import history)' '
test_oid_init &&
git fast-import < "$TEST_DIRECTORY"/t4211/history.export &&
git reset --hard
'
canned_test_1 () {
test_expect_$1 "$2" "
git log $2 >actual &&
test_cmp \"\$TEST_DIRECTORY\"/t4211/$(test_oid algo)/expect.$3 actual
"
}
canned_test () {
canned_test_1 success "$@"
}
canned_test_failure () {
canned_test_1 failure "$@"
}
test_bad_opts () {
test_expect_success "invalid args: $1" "
test_must_fail git log $1 2>errors &&
test_i18ngrep '$2' errors
"
}
canned_test "-L 4,12:a.c simple" simple-f
canned_test "-L 4,+9:a.c simple" simple-f
canned_test "-L '/long f/,/^}/:a.c' simple" simple-f
canned_test "-L :f:a.c simple" simple-f-to-main
canned_test "-L '/main/,/^}/:a.c' simple" simple-main
canned_test "-L :main:a.c simple" simple-main-to-end
canned_test "-L 1,+4:a.c simple" beginning-of-file
canned_test "-L 20:a.c simple" end-of-file
canned_test "-L '/long f/',/^}/:a.c -L /main/,/^}/:a.c simple" two-ranges
canned_test "-L 24,+1:a.c simple" vanishes-early
canned_test "-M -L '/long f/,/^}/:b.c' move-support" move-support-f
canned_test "-M -L ':f:b.c' parallel-change" parallel-change-f-to-main
canned_test "-L 4,12:a.c -L :main:a.c simple" multiple
canned_test "-L 4,18:a.c -L ^:main:a.c simple" multiple-overlapping
canned_test "-L :main:a.c -L 4,18:a.c simple" multiple-overlapping
canned_test "-L 4:a.c -L 8,12:a.c simple" multiple-superset
canned_test "-L 8,12:a.c -L 4:a.c simple" multiple-superset
test_bad_opts "-L" "switch.*requires a value"
test_bad_opts "-L b.c" "argument not .start,end:file"
test_bad_opts "-L 1:" "argument not .start,end:file"
test_bad_opts "-L 1:nonexistent" "There is no path"
test_bad_opts "-L 1:simple" "There is no path"
test_bad_opts "-L '/foo:b.c'" "argument not .start,end:file"
test_bad_opts "-L 1000:b.c" "has only.*lines"
test_bad_opts "-L :b.c" "argument not .start,end:file"
test_bad_opts "-L :foo:b.c" "no match"
test_expect_success '-L X (X == nlines)' '
n=$(wc -l <b.c) &&
git log -L $n:b.c
'
test_expect_success '-L X (X == nlines + 1)' '
n=$(expr $(wc -l <b.c) + 1) &&
test_must_fail git log -L $n:b.c
'
test_expect_success '-L X (X == nlines + 2)' '
n=$(expr $(wc -l <b.c) + 2) &&
test_must_fail git log -L $n:b.c
'
test_expect_success '-L ,Y (Y == nlines)' '
n=$(printf "%d" $(wc -l <b.c)) &&
git log -L ,$n:b.c
'
test_expect_success '-L ,Y (Y == nlines + 1)' '
n=$(expr $(wc -l <b.c) + 1) &&
git log -L ,$n:b.c
'
test_expect_success '-L ,Y (Y == nlines + 2)' '
n=$(expr $(wc -l <b.c) + 2) &&
git log -L ,$n:b.c
'
test_expect_success '-L with --first-parent and a merge' '
git checkout parallel-change &&
git log --first-parent -L 1,1:b.c
'
test_expect_success '-L with --output' '
git checkout parallel-change &&
git log --output=log -L :main:b.c >output &&
test_must_be_empty output &&
test_line_count = 70 log
'
test_expect_success 'range_set_union' '
test_seq 500 > c.c &&
git add c.c &&
git commit -m "many lines" &&
test_seq 1000 > c.c &&
git add c.c &&
git commit -m "modify many lines" &&
git log $(for x in $(test_seq 200); do echo -L $((2*x)),+1:c.c; done)
'
test_expect_success '-s shows only line-log commits' '
git log --format="commit %s" -L1,24:b.c >expect.raw &&
grep ^commit expect.raw >expect &&
git log --format="commit %s" -L1,24:b.c -s >actual &&
test_cmp expect actual
'
test_expect_success '-p shows the default patch output' '
git log -L1,24:b.c >expect &&
git log -L1,24:b.c -p >actual &&
test_cmp expect actual
'
test_expect_success '--raw is forbidden' '
test_must_fail git log -L1,24:b.c --raw
'
test_expect_success 'setup for checking fancy rename following' '
git checkout --orphan moves-start &&
git reset --hard &&
printf "%s\n" 12 13 14 15 b c d e >file-1 &&
printf "%s\n" 22 23 24 25 B C D E >file-2 &&
git add file-1 file-2 &&
test_tick &&
git commit -m "Add file-1 and file-2" &&
oid_add_f1_f2=$(git rev-parse --short HEAD) &&
git checkout -b moves-main &&
printf "%s\n" 11 12 13 14 15 b c d e >file-1 &&
git commit -a -m "Modify file-1 on main" &&
oid_mod_f1_main=$(git rev-parse --short HEAD) &&
printf "%s\n" 21 22 23 24 25 B C D E >file-2 &&
git commit -a -m "Modify file-2 on main #1" &&
oid_mod_f2_main_1=$(git rev-parse --short HEAD) &&
git mv file-1 renamed-1 &&
git commit -m "Rename file-1 to renamed-1 on main" &&
printf "%s\n" 11 12 13 14 15 b c d e f >renamed-1 &&
git commit -a -m "Modify renamed-1 on main" &&
oid_mod_r1_main=$(git rev-parse --short HEAD) &&
printf "%s\n" 21 22 23 24 25 B C D E F >file-2 &&
git commit -a -m "Modify file-2 on main #2" &&
oid_mod_f2_main_2=$(git rev-parse --short HEAD) &&
git checkout -b moves-side moves-start &&
printf "%s\n" 12 13 14 15 16 b c d e >file-1 &&
git commit -a -m "Modify file-1 on side #1" &&
oid_mod_f1_side_1=$(git rev-parse --short HEAD) &&
printf "%s\n" 22 23 24 25 26 B C D E >file-2 &&
git commit -a -m "Modify file-2 on side" &&
oid_mod_f2_side=$(git rev-parse --short HEAD) &&
git mv file-2 renamed-2 &&
git commit -m "Rename file-2 to renamed-2 on side" &&
printf "%s\n" 12 13 14 15 16 a b c d e >file-1 &&
git commit -a -m "Modify file-1 on side #2" &&
oid_mod_f1_side_2=$(git rev-parse --short HEAD) &&
printf "%s\n" 22 23 24 25 26 A B C D E >renamed-2 &&
git commit -a -m "Modify renamed-2 on side" &&
oid_mod_r2_side=$(git rev-parse --short HEAD) &&
git checkout moves-main &&
git merge moves-side &&
oid_merge=$(git rev-parse --short HEAD)
'
test_expect_success 'fancy rename following #1' '
cat >expect <<-EOF &&
$oid_merge Merge branch '\''moves-side'\'' into moves-main
$oid_mod_f1_side_2 Modify file-1 on side #2
$oid_mod_f1_side_1 Modify file-1 on side #1
$oid_mod_r1_main Modify renamed-1 on main
$oid_mod_f1_main Modify file-1 on main
$oid_add_f1_f2 Add file-1 and file-2
EOF
git log -L1:renamed-1 --oneline --no-patch >actual &&
test_cmp expect actual
'
test_expect_success 'fancy rename following #2' '
cat >expect <<-EOF &&
$oid_merge Merge branch '\''moves-side'\'' into moves-main
$oid_mod_r2_side Modify renamed-2 on side
$oid_mod_f2_side Modify file-2 on side
$oid_mod_f2_main_2 Modify file-2 on main #2
$oid_mod_f2_main_1 Modify file-2 on main #1
$oid_add_f1_f2 Add file-1 and file-2
EOF
git log -L1:renamed-2 --oneline --no-patch >actual &&
test_cmp expect actual
'
# Create the following linear history, where each commit does what its
# subject line promises:
#
# * 66c6410 Modify func2() in file.c
# * 50834e5 Modify other-file
# * fe5851c Modify func1() in file.c
# * 8c7c7dd Add other-file
# * d5f4417 Add func1() and func2() in file.c
test_expect_success 'setup for checking line-log and parent oids' '
git checkout --orphan parent-oids &&
git reset --hard &&
cat >file.c <<-\EOF &&
int func1()
{
return F1;
}
int func2()
{
return F2;
}
EOF
git add file.c &&
test_tick &&
git commit -m "Add func1() and func2() in file.c" &&
echo 1 >other-file &&
git add other-file &&
git commit -m "Add other-file" &&
sed -e "s/F1/F1 + 1/" file.c >tmp &&
mv tmp file.c &&
git commit -a -m "Modify func1() in file.c" &&
echo 2 >other-file &&
git commit -a -m "Modify other-file" &&
sed -e "s/F2/F2 + 2/" file.c >tmp &&
mv tmp file.c &&
git commit -a -m "Modify func2() in file.c" &&
head_oid=$(git rev-parse --short HEAD) &&
prev_oid=$(git rev-parse --short HEAD^) &&
root_oid=$(git rev-parse --short HEAD~4)
'
# Parent oid should be from immediate parent.
test_expect_success 'parent oids without parent rewriting' '
cat >expect <<-EOF &&
$head_oid $prev_oid Modify func2() in file.c
$root_oid Add func1() and func2() in file.c
EOF
git log --format="%h %p %s" --no-patch -L:func2:file.c >actual &&
test_cmp expect actual
'
# Parent oid should be from the most recent ancestor touching func2(),
# i.e. in this case from the root commit.
test_expect_success 'parent oids with parent rewriting' '
cat >expect <<-EOF &&
$head_oid $root_oid Modify func2() in file.c
$root_oid Add func1() and func2() in file.c
EOF
git log --format="%h %p %s" --no-patch -L:func2:file.c --parents >actual &&
test_cmp expect actual
'
test_done
View Git Blame Copy Permalink