From 738fab524ce9d666336f90034a3f600876529ffa Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 18 Jul 2024 19:55:45 +0000 Subject: [PATCH 1/2] t5319: add failing test case for repack/expire Git 2.45.0 included the change b7d6f23a171 (midx-write.c: use `--stdin-packs` when repacking, 2024-04-01) which caused the 'git multi-pack-index repack' command to use 'git pack-objects --stdin-packs' instead of listing the objects to repack. While this change was motivated by efficient cross-process communication and the ability to improve delta compression, it breaks a fundamental function of the 'incremental-repack' task that is enabled by default in Scalar clones or Git repositories that run 'git maintenance start'. The 'incremental-repack' task performs a two-step process of the 'expire' and 'repack' subcommands of the 'git multi-pack-index' builtin. The 'expire' command removes any pack-files listed in the multi-pack-index but without any referenced objects. The 'repack' task then finds a batch of pack-files to repack and sends their objects to 'git pack-objects'. Both the pack-files chosen for the batch and the objects chosen to repack are based on the ones that the multi-pack-index references. Objects that appear in a pack-file but have a duplicate copy in a newer pack-file are not considered in this case. Since the multi-pack-index references only the newest copy of an object, this allows the next 'incremental-repack' task to remove the pack-files in the next 'expire' task. This delay is intentional due to how Windows handles may block deletion of files with open read handles. However, the mentioned commit changed this behavior to divorce the set of objects referenced by the multi-pack-index and instead use a set of "included" and "excluded" pack-files in the 'git pack-objects' builtin. When a pack-file is selected as "included", only the objects it contains but are not in any "excluded" pack-files are considered for repacking. This has led to client repositories failing to remove old pack-files as they still have some referenced objects. This grows over time until the point that Git is trying to repack the same pack-files over and over. For now, create a test case that demonstrates the expected behavior, but also fails in its final line. The setup here it attempting to recreate a typical situation for a repository that uses a blobless partial clone. There would be a large initial pack-file from the clone that is never selected in the 'repack' batch. There are other pack-files that have a combination of new objects from incremental fetches and possibly blobs that are not connected to those incremental fetches; these blobs could be filled in from commands like 'git checkout' or 'git blame'. The pack-files also have some overlap on purpose so test-1 has some duplicates in test-2 and test-2 has some duplicates in test-3. At the end of the test, the test-2 pack-file still exists though it should have been expired. This test will pass when reverting the offending commit. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- t/t5319-multi-pack-index.sh | 55 +++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index dd09134db0..327376233c 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -1004,6 +1004,61 @@ test_expect_success 'repack --batch-size= repacks everything' ' ) ' +test_expect_failure 'repack/expire loop' ' + git init repack-expire && + test_when_finished "rm -fr repack-expire" && + ( + cd repack-expire && + + test_commit_bulk 5 && + + # Create three overlapping pack-files + git rev-list --objects HEAD~3 >in-1 && + git rev-list --objects HEAD~4..HEAD~2 >in-2 && + git rev-list --objects HEAD~3..HEAD >in-3 && + + # Create disconnected blobs + obj1=$(git hash-object -w in-1) && + obj2=$(git hash-object -w in-2) && + obj3=$(git hash-object -w in-3) && + + echo $obj2 >>in-2 && + echo $obj3 >>in-3 && + + for i in $(test_seq 3) + do + git pack-objects .git/objects/pack/test-$i Date: Thu, 18 Jul 2024 19:55:46 +0000 Subject: [PATCH 2/2] midx-write: revert use of --stdin-packs This reverts b7d6f23a171 (midx-write.c: use `--stdin-packs` when repacking, 2024-04-01) and then marks the test created in the previous change as passing. The fundamental issue with the reverted change is that the focus on pack-files separates the object selection from how the multi-pack-index selects a single pack-file for an object ID with multiple copies among the tracked pack-files. The change was made with the intention of improving delta compression in the resulting pack-file, but that can be resolved with the existing object list mechanism. There are other potential pitfalls of doing an object walk at this time if the repository is a blobless partial clone, and that will require additional testing on top of the one that changes here. Signed-off-by: Derrick Stolee Acked-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx-write.c | 18 +++++++++--------- t/t5319-multi-pack-index.sh | 2 +- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/midx-write.c b/midx-write.c index 65e69d2de7..960cc46250 100644 --- a/midx-write.c +++ b/midx-write.c @@ -1474,8 +1474,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, repo_config_get_bool(r, "repack.usedeltabaseoffset", &delta_base_offset); repo_config_get_bool(r, "repack.usedeltaislands", &use_delta_islands); - strvec_pushl(&cmd.args, "pack-objects", "--stdin-packs", "--non-empty", - NULL); + strvec_push(&cmd.args, "pack-objects"); strvec_pushf(&cmd.args, "%s/pack/pack", object_dir); @@ -1499,15 +1498,16 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, } cmd_in = xfdopen(cmd.in, "w"); - for (i = 0; i < m->num_packs; i++) { - struct packed_git *p = m->packs[i]; - if (!p) + + for (i = 0; i < m->num_objects; i++) { + struct object_id oid; + uint32_t pack_int_id = nth_midxed_pack_int_id(m, i); + + if (!include_pack[pack_int_id]) continue; - if (include_pack[i]) - fprintf(cmd_in, "%s\n", pack_basename(p)); - else - fprintf(cmd_in, "^%s\n", pack_basename(p)); + nth_midxed_object_oid(&oid, m, i); + fprintf(cmd_in, "%s\n", oid_to_hex(&oid)); } fclose(cmd_in); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 327376233c..5cce0be19e 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -1004,7 +1004,7 @@ test_expect_success 'repack --batch-size= repacks everything' ' ) ' -test_expect_failure 'repack/expire loop' ' +test_expect_success 'repack/expire loop' ' git init repack-expire && test_when_finished "rm -fr repack-expire" && (