mirror of
https://github.com/git/git.git
synced 2024-05-09 07:56:11 +02:00
Merge branch 'tb/pseudo-merge-reachability-bitmap' into seen
The pack-bitmap machinery learned to write pseudo-merge bitmaps, which act as imaginary octopus merges covering un-bitmapped reference tips. This enhances bitmap coverage, and thus, performance, for repositories with many references using bitmaps. * tb/pseudo-merge-reachability-bitmap: (24 commits) t/perf: implement performace tests for pseudo-merge bitmaps pseudo-merge: implement support for finding existing merges ewah: `bitmap_equals_ewah()` pack-bitmap: extra trace2 information pack-bitmap.c: use pseudo-merges during traversal t/test-lib-functions.sh: support `--date` in `test_commit_bulk()` pack-bitmap: implement test helpers for pseudo-merge ewah: implement `ewah_bitmap_popcount()` pseudo-merge: implement support for reading pseudo-merge commits pack-bitmap.c: read pseudo-merge extension pseudo-merge: scaffolding for reads pack-bitmap: extract `read_bitmap()` function pack-bitmap-write.c: write pseudo-merge table pack-bitmap-write.c: select pseudo-merge commits pseudo-merge: implement support for selecting pseudo-merge commits pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()` pack-bitmap-write: support storing pseudo-merge commits pseudo-merge.ch: initial commit pack-bitmap: move some initialization to `bitmap_writer_init()` ...
This commit is contained in:
Junio C Hamano
commit
05e0a59f43
|
|
@ -393,6 +393,8 @@ include::config/apply.txt[]
|
|||
|
||||
include::config/attr.txt[]
|
||||
|
||||
include::config/bitmap-pseudo-merge.txt[]
|
||||
|
||||
include::config/blame.txt[]
|
||||
|
||||
include::config/branch.txt[]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,75 @@
|
|||
bitmapPseudoMerge.<name>.pattern::
|
||||
Regular expression used to match reference names. Commits
|
||||
pointed to by references matching this pattern (and meeting
|
||||
the below criteria, like `bitmapPseudoMerge.<name>.sampleRate`
|
||||
and `bitmapPseudoMerge.<name>.threshold`) will be considered
|
||||
for inclusion in a pseudo-merge bitmap.
|
||||
+
|
||||
Commits are grouped into pseudo-merge groups based on whether or not
|
||||
any reference(s) that point at a given commit match the pattern, which
|
||||
is an extended regular expression.
|
||||
+
|
||||
Within a pseudo-merge group, commits may be further grouped into
|
||||
sub-groups based on the capture groups in the pattern. These
|
||||
sub-groupings are formed from the regular expressions by concatenating
|
||||
any capture groups from the regular expression, with a '-' dash in
|
||||
between.
|
||||
+
|
||||
For example, if the pattern is `refs/tags/`, then all tags (provided
|
||||
they meet the below criteria) will be considered candidates for the
|
||||
same pseudo-merge group. However, if the pattern is instead
|
||||
`refs/remotes/([0-9])+/tags/`, then tags from different remotes will
|
||||
be grouped into separate pseudo-merge groups, based on the remote
|
||||
number.
|
||||
|
||||
bitmapPseudoMerge.<name>.decay::
|
||||
Determines the rate at which consecutive pseudo-merge bitmap
|
||||
groups decrease in size. Must be non-negative. This parameter
|
||||
can be thought of as `k` in the function `f(n) = C *
|
||||
n^(-k/100)`, where `f(n)` is the size of the `n`th group.
|
||||
+
|
||||
Setting the decay rate equal to `0` will cause all groups to be the
|
||||
same size. Setting the decay rate equal to `100` will cause the `n`th
|
||||
group to be `1/n` the size of the initial group. Higher values of the
|
||||
decay rate cause consecutive groups to shrink at an increasing rate.
|
||||
The default is `100`.
|
||||
|
||||
bitmapPseudoMerge.<name>.sampleRate::
|
||||
Determines the proportion of non-bitmapped commits (among
|
||||
reference tips) which are selected for inclusion in an
|
||||
unstable pseudo-merge bitmap. Must be between `0` and `100`
|
||||
(inclusive). The default is `100`.
|
||||
|
||||
bitmapPseudoMerge.<name>.threshold::
|
||||
Determines the minimum age of non-bitmapped commits (among
|
||||
reference tips, as above) which are candidates for inclusion
|
||||
in an unstable pseudo-merge bitmap. The default is
|
||||
`1.week.ago`.
|
||||
|
||||
bitmapPseudoMerge.<name>.maxMerges::
|
||||
Determines the maximum number of pseudo-merge commits among
|
||||
which commits may be distributed.
|
||||
+
|
||||
For pseudo-merge groups whose pattern does not contain any capture
|
||||
groups, this setting is applied for all commits matching the regular
|
||||
expression. For patterns that have one or more capture groups, this
|
||||
setting is applied for each distinct capture group.
|
||||
+
|
||||
For example, if your capture group is `refs/tags/`, then this setting
|
||||
will distribute all tags into a maximum of `maxMerges` pseudo-merge
|
||||
commits. However, if your capture group is, say,
|
||||
`refs/remotes/([0-9]+)/tags/`, then this setting will be applied to
|
||||
each remote's set of tags individually.
|
||||
+
|
||||
Must be non-negative. The default value is 64.
|
||||
|
||||
bitmapPseudoMerge.<name>.stableThreshold::
|
||||
Determines the minimum age of commits (among reference tips,
|
||||
as above, however stable commits are still considered
|
||||
candidates even when they have been covered by a bitmap) which
|
||||
are candidates for a stable a pseudo-merge bitmap. The default
|
||||
is `1.month.ago`.
|
||||
|
||||
bitmapPseudoMerge.<name>.stableSize::
|
||||
Determines the size (in number of commits) of a stable
|
||||
psuedo-merge bitmap. The default is `512`.
|
||||
|
|
@ -255,3 +255,208 @@ triplet is -
|
|||
xor_row (4 byte integer, network byte order): ::
|
||||
The position of the triplet whose bitmap is used to compress
|
||||
this one, or `0xffffffff` if no such bitmap exists.
|
||||
|
||||
Pseudo-merge bitmaps
|
||||
--------------------
|
||||
|
||||
If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of
|
||||
bytes (preceding the name-hash cache, commit lookup table, and trailing
|
||||
checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.
|
||||
|
||||
A "pseudo-merge bitmap" is used to refer to a pair of bitmaps, as
|
||||
follows:
|
||||
|
||||
Commit bitmap::
|
||||
|
||||
A bitmap whose set bits describe the set of commits included in the
|
||||
pseudo-merge's "merge" bitmap (as below).
|
||||
|
||||
Merge bitmap::
|
||||
|
||||
A bitmap whose set bits describe the reachability closure over the set
|
||||
of commits in the pseudo-merge's "commits" bitmap (as above). An
|
||||
identical bitmap would be generated for an octopus merge with the same
|
||||
set of parents as described in the commits bitmap.
|
||||
|
||||
Pseudo-merge bitmaps can accelerate bitmap traversals when all commits
|
||||
for a given pseudo-merge are listed on either side of the traversal,
|
||||
either directly (by explicitly asking for them as part of the `HAVES`
|
||||
or `WANTS`) or indirectly (by encountering them during a fill-in
|
||||
traversal).
|
||||
|
||||
=== Use-cases
|
||||
|
||||
For example, suppose there exists a pseudo-merge bitmap with a large
|
||||
number of commits, all of which are listed in the `WANTS` section of
|
||||
some bitmap traversal query. When pseudo-merge bitmaps are enabled, the
|
||||
bitmap machinery can quickly determine there is a pseudo-merge which
|
||||
satisfies some subset of the wanted objects on either side of the query.
|
||||
Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the
|
||||
resulting bitmap. By contrast, without pseudo-merge bitmaps, we would
|
||||
have to repeat the decompression and `OR`-ing step over a potentially
|
||||
large number of individual bitmaps, which can take proportionally more
|
||||
time.
|
||||
|
||||
Another benefit of pseudo-merges arises when there is some combination
|
||||
of (a) a large number of references, with (b) poor bitmap coverage, and
|
||||
(c) deep, nested trees, making fill-in traversal relatively expensive.
|
||||
For example, suppose that there are a large enough number of tags where
|
||||
bitmapping each of the tags individually is infeasible. Without
|
||||
pseudo-merge bitmaps, computing the result of, say, `git rev-list
|
||||
--use-bitmap-index --count --objects --tags` would likely require a
|
||||
large amount of fill-in traversal. But when a large quantity of those
|
||||
tags are stored together in a pseudo-merge bitmap, the bitmap machinery
|
||||
can take advantage of the fact that we only care about the union of
|
||||
objects reachable from all of those tags, and answer the query much
|
||||
faster.
|
||||
|
||||
=== File format
|
||||
|
||||
If enabled, pseudo-merge bitmaps are stored in an optional section at
|
||||
the end of a `.bitmap` file. The format is as follows:
|
||||
|
||||
....
|
||||
+-------------------------------------------+
|
||||
| .bitmap File |
|
||||
+-------------------------------------------+
|
||||
| |
|
||||
| Pseudo-merge bitmaps (Variable Length) |
|
||||
| +---------------------------+ |
|
||||
| | commits_bitmap (EWAH) | |
|
||||
| +---------------------------+ |
|
||||
| | merge_bitmap (EWAH) | |
|
||||
| +---------------------------+ |
|
||||
| |
|
||||
+-------------------------------------------+
|
||||
| |
|
||||
| Lookup Table |
|
||||
| +------------+--------------+ |
|
||||
| | commit_pos | offset | |
|
||||
| +------------+--------------+ |
|
||||
| | 4 bytes | 8 bytes | |
|
||||
| +------------+--------------+ |
|
||||
| |
|
||||
| Offset Cases: |
|
||||
| ------------- |
|
||||
| |
|
||||
| 1. MSB Unset: single pseudo-merge bitmap |
|
||||
| + offset to pseudo-merge bitmap |
|
||||
| |
|
||||
| 2. MSB Set: multiple pseudo-merges |
|
||||
| + offset to extended lookup table |
|
||||
| |
|
||||
+-------------------------------------------+
|
||||
| |
|
||||
| Extended Lookup Table (Optional) |
|
||||
| |
|
||||
| +----+----------+----------+----------+ |
|
||||
| | N | Offset 1 | .... | Offset N | |
|
||||
| +----+----------+----------+----------+ |
|
||||
| | | 8 bytes | .... | 8 bytes | |
|
||||
| +----+----------+----------+----------+ |
|
||||
| |
|
||||
+-------------------------------------------+
|
||||
| |
|
||||
| Pseudo-merge Metadata |
|
||||
| +------------------+----------------+ |
|
||||
| | # pseudo-merges | # Commits | |
|
||||
| +------------------+----------------+ |
|
||||
| | 4 bytes | 4 bytes | |
|
||||
| +------------------+----------------+ |
|
||||
| |
|
||||
| +------------------+----------------+ |
|
||||
| | Lookup offset | Extension size | |
|
||||
| +------------------+----------------+ |
|
||||
| | 8 bytes | 8 bytes | |
|
||||
| +------------------+----------------+ |
|
||||
| |
|
||||
+-------------------------------------------+
|
||||
....
|
||||
|
||||
* One or more pseudo-merge bitmaps, each containing:
|
||||
|
||||
** `commits_bitmap`, an EWAH-compressed bitmap describing the set of
|
||||
commits included in the this psuedo-merge.
|
||||
|
||||
** `merge_bitmap`, an EWAH-compressed bitmap describing the union of
|
||||
the set of objects reachable from all commits listed in the
|
||||
`commits_bitmap`.
|
||||
|
||||
* A lookup table, mapping pseudo-merged commits to the pseudo-merges
|
||||
they belong to. Entries appear in increasing order of each commit's
|
||||
bit position. Each entry is 12 bytes wide, and is comprised of the
|
||||
following:
|
||||
|
||||
** `commit_pos`, a 4-byte unsigned value (in network byte-order)
|
||||
containing the bit position for this commit.
|
||||
|
||||
** `offset`, an 8-byte unsigned value (also in network byte-order)
|
||||
containing either one of two possible offsets, depending on whether or
|
||||
not the most-significant bit is set.
|
||||
|
||||
*** If unset (i.e. `offset & ((uint64_t)1<<63) == 0`), the offset
|
||||
(relative to the beginning of the `.bitmap` file) at which the
|
||||
pseudo-merge bitmap for this commit can be read. This indicates
|
||||
only a single pseudo-merge bitmap contains this commit.
|
||||
|
||||
*** If set (i.e. `offset & ((uint64_t)1<<63) != 0`), the offset
|
||||
(again relative to the beginning of the `.bitmap` file) at which
|
||||
the extended offset table can be located describing the set of
|
||||
pseudo-merge bitmaps which contain this commit. This indicates
|
||||
that multiple pseudo-merge bitmaps contain this commit.
|
||||
|
||||
* An (optional) extended lookup table (written if and only if there is
|
||||
at least one commit which appears in more than one pseudo-merge).
|
||||
There are as many entries as commits which appear in multiple
|
||||
pseudo-merges. Each entry contains the following:
|
||||
|
||||
** `N`, a 4-byte unsigned value equal to the number of pseudo-merges
|
||||
which contain a given commit.
|
||||
|
||||
** An array of `N` 8-byte unsigned values, each of which is
|
||||
interpreted as an offset (relative to the beginning of the
|
||||
`.bitmap` file) at which a pseudo-merge bitmap for this commit can
|
||||
be read. These values occur in no particular order.
|
||||
|
||||
* Positions for all pseudo-merges, each stored as an 8-byte unsigned
|
||||
value (in network byte-order) containing the offset (relative to the
|
||||
beginnign of the `.bitmap` file) of each consecutive pseudo-merge.
|
||||
|
||||
* A 4-byte unsigned value (in network byte-order) equal to the number of
|
||||
pseudo-merges.
|
||||
|
||||
* A 4-byte unsigned value (in network byte-order) equal to the number of
|
||||
unique commits which appear in any pseudo-merge.
|
||||
|
||||
* An 8-byte unsigned value (in network byte-order) equal to the number
|
||||
of bytes between the start of the pseudo-merge section and the
|
||||
beginning of the lookup table.
|
||||
|
||||
* An 8-byte unsigned value (in network byte-order) equal to the number
|
||||
of bytes in the pseudo-merge section (including this field).
|
||||
|
||||
=== Pseudo-merge selection
|
||||
|
||||
Pseudo-merge commits are selected among non-bitmapped commits at the
|
||||
tip of one or more reference(s). In addition, there are a handful of
|
||||
constraints to further refine this selection:
|
||||
|
||||
`pack.bitmapPseudoMergeDecay`:: Defines the "decay rate", which
|
||||
corresponds to how quickly (or not) consecutive pseudo-merges decrease
|
||||
in size relative to one another.
|
||||
|
||||
`pack.bitmapPseudoMergeGroups`:: Defines the maximum number of
|
||||
pseudo-merge groups.
|
||||
|
||||
`pack.bitmapPseudoMergeSampleRate`:: Defines the percentage of commits
|
||||
(matching the above criteria) which are selected.
|
||||
|
||||
`pack.bitmapPseudoMergeThreshold`:: Defines the minimum age of a commit
|
||||
in order to be considered for inclusion within one or more pseudo-merge
|
||||
bitmaps.
|
||||
|
||||
The size of consecutive pseudo-merge groups decays according to a
|
||||
power-law decay function, where the size of the `n`-th group is `f(n) =
|
||||
C*n^-k`. The value of `C` is chosen accordingly to match the number of
|
||||
desired groups, and `k` is 1/100th of the value of
|
||||
`pack.bitmapPseudoMergeDecay`.
|
||||
|
|
|
|||
1
Makefile
1
Makefile
|
|
@ -1104,6 +1104,7 @@ LIB_OBJS += prompt.o
|
|||
LIB_OBJS += protocol.o
|
||||
LIB_OBJS += protocol-caps.o
|
||||
LIB_OBJS += prune-packed.o
|
||||
LIB_OBJS += pseudo-merge.o
|
||||
LIB_OBJS += quote.o
|
||||
LIB_OBJS += range-diff.o
|
||||
LIB_OBJS += reachable.o
|
||||
|
|
|
|||
|
|
@ -1339,6 +1339,7 @@ static void write_pack_file(void)
|
|||
hash_to_hex(hash));
|
||||
|
||||
if (write_bitmap_index) {
|
||||
bitmap_writer_init(the_repository);
|
||||
bitmap_writer_set_checksum(hash);
|
||||
bitmap_writer_build_type_index(
|
||||
&to_pack, written_list, nr_written);
|
||||
|
|
@ -1359,7 +1360,7 @@ static void write_pack_file(void)
|
|||
stop_progress(&progress_state);
|
||||
|
||||
bitmap_writer_show_progress(progress);
|
||||
bitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);
|
||||
bitmap_writer_select_commits(indexed_commits, indexed_commits_nr);
|
||||
if (bitmap_writer_build(&to_pack) < 0)
|
||||
die(_("failed to write bitmap index"));
|
||||
bitmap_writer_finish(written_list, nr_written,
|
||||
|
|
|
|||
18
config.c
18
config.c
|
|
@ -2654,6 +2654,24 @@ int repo_config_get_pathname(struct repository *repo,
|
|||
return ret;
|
||||
}
|
||||
|
||||
int repo_config_get_expiry(struct repository *repo,
|
||||
const char *key, const char **dest)
|
||||
{
|
||||
int ret;
|
||||
|
||||
git_config_check_init(repo);
|
||||
|
||||
ret = repo_config_get_string(repo, key, (char **)dest);
|
||||
if (ret)
|
||||
return ret;
|
||||
if (strcmp(*dest, "now")) {
|
||||
timestamp_t now = approxidate("now");
|
||||
if (approxidate(*dest) >= now)
|
||||
git_die_config(key, _("Invalid %s: '%s'"), key, *dest);
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
/* Read values into protected_config. */
|
||||
static void read_protected_config(void)
|
||||
{
|
||||
|
|
|
|||
2
config.h
2
config.h
|
|
@ -578,6 +578,8 @@ int repo_config_get_maybe_bool(struct repository *repo,
|
|||
const char *key, int *dest);
|
||||
int repo_config_get_pathname(struct repository *repo,
|
||||
const char *key, const char **dest);
|
||||
int repo_config_get_expiry(struct repository *repo,
|
||||
const char *key, const char **dest);
|
||||
|
||||
/*
|
||||
* Functions for reading protected config. By definition, protected
|
||||
|
|
|
|||
|
|
@ -138,6 +138,49 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other)
|
|||
self->words[i] |= other->words[i];
|
||||
}
|
||||
|
||||
int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other)
|
||||
{
|
||||
struct ewah_iterator it;
|
||||
eword_t word;
|
||||
size_t i;
|
||||
|
||||
ewah_iterator_init(&it, self);
|
||||
|
||||
for (i = 0; i < other->word_alloc; i++) {
|
||||
if (!ewah_iterator_next(&word, &it)) {
|
||||
/*
|
||||
* If we reached the end of `self`, and haven't
|
||||
* rejected `self` as a possible subset of
|
||||
* `other` yet, then we are done and `self` is
|
||||
* indeed a subset of `other`.
|
||||
*/
|
||||
return 1;
|
||||
}
|
||||
if (word & ~other->words[i]) {
|
||||
/*
|
||||
* Otherwise, compare the next two pairs of
|
||||
* words. If the word from `self` has bit(s) not
|
||||
* in the word from `other`, `self` is not a
|
||||
* proper subset of `other`.
|
||||
*/
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* If we got to this point, there may be zero or more words
|
||||
* remaining in `self`, with no remaining words left in `other`.
|
||||
* If there are any bits set in the remaining word(s) in `self`,
|
||||
* then `self` is not a proper subset of `other`.
|
||||
*/
|
||||
while (ewah_iterator_next(&word, &it))
|
||||
if (word)
|
||||
return 0;
|
||||
|
||||
/* `self` is definitely a subset of `other` */
|
||||
return 1;
|
||||
}
|
||||
|
||||
void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)
|
||||
{
|
||||
size_t original_size = self->word_alloc;
|
||||
|
|
@ -169,6 +212,20 @@ size_t bitmap_popcount(struct bitmap *self)
|
|||
return count;
|
||||
}
|
||||
|
||||
size_t ewah_bitmap_popcount(struct ewah_bitmap *self)
|
||||
{
|
||||
struct ewah_iterator it;
|
||||
eword_t word;
|
||||
size_t count = 0;
|
||||
|
||||
ewah_iterator_init(&it, self);
|
||||
|
||||
while (ewah_iterator_next(&word, &it))
|
||||
count += ewah_bit_popcount64(word);
|
||||
|
||||
return count;
|
||||
}
|
||||
|
||||
int bitmap_is_empty(struct bitmap *self)
|
||||
{
|
||||
size_t i;
|
||||
|
|
@ -204,6 +261,25 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)
|
|||
return 1;
|
||||
}
|
||||
|
||||
int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other)
|
||||
{
|
||||
struct ewah_iterator it;
|
||||
eword_t word;
|
||||
size_t i = 0;
|
||||
|
||||
ewah_iterator_init(&it, other);
|
||||
|
||||
while (ewah_iterator_next(&word, &it))
|
||||
if (word != (i < self->word_alloc ? self->words[i++] : 0))
|
||||
return 0;
|
||||
|
||||
for (; i < self->word_alloc; i++)
|
||||
if (self->words[i])
|
||||
return 0;
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
int bitmap_is_subset(struct bitmap *self, struct bitmap *other)
|
||||
{
|
||||
size_t common_size, i;
|
||||
|
|
|
|||
|
|
@ -179,7 +179,9 @@ void bitmap_unset(struct bitmap *self, size_t pos);
|
|||
int bitmap_get(struct bitmap *self, size_t pos);
|
||||
void bitmap_free(struct bitmap *self);
|
||||
int bitmap_equals(struct bitmap *self, struct bitmap *other);
|
||||
int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other);
|
||||
int bitmap_is_subset(struct bitmap *self, struct bitmap *other);
|
||||
int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other);
|
||||
|
||||
struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);
|
||||
struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);
|
||||
|
|
@ -189,6 +191,7 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other);
|
|||
void bitmap_or(struct bitmap *self, const struct bitmap *other);
|
||||
|
||||
size_t bitmap_popcount(struct bitmap *self);
|
||||
size_t ewah_bitmap_popcount(struct ewah_bitmap *self);
|
||||
int bitmap_is_empty(struct bitmap *self);
|
||||
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -819,6 +819,7 @@ static int write_midx_bitmap(const char *midx_name,
|
|||
for (i = 0; i < pdata->nr_objects; i++)
|
||||
index[i] = &pdata->objects[i].idx;
|
||||
|
||||
bitmap_writer_init(the_repository);
|
||||
bitmap_writer_show_progress(flags & MIDX_PROGRESS);
|
||||
bitmap_writer_build_type_index(pdata, index, pdata->nr_objects);
|
||||
|
||||
|
|
@ -838,7 +839,7 @@ static int write_midx_bitmap(const char *midx_name,
|
|||
for (i = 0; i < pdata->nr_objects; i++)
|
||||
index[pack_order[i]] = &pdata->objects[i].idx;
|
||||
|
||||
bitmap_writer_select_commits(commits, commits_nr, -1);
|
||||
bitmap_writer_select_commits(commits, commits_nr);
|
||||
ret = bitmap_writer_build(pdata);
|
||||
if (ret < 0)
|
||||
goto cleanup;
|
||||
|
|
|
|||
|
|
@ -17,6 +17,12 @@
|
|||
#include "trace2.h"
|
||||
#include "tree.h"
|
||||
#include "tree-walk.h"
|
||||
#include "pseudo-merge.h"
|
||||
#include "oid-array.h"
|
||||
#include "config.h"
|
||||
#include "alloc.h"
|
||||
#include "refs.h"
|
||||
#include "strmap.h"
|
||||
|
||||
struct bitmapped_commit {
|
||||
struct commit *commit;
|
||||
|
|
@ -24,7 +30,7 @@ struct bitmapped_commit {
|
|||
struct ewah_bitmap *write_as;
|
||||
int flags;
|
||||
int xor_offset;
|
||||
uint32_t commit_pos;
|
||||
unsigned pseudo_merge : 1;
|
||||
};
|
||||
|
||||
struct bitmap_writer {
|
||||
|
|
@ -39,6 +45,10 @@ struct bitmap_writer {
|
|||
struct bitmapped_commit *selected;
|
||||
unsigned int selected_nr, selected_alloc;
|
||||
|
||||
struct string_list pseudo_merge_groups;
|
||||
kh_oid_map_t *pseudo_merge_commits; /* oid -> pseudo merge(s) */
|
||||
uint32_t pseudo_merges_nr;
|
||||
|
||||
struct progress *progress;
|
||||
int show_progress;
|
||||
unsigned char pack_checksum[GIT_MAX_RAWSZ];
|
||||
|
|
@ -46,6 +56,21 @@ struct bitmap_writer {
|
|||
|
||||
static struct bitmap_writer writer;
|
||||
|
||||
static inline int bitmap_writer_selected_nr(void)
|
||||
{
|
||||
return writer.selected_nr - writer.pseudo_merges_nr;
|
||||
}
|
||||
|
||||
void bitmap_writer_init(struct repository *r)
|
||||
{
|
||||
writer.bitmaps = kh_init_oid_map();
|
||||
writer.pseudo_merge_commits = kh_init_oid_map();
|
||||
|
||||
string_list_init_dup(&writer.pseudo_merge_groups);
|
||||
|
||||
load_pseudo_merges_from_config(&writer.pseudo_merge_groups);
|
||||
}
|
||||
|
||||
void bitmap_writer_show_progress(int show)
|
||||
{
|
||||
writer.show_progress = show;
|
||||
|
|
@ -111,20 +136,39 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,
|
|||
}
|
||||
}
|
||||
|
||||
int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid)
|
||||
{
|
||||
return kh_get_oid_map(writer.bitmaps, *oid) != kh_end(writer.bitmaps);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the actual bitmaps
|
||||
*/
|
||||
|
||||
static inline void push_bitmapped_commit(struct commit *commit)
|
||||
void bitmap_writer_push_bitmapped_commit(struct commit *commit,
|
||||
unsigned pseudo_merge)
|
||||
{
|
||||
if (writer.selected_nr >= writer.selected_alloc) {
|
||||
writer.selected_alloc = (writer.selected_alloc + 32) * 2;
|
||||
REALLOC_ARRAY(writer.selected, writer.selected_alloc);
|
||||
}
|
||||
|
||||
if (!pseudo_merge) {
|
||||
int hash_ret;
|
||||
khiter_t hash_pos = kh_put_oid_map(writer.bitmaps,
|
||||
commit->object.oid,
|
||||
&hash_ret);
|
||||
|
||||
if (!hash_ret)
|
||||
die(_("duplicate entry when writing bitmap index: %s"),
|
||||
oid_to_hex(&commit->object.oid));
|
||||
kh_value(writer.bitmaps, hash_pos) = NULL;
|
||||
}
|
||||
|
||||
writer.selected[writer.selected_nr].commit = commit;
|
||||
writer.selected[writer.selected_nr].bitmap = NULL;
|
||||
writer.selected[writer.selected_nr].flags = 0;
|
||||
writer.selected[writer.selected_nr].pseudo_merge = pseudo_merge;
|
||||
|
||||
writer.selected_nr++;
|
||||
}
|
||||
|
|
@ -154,16 +198,20 @@ static void compute_xor_offsets(void)
|
|||
|
||||
while (next < writer.selected_nr) {
|
||||
struct bitmapped_commit *stored = &writer.selected[next];
|
||||
|
||||
int best_offset = 0;
|
||||
struct ewah_bitmap *best_bitmap = stored->bitmap;
|
||||
struct ewah_bitmap *test_xor;
|
||||
|
||||
if (stored->pseudo_merge)
|
||||
goto next;
|
||||
|
||||
for (i = 1; i <= MAX_XOR_OFFSET_SEARCH; ++i) {
|
||||
int curr = next - i;
|
||||
|
||||
if (curr < 0)
|
||||
break;
|
||||
if (writer.selected[curr].pseudo_merge)
|
||||
continue;
|
||||
|
||||
test_xor = ewah_pool_new();
|
||||
ewah_xor(writer.selected[curr].bitmap, stored->bitmap, test_xor);
|
||||
|
|
@ -179,6 +227,7 @@ static void compute_xor_offsets(void)
|
|||
}
|
||||
}
|
||||
|
||||
next:
|
||||
stored->xor_offset = best_offset;
|
||||
stored->write_as = best_bitmap;
|
||||
|
||||
|
|
@ -191,7 +240,8 @@ struct bb_commit {
|
|||
struct bitmap *commit_mask;
|
||||
struct bitmap *bitmap;
|
||||
unsigned selected:1,
|
||||
maximal:1;
|
||||
maximal:1,
|
||||
pseudo_merge:1;
|
||||
unsigned idx; /* within selected array */
|
||||
};
|
||||
|
||||
|
|
@ -229,17 +279,18 @@ static void bitmap_builder_init(struct bitmap_builder *bb,
|
|||
revs.first_parent_only = 1;
|
||||
|
||||
for (i = 0; i < writer->selected_nr; i++) {
|
||||
struct commit *c = writer->selected[i].commit;
|
||||
struct bb_commit *ent = bb_data_at(&bb->data, c);
|
||||
struct bitmapped_commit *bc = &writer->selected[i];
|
||||
struct bb_commit *ent = bb_data_at(&bb->data, bc->commit);
|
||||
|
||||
ent->selected = 1;
|
||||
ent->maximal = 1;
|
||||
ent->pseudo_merge = bc->pseudo_merge;
|
||||
ent->idx = i;
|
||||
|
||||
ent->commit_mask = bitmap_new();
|
||||
bitmap_set(ent->commit_mask, i);
|
||||
|
||||
add_pending_object(&revs, &c->object, "");
|
||||
add_pending_object(&revs, &bc->commit->object, "");
|
||||
}
|
||||
|
||||
if (prepare_revision_walk(&revs))
|
||||
|
|
@ -396,6 +447,7 @@ static int fill_bitmap_tree(struct bitmap *bitmap,
|
|||
}
|
||||
|
||||
static int reused_bitmaps_nr;
|
||||
static int reused_pseudo_merge_bitmaps_nr;
|
||||
|
||||
static int fill_bitmap_commit(struct bb_commit *ent,
|
||||
struct commit *commit,
|
||||
|
|
@ -416,8 +468,13 @@ static int fill_bitmap_commit(struct bb_commit *ent,
|
|||
struct commit *c = prio_queue_get(queue);
|
||||
|
||||
if (old_bitmap && mapping) {
|
||||
struct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);
|
||||
struct ewah_bitmap *old;
|
||||
struct bitmap *remapped = bitmap_new();
|
||||
|
||||
if (commit->object.flags & BITMAP_PSEUDO_MERGE)
|
||||
old = pseudo_merge_bitmap_for_commit(old_bitmap, c);
|
||||
else
|
||||
old = bitmap_for_commit(old_bitmap, c);
|
||||
/*
|
||||
* If this commit has an old bitmap, then translate that
|
||||
* bitmap and add its bits to this one. No need to walk
|
||||
|
|
@ -426,7 +483,10 @@ static int fill_bitmap_commit(struct bb_commit *ent,
|
|||
if (old && !rebuild_bitmap(mapping, old, remapped)) {
|
||||
bitmap_or(ent->bitmap, remapped);
|
||||
bitmap_free(remapped);
|
||||
reused_bitmaps_nr++;
|
||||
if (commit->object.flags & BITMAP_PSEUDO_MERGE)
|
||||
reused_pseudo_merge_bitmaps_nr++;
|
||||
else
|
||||
reused_bitmaps_nr++;
|
||||
continue;
|
||||
}
|
||||
bitmap_free(remapped);
|
||||
|
|
@ -436,12 +496,14 @@ static int fill_bitmap_commit(struct bb_commit *ent,
|
|||
* Mark ourselves and queue our tree. The commit
|
||||
* walk ensures we cover all parents.
|
||||
*/
|
||||
pos = find_object_pos(&c->object.oid, &found);
|
||||
if (!found)
|
||||
return -1;
|
||||
bitmap_set(ent->bitmap, pos);
|
||||
prio_queue_put(tree_queue,
|
||||
repo_get_commit_tree(the_repository, c));
|
||||
if (!(c->object.flags & BITMAP_PSEUDO_MERGE)) {
|
||||
pos = find_object_pos(&c->object.oid, &found);
|
||||
if (!found)
|
||||
return -1;
|
||||
bitmap_set(ent->bitmap, pos);
|
||||
prio_queue_put(tree_queue,
|
||||
repo_get_commit_tree(the_repository, c));
|
||||
}
|
||||
|
||||
for (p = c->parents; p; p = p->next) {
|
||||
pos = find_object_pos(&p->item->object.oid, &found);
|
||||
|
|
@ -466,14 +528,17 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)
|
|||
{
|
||||
struct bitmapped_commit *stored = &writer.selected[ent->idx];
|
||||
khiter_t hash_pos;
|
||||
int hash_ret;
|
||||
|
||||
stored->bitmap = bitmap_to_ewah(ent->bitmap);
|
||||
|
||||
hash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);
|
||||
if (hash_ret == 0)
|
||||
die("Duplicate entry when writing index: %s",
|
||||
if (ent->pseudo_merge)
|
||||
return;
|
||||
|
||||
hash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);
|
||||
if (hash_pos == kh_end(writer.bitmaps))
|
||||
die(_("attempted to store non-selected commit: '%s'"),
|
||||
oid_to_hex(&commit->object.oid));
|
||||
|
||||
kh_value(writer.bitmaps, hash_pos) = stored;
|
||||
}
|
||||
|
||||
|
|
@ -488,7 +553,6 @@ int bitmap_writer_build(struct packing_data *to_pack)
|
|||
uint32_t *mapping;
|
||||
int closed = 1; /* until proven otherwise */
|
||||
|
||||
writer.bitmaps = kh_init_oid_map();
|
||||
writer.to_pack = to_pack;
|
||||
|
||||
if (writer.show_progress)
|
||||
|
|
@ -548,6 +612,9 @@ int bitmap_writer_build(struct packing_data *to_pack)
|
|||
the_repository);
|
||||
trace2_data_intmax("pack-bitmap-write", the_repository,
|
||||
"building_bitmaps_reused", reused_bitmaps_nr);
|
||||
trace2_data_intmax("pack-bitmap-write", the_repository,
|
||||
"building_bitmaps_pseudo_merge_reused",
|
||||
reused_pseudo_merge_bitmaps_nr);
|
||||
|
||||
stop_progress(&writer.progress);
|
||||
|
||||
|
|
@ -591,8 +658,7 @@ static int date_compare(const void *_a, const void *_b)
|
|||
}
|
||||
|
||||
void bitmap_writer_select_commits(struct commit **indexed_commits,
|
||||
unsigned int indexed_commits_nr,
|
||||
int max_bitmaps)
|
||||
unsigned int indexed_commits_nr)
|
||||
{
|
||||
unsigned int i = 0, j, next;
|
||||
|
||||
|
|
@ -600,7 +666,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,
|
|||
|
||||
if (indexed_commits_nr < 100) {
|
||||
for (i = 0; i < indexed_commits_nr; ++i)
|
||||
push_bitmapped_commit(indexed_commits[i]);
|
||||
bitmap_writer_push_bitmapped_commit(indexed_commits[i], 0);
|
||||
return;
|
||||
}
|
||||
|
||||
|
|
@ -615,11 +681,6 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,
|
|||
if (i + next >= indexed_commits_nr)
|
||||
break;
|
||||
|
||||
if (max_bitmaps > 0 && writer.selected_nr >= max_bitmaps) {
|
||||
writer.selected_nr = max_bitmaps;
|
||||
break;
|
||||
}
|
||||
|
||||
if (next == 0) {
|
||||
chosen = indexed_commits[i];
|
||||
} else {
|
||||
|
|
@ -638,13 +699,19 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,
|
|||
}
|
||||
}
|
||||
|
||||
push_bitmapped_commit(chosen);
|
||||
bitmap_writer_push_bitmapped_commit(chosen, 0);
|
||||
|
||||
i += next + 1;
|
||||
display_progress(writer.progress, i);
|
||||
}
|
||||
|
||||
stop_progress(&writer.progress);
|
||||
|
||||
select_pseudo_merges(&writer.pseudo_merge_groups,
|
||||
indexed_commits, indexed_commits_nr,
|
||||
writer.pseudo_merge_commits,
|
||||
&writer.pseudo_merges_nr,
|
||||
writer.show_progress);
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -676,8 +743,11 @@ static void write_selected_commits_v1(struct hashfile *f,
|
|||
{
|
||||
int i;
|
||||
|
||||
for (i = 0; i < writer.selected_nr; ++i) {
|
||||
for (i = 0; i < bitmap_writer_selected_nr(); ++i) {
|
||||
struct bitmapped_commit *stored = &writer.selected[i];
|
||||
if (stored->pseudo_merge)
|
||||
BUG("unexpected pseudo-merge among selected: %s",
|
||||
oid_to_hex(&stored->commit->object.oid));
|
||||
|
||||
if (offsets)
|
||||
offsets[i] = hashfile_total(f);
|
||||
|
|
@ -690,6 +760,127 @@ static void write_selected_commits_v1(struct hashfile *f,
|
|||
}
|
||||
}
|
||||
|
||||
static void write_pseudo_merges(struct hashfile *f)
|
||||
{
|
||||
struct oid_array commits = OID_ARRAY_INIT;
|
||||
struct bitmap **commits_bitmap = NULL;
|
||||
off_t *pseudo_merge_ofs = NULL;
|
||||
off_t start, table_start, next_ext;
|
||||
|
||||
uint32_t base = bitmap_writer_selected_nr();
|
||||
size_t i, j = 0;
|
||||
|
||||
CALLOC_ARRAY(commits_bitmap, writer.pseudo_merges_nr);
|
||||
CALLOC_ARRAY(pseudo_merge_ofs, writer.pseudo_merges_nr);
|
||||
|
||||
for (i = 0; i < writer.pseudo_merges_nr; i++) {
|
||||
struct bitmapped_commit *merge = &writer.selected[base + i];
|
||||
struct commit_list *p;
|
||||
|
||||
if (!merge->pseudo_merge)
|
||||
BUG("found non-pseudo merge commit at %"PRIuMAX, (uintmax_t)i);
|
||||
|
||||
commits_bitmap[i] = bitmap_new();
|
||||
|
||||
for (p = merge->commit->parents; p; p = p->next)
|
||||
bitmap_set(commits_bitmap[i],
|
||||
find_object_pos(&p->item->object.oid, NULL));
|
||||
}
|
||||
|
||||
start = hashfile_total(f);
|
||||
|
||||
for (i = 0; i < writer.pseudo_merges_nr; i++) {
|
||||
struct ewah_bitmap *commits_ewah = bitmap_to_ewah(commits_bitmap[i]);
|
||||
|
||||
pseudo_merge_ofs[i] = hashfile_total(f);
|
||||
|
||||
dump_bitmap(f, commits_ewah);
|
||||
dump_bitmap(f, writer.selected[base+i].write_as);
|
||||
|
||||
ewah_free(commits_ewah);
|
||||
}
|
||||
|
||||
next_ext = st_add(hashfile_total(f),
|
||||
st_mult(kh_size(writer.pseudo_merge_commits),
|
||||
sizeof(uint64_t)));
|
||||
|
||||
table_start = hashfile_total(f);
|
||||
|
||||
commits.alloc = kh_size(writer.pseudo_merge_commits);
|
||||
CALLOC_ARRAY(commits.oid, commits.alloc);
|
||||
|
||||
for (i = kh_begin(writer.pseudo_merge_commits); i != kh_end(writer.pseudo_merge_commits); i++) {
|
||||
if (!kh_exist(writer.pseudo_merge_commits, i))
|
||||
continue;
|
||||
oid_array_append(&commits, &kh_key(writer.pseudo_merge_commits, i));
|
||||
}
|
||||
|
||||
oid_array_sort(&commits);
|
||||
|
||||
/* write lookup table (non-extended) */
|
||||
for (i = 0; i < commits.nr; i++) {
|
||||
int hash_pos;
|
||||
struct pseudo_merge_commit_idx *c;
|
||||
|
||||
hash_pos = kh_get_oid_map(writer.pseudo_merge_commits,
|
||||
commits.oid[i]);
|
||||
if (hash_pos == kh_end(writer.pseudo_merge_commits))
|
||||
BUG("could not find pseudo-merge commit %s",
|
||||
oid_to_hex(&commits.oid[i]));
|
||||
|
||||
c = kh_value(writer.pseudo_merge_commits, hash_pos);
|
||||
|
||||
hashwrite_be32(f, find_object_pos(&commits.oid[i], NULL));
|
||||
if (c->nr == 1)
|
||||
hashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[0]]);
|
||||
else if (c->nr > 1) {
|
||||
if (next_ext & ((uint64_t)1<<63))
|
||||
die(_("too many pseudo-merges"));
|
||||
hashwrite_be64(f, next_ext | ((uint64_t)1<<63));
|
||||
next_ext = st_add3(next_ext,
|
||||
sizeof(uint32_t),
|
||||
st_mult(c->nr, sizeof(uint64_t)));
|
||||
} else
|
||||
BUG("expected commit '%s' to have at least one "
|
||||
"pseudo-merge", oid_to_hex(&commits.oid[i]));
|
||||
}
|
||||
|
||||
/* write lookup table (extended) */
|
||||
for (i = 0; i < commits.nr; i++) {
|
||||
int hash_pos;
|
||||
struct pseudo_merge_commit_idx *c;
|
||||
|
||||
hash_pos = kh_get_oid_map(writer.pseudo_merge_commits,
|
||||
commits.oid[i]);
|
||||
if (hash_pos == kh_end(writer.pseudo_merge_commits))
|
||||
BUG("could not find pseudo-merge commit %s",
|
||||
oid_to_hex(&commits.oid[i]));
|
||||
|
||||
c = kh_value(writer.pseudo_merge_commits, hash_pos);
|
||||
if (c->nr == 1)
|
||||
continue;
|
||||
|
||||
hashwrite_be32(f, c->nr);
|
||||
for (j = 0; j < c->nr; j++)
|
||||
hashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[j]]);
|
||||
}
|
||||
|
||||
/* write positions for all pseudo merges */
|
||||
for (i = 0; i < writer.pseudo_merges_nr; i++)
|
||||
hashwrite_be64(f, pseudo_merge_ofs[i]);
|
||||
|
||||
hashwrite_be32(f, writer.pseudo_merges_nr);
|
||||
hashwrite_be32(f, kh_size(writer.pseudo_merge_commits));
|
||||
hashwrite_be64(f, table_start - start);
|
||||
hashwrite_be64(f, hashfile_total(f) - start + sizeof(uint64_t));
|
||||
|
||||
for (i = 0; i < writer.pseudo_merges_nr; i++)
|
||||
bitmap_free(commits_bitmap[i]);
|
||||
|
||||
free(pseudo_merge_ofs);
|
||||
free(commits_bitmap);
|
||||
}
|
||||
|
||||
static int table_cmp(const void *_va, const void *_vb, void *_data)
|
||||
{
|
||||
uint32_t *commit_positions = _data;
|
||||
|
|
@ -711,10 +902,10 @@ static void write_lookup_table(struct hashfile *f,
|
|||
uint32_t i;
|
||||
uint32_t *table, *table_inv;
|
||||
|
||||
ALLOC_ARRAY(table, writer.selected_nr);
|
||||
ALLOC_ARRAY(table_inv, writer.selected_nr);
|
||||
ALLOC_ARRAY(table, bitmap_writer_selected_nr());
|
||||
ALLOC_ARRAY(table_inv, bitmap_writer_selected_nr());
|
||||
|
||||
for (i = 0; i < writer.selected_nr; i++)
|
||||
for (i = 0; i < bitmap_writer_selected_nr(); i++)
|
||||
table[i] = i;
|
||||
|
||||
/*
|
||||
|
|
@ -722,16 +913,16 @@ static void write_lookup_table(struct hashfile *f,
|
|||
* bitmap corresponds to j'th bitmapped commit (among the selected
|
||||
* commits) in lex order of OIDs.
|
||||
*/
|
||||
QSORT_S(table, writer.selected_nr, table_cmp, commit_positions);
|
||||
QSORT_S(table, bitmap_writer_selected_nr(), table_cmp, commit_positions);
|
||||
|
||||
/* table_inv helps us discover that relationship (i'th bitmap
|
||||
* to j'th commit by j = table_inv[i])
|
||||
*/
|
||||
for (i = 0; i < writer.selected_nr; i++)
|
||||
for (i = 0; i < bitmap_writer_selected_nr(); i++)
|
||||
table_inv[table[i]] = i;
|
||||
|
||||
trace2_region_enter("pack-bitmap-write", "writing_lookup_table", the_repository);
|
||||
for (i = 0; i < writer.selected_nr; i++) {
|
||||
for (i = 0; i < bitmap_writer_selected_nr(); i++) {
|
||||
struct bitmapped_commit *selected = &writer.selected[table[i]];
|
||||
uint32_t xor_offset = selected->xor_offset;
|
||||
uint32_t xor_row;
|
||||
|
|
@ -797,12 +988,15 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
|
|||
|
||||
int fd = odb_mkstemp(&tmp_file, "pack/tmp_bitmap_XXXXXX");
|
||||
|
||||
if (writer.pseudo_merges_nr)
|
||||
options |= BITMAP_OPT_PSEUDO_MERGES;
|
||||
|
||||
f = hashfd(fd, tmp_file.buf);
|
||||
|
||||
memcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));
|
||||
header.version = htons(default_version);
|
||||
header.options = htons(flags | options);
|
||||
header.entry_count = htonl(writer.selected_nr);
|
||||
header.entry_count = htonl(bitmap_writer_selected_nr());
|
||||
hashcpy(header.checksum, writer.pack_checksum);
|
||||
|
||||
hashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz);
|
||||
|
|
@ -814,9 +1008,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
|
|||
if (options & BITMAP_OPT_LOOKUP_TABLE)
|
||||
CALLOC_ARRAY(offsets, index_nr);
|
||||
|
||||
ALLOC_ARRAY(commit_positions, writer.selected_nr);
|
||||
ALLOC_ARRAY(commit_positions, bitmap_writer_selected_nr());
|
||||
|
||||
for (i = 0; i < writer.selected_nr; i++) {
|
||||
for (i = 0; i < bitmap_writer_selected_nr(); i++) {
|
||||
struct bitmapped_commit *stored = &writer.selected[i];
|
||||
int commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);
|
||||
|
||||
|
|
@ -828,6 +1022,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
|
|||
|
||||
write_selected_commits_v1(f, commit_positions, offsets);
|
||||
|
||||
if (options & BITMAP_OPT_PSEUDO_MERGES)
|
||||
write_pseudo_merges(f);
|
||||
|
||||
if (options & BITMAP_OPT_LOOKUP_TABLE)
|
||||
write_lookup_table(f, commit_positions, offsets);
|
||||
|
||||
|
|
|
|||
359
pack-bitmap.c
359
pack-bitmap.c
|
|
@ -20,6 +20,7 @@
|
|||
#include "list-objects-filter-options.h"
|
||||
#include "midx.h"
|
||||
#include "config.h"
|
||||
#include "pseudo-merge.h"
|
||||
|
||||
/*
|
||||
* An entry on the bitmap index, representing the bitmap for a given
|
||||
|
|
@ -86,6 +87,9 @@ struct bitmap_index {
|
|||
*/
|
||||
unsigned char *table_lookup;
|
||||
|
||||
/* This contains the pseudo-merge cache within 'map' (if found). */
|
||||
struct pseudo_merge_map pseudo_merges;
|
||||
|
||||
/*
|
||||
* Extended index.
|
||||
*
|
||||
|
|
@ -110,6 +114,13 @@ struct bitmap_index {
|
|||
unsigned int version;
|
||||
};
|
||||
|
||||
static int pseudo_merges_satisfied_nr;
|
||||
static int pseudo_merges_cascades_nr;
|
||||
static int existing_bitmaps_hits_nr;
|
||||
static int existing_bitmaps_misses_nr;
|
||||
static int roots_with_bitmaps_nr;
|
||||
static int roots_without_bitmaps_nr;
|
||||
|
||||
static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)
|
||||
{
|
||||
struct ewah_bitmap *parent;
|
||||
|
|
@ -129,17 +140,13 @@ static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)
|
|||
return composed;
|
||||
}
|
||||
|
||||
/*
|
||||
* Read a bitmap from the current read position on the mmaped
|
||||
* index, and increase the read position accordingly
|
||||
*/
|
||||
static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)
|
||||
struct ewah_bitmap *read_bitmap(const unsigned char *map,
|
||||
size_t map_size, size_t *map_pos)
|
||||
{
|
||||
struct ewah_bitmap *b = ewah_pool_new();
|
||||
|
||||
ssize_t bitmap_size = ewah_read_mmap(b,
|
||||
index->map + index->map_pos,
|
||||
index->map_size - index->map_pos);
|
||||
ssize_t bitmap_size = ewah_read_mmap(b, map + *map_pos,
|
||||
map_size - *map_pos);
|
||||
|
||||
if (bitmap_size < 0) {
|
||||
error(_("failed to load bitmap index (corrupted?)"));
|
||||
|
|
@ -147,10 +154,20 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)
|
|||
return NULL;
|
||||
}
|
||||
|
||||
index->map_pos += bitmap_size;
|
||||
*map_pos += bitmap_size;
|
||||
|
||||
return b;
|
||||
}
|
||||
|
||||
/*
|
||||
* Read a bitmap from the current read position on the mmaped
|
||||
* index, and increase the read position accordingly
|
||||
*/
|
||||
static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)
|
||||
{
|
||||
return read_bitmap(index->map, index->map_size, &index->map_pos);
|
||||
}
|
||||
|
||||
static uint32_t bitmap_num_objects(struct bitmap_index *index)
|
||||
{
|
||||
if (index->midx)
|
||||
|
|
@ -199,6 +216,41 @@ static int load_bitmap_header(struct bitmap_index *index)
|
|||
index->table_lookup = (void *)(index_end - table_size);
|
||||
index_end -= table_size;
|
||||
}
|
||||
|
||||
if (flags & BITMAP_OPT_PSEUDO_MERGES) {
|
||||
unsigned char *pseudo_merge_ofs;
|
||||
size_t table_size;
|
||||
uint32_t i;
|
||||
|
||||
if (sizeof(table_size) > index_end - index->map - header_size)
|
||||
return error(_("corrupted bitmap index file (too short to fit pseudo-merge table header)"));
|
||||
|
||||
table_size = get_be64(index_end - 8);
|
||||
if (table_size > index_end - index->map - header_size)
|
||||
return error(_("corrupted bitmap index file (too short to fit pseudo-merge table)"));
|
||||
|
||||
if (git_env_bool("GIT_TEST_USE_PSEUDO_MERGES", 1)) {
|
||||
const unsigned char *ext = (index_end - table_size);
|
||||
|
||||
index->pseudo_merges.map = index->map;
|
||||
index->pseudo_merges.map_size = index->map_size;
|
||||
index->pseudo_merges.commits = ext + get_be64(index_end - 16);
|
||||
index->pseudo_merges.commits_nr = get_be32(index_end - 20);
|
||||
index->pseudo_merges.nr = get_be32(index_end - 24);
|
||||
|
||||
CALLOC_ARRAY(index->pseudo_merges.v,
|
||||
index->pseudo_merges.nr);
|
||||
|
||||
pseudo_merge_ofs = index_end - 24 -
|
||||
(index->pseudo_merges.nr * sizeof(uint64_t));
|
||||
for (i = 0; i < index->pseudo_merges.nr; i++) {
|
||||
index->pseudo_merges.v[i].at = get_be64(pseudo_merge_ofs);
|
||||
pseudo_merge_ofs += sizeof(uint64_t);
|
||||
}
|
||||
}
|
||||
|
||||
index_end -= table_size;
|
||||
}
|
||||
}
|
||||
|
||||
index->entry_count = ntohl(header->entry_count);
|
||||
|
|
@ -961,6 +1013,22 @@ static void show_commit(struct commit *commit UNUSED,
|
|||
{
|
||||
}
|
||||
|
||||
static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git,
|
||||
struct bitmap *result,
|
||||
struct commit *commit,
|
||||
uint32_t commit_pos)
|
||||
{
|
||||
int ret;
|
||||
|
||||
ret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,
|
||||
result, commit, commit_pos);
|
||||
|
||||
if (ret)
|
||||
pseudo_merges_satisfied_nr += ret;
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
static int add_to_include_set(struct bitmap_index *bitmap_git,
|
||||
struct include_data *data,
|
||||
struct commit *commit,
|
||||
|
|
@ -976,11 +1044,19 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,
|
|||
|
||||
partial = bitmap_for_commit(bitmap_git, commit);
|
||||
if (partial) {
|
||||
existing_bitmaps_hits_nr++;
|
||||
|
||||
bitmap_or_ewah(data->base, partial);
|
||||
return 0;
|
||||
}
|
||||
|
||||
existing_bitmaps_misses_nr++;
|
||||
|
||||
bitmap_set(data->base, bitmap_pos);
|
||||
if (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,
|
||||
bitmap_pos))
|
||||
return 0;
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
|
|
@ -1031,8 +1107,12 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,
|
|||
{
|
||||
struct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);
|
||||
|
||||
if (!or_with)
|
||||
if (!or_with) {
|
||||
existing_bitmaps_misses_nr++;
|
||||
return 0;
|
||||
}
|
||||
|
||||
existing_bitmaps_hits_nr++;
|
||||
|
||||
if (!*base)
|
||||
*base = ewah_to_bitmap(or_with);
|
||||
|
|
@ -1106,6 +1186,20 @@ static void show_boundary_object(struct object *object UNUSED,
|
|||
BUG("should not be called");
|
||||
}
|
||||
|
||||
static unsigned cascade_pseudo_merges_1(struct bitmap_index *bitmap_git,
|
||||
struct bitmap *result,
|
||||
struct bitmap *roots)
|
||||
{
|
||||
int ret = cascade_pseudo_merges(&bitmap_git->pseudo_merges,
|
||||
result, roots);
|
||||
if (ret) {
|
||||
pseudo_merges_cascades_nr++;
|
||||
pseudo_merges_satisfied_nr += ret;
|
||||
}
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,
|
||||
struct rev_info *revs,
|
||||
struct object_list *roots)
|
||||
|
|
@ -1115,6 +1209,7 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,
|
|||
unsigned int i;
|
||||
unsigned int tmp_blobs, tmp_trees, tmp_tags;
|
||||
int any_missing = 0;
|
||||
int existing_bitmaps = 0;
|
||||
|
||||
cb.bitmap_git = bitmap_git;
|
||||
cb.base = bitmap_new();
|
||||
|
|
@ -1122,6 +1217,25 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,
|
|||
|
||||
revs->ignore_missing_links = 1;
|
||||
|
||||
if (bitmap_git->pseudo_merges.nr) {
|
||||
struct bitmap *roots_bitmap = bitmap_new();
|
||||
struct object_list *objects = NULL;
|
||||
|
||||
for (objects = roots; objects; objects = objects->next) {
|
||||
struct object *object = objects->item;
|
||||
int pos;
|
||||
|
||||
pos = bitmap_position(bitmap_git, &object->oid);
|
||||
if (pos < 0)
|
||||
continue;
|
||||
|
||||
bitmap_set(roots_bitmap, pos);
|
||||
}
|
||||
|
||||
if (!cascade_pseudo_merges_1(bitmap_git, cb.base, roots_bitmap))
|
||||
bitmap_free(roots_bitmap);
|
||||
}
|
||||
|
||||
/*
|
||||
* OR in any existing reachability bitmaps among `roots` into
|
||||
* `cb.base`.
|
||||
|
|
@ -1133,8 +1247,10 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,
|
|||
continue;
|
||||
|
||||
if (add_commit_to_bitmap(bitmap_git, &cb.base,
|
||||
(struct commit *)object))
|
||||
(struct commit *)object)) {
|
||||
existing_bitmaps = 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
any_missing = 1;
|
||||
}
|
||||
|
|
@ -1142,6 +1258,9 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,
|
|||
if (!any_missing)
|
||||
goto cleanup;
|
||||
|
||||
if (existing_bitmaps)
|
||||
cascade_pseudo_merges_1(bitmap_git, cb.base, NULL);
|
||||
|
||||
tmp_blobs = revs->blob_objects;
|
||||
tmp_trees = revs->tree_objects;
|
||||
tmp_tags = revs->blob_objects;
|
||||
|
|
@ -1197,6 +1316,44 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,
|
|||
return cb.base;
|
||||
}
|
||||
|
||||
struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,
|
||||
struct commit *commit)
|
||||
{
|
||||
struct commit_list *p;
|
||||
struct bitmap *parents;
|
||||
struct pseudo_merge *match = NULL;
|
||||
|
||||
if (!bitmap_git->pseudo_merges.nr)
|
||||
return NULL;
|
||||
|
||||
parents = bitmap_new();
|
||||
|
||||
for (p = commit->parents; p; p = p->next) {
|
||||
int pos = bitmap_position(bitmap_git, &p->item->object.oid);
|
||||
if (pos < 0 || pos >= bitmap_num_objects(bitmap_git))
|
||||
goto done;
|
||||
|
||||
bitmap_set(parents, pos);
|
||||
}
|
||||
|
||||
match = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,
|
||||
parents);
|
||||
|
||||
done:
|
||||
bitmap_free(parents);
|
||||
if (match)
|
||||
return pseudo_merge_bitmap(&bitmap_git->pseudo_merges, match);
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)
|
||||
{
|
||||
uint32_t i;
|
||||
for (i = 0; i < bitmap_git->pseudo_merges.nr; i++)
|
||||
bitmap_git->pseudo_merges.v[i].satisfied = 0;
|
||||
}
|
||||
|
||||
static struct bitmap *find_objects(struct bitmap_index *bitmap_git,
|
||||
struct rev_info *revs,
|
||||
struct object_list *roots,
|
||||
|
|
@ -1204,9 +1361,32 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,
|
|||
{
|
||||
struct bitmap *base = NULL;
|
||||
int needs_walk = 0;
|
||||
unsigned existing_bitmaps = 0;
|
||||
|
||||
struct object_list *not_mapped = NULL;
|
||||
|
||||
unsatisfy_all_pseudo_merges(bitmap_git);
|
||||
|
||||
if (bitmap_git->pseudo_merges.nr) {
|
||||
struct bitmap *roots_bitmap = bitmap_new();
|
||||
struct object_list *objects = NULL;
|
||||
|
||||
for (objects = roots; objects; objects = objects->next) {
|
||||
struct object *object = objects->item;
|
||||
int pos;
|
||||
|
||||
pos = bitmap_position(bitmap_git, &object->oid);
|
||||
if (pos < 0)
|
||||
continue;
|
||||
|
||||
bitmap_set(roots_bitmap, pos);
|
||||
}
|
||||
|
||||
base = bitmap_new();
|
||||
if (!cascade_pseudo_merges_1(bitmap_git, base, roots_bitmap))
|
||||
bitmap_free(roots_bitmap);
|
||||
}
|
||||
|
||||
/*
|
||||
* Go through all the roots for the walk. The ones that have bitmaps
|
||||
* on the bitmap index will be `or`ed together to form an initial
|
||||
|
|
@ -1217,11 +1397,21 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,
|
|||
*/
|
||||
while (roots) {
|
||||
struct object *object = roots->item;
|
||||
|
||||
roots = roots->next;
|
||||
|
||||
if (base) {
|
||||
int pos = bitmap_position(bitmap_git, &object->oid);
|
||||
if (pos > 0 && bitmap_get(base, pos)) {
|
||||
object->flags |= SEEN;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
if (object->type == OBJ_COMMIT &&
|
||||
add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {
|
||||
object->flags |= SEEN;
|
||||
existing_bitmaps = 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
|
|
@ -1237,6 +1427,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,
|
|||
|
||||
roots = not_mapped;
|
||||
|
||||
if (existing_bitmaps)
|
||||
cascade_pseudo_merges_1(bitmap_git, base, NULL);
|
||||
|
||||
/*
|
||||
* Let's iterate through all the roots that don't have bitmaps to
|
||||
* check if we can determine them to be reachable from the existing
|
||||
|
|
@ -1257,8 +1450,12 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,
|
|||
object->flags &= ~UNINTERESTING;
|
||||
add_pending_object(revs, object, "");
|
||||
needs_walk = 1;
|
||||
|
||||
roots_without_bitmaps_nr++;
|
||||
} else {
|
||||
object->flags |= SEEN;
|
||||
|
||||
roots_with_bitmaps_nr++;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1821,6 +2018,19 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,
|
|||
object_list_free(&wants);
|
||||
object_list_free(&haves);
|
||||
|
||||
trace2_data_intmax("bitmap", the_repository, "pseudo_merges_satisfied",
|
||||
pseudo_merges_satisfied_nr);
|
||||
trace2_data_intmax("bitmap", the_repository, "pseudo_merges_cascades",
|
||||
pseudo_merges_cascades_nr);
|
||||
trace2_data_intmax("bitmap", the_repository, "bitmap/hits",
|
||||
existing_bitmaps_hits_nr);
|
||||
trace2_data_intmax("bitmap", the_repository, "bitmap/misses",
|
||||
existing_bitmaps_misses_nr);
|
||||
trace2_data_intmax("bitmap", the_repository, "bitmap/roots_with_bitmap",
|
||||
roots_with_bitmaps_nr);
|
||||
trace2_data_intmax("bitmap", the_repository, "bitmap/roots_without_bitmap",
|
||||
roots_without_bitmaps_nr);
|
||||
|
||||
return bitmap_git;
|
||||
|
||||
cleanup:
|
||||
|
|
@ -2398,6 +2608,132 @@ int test_bitmap_hashes(struct repository *r)
|
|||
return 0;
|
||||
}
|
||||
|
||||
static void bit_pos_to_object_id(struct bitmap_index *bitmap_git,
|
||||
uint32_t bit_pos,
|
||||
struct object_id *oid)
|
||||
{
|
||||
uint32_t index_pos;
|
||||
|
||||
if (bitmap_is_midx(bitmap_git))
|
||||
index_pos = pack_pos_to_midx(bitmap_git->midx, bit_pos);
|
||||
else
|
||||
index_pos = pack_pos_to_index(bitmap_git->pack, bit_pos);
|
||||
|
||||
nth_bitmap_object_oid(bitmap_git, oid, index_pos);
|
||||
}
|
||||
|
||||
int test_bitmap_pseudo_merges(struct repository *r)
|
||||
{
|
||||
struct bitmap_index *bitmap_git;
|
||||
uint32_t i;
|
||||
|
||||
bitmap_git = prepare_bitmap_git(r);
|
||||
if (!bitmap_git || !bitmap_git->pseudo_merges.nr)
|
||||
goto cleanup;
|
||||
|
||||
for (i = 0; i < bitmap_git->pseudo_merges.nr; i++) {
|
||||
struct pseudo_merge *merge;
|
||||
struct ewah_bitmap *commits_bitmap, *merge_bitmap;
|
||||
|
||||
merge = use_pseudo_merge(&bitmap_git->pseudo_merges,
|
||||
&bitmap_git->pseudo_merges.v[i]);
|
||||
commits_bitmap = merge->commits;
|
||||
merge_bitmap = pseudo_merge_bitmap(&bitmap_git->pseudo_merges,
|
||||
merge);
|
||||
|
||||
printf("at=%"PRIuMAX", commits=%"PRIuMAX", objects=%"PRIuMAX"\n",
|
||||
(uintmax_t)merge->at,
|
||||
(uintmax_t)ewah_bitmap_popcount(commits_bitmap),
|
||||
(uintmax_t)ewah_bitmap_popcount(merge_bitmap));
|
||||
}
|
||||
|
||||
cleanup:
|
||||
free_bitmap_index(bitmap_git);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void dump_ewah_object_ids(struct bitmap_index *bitmap_git,
|
||||
struct ewah_bitmap *bitmap)
|
||||
|
||||
{
|
||||
struct ewah_iterator it;
|
||||
eword_t word;
|
||||
uint32_t pos = 0;
|
||||
|
||||
ewah_iterator_init(&it, bitmap);
|
||||
|
||||
while (ewah_iterator_next(&word, &it)) {
|
||||
struct object_id oid;
|
||||
uint32_t offset;
|
||||
|
||||
for (offset = 0; offset < BITS_IN_EWORD; offset++) {
|
||||
if (!(word >> offset))
|
||||
break;
|
||||
|
||||
offset += ewah_bit_ctz64(word >> offset);
|
||||
|
||||
bit_pos_to_object_id(bitmap_git, pos + offset, &oid);
|
||||
printf("%s\n", oid_to_hex(&oid));
|
||||
}
|
||||
pos += BITS_IN_EWORD;
|
||||
}
|
||||
}
|
||||
|
||||
int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n)
|
||||
{
|
||||
struct bitmap_index *bitmap_git;
|
||||
struct pseudo_merge *merge;
|
||||
int ret = 0;
|
||||
|
||||
bitmap_git = prepare_bitmap_git(r);
|
||||
if (!bitmap_git || !bitmap_git->pseudo_merges.nr)
|
||||
goto cleanup;
|
||||
|
||||
if (n >= bitmap_git->pseudo_merges.nr) {
|
||||
ret = error(_("pseudo-merge index out of range "
|
||||
"(%"PRIu32" >= %"PRIuMAX")"),
|
||||
n, (uintmax_t)bitmap_git->pseudo_merges.nr);
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
merge = use_pseudo_merge(&bitmap_git->pseudo_merges,
|
||||
&bitmap_git->pseudo_merges.v[n]);
|
||||
dump_ewah_object_ids(bitmap_git, merge->commits);
|
||||
|
||||
cleanup:
|
||||
free_bitmap_index(bitmap_git);
|
||||
return ret;
|
||||
}
|
||||
|
||||
int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n)
|
||||
{
|
||||
struct bitmap_index *bitmap_git;
|
||||
struct pseudo_merge *merge;
|
||||
int ret = 0;
|
||||
|
||||
bitmap_git = prepare_bitmap_git(r);
|
||||
if (!bitmap_git || !bitmap_git->pseudo_merges.nr)
|
||||
goto cleanup;
|
||||
|
||||
if (n >= bitmap_git->pseudo_merges.nr) {
|
||||
ret = error(_("pseudo-merge index out of range "
|
||||
"(%"PRIu32" >= %"PRIuMAX")"),
|
||||
n, (uintmax_t)bitmap_git->pseudo_merges.nr);
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
merge = use_pseudo_merge(&bitmap_git->pseudo_merges,
|
||||
&bitmap_git->pseudo_merges.v[n]);
|
||||
|
||||
dump_ewah_object_ids(bitmap_git,
|
||||
pseudo_merge_bitmap(&bitmap_git->pseudo_merges,
|
||||
merge));
|
||||
|
||||
cleanup:
|
||||
free_bitmap_index(bitmap_git);
|
||||
return ret;
|
||||
}
|
||||
|
||||
int rebuild_bitmap(const uint32_t *reposition,
|
||||
struct ewah_bitmap *source,
|
||||
struct bitmap *dest)
|
||||
|
|
@ -2504,6 +2840,7 @@ void free_bitmap_index(struct bitmap_index *b)
|
|||
*/
|
||||
close_midx_revindex(b->midx);
|
||||
}
|
||||
free_pseudo_merge_map(&b->pseudo_merges);
|
||||
free(b);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ struct bitmap_disk_header {
|
|||
unsigned char checksum[GIT_MAX_RAWSZ];
|
||||
};
|
||||
|
||||
#define BITMAP_PSEUDO_MERGE (1u<<21)
|
||||
#define NEEDS_BITMAP (1u<<22)
|
||||
|
||||
/*
|
||||
|
|
@ -36,6 +37,7 @@ enum pack_bitmap_opts {
|
|||
BITMAP_OPT_FULL_DAG = 0x1,
|
||||
BITMAP_OPT_HASH_CACHE = 0x4,
|
||||
BITMAP_OPT_LOOKUP_TABLE = 0x10,
|
||||
BITMAP_OPT_PSEUDO_MERGES = 0x20,
|
||||
};
|
||||
|
||||
enum pack_bitmap_flags {
|
||||
|
|
@ -71,6 +73,9 @@ void traverse_bitmap_commit_list(struct bitmap_index *,
|
|||
void test_bitmap_walk(struct rev_info *revs);
|
||||
int test_bitmap_commits(struct repository *r);
|
||||
int test_bitmap_hashes(struct repository *r);
|
||||
int test_bitmap_pseudo_merges(struct repository *r);
|
||||
int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n);
|
||||
int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);
|
||||
|
||||
#define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \
|
||||
"GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL"
|
||||
|
|
@ -97,6 +102,11 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *, const struct object_i
|
|||
|
||||
off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *);
|
||||
|
||||
int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid);
|
||||
void bitmap_writer_push_bitmapped_commit(struct commit *commit,
|
||||
unsigned pseudo_merge);
|
||||
|
||||
void bitmap_writer_init(struct repository *r);
|
||||
void bitmap_writer_show_progress(int show);
|
||||
void bitmap_writer_set_checksum(const unsigned char *sha1);
|
||||
void bitmap_writer_build_type_index(struct packing_data *to_pack,
|
||||
|
|
@ -109,8 +119,10 @@ int rebuild_bitmap(const uint32_t *reposition,
|
|||
struct bitmap *dest);
|
||||
struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,
|
||||
struct commit *commit);
|
||||
struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,
|
||||
struct commit *commit);
|
||||
void bitmap_writer_select_commits(struct commit **indexed_commits,
|
||||
unsigned int indexed_commits_nr, int max_bitmaps);
|
||||
unsigned int indexed_commits_nr);
|
||||
int bitmap_writer_build(struct packing_data *to_pack);
|
||||
void bitmap_writer_finish(struct pack_idx_entry **index,
|
||||
uint32_t index_nr,
|
||||
|
|
@ -126,4 +138,6 @@ int bitmap_is_preferred_refname(struct repository *r, const char *refname);
|
|||
|
||||
int verify_bitmap_files(struct repository *r);
|
||||
|
||||
struct ewah_bitmap *read_bitmap(const unsigned char *map,
|
||||
size_t map_size, size_t *map_pos);
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -0,0 +1,739 @@
|
|||
#include "git-compat-util.h"
|
||||
#include "pseudo-merge.h"
|
||||
#include "date.h"
|
||||
#include "oid-array.h"
|
||||
#include "strbuf.h"
|
||||
#include "config.h"
|
||||
#include "string-list.h"
|
||||
#include "refs.h"
|
||||
#include "pack-bitmap.h"
|
||||
#include "commit.h"
|
||||
#include "alloc.h"
|
||||
#include "progress.h"
|
||||
#include "hex.h"
|
||||
|
||||
#define DEFAULT_PSEUDO_MERGE_DECAY 1.0f
|
||||
#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64
|
||||
#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 100
|
||||
#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate("1.week.ago")
|
||||
#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate("1.month.ago")
|
||||
#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512
|
||||
|
||||
static float gitexp(float base, int exp)
|
||||
{
|
||||
float result = 1;
|
||||
while (1) {
|
||||
if (exp % 2)
|
||||
result *= base;
|
||||
exp >>= 1;
|
||||
if (!exp)
|
||||
break;
|
||||
base *= base;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group,
|
||||
const struct pseudo_merge_matches *matches,
|
||||
uint32_t i)
|
||||
{
|
||||
float C = 0.0f;
|
||||
uint32_t n;
|
||||
|
||||
/*
|
||||
* The size of pseudo-merge groups decays according to a power series,
|
||||
* which looks like:
|
||||
*
|
||||
* f(n) = C * n^-k
|
||||
*
|
||||
* , where 'n' is the n-th pseudo-merge group, 'f(n)' is its size, 'k'
|
||||
* is the decay rate, and 'C' is a scaling value.
|
||||
*
|
||||
* The value of C depends on the number of groups, decay rate, and total
|
||||
* number of commits. It is computed such that if there are M and N
|
||||
* total groups and commits, respectively, that:
|
||||
*
|
||||
* N = f(0) + f(1) + ... f(M-1)
|
||||
*
|
||||
* Rearranging to isolate C, we get:
|
||||
*
|
||||
* N = \sum_{n=1}^M C / n^k
|
||||
*
|
||||
* N / C = \sum_{n=1}^M n^-k
|
||||
*
|
||||
* C = N / \sum_{n=1}^M n^-k
|
||||
*
|
||||
* For example, if we have a decay rate of 'k' being equal to 1.5, 'N'
|
||||
* total commits equal to 10,000, and 'M' being equal to 6 groups, then
|
||||
* the (rounded) group sizes are:
|
||||
*
|
||||
* { 5469, 1934, 1053, 684, 489, 372 }
|
||||
*
|
||||
* increasing the number of total groups, say to 10, scales the group
|
||||
* sizes appropriately:
|
||||
*
|
||||
* { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 }
|
||||
*/
|
||||
for (n = 0; n < group->max_merges; n++)
|
||||
C += 1.0f / gitexp(n + 1, group->decay);
|
||||
C = matches->unstable_nr / C;
|
||||
|
||||
return (int)((C / gitexp(i + 1, group->decay)) + 0.5);
|
||||
}
|
||||
|
||||
static void init_pseudo_merge_group(struct pseudo_merge_group *group)
|
||||
{
|
||||
memset(group, 0, sizeof(struct pseudo_merge_group));
|
||||
|
||||
strmap_init_with_options(&group->matches, NULL, 0);
|
||||
|
||||
group->decay = DEFAULT_PSEUDO_MERGE_DECAY;
|
||||
group->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;
|
||||
group->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;
|
||||
group->threshold = DEFAULT_PSEUDO_MERGE_THRESHOLD;
|
||||
group->stable_threshold = DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD;
|
||||
group->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;
|
||||
}
|
||||
|
||||
static int pseudo_merge_config(const char *var, const char *value,
|
||||
const struct config_context *ctx,
|
||||
void *cb_data)
|
||||
{
|
||||
struct string_list *list = cb_data;
|
||||
struct string_list_item *item;
|
||||
struct pseudo_merge_group *group;
|
||||
struct strbuf buf = STRBUF_INIT;
|
||||
const char *sub, *key;
|
||||
size_t sub_len;
|
||||
|
||||
if (parse_config_key(var, "bitmappseudomerge", &sub, &sub_len, &key))
|
||||
return 0;
|
||||
|
||||
if (!sub_len)
|
||||
return 0;
|
||||
|
||||
strbuf_add(&buf, sub, sub_len);
|
||||
|
||||
item = string_list_lookup(list, buf.buf);
|
||||
if (!item) {
|
||||
item = string_list_insert(list, buf.buf);
|
||||
|
||||
item->util = xmalloc(sizeof(struct pseudo_merge_group));
|
||||
init_pseudo_merge_group(item->util);
|
||||
}
|
||||
|
||||
group = item->util;
|
||||
|
||||
if (!strcmp(key, "pattern")) {
|
||||
struct strbuf re = STRBUF_INIT;
|
||||
|
||||
free(group->pattern);
|
||||
if (*value != '^')
|
||||
strbuf_addch(&re, '^');
|
||||
strbuf_addstr(&re, value);
|
||||
|
||||
group->pattern = xcalloc(1, sizeof(regex_t));
|
||||
if (regcomp(group->pattern, re.buf, REG_EXTENDED))
|
||||
die(_("failed to load pseudo-merge regex for %s: '%s'"),
|
||||
sub, re.buf);
|
||||
|
||||
strbuf_release(&re);
|
||||
} else if (!strcmp(key, "decay")) {
|
||||
group->decay = git_config_int(var, value, ctx->kvi);
|
||||
if (group->decay < 0) {
|
||||
warning(_("%s must be non-negative, using default"), var);
|
||||
group->decay = DEFAULT_PSEUDO_MERGE_DECAY;
|
||||
}
|
||||
} else if (!strcmp(key, "samplerate")) {
|
||||
group->sample_rate = git_config_int(var, value, ctx->kvi);
|
||||
if (!(0 <= group->sample_rate && group->sample_rate <= 100)) {
|
||||
warning(_("%s must be between 0 and 100, using default"), var);
|
||||
group->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;
|
||||
}
|
||||
} else if (!strcmp(key, "threshold")) {
|
||||
if (git_config_expiry_date(&group->threshold, var, value)) {
|
||||
strbuf_release(&buf);
|
||||
return -1;
|
||||
}
|
||||
} else if (!strcmp(key, "maxmerges")) {
|
||||
group->max_merges = git_config_int(var, value, ctx->kvi);
|
||||
if (group->max_merges < 0) {
|
||||
warning(_("%s must be non-negative, using default"), var);
|
||||
group->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;
|
||||
}
|
||||
} else if (!strcmp(key, "stablethreshold")) {
|
||||
if (git_config_expiry_date(&group->stable_threshold, var, value)) {
|
||||
strbuf_release(&buf);
|
||||
return -1;
|
||||
}
|
||||
} else if (!strcmp(key, "stablesize")) {
|
||||
group->stable_size = git_config_int(var, value, ctx->kvi);
|
||||
if (group->stable_size <= 0) {
|
||||
warning(_("%s must be positive, using default"), var);
|
||||
group->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;
|
||||
}
|
||||
}
|
||||
|
||||
strbuf_release(&buf);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
void load_pseudo_merges_from_config(struct string_list *list)
|
||||
{
|
||||
struct string_list_item *item;
|
||||
|
||||
git_config(pseudo_merge_config, list);
|
||||
|
||||
for_each_string_list_item(item, list) {
|
||||
struct pseudo_merge_group *group = item->util;
|
||||
if (!group->pattern)
|
||||
die(_("pseudo-merge group '%s' missing required pattern"),
|
||||
item->string);
|
||||
if (group->threshold < group->stable_threshold)
|
||||
die(_("pseudo-merge group '%s' has unstable threshold "
|
||||
"before stable one"), item->string);
|
||||
}
|
||||
}
|
||||
|
||||
static int find_pseudo_merge_group_for_ref(const char *refname,
|
||||
const struct object_id *oid,
|
||||
int flags UNUSED,
|
||||
void *_data)
|
||||
{
|
||||
struct string_list *list = _data;
|
||||
struct object_id peeled;
|
||||
struct commit *c;
|
||||
uint32_t i;
|
||||
int has_bitmap;
|
||||
|
||||
if (!peel_iterated_oid(oid, &peeled))
|
||||
oid = &peeled;
|
||||
|
||||
c = lookup_commit(the_repository, oid);
|
||||
if (!c)
|
||||
return 0;
|
||||
|
||||
has_bitmap = bitmap_writer_has_bitmapped_object_id(oid);
|
||||
|
||||
for (i = 0; i < list->nr; i++) {
|
||||
struct pseudo_merge_group *group;
|
||||
struct pseudo_merge_matches *matches;
|
||||
struct strbuf group_name = STRBUF_INIT;
|
||||
regmatch_t captures[16];
|
||||
size_t j;
|
||||
|
||||
group = list->items[i].util;
|
||||
if (regexec(group->pattern, refname, ARRAY_SIZE(captures),
|
||||
captures, 0))
|
||||
continue;
|
||||
|
||||
if (captures[ARRAY_SIZE(captures) - 1].rm_so != -1)
|
||||
warning(_("pseudo-merge regex from config has too many capture "
|
||||
"groups (max=%"PRIuMAX")"),
|
||||
(uintmax_t)ARRAY_SIZE(captures) - 2);
|
||||
|
||||
for (j = !!group->pattern->re_nsub; j < ARRAY_SIZE(captures); j++) {
|
||||
regmatch_t *match = &captures[j];
|
||||
if (match->rm_so == -1)
|
||||
continue;
|
||||
|
||||
if (group_name.len)
|
||||
strbuf_addch(&group_name, '-');
|
||||
|
||||
strbuf_add(&group_name, refname + match->rm_so,
|
||||
match->rm_eo - match->rm_so);
|
||||
}
|
||||
|
||||
matches = strmap_get(&group->matches, group_name.buf);
|
||||
if (!matches) {
|
||||
matches = xcalloc(1, sizeof(*matches));
|
||||
strmap_put(&group->matches, strbuf_detach(&group_name, NULL),
|
||||
matches);
|
||||
}
|
||||
|
||||
if (c->date <= group->stable_threshold) {
|
||||
ALLOC_GROW(matches->stable, matches->stable_nr + 1,
|
||||
matches->stable_alloc);
|
||||
matches->stable[matches->stable_nr++] = c;
|
||||
} else if (c->date <= group->threshold && !has_bitmap) {
|
||||
ALLOC_GROW(matches->unstable, matches->unstable_nr + 1,
|
||||
matches->unstable_alloc);
|
||||
matches->unstable[matches->unstable_nr++] = c;
|
||||
}
|
||||
|
||||
strbuf_release(&group_name);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static struct commit *push_pseudo_merge(struct pseudo_merge_group *group)
|
||||
{
|
||||
struct commit *merge;
|
||||
|
||||
ALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc);
|
||||
|
||||
merge = alloc_commit_node(the_repository);
|
||||
merge->object.parsed = 1;
|
||||
merge->object.flags |= BITMAP_PSEUDO_MERGE;
|
||||
|
||||
group->merges[group->merges_nr++] = merge;
|
||||
|
||||
return merge;
|
||||
}
|
||||
|
||||
static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,
|
||||
const struct object_id *oid)
|
||||
|
||||
{
|
||||
struct pseudo_merge_commit_idx *pmc;
|
||||
khiter_t hash_pos;
|
||||
|
||||
hash_pos = kh_get_oid_map(pseudo_merge_commits, *oid);
|
||||
if (hash_pos == kh_end(pseudo_merge_commits)) {
|
||||
int hash_ret;
|
||||
hash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, &hash_ret);
|
||||
|
||||
CALLOC_ARRAY(pmc, 1);
|
||||
|
||||
kh_value(pseudo_merge_commits, hash_pos) = pmc;
|
||||
} else {
|
||||
pmc = kh_value(pseudo_merge_commits, hash_pos);
|
||||
}
|
||||
|
||||
return pmc;
|
||||
}
|
||||
|
||||
#define MIN_PSEUDO_MERGE_SIZE 8
|
||||
|
||||
static void select_pseudo_merges_1(struct pseudo_merge_group *group,
|
||||
struct pseudo_merge_matches *matches,
|
||||
kh_oid_map_t *pseudo_merge_commits,
|
||||
uint32_t *pseudo_merges_nr)
|
||||
{
|
||||
uint32_t i, j;
|
||||
uint32_t stable_merges_nr;
|
||||
|
||||
if (!matches->stable_nr && !matches->unstable_nr)
|
||||
return; /* all tips in this group already have bitmaps */
|
||||
|
||||
stable_merges_nr = matches->stable_nr / group->stable_size;
|
||||
if (matches->stable_nr % group->stable_size)
|
||||
stable_merges_nr++;
|
||||
|
||||
/* make stable_merges_nr pseudo merges for stable commits */
|
||||
for (i = 0, j = 0; i < stable_merges_nr; i++) {
|
||||
struct commit *merge;
|
||||
struct commit_list **p;
|
||||
|
||||
merge = push_pseudo_merge(group);
|
||||
p = &merge->parents;
|
||||
|
||||
do {
|
||||
struct commit *c;
|
||||
struct pseudo_merge_commit_idx *pmc;
|
||||
|
||||
if (j >= matches->stable_nr)
|
||||
break;
|
||||
|
||||
c = matches->stable[j++];
|
||||
pmc = pseudo_merge_idx(pseudo_merge_commits,
|
||||
&c->object.oid);
|
||||
|
||||
ALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);
|
||||
|
||||
pmc->pseudo_merge[pmc->nr++] = *pseudo_merges_nr;
|
||||
p = commit_list_append(c, p);
|
||||
} while (j % group->stable_size);
|
||||
|
||||
bitmap_writer_push_bitmapped_commit(merge, 1);
|
||||
(*pseudo_merges_nr)++;
|
||||
}
|
||||
|
||||
/* make up to group->max_merges pseudo merges for unstable commits */
|
||||
for (i = 0, j = 0; i < group->max_merges; i++) {
|
||||
struct commit *merge;
|
||||
struct commit_list **p;
|
||||
uint32_t size, end;
|
||||
|
||||
merge = push_pseudo_merge(group);
|
||||
p = &merge->parents;
|
||||
|
||||
size = pseudo_merge_group_size(group, matches, i);
|
||||
end = size < MIN_PSEUDO_MERGE_SIZE ? matches->unstable_nr : j + size;
|
||||
|
||||
for (; j < end && j < matches->unstable_nr; j++) {
|
||||
struct commit *c = matches->unstable[j];
|
||||
struct pseudo_merge_commit_idx *pmc;
|
||||
|
||||
if (j % (100 / group->sample_rate))
|
||||
continue;
|
||||
|
||||
pmc = pseudo_merge_idx(pseudo_merge_commits,
|
||||
&c->object.oid);
|
||||
|
||||
ALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);
|
||||
|
||||
pmc->pseudo_merge[pmc->nr++] = *pseudo_merges_nr;
|
||||
p = commit_list_append(c, p);
|
||||
}
|
||||
|
||||
bitmap_writer_push_bitmapped_commit(merge, 1);
|
||||
(*pseudo_merges_nr)++;
|
||||
if (end >= matches->unstable_nr)
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
static int commit_date_cmp(const void *va, const void *vb)
|
||||
{
|
||||
timestamp_t a = (*(const struct commit **)va)->date;
|
||||
timestamp_t b = (*(const struct commit **)vb)->date;
|
||||
|
||||
if (a < b)
|
||||
return -1;
|
||||
else if (a > b)
|
||||
return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void sort_pseudo_merge_matches(struct pseudo_merge_matches *matches)
|
||||
{
|
||||
QSORT(matches->stable, matches->stable_nr, commit_date_cmp);
|
||||
QSORT(matches->unstable, matches->unstable_nr, commit_date_cmp);
|
||||
}
|
||||
|
||||
void select_pseudo_merges(struct string_list *list,
|
||||
struct commit **commits, size_t commits_nr,
|
||||
kh_oid_map_t *pseudo_merge_commits,
|
||||
uint32_t *pseudo_merges_nr,
|
||||
unsigned show_progress)
|
||||
{
|
||||
struct progress *progress = NULL;
|
||||
uint32_t i;
|
||||
|
||||
if (!list->nr)
|
||||
return;
|
||||
|
||||
if (show_progress)
|
||||
progress = start_progress("Selecting pseudo-merge commits", list->nr);
|
||||
|
||||
for_each_ref(find_pseudo_merge_group_for_ref, list);
|
||||
|
||||
for (i = 0; i < list->nr; i++) {
|
||||
struct pseudo_merge_group *group;
|
||||
struct hashmap_iter iter;
|
||||
struct strmap_entry *e;
|
||||
|
||||
group = list->items[i].util;
|
||||
strmap_for_each_entry(&group->matches, &iter, e) {
|
||||
struct pseudo_merge_matches *matches = e->value;
|
||||
|
||||
sort_pseudo_merge_matches(matches);
|
||||
|
||||
select_pseudo_merges_1(group, matches,
|
||||
pseudo_merge_commits,
|
||||
pseudo_merges_nr);
|
||||
}
|
||||
|
||||
display_progress(progress, i + 1);
|
||||
}
|
||||
|
||||
stop_progress(&progress);
|
||||
}
|
||||
|
||||
void free_pseudo_merge_map(struct pseudo_merge_map *pm)
|
||||
{
|
||||
uint32_t i;
|
||||
for (i = 0; i < pm->nr; i++) {
|
||||
ewah_pool_free(pm->v[i].commits);
|
||||
ewah_pool_free(pm->v[i].bitmap);
|
||||
}
|
||||
free(pm->v);
|
||||
}
|
||||
|
||||
struct pseudo_merge_commit_ext {
|
||||
uint32_t nr;
|
||||
const unsigned char *ptr;
|
||||
};
|
||||
|
||||
static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm,
|
||||
struct pseudo_merge_commit_ext *ext, size_t at)
|
||||
{
|
||||
if (at >= pm->map_size)
|
||||
return error(_("extended pseudo-merge read out-of-bounds "
|
||||
"(%"PRIuMAX" >= %"PRIuMAX")"),
|
||||
(uintmax_t)at, (uintmax_t)pm->map_size);
|
||||
|
||||
ext->nr = get_be32(pm->map + at);
|
||||
ext->ptr = pm->map + at + sizeof(uint32_t);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,
|
||||
struct pseudo_merge *merge)
|
||||
{
|
||||
if (!merge->loaded_commits)
|
||||
BUG("cannot use unloaded pseudo-merge bitmap");
|
||||
|
||||
if (!merge->loaded_bitmap) {
|
||||
size_t at = merge->bitmap_at;
|
||||
|
||||
merge->bitmap = read_bitmap(pm->map, pm->map_size, &at);
|
||||
merge->loaded_bitmap = 1;
|
||||
}
|
||||
|
||||
return merge->bitmap;
|
||||
}
|
||||
|
||||
struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,
|
||||
struct pseudo_merge *merge)
|
||||
{
|
||||
if (!merge->loaded_commits) {
|
||||
size_t pos = merge->at;
|
||||
|
||||
merge->commits = read_bitmap(pm->map, pm->map_size, &pos);
|
||||
merge->bitmap_at = pos;
|
||||
merge->loaded_commits = 1;
|
||||
}
|
||||
return merge;
|
||||
}
|
||||
|
||||
static struct pseudo_merge *pseudo_merge_at(const struct pseudo_merge_map *pm,
|
||||
struct object_id *oid,
|
||||
size_t want)
|
||||
{
|
||||
size_t lo = 0;
|
||||
size_t hi = pm->nr;
|
||||
|
||||
while (lo < hi) {
|
||||
size_t mi = lo + (hi - lo) / 2;
|
||||
size_t got = pm->v[mi].at;
|
||||
|
||||
if (got == want)
|
||||
return use_pseudo_merge(pm, &pm->v[mi]);
|
||||
else if (got < want)
|
||||
hi = mi;
|
||||
else
|
||||
lo = mi + 1;
|
||||
}
|
||||
|
||||
warning(_("could not find pseudo-merge for commit %s at offset %"PRIuMAX),
|
||||
oid_to_hex(oid), (uintmax_t)want);
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
struct pseudo_merge_commit {
|
||||
uint32_t commit_pos;
|
||||
uint64_t pseudo_merge_ofs;
|
||||
};
|
||||
|
||||
#define PSEUDO_MERGE_COMMIT_RAWSZ (sizeof(uint32_t)+sizeof(uint64_t))
|
||||
|
||||
static void read_pseudo_merge_commit_at(struct pseudo_merge_commit *merge,
|
||||
const unsigned char *at)
|
||||
{
|
||||
merge->commit_pos = get_be32(at);
|
||||
merge->pseudo_merge_ofs = get_be64(at + sizeof(uint32_t));
|
||||
}
|
||||
|
||||
static int nth_pseudo_merge_ext(const struct pseudo_merge_map *pm,
|
||||
struct pseudo_merge_commit_ext *ext,
|
||||
struct pseudo_merge_commit *merge,
|
||||
uint32_t n)
|
||||
{
|
||||
size_t ofs;
|
||||
|
||||
if (n >= ext->nr)
|
||||
return error(_("extended pseudo-merge lookup out-of-bounds "
|
||||
"(%"PRIu32" >= %"PRIu32")"), n, ext->nr);
|
||||
|
||||
ofs = get_be64(ext->ptr + st_mult(n, sizeof(uint64_t)));
|
||||
if (ofs >= pm->map_size)
|
||||
return error(_("out-of-bounds read: (%"PRIuMAX" >= %"PRIuMAX")"),
|
||||
(uintmax_t)ofs, (uintmax_t)pm->map_size);
|
||||
|
||||
read_pseudo_merge_commit_at(merge, pm->map + ofs);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static unsigned apply_pseudo_merge(const struct pseudo_merge_map *pm,
|
||||
struct pseudo_merge *merge,
|
||||
struct bitmap *result,
|
||||
struct bitmap *roots)
|
||||
{
|
||||
if (merge->satisfied)
|
||||
return 0;
|
||||
|
||||
if (!ewah_bitmap_is_subset(merge->commits, roots ? roots : result))
|
||||
return 0;
|
||||
|
||||
bitmap_or_ewah(result, pseudo_merge_bitmap(pm, merge));
|
||||
if (roots)
|
||||
bitmap_or_ewah(roots, pseudo_merge_bitmap(pm, merge));
|
||||
merge->satisfied = 1;
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
static int pseudo_merge_commit_cmp(const void *va, const void *vb)
|
||||
{
|
||||
struct pseudo_merge_commit merge;
|
||||
uint32_t key = *(uint32_t*)va;
|
||||
|
||||
read_pseudo_merge_commit_at(&merge, vb);
|
||||
|
||||
if (key < merge.commit_pos)
|
||||
return -1;
|
||||
if (key > merge.commit_pos)
|
||||
return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static struct pseudo_merge_commit *find_pseudo_merge(const struct pseudo_merge_map *pm,
|
||||
uint32_t pos)
|
||||
{
|
||||
if (!pm->commits_nr)
|
||||
return NULL;
|
||||
|
||||
return bsearch(&pos, pm->commits, pm->commits_nr,
|
||||
PSEUDO_MERGE_COMMIT_RAWSZ, pseudo_merge_commit_cmp);
|
||||
}
|
||||
|
||||
int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,
|
||||
struct bitmap *result,
|
||||
struct commit *commit, uint32_t commit_pos)
|
||||
{
|
||||
struct pseudo_merge *merge;
|
||||
struct pseudo_merge_commit *merge_commit;
|
||||
int ret = 0;
|
||||
|
||||
merge_commit = find_pseudo_merge(pm, commit_pos);
|
||||
if (!merge_commit)
|
||||
return 0;
|
||||
|
||||
if (merge_commit->pseudo_merge_ofs & ((uint64_t)1<<63)) {
|
||||
struct pseudo_merge_commit_ext ext = { 0 };
|
||||
off_t ofs = merge_commit->pseudo_merge_ofs & ~((uint64_t)1<<63);
|
||||
uint32_t i;
|
||||
|
||||
if (pseudo_merge_ext_at(pm, &ext, ofs) < -1) {
|
||||
warning(_("could not read extended pseudo-merge table "
|
||||
"for commit %s"),
|
||||
oid_to_hex(&commit->object.oid));
|
||||
return ret;
|
||||
}
|
||||
|
||||
for (i = 0; i < ext.nr; i++) {
|
||||
if (nth_pseudo_merge_ext(pm, &ext, merge_commit, i) < 0)
|
||||
return ret;
|
||||
|
||||
merge = pseudo_merge_at(pm, &commit->object.oid,
|
||||
merge_commit->pseudo_merge_ofs);
|
||||
|
||||
if (!merge)
|
||||
return ret;
|
||||
|
||||
if (apply_pseudo_merge(pm, merge, result, NULL))
|
||||
ret++;
|
||||
}
|
||||
} else {
|
||||
merge = pseudo_merge_at(pm, &commit->object.oid,
|
||||
merge_commit->pseudo_merge_ofs);
|
||||
|
||||
if (!merge)
|
||||
return ret;
|
||||
|
||||
if (apply_pseudo_merge(pm, merge, result, NULL))
|
||||
ret++;
|
||||
}
|
||||
|
||||
if (ret)
|
||||
cascade_pseudo_merges(pm, result, NULL);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
int cascade_pseudo_merges(const struct pseudo_merge_map *pm,
|
||||
struct bitmap *result,
|
||||
struct bitmap *roots)
|
||||
{
|
||||
unsigned any_satisfied;
|
||||
int ret = 0;
|
||||
|
||||
do {
|
||||
struct pseudo_merge *merge;
|
||||
uint32_t i;
|
||||
|
||||
any_satisfied = 0;
|
||||
|
||||
for (i = 0; i < pm->nr; i++) {
|
||||
merge = use_pseudo_merge(pm, &pm->v[i]);
|
||||
if (apply_pseudo_merge(pm, merge, result, roots)) {
|
||||
any_satisfied |= 1;
|
||||
ret++;
|
||||
}
|
||||
}
|
||||
} while (any_satisfied);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,
|
||||
struct bitmap *parents)
|
||||
{
|
||||
struct pseudo_merge *match = NULL;
|
||||
size_t i;
|
||||
|
||||
if (!pm->nr)
|
||||
return NULL;
|
||||
|
||||
/*
|
||||
* NOTE: this loop is quadratic in the worst-case (where no
|
||||
* matching pseudo-merge bitmaps are found), but in practice
|
||||
* this is OK for a few reasons:
|
||||
*
|
||||
* - Rejecting pseudo-merge bitmaps that do not match the
|
||||
* given commit is done quickly (i.e. `bitmap_equals_ewah()`
|
||||
* returns early when we know the two bitmaps aren't equal.
|
||||
*
|
||||
* - Already matched pseudo-merge bitmaps (which we track with
|
||||
* the `->satisfied` bit here) are skipped as potential
|
||||
* candidates.
|
||||
*
|
||||
* - The number of pseudo-merges should be small (in the
|
||||
* hundreds for most repositories).
|
||||
*
|
||||
* If in the future this semi-quadratic behavior does become a
|
||||
* problem, another approach would be to keep track of which
|
||||
* pseudo-merges are still "viable" after enumerating the
|
||||
* pseudo-merge commit's parents:
|
||||
*
|
||||
* - A pseudo-merge bitmap becomes non-viable when the bit(s)
|
||||
* corresponding to one or more parent(s) of the given
|
||||
* commit are not set in a candidate pseudo-merge's commits
|
||||
* bitmap.
|
||||
*
|
||||
* - After processing all bits, enumerate the remaining set of
|
||||
* viable pseudo-merge bitmaps, and check that their
|
||||
* popcount() matches the number of parents in the given
|
||||
* commit.
|
||||
*/
|
||||
for (i = 0; i < pm->nr; i++) {
|
||||
struct pseudo_merge *candidate = use_pseudo_merge(pm, &pm->v[i]);
|
||||
if (!candidate || candidate->satisfied)
|
||||
continue;
|
||||
if (!bitmap_equals_ewah(parents, candidate->commits))
|
||||
continue;
|
||||
|
||||
match = candidate;
|
||||
match->satisfied = 1;
|
||||
break;
|
||||
}
|
||||
|
||||
return match;
|
||||
}
|
||||
|
|
@ -0,0 +1,218 @@
|
|||
#ifndef PSEUDO_MERGE_H
|
||||
#define PSEUDO_MERGE_H
|
||||
|
||||
#include "git-compat-util.h"
|
||||
#include "strmap.h"
|
||||
#include "khashl.h"
|
||||
#include "ewah/ewok.h"
|
||||
|
||||
struct commit;
|
||||
struct string_list;
|
||||
struct bitmap_index;
|
||||
|
||||
/*
|
||||
* A pseudo-merge group tracks the set of non-bitmapped reference tips
|
||||
* that match the given pattern.
|
||||
*
|
||||
* Within those matches, they are further segmented by separating
|
||||
* consecutive capture groups with '-' dash character capture groups
|
||||
* with '-' dash characters.
|
||||
*
|
||||
* Those groups are then ordered by committer date and partitioned
|
||||
* into individual pseudo-merge(s) according to the decay, max_merges,
|
||||
* sample_rate, and threshold parameters.
|
||||
*/
|
||||
struct pseudo_merge_group {
|
||||
regex_t *pattern;
|
||||
|
||||
/* capture group(s) -> struct pseudo_merge_matches */
|
||||
struct strmap matches;
|
||||
|
||||
/*
|
||||
* The individual pseudo-merge(s) that are generated from the
|
||||
* above array of matches, partitioned according to the below
|
||||
* parameters.
|
||||
*/
|
||||
struct commit **merges;
|
||||
size_t merges_nr;
|
||||
size_t merges_alloc;
|
||||
|
||||
/*
|
||||
* Pseudo-merge grouping parameters. See git-config(1) for
|
||||
* more information.
|
||||
*/
|
||||
float decay;
|
||||
int max_merges;
|
||||
int sample_rate;
|
||||
int stable_size;
|
||||
timestamp_t threshold;
|
||||
timestamp_t stable_threshold;
|
||||
};
|
||||
|
||||
struct pseudo_merge_matches {
|
||||
struct commit **stable;
|
||||
struct commit **unstable;
|
||||
size_t stable_nr, stable_alloc;
|
||||
size_t unstable_nr, unstable_alloc;
|
||||
};
|
||||
|
||||
/*
|
||||
* Read the repository's configuration:
|
||||
*
|
||||
* - bitmapPseudoMerge.<name>.pattern
|
||||
* - bitmapPseudoMerge.<name>.decay
|
||||
* - bitmapPseudoMerge.<name>.sampleRate
|
||||
* - bitmapPseudoMerge.<name>.threshold
|
||||
* - bitmapPseudoMerge.<name>.maxMerges
|
||||
* - bitmapPseudoMerge.<name>.stableThreshold
|
||||
* - bitmapPseudoMerge.<name>.stableSize
|
||||
*
|
||||
* and populates the given `list` with pseudo-merge groups. String
|
||||
* entry keys are the pseudo-merge group names, and the values are
|
||||
* pointers to the pseudo_merge_group structure itself.
|
||||
*/
|
||||
void load_pseudo_merges_from_config(struct string_list *list);
|
||||
|
||||
/*
|
||||
* A pseudo-merge commit index (pseudo_merge_commit_idx) maps a
|
||||
* particular (non-pseudo-merge) commit to the list of pseudo-merge(s)
|
||||
* it appears in.
|
||||
*/
|
||||
struct pseudo_merge_commit_idx {
|
||||
uint32_t *pseudo_merge;
|
||||
size_t nr, alloc;
|
||||
};
|
||||
|
||||
/*
|
||||
* Selects pseudo-merges from a list of commits, populating the given
|
||||
* string_list of pseudo-merge groups.
|
||||
*
|
||||
* Populates the pseudo_merge_commits map with a commit_idx
|
||||
* corresponding to each commit in the list. Counts the total number
|
||||
* of pseudo-merges generated.
|
||||
*
|
||||
* Optionally shows a progress meter.
|
||||
*/
|
||||
void select_pseudo_merges(struct string_list *list,
|
||||
struct commit **commits, size_t commits_nr,
|
||||
kh_oid_map_t *pseudo_merge_commits,
|
||||
uint32_t *pseudo_merges_nr,
|
||||
unsigned show_progress);
|
||||
|
||||
/*
|
||||
* Represents a serialized view of a file containing pseudo-merge(s)
|
||||
* (see Documentation/technical/bitmap-format.txt for a specification
|
||||
* of the format).
|
||||
*/
|
||||
struct pseudo_merge_map {
|
||||
/*
|
||||
* An array of pseudo-merge(s), lazily loaded from the .bitmap
|
||||
* file.
|
||||
*/
|
||||
struct pseudo_merge *v;
|
||||
size_t nr;
|
||||
size_t commits_nr;
|
||||
|
||||
/*
|
||||
* Pointers into a memory-mapped view of the .bitmap file:
|
||||
*
|
||||
* - map: the beginning of the .bitmap file
|
||||
* - commits: the beginning of the pseudo-merge commit index
|
||||
* - map_size: the size of the .bitmap file
|
||||
*/
|
||||
const unsigned char *map;
|
||||
const unsigned char *commits;
|
||||
|
||||
size_t map_size;
|
||||
};
|
||||
|
||||
/*
|
||||
* An individual pseudo-merge, storing a pair of lazily-loaded
|
||||
* bitmaps:
|
||||
*
|
||||
* - commits: the set of commit(s) that are part of the pseudo-merge
|
||||
* - bitmap: the set of object(s) reachable from the above set of
|
||||
* commits.
|
||||
*
|
||||
* The `at` and `bitmap_at` fields are used to store the locations of
|
||||
* each of the above bitmaps in the .bitmap file.
|
||||
*/
|
||||
struct pseudo_merge {
|
||||
struct ewah_bitmap *commits;
|
||||
struct ewah_bitmap *bitmap;
|
||||
|
||||
off_t at;
|
||||
off_t bitmap_at;
|
||||
|
||||
/*
|
||||
* `satisfied` indicates whether the given pseudo-merge has been
|
||||
* used.
|
||||
*
|
||||
* `loaded_commits` and `loaded_bitmap` indicate whether the
|
||||
* respective bitmaps have been loaded and read from the
|
||||
* .bitmap file.
|
||||
*/
|
||||
unsigned satisfied : 1,
|
||||
loaded_commits : 1,
|
||||
loaded_bitmap : 1;
|
||||
};
|
||||
|
||||
/*
|
||||
* Frees the given pseudo-merge map, releasing any memory held by (a)
|
||||
* parsed EWAH bitmaps, or (b) the array of pseudo-merges itself. Does
|
||||
* not free the memory-mapped view of the .bitmap file.
|
||||
*/
|
||||
void free_pseudo_merge_map(struct pseudo_merge_map *pm);
|
||||
|
||||
/*
|
||||
* Loads the bitmap corresponding to the given pseudo-merge from the
|
||||
* map, if it has not already been loaded.
|
||||
*/
|
||||
struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,
|
||||
struct pseudo_merge *merge);
|
||||
|
||||
/*
|
||||
* Loads the pseudo-merge and its commits bitmap from the given
|
||||
* pseudo-merge map, if it has not already been loaded.
|
||||
*/
|
||||
struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,
|
||||
struct pseudo_merge *merge);
|
||||
|
||||
/*
|
||||
* Applies pseudo-merge(s) containing the given commit to the bitmap
|
||||
* "result".
|
||||
*
|
||||
* If any pseudo-merge(s) were satisfied, returns the number
|
||||
* satisfied, otherwise returns 0. If any were satisfied, the
|
||||
* remaining unsatisfied pseudo-merges are cascaded (see below).
|
||||
*/
|
||||
int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,
|
||||
struct bitmap *result,
|
||||
struct commit *commit, uint32_t commit_pos);
|
||||
|
||||
/*
|
||||
* Applies pseudo-merge(s) which are satisfied according to the
|
||||
* current bitmap in result (or roots, see below). If any
|
||||
* pseudo-merges were satisfied, repeat the process over unsatisfied
|
||||
* pseudo-merge commits until no more pseudo-merges are satisfied.
|
||||
*
|
||||
* Result is the bitmap to which the pseudo-merge(s) are applied.
|
||||
* Roots (if given) is a bitmap of the traversal tip(s) for either
|
||||
* side of a reachability traversal.
|
||||
*
|
||||
* Roots may given instead of a populated results bitmap at the
|
||||
* beginning of a traversal on either side where the reachability
|
||||
* closure over tips is not yet known.
|
||||
*/
|
||||
int cascade_pseudo_merges(const struct pseudo_merge_map *pm,
|
||||
struct bitmap *result,
|
||||
struct bitmap *roots);
|
||||
|
||||
/*
|
||||
* Returns a pseudo-merge which contains the exact set of commits
|
||||
* listed in the "parents" bitamp, or NULL if none could be found.
|
||||
*/
|
||||
struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,
|
||||
struct bitmap *parents);
|
||||
|
||||
#endif
|
||||
|
|
@ -13,21 +13,41 @@ static int bitmap_dump_hashes(void)
|
|||
return test_bitmap_hashes(the_repository);
|
||||
}
|
||||
|
||||
static int bitmap_dump_pseudo_merges(void)
|
||||
{
|
||||
return test_bitmap_pseudo_merges(the_repository);
|
||||
}
|
||||
|
||||
static int bitmap_dump_pseudo_merge_commits(uint32_t n)
|
||||
{
|
||||
return test_bitmap_pseudo_merge_commits(the_repository, n);
|
||||
}
|
||||
|
||||
static int bitmap_dump_pseudo_merge_objects(uint32_t n)
|
||||
{
|
||||
return test_bitmap_pseudo_merge_objects(the_repository, n);
|
||||
}
|
||||
|
||||
int cmd__bitmap(int argc, const char **argv)
|
||||
{
|
||||
setup_git_directory();
|
||||
|
||||
if (argc != 2)
|
||||
goto usage;
|
||||
|
||||
if (!strcmp(argv[1], "list-commits"))
|
||||
if (argc == 2 && !strcmp(argv[1], "list-commits"))
|
||||
return bitmap_list_commits();
|
||||
if (!strcmp(argv[1], "dump-hashes"))
|
||||
if (argc == 2 && !strcmp(argv[1], "dump-hashes"))
|
||||
return bitmap_dump_hashes();
|
||||
if (argc == 2 && !strcmp(argv[1], "dump-pseudo-merges"))
|
||||
return bitmap_dump_pseudo_merges();
|
||||
if (argc == 3 && !strcmp(argv[1], "dump-pseudo-merge-commits"))
|
||||
return bitmap_dump_pseudo_merge_commits(atoi(argv[2]));
|
||||
if (argc == 3 && !strcmp(argv[1], "dump-pseudo-merge-objects"))
|
||||
return bitmap_dump_pseudo_merge_objects(atoi(argv[2]));
|
||||
|
||||
usage:
|
||||
usage("\ttest-tool bitmap list-commits\n"
|
||||
"\ttest-tool bitmap dump-hashes");
|
||||
"\ttest-tool bitmap dump-hashes\n"
|
||||
"\ttest-tool bitmap dump-pseudo-merges\n"
|
||||
"\ttest-tool bitmap dump-pseudo-merge-commits <n>\n"
|
||||
"\ttest-tool bitmap dump-pseudo-merge-objects <n>");
|
||||
|
||||
return -1;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,32 @@
|
|||
#!/bin/sh
|
||||
|
||||
test_description='pseudo-merge bitmaps'
|
||||
. ./perf-lib.sh
|
||||
|
||||
test_perf_large_repo
|
||||
|
||||
test_expect_success 'setup' '
|
||||
git \
|
||||
-c bitmapPseudoMerge.all.pattern="refs/" \
|
||||
-c bitmapPseudoMerge.all.threshold=now \
|
||||
-c bitmapPseudoMerge.all.stableThreshold=never \
|
||||
-c bitmapPseudoMerge.all.maxMerges=64 \
|
||||
-c pack.writeBitmapLookupTable=true \
|
||||
repack -adb
|
||||
'
|
||||
|
||||
test_perf 'git rev-list --count --all --objects (no bitmaps)' '
|
||||
git rev-list --objects --all
|
||||
'
|
||||
|
||||
test_perf 'git rev-list --count --all --objects (no pseudo-merges)' '
|
||||
GIT_TEST_USE_PSEDUO_MERGES=0 \
|
||||
git rev-list --objects --all --use-bitmap-index
|
||||
'
|
||||
|
||||
test_perf 'git rev-list --count --all --objects (with pseudo-merges)' '
|
||||
GIT_TEST_USE_PSEDUO_MERGES=1 \
|
||||
git rev-list --objects --all --use-bitmap-index
|
||||
'
|
||||
|
||||
test_done
|
||||
|
|
@ -0,0 +1,389 @@
|
|||
#!/bin/sh
|
||||
|
||||
test_description='pseudo-merge bitmaps'
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
|
||||
|
||||
. ./test-lib.sh
|
||||
|
||||
test_pseudo_merges () {
|
||||
test-tool bitmap dump-pseudo-merges
|
||||
}
|
||||
|
||||
test_pseudo_merge_commits () {
|
||||
test-tool bitmap dump-pseudo-merge-commits "$1"
|
||||
}
|
||||
|
||||
test_pseudo_merges_satisfied () {
|
||||
test_trace2_data bitmap pseudo_merges_satisfied "$1"
|
||||
}
|
||||
|
||||
test_pseudo_merges_cascades () {
|
||||
test_trace2_data bitmap pseudo_merges_cascades "$1"
|
||||
}
|
||||
|
||||
test_pseudo_merges_reused () {
|
||||
test_trace2_data pack-bitmap-write building_bitmaps_pseudo_merge_reused "$1"
|
||||
}
|
||||
|
||||
tag_everything () {
|
||||
git rev-list --all --no-object-names >in &&
|
||||
perl -lne '
|
||||
print "create refs/tags/" . $. . " " . $1 if /([0-9a-f]+)/
|
||||
' <in | git update-ref --stdin
|
||||
}
|
||||
|
||||
test_expect_success 'setup' '
|
||||
test_commit_bulk 512 &&
|
||||
tag_everything
|
||||
'
|
||||
|
||||
test_expect_success 'bitmap traversal without pseudo-merges' '
|
||||
git repack -adb &&
|
||||
|
||||
git rev-list --count --all --objects >expect &&
|
||||
|
||||
: >trace2.txt &&
|
||||
GIT_TRACE2_EVENT=$PWD/trace2.txt \
|
||||
git rev-list --count --all --objects --use-bitmap-index >actual &&
|
||||
|
||||
test_pseudo_merges_satisfied 0 <trace2.txt &&
|
||||
test_pseudo_merges_cascades 0 <trace2.txt &&
|
||||
test_pseudo_merges >merges &&
|
||||
test_must_be_empty merges &&
|
||||
test_cmp expect actual
|
||||
'
|
||||
|
||||
test_expect_success 'pseudo-merges accurately represent their objects' '
|
||||
test_config bitmapPseudoMerge.test.pattern "refs/tags/" &&
|
||||
test_config bitmapPseudoMerge.test.maxMerges 8 &&
|
||||
test_config bitmapPseudoMerge.test.stableThreshold never &&
|
||||
|
||||
git repack -adb &&
|
||||
|
||||
test_pseudo_merges >merges &&
|
||||
test_line_count = 8 merges &&
|
||||
|
||||
for i in $(test_seq 0 $(($(wc -l <merges)-1)))
|
||||
do
|
||||
test-tool bitmap dump-pseudo-merge-commits $i >commits &&
|
||||
|
||||
git rev-list --objects --no-object-names --stdin <commits >expect.raw &&
|
||||
test-tool bitmap dump-pseudo-merge-objects $i >actual.raw &&
|
||||
|
||||
sort -u <expect.raw >expect &&
|
||||
sort -u <actual.raw >actual &&
|
||||
|
||||
test_cmp expect actual || return 1
|
||||
done
|
||||
'
|
||||
|
||||
test_expect_success 'bitmap traversal with pseudo-merges' '
|
||||
: >trace2.txt &&
|
||||
GIT_TRACE2_EVENT=$PWD/trace2.txt \
|
||||
git rev-list --count --all --objects --use-bitmap-index >actual &&
|
||||
git rev-list --count --all --objects >expect &&
|
||||
|
||||
test_pseudo_merges_satisfied 8 <trace2.txt &&
|
||||
test_pseudo_merges_cascades 1 <trace2.txt &&
|
||||
test_cmp expect actual
|
||||
'
|
||||
|
||||
test_expect_success 'stale bitmap traversal with pseudo-merges' '
|
||||
test_commit other &&
|
||||
|
||||
: >trace2.txt &&
|
||||
GIT_TRACE2_EVENT=$PWD/trace2.txt \
|
||||
git rev-list --count --all --objects --use-bitmap-index >actual &&
|
||||
git rev-list --count --all --objects >expect &&
|
||||
|
||||
test_pseudo_merges_satisfied 8 <trace2.txt &&
|
||||
test_pseudo_merges_cascades 1 <trace2.txt &&
|
||||
test_cmp expect actual
|
||||
'
|
||||
|
||||
test_expect_success 'bitmapPseudoMerge.sampleRate adjusts commit selection rate' '
|
||||
test_config bitmapPseudoMerge.test.pattern "refs/tags/" &&
|
||||
test_config bitmapPseudoMerge.test.maxMerges 8 &&
|
||||
test_config bitmapPseudoMerge.test.stableThreshold never &&
|
||||
|
||||
commits_nr=$(git rev-list --all --count) &&
|
||||
|
||||
for rate in 100 50 10
|
||||
do
|
||||
git -c bitmapPseudoMerge.test.sampleRate=$rate repack -adb &&
|
||||
|
||||
test_pseudo_merges >merges &&
|
||||
for i in $(test_seq 0 $(($(wc -l <merges)-1)))
|
||||
do
|
||||
test_pseudo_merge_commits $i || return 1
|
||||
done >commits &&
|
||||
|
||||
test-tool bitmap list-commits >bitmaps &&
|
||||
bitmaps_nr="$(wc -l <bitmaps)" &&
|
||||
|
||||
perl -MPOSIX -e "print ceil((\$ARGV[0]/100)*(\$ARGV[1]-\$ARGV[2]))" \
|
||||
"$rate" "$commits_nr" "$bitmaps_nr" >expect &&
|
||||
|
||||
test $(cat expect) -eq $(wc -l <commits) || return 1
|
||||
done
|
||||
'
|
||||
|
||||
test_expect_success 'bitmapPseudoMerge.threshold excludes newer commits' '
|
||||
git init pseudo-merge-threshold &&
|
||||
(
|
||||
cd pseudo-merge-threshold &&
|
||||
|
||||
new="1672549200" && # 2023-01-01
|
||||
old="1641013200" && # 2022-01-01
|
||||
|
||||
test_commit_bulk --message="new" --date "$new +0000" 128 &&
|
||||
test_commit_bulk --message="old" --date "$old +0000" 128 &&
|
||||
test_tick &&
|
||||
|
||||
tag_everything &&
|
||||
|
||||
git \
|
||||
-c bitmapPseudoMerge.test.pattern="refs/tags/" \
|
||||
-c bitmapPseudoMerge.test.maxMerges=1 \
|
||||
-c bitmapPseudoMerge.test.threshold=$(($new - 1)) \
|
||||
-c bitmapPseudoMerge.test.stableThreshold=never \
|
||||
repack -adb &&
|
||||
|
||||
test_pseudo_merges >merges &&
|
||||
test_line_count = 1 merges &&
|
||||
|
||||
test_pseudo_merge_commits 0 >oids &&
|
||||
git cat-file --batch <oids >commits &&
|
||||
|
||||
test $(wc -l <oids) = $(grep -c "^committer.*$old +0000$" commits)
|
||||
)
|
||||
'
|
||||
|
||||
test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' '
|
||||
(
|
||||
cd pseudo-merge-threshold &&
|
||||
|
||||
new="1672549200" && # 2023-01-01
|
||||
mid="1654059600" && # 2022-06-01
|
||||
old="1641013200" && # 2022-01-01
|
||||
|
||||
test_commit_bulk --message="mid" --date "$mid +0000" 128 &&
|
||||
test_tick &&
|
||||
|
||||
git for-each-ref --format="delete %(refname)" refs/tags >in &&
|
||||
git update-ref --stdin <in &&
|
||||
|
||||
tag_everything &&
|
||||
|
||||
git \
|
||||
-c bitmapPseudoMerge.test.pattern="refs/tags/" \
|
||||
-c bitmapPseudoMerge.test.maxMerges=1 \
|
||||
-c bitmapPseudoMerge.test.threshold=$(($new - 1)) \
|
||||
-c bitmapPseudoMerge.test.stableThreshold=$(($mid - 1)) \
|
||||
-c bitmapPseudoMerge.test.stableSize=10 \
|
||||
repack -adb &&
|
||||
|
||||
test_pseudo_merges >merges &&
|
||||
merges_nr="$(wc -l <merges)" &&
|
||||
|
||||
for i in $(test_seq $(($merges_nr - 1)))
|
||||
do
|
||||
test_pseudo_merge_commits 0 >oids &&
|
||||
git cat-file --batch <oids >commits &&
|
||||
|
||||
expect="$(grep -c "^committer.*$old +0000$" commits)" &&
|
||||
actual="$(wc -l <oids)" &&
|
||||
|
||||
test $expect = $actual || return 1
|
||||
done &&
|
||||
|
||||
test_pseudo_merge_commits $(($merges_nr - 1)) >oids &&
|
||||
git cat-file --batch <oids >commits &&
|
||||
test $(wc -l <oids) = $(grep -c "^committer.*$mid +0000$" commits)
|
||||
)
|
||||
'
|
||||
|
||||
test_expect_success 'out of order thresholds are rejected' '
|
||||
test_must_fail git \
|
||||
-c bitmapPseudoMerge.test.pattern="refs/*" \
|
||||
-c bitmapPseudoMerge.test.threshold=1.month.ago \
|
||||
-c bitmapPseudoMerge.test.stableThreshold=1.week.ago \
|
||||
repack -adb 2>err &&
|
||||
|
||||
cat >expect <<-EOF &&
|
||||
fatal: pseudo-merge group ${SQ}test${SQ} has unstable threshold before stable one
|
||||
EOF
|
||||
|
||||
test_cmp expect err
|
||||
'
|
||||
|
||||
test_expect_success 'pseudo-merge pattern with capture groups' '
|
||||
git init pseudo-merge-captures &&
|
||||
(
|
||||
cd pseudo-merge-captures &&
|
||||
|
||||
test_commit_bulk 128 &&
|
||||
tag_everything &&
|
||||
|
||||
for r in $(test_seq 8)
|
||||
do
|
||||
test_commit_bulk 16 &&
|
||||
|
||||
git rev-list HEAD~16.. >in &&
|
||||
|
||||
perl -lne "print \"create refs/remotes/$r/tags/\$. \$_\"" <in |
|
||||
git update-ref --stdin || return 1
|
||||
done &&
|
||||
|
||||
git \
|
||||
-c bitmapPseudoMerge.tags.pattern="refs/remotes/([0-9]+)/tags/" \
|
||||
-c bitmapPseudoMerge.tags.maxMerges=1 \
|
||||
repack -adb &&
|
||||
|
||||
git for-each-ref --format="%(objectname) %(refname)" >refs &&
|
||||
|
||||
test_pseudo_merges >merges &&
|
||||
for m in $(test_seq 0 $(($(wc -l <merges) - 1)))
|
||||
do
|
||||
test_pseudo_merge_commits $m >oids &&
|
||||
grep -f oids refs |
|
||||
perl -lne "print \$1 if /refs\/remotes\/([0-9]+)/" |
|
||||
sort -u || return 1
|
||||
done >remotes &&
|
||||
|
||||
test $(wc -l <remotes) -eq $(sort -u <remotes | wc -l)
|
||||
)
|
||||
'
|
||||
|
||||
test_expect_success 'pseudo-merge overlap setup' '
|
||||
git init pseudo-merge-overlap &&
|
||||
(
|
||||
cd pseudo-merge-overlap &&
|
||||
|
||||
test_commit_bulk 256 &&
|
||||
tag_everything &&
|
||||
|
||||
git \
|
||||
-c bitmapPseudoMerge.all.pattern="refs/" \
|
||||
-c bitmapPseudoMerge.all.maxMerges=1 \
|
||||
-c bitmapPseudoMerge.all.stableThreshold=never \
|
||||
-c bitmapPseudoMerge.tags.pattern="refs/tags/" \
|
||||
-c bitmapPseudoMerge.tags.maxMerges=1 \
|
||||
-c bitmapPseudoMerge.tags.stableThreshold=never \
|
||||
repack -adb
|
||||
)
|
||||
'
|
||||
|
||||
test_expect_success 'pseudo-merge overlap generates overlapping groups' '
|
||||
(
|
||||
cd pseudo-merge-overlap &&
|
||||
|
||||
test_pseudo_merges >merges &&
|
||||
test_line_count = 2 merges &&
|
||||
|
||||
test_pseudo_merge_commits 0 >commits-0.raw &&
|
||||
test_pseudo_merge_commits 1 >commits-1.raw &&
|
||||
|
||||
sort commits-0.raw >commits-0 &&
|
||||
sort commits-1.raw >commits-1 &&
|
||||
|
||||
comm -12 commits-0 commits-1 >overlap &&
|
||||
|
||||
test_line_count -gt 0 overlap
|
||||
)
|
||||
'
|
||||
|
||||
test_expect_success 'pseudo-merge overlap traversal' '
|
||||
(
|
||||
cd pseudo-merge-overlap &&
|
||||
|
||||
: >trace2.txt &&
|
||||
GIT_TRACE2_EVENT=$PWD/trace2.txt \
|
||||
git rev-list --count --all --objects --use-bitmap-index >actual &&
|
||||
git rev-list --count --all --objects >expect &&
|
||||
|
||||
test_pseudo_merges_satisfied 2 <trace2.txt &&
|
||||
test_pseudo_merges_cascades 1 <trace2.txt &&
|
||||
test_cmp expect actual
|
||||
)
|
||||
'
|
||||
|
||||
test_expect_success 'pseudo-merge overlap stale traversal' '
|
||||
(
|
||||
cd pseudo-merge-overlap &&
|
||||
|
||||
test_commit other &&
|
||||
|
||||
: >trace2.txt &&
|
||||
GIT_TRACE2_EVENT=$PWD/trace2.txt \
|
||||
git rev-list --count --all --objects --use-bitmap-index >actual &&
|
||||
git rev-list --count --all --objects >expect &&
|
||||
|
||||
test_pseudo_merges_satisfied 2 <trace2.txt &&
|
||||
test_pseudo_merges_cascades 1 <trace2.txt &&
|
||||
test_cmp expect actual
|
||||
)
|
||||
'
|
||||
|
||||
test_expect_success 'pseudo-merge reuse' '
|
||||
git init pseudo-merge-reuse &&
|
||||
(
|
||||
cd pseudo-merge-reuse &&
|
||||
|
||||
stable="1641013200" && # 2022-01-01
|
||||
unstable="1672549200" && # 2023-01-01
|
||||
|
||||
for date in $stable $unstable
|
||||
do
|
||||
test_commit_bulk --date "$date +0000" 128 &&
|
||||
test_tick || return 1
|
||||
done &&
|
||||
|
||||
tag_everything &&
|
||||
|
||||
git \
|
||||
-c bitmapPseudoMerge.test.pattern="refs/tags/" \
|
||||
-c bitmapPseudoMerge.test.maxMerges=1 \
|
||||
-c bitmapPseudoMerge.test.threshold=now \
|
||||
-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \
|
||||
-c bitmapPseudoMerge.test.stableSize=512 \
|
||||
repack -adb &&
|
||||
|
||||
test_pseudo_merges >merges &&
|
||||
test_line_count = 2 merges &&
|
||||
|
||||
test_pseudo_merge_commits 0 >stable-oids.before &&
|
||||
test_pseudo_merge_commits 1 >unstable-oids.before &&
|
||||
|
||||
: >trace2.txt &&
|
||||
GIT_TRACE2_EVENT=$PWD/trace2.txt git \
|
||||
-c bitmapPseudoMerge.test.pattern="refs/tags/" \
|
||||
-c bitmapPseudoMerge.test.maxMerges=2 \
|
||||
-c bitmapPseudoMerge.test.threshold=now \
|
||||
-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \
|
||||
-c bitmapPseudoMerge.test.stableSize=512 \
|
||||
repack -adb &&
|
||||
|
||||
test_pseudo_merges_reused 1 <trace2.txt &&
|
||||
|
||||
test_pseudo_merges >merges &&
|
||||
test_line_count = 3 merges &&
|
||||
|
||||
test_pseudo_merge_commits 0 >stable-oids.after &&
|
||||
for i in 1 2
|
||||
do
|
||||
test_pseudo_merge_commits $i || return 1
|
||||
done >unstable-oids.after &&
|
||||
|
||||
sort -u <stable-oids.before >expect &&
|
||||
sort -u <stable-oids.after >actual &&
|
||||
test_cmp expect actual &&
|
||||
|
||||
sort -u <unstable-oids.before >expect &&
|
||||
sort -u <unstable-oids.after >actual &&
|
||||
test_cmp expect actual
|
||||
)
|
||||
'
|
||||
|
||||
test_done
|
||||
|
|
@ -458,6 +458,7 @@ test_commit_bulk () {
|
|||
indir=.
|
||||
ref=HEAD
|
||||
n=1
|
||||
notick=
|
||||
message='commit %s'
|
||||
filename='%s.t'
|
||||
contents='content %s'
|
||||
|
|
@ -488,6 +489,12 @@ test_commit_bulk () {
|
|||
filename="${1#--*=}-%s.t"
|
||||
contents="${1#--*=} %s"
|
||||
;;
|
||||
--date)
|
||||
notick=yes
|
||||
GIT_COMMITTER_DATE="$2"
|
||||
GIT_AUTHOR_DATE="$2"
|
||||
shift
|
||||
;;
|
||||
-*)
|
||||
BUG "invalid test_commit_bulk option: $1"
|
||||
;;
|
||||
|
|
@ -507,7 +514,10 @@ test_commit_bulk () {
|
|||
|
||||
while test "$total" -gt 0
|
||||
do
|
||||
test_tick &&
|
||||
if test -z "$notick"
|
||||
then
|
||||
test_tick
|
||||
fi &&
|
||||
echo "commit $ref"
|
||||
printf 'author %s <%s> %s\n' \
|
||||
"$GIT_AUTHOR_NAME" \
|
||||
|
|
|
|||
Loading…
Reference in New Issue