1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-05-22 17:56:09 +02:00
git/repository.h
Taylor Blau b0afdce5da pack-bitmap.c: use commit boundary during bitmap traversal
When reachability bitmap coverage exists in a repository, Git will use a
different (and hopefully faster) traversal to compute revision walks.

Consider a set of positive and negative tips (which we'll refer to with
their standard bitmap parlance by "wants", and "haves"). In order to
figure out what objects exist between the tips, the existing traversal
in `prepare_bitmap_walk()` does something like:

  1. Consider if we can even compute the set of objects with bitmaps,
     and fall back to the usual traversal if we cannot. For example,
     pathspec limiting traversals can't be computed using bitmaps (since
     they don't know which objects are at which paths). The same is true
     of certain kinds of non-trivial object filters.

  2. If we can compute the traversal with bitmaps, partition the
     (dereferenced) tips into two object lists, "haves", and "wants",
     based on whether or not the objects have the UNINTERESTING flag,
     respectively.

  3. Fall back to the ordinary object traversal if either (a) there are
     more than zero haves, none of which are in the bitmapped pack or
     MIDX, or (b) there are no wants.

  4. Construct a reachability bitmap for the "haves" side by walking
     from the revision tips down to any existing bitmaps, OR-ing in any
     bitmaps as they are found.

  5. Then do the same for the "wants" side, stopping at any objects that
     appear in the "haves" bitmap.

  6. Filter the results if any object filter (that can be easily
     computed with bitmaps alone) was given, and then return back to the
     caller.

When there is good bitmap coverage relative to the traversal tips, this
walk is often significantly faster than an ordinary object traversal
because it can visit far fewer objects.

But in certain cases, it can be significantly *slower* than the usual
object traversal. Why? Because we need to compute complete bitmaps on
either side of the walk. If either one (or both) of the sides require
walking many (or all!) objects before they get to an existing bitmap,
the extra bitmap machinery is mostly or all overhead.

One of the benefits, however, is that even if the walk is slower, bitmap
traversals are guaranteed to provide an *exact* answer. Unlike the
traditional object traversal algorithm, which can over-count the results
by not opening trees for older commits, the bitmap walk builds an exact
reachability bitmap for either side, meaning the results are never
over-counted.

But producing non-exact results is OK for our traversal here (both in
the bitmap case and not), as long as the results are over-counted, not
under.

Relaxing the bitmap traversal to allow it to produce over-counted
results gives us the opportunity to make some significant improvements.
Instead of the above, the new algorithm only has to walk from the
*boundary* down to the nearest bitmap, instead of from each of the
UNINTERESTING tips.

The boundary-based approach still has degenerate cases, but we'll show
in a moment that it is often a significant improvement.

The new algorithm works as follows:

  1. Build a (partial) bitmap of the haves side by first OR-ing any
     bitmap(s) that already exist for UNINTERESTING commits between the
     haves and the boundary.

  2. For each commit along the boundary, add it as a fill-in traversal
     tip (where the traversal terminates once an existing bitmap is
     found), and perform fill-in traversal.

  3. Build up a complete bitmap of the wants side as usual, stopping any
     time we intersect the (partial) haves side.

  4. Return the results.

And is more-or-less equivalent to using the *old* algorithm with this
invocation:

    $ git rev-list --objects --use-bitmap-index $WANTS --not \
        $(git rev-list --objects --boundary $WANTS --not $HAVES |
          perl -lne 'print $1 if /^-(.*)/')

The new result performs significantly better in many cases, particularly
when the distance from the boundary commit(s) to an existing bitmap is
shorter than the distance from (all of) the have tips to the nearest
bitmapped commit.

Note that when using the old bitmap traversal algorithm, the results can
be *slower* than without bitmaps! Under the new algorithm, the result is
computed faster with bitmaps than without (at the cost of over-counting
the true number of objects in a similar fashion as the non-bitmap
traversal):

    # (Computing the number of tagged objects not on any branches
    # without bitmaps).
    $ time git rev-list --count --objects --tags --not --branches
    20

    real	0m1.388s
    user	0m1.092s
    sys	0m0.296s

    # (Computing the same query using the old bitmap traversal).
    $ time git rev-list --count --objects --tags --not --branches --use-bitmap-index
    19

    real	0m22.709s
    user	0m21.628s
    sys	0m1.076s

    # (this commit)
    $ time git.compile rev-list --count --objects --tags --not --branches --use-bitmap-index
    19

    real	0m1.518s
    user	0m1.234s
    sys	0m0.284s

The new algorithm is still slower than not using bitmaps at all, but it
is nearly a 15-fold improvement over the existing traversal.

In a more realistic setting (using my local copy of git.git), I can
observe a similar (if more modest) speed-up:

    $ argv="--count --objects --branches --not --tags"
    hyperfine \
      -n 'no bitmaps' "git.compile rev-list $argv" \
      -n 'existing traversal' "git.compile rev-list --use-bitmap-index $argv" \
      -n 'boundary traversal' "git.compile -c pack.useBitmapBoundaryTraversal=true rev-list --use-bitmap-index $argv"
    Benchmark 1: no bitmaps
      Time (mean ± σ):     124.6 ms ±   2.1 ms    [User: 103.7 ms, System: 20.8 ms]
      Range (min … max):   122.6 ms … 133.1 ms    22 runs

    Benchmark 2: existing traversal
      Time (mean ± σ):     368.6 ms ±   3.0 ms    [User: 325.3 ms, System: 43.1 ms]
      Range (min … max):   365.1 ms … 374.8 ms    10 runs

    Benchmark 3: boundary traversal
      Time (mean ± σ):     167.6 ms ±   0.9 ms    [User: 139.5 ms, System: 27.9 ms]
      Range (min … max):   166.1 ms … 169.2 ms    17 runs

    Summary
      'no bitmaps' ran
        1.34 ± 0.02 times faster than 'boundary traversal'
        2.96 ± 0.05 times faster than 'existing traversal'

Here, the new algorithm is also still slower than not using bitmaps, but
represents a more than 2-fold improvement over the existing traversal in
a more modest example.

Since this algorithm was originally written (nearly a year and a half
ago, at the time of writing), the bitmap lookup table shipped, making
the new algorithm's result more competitive. A few other future
directions for improving bitmap traversal times beyond not using bitmaps
at all:

  - Decrease the cost to decompress and OR together many bitmaps
    together (particularly when enumerating the uninteresting side of
    the walk). Here we could explore more efficient bitmap storage
    techniques, like Roaring+Run and/or use SIMD instructions to speed
    up ORing them together.

  - Store pseudo-merge bitmaps, which could allow us to OR together
    fewer "summary" bitmaps (which would also help with the above).

Helped-by: Jeff King <peff@peff.net>
Helped-by: Derrick Stolee <derrickstolee@github.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-05-08 12:05:55 -07:00

243 lines
6.5 KiB
C

#ifndef REPOSITORY_H
#define REPOSITORY_H
#include "path.h"
struct config_set;
struct fsmonitor_settings;
struct git_hash_algo;
struct index_state;
struct lock_file;
struct pathspec;
struct raw_object_store;
struct submodule_cache;
struct promisor_remote_config;
struct remote_state;
enum untracked_cache_setting {
UNTRACKED_CACHE_KEEP,
UNTRACKED_CACHE_REMOVE,
UNTRACKED_CACHE_WRITE,
};
enum fetch_negotiation_setting {
FETCH_NEGOTIATION_CONSECUTIVE,
FETCH_NEGOTIATION_SKIPPING,
FETCH_NEGOTIATION_NOOP,
};
struct repo_settings {
int initialized;
int core_commit_graph;
int commit_graph_generation_version;
int commit_graph_read_changed_paths;
int gc_write_commit_graph;
int fetch_write_commit_graph;
int command_requires_full_index;
int sparse_index;
int pack_read_reverse_index;
int pack_use_bitmap_boundary_traversal;
struct fsmonitor_settings *fsmonitor; /* lazily loaded */
int index_version;
int index_skip_hash;
enum untracked_cache_setting core_untracked_cache;
int pack_use_sparse;
enum fetch_negotiation_setting fetch_negotiation_algorithm;
int core_multi_pack_index;
};
struct repo_path_cache {
char *squash_msg;
char *merge_msg;
char *merge_rr;
char *merge_mode;
char *merge_head;
char *merge_autostash;
char *auto_merge;
char *fetch_head;
char *shallow;
};
struct repository {
/* Environment */
/*
* Path to the git directory.
* Cannot be NULL after initialization.
*/
char *gitdir;
/*
* Path to the common git directory.
* Cannot be NULL after initialization.
*/
char *commondir;
/*
* Holds any information related to accessing the raw object content.
*/
struct raw_object_store *objects;
/*
* All objects in this repository that have been parsed. This structure
* owns all objects it references, so users of "struct object *"
* generally do not need to free them; instead, when a repository is no
* longer used, call parsed_object_pool_clear() on this structure, which
* is called by the repositories repo_clear on its desconstruction.
*/
struct parsed_object_pool *parsed_objects;
/*
* The store in which the refs are held. This should generally only be
* accessed via get_main_ref_store(), as that will lazily initialize
* the ref object.
*/
struct ref_store *refs_private;
/*
* Contains path to often used file names.
*/
struct repo_path_cache cached_paths;
/*
* Path to the repository's graft file.
* Cannot be NULL after initialization.
*/
char *graft_file;
/*
* Path to the current worktree's index file.
* Cannot be NULL after initialization.
*/
char *index_file;
/*
* Path to the working directory.
* A NULL value indicates that there is no working directory.
*/
char *worktree;
/*
* Path from the root of the top-level superproject down to this
* repository. This is only non-NULL if the repository is initialized
* as a submodule of another repository.
*/
char *submodule_prefix;
struct repo_settings settings;
/* Subsystems */
/*
* Repository's config which contains key-value pairs from the usual
* set of config files (i.e. repo specific .git/config, user wide
* ~/.gitconfig, XDG config file and the global /etc/gitconfig)
*/
struct config_set *config;
/* Repository's submodule config as defined by '.gitmodules' */
struct submodule_cache *submodule_cache;
/*
* Repository's in-memory index.
* 'repo_read_index()' can be used to populate 'index'.
*/
struct index_state *index;
/* Repository's remotes and associated structures. */
struct remote_state *remote_state;
/* Repository's current hash algorithm, as serialized on disk. */
const struct git_hash_algo *hash_algo;
/* A unique-id for tracing purposes. */
int trace2_repo_id;
/* True if commit-graph has been disabled within this process. */
int commit_graph_disabled;
/* Configurations related to promisor remotes. */
char *repository_format_partial_clone;
struct promisor_remote_config *promisor_remote_config;
/* Configurations */
/* Indicate if a repository has a different 'commondir' from 'gitdir' */
unsigned different_commondir:1;
};
extern struct repository *the_repository;
/*
* Define a custom repository layout. Any field can be NULL, which
* will default back to the path according to the default layout.
*/
struct set_gitdir_args {
const char *commondir;
const char *object_dir;
const char *graft_file;
const char *index_file;
const char *alternate_db;
int disable_ref_updates;
};
void repo_set_gitdir(struct repository *repo, const char *root,
const struct set_gitdir_args *extra_args);
void repo_set_worktree(struct repository *repo, const char *path);
void repo_set_hash_algo(struct repository *repo, int algo);
void initialize_the_repository(void);
RESULT_MUST_BE_USED
int repo_init(struct repository *r, const char *gitdir, const char *worktree);
/*
* Initialize the repository 'subrepo' as the submodule at the given path. If
* the submodule's gitdir cannot be found at <path>/.git, this function calls
* submodule_from_path() to try to find it. treeish_name is only used if
* submodule_from_path() needs to be called; see its documentation for more
* information.
* Return 0 upon success and a non-zero value upon failure.
*/
struct object_id;
RESULT_MUST_BE_USED
int repo_submodule_init(struct repository *subrepo,
struct repository *superproject,
const char *path,
const struct object_id *treeish_name);
void repo_clear(struct repository *repo);
/*
* Populates the repository's index from its index_file, an index struct will
* be allocated if needed.
*
* Return the number of index entries in the populated index or a value less
* than zero if an error occurred. If the repository's index has already been
* populated then the number of entries will simply be returned.
*/
int repo_read_index(struct repository *repo);
int repo_hold_locked_index(struct repository *repo,
struct lock_file *lf,
int flags);
int repo_read_index_preload(struct repository *,
const struct pathspec *pathspec,
unsigned refresh_flags);
int repo_read_index_unmerged(struct repository *);
/*
* Opportunistically update the index but do not complain if we can't.
* The lockfile is always committed or rolled back.
*/
void repo_update_index_if_able(struct repository *, struct lock_file *);
void prepare_repo_settings(struct repository *r);
/*
* Return 1 if upgrade repository format to target_version succeeded,
* 0 if no upgrade is necessary, and -1 when upgrade is not possible.
*/
int upgrade_repository_format(int target_version);
#endif /* REPOSITORY_H */