mirror of
https://github.com/git/git.git
synced 2024-03-28 16:49:58 +01:00
Merge branch 'ds/sparse-index' into next
Both in-core and on-disk index has been updated to optionally omit individual entries and replace them with the tree object that corresponds to the directory that contains them when the "cone" mode of sparse checkout is in use. * ds/sparse-index: (21 commits) p2000: add sparse-index repos sparse-index: loose integration with cache_tree_verify() cache-tree: integrate with sparse directory entries sparse-checkout: disable sparse-index sparse-checkout: toggle sparse index from builtin sparse-index: add index.sparse config option sparse-index: check index conversion happens unpack-trees: allow sparse directories submodule: sparse-index should not collapse links sparse-index: convert from full to sparse sparse-index: add 'sdir' index extension sparse-checkout: hold pattern list in index unpack-trees: ensure full index test-tool: don't force full index test-read-cache: print cache entries with --table t1092: compare sparse-checkout to sparse-index sparse-index: implement ensure_full_index() sparse-index: add guard to ensure full index t1092: clean up script quoting t/perf: add performance test for sparse operations ...
This commit is contained in:
commit
f1290a7929
|
@ -14,6 +14,11 @@ index.recordOffsetTable::
|
|||
Defaults to 'true' if index.threads has been explicitly enabled,
|
||||
'false' otherwise.
|
||||
|
||||
index.sparse::
|
||||
When enabled, write the index using sparse-directory entries. This
|
||||
has no effect unless `core.sparseCheckout` and
|
||||
`core.sparseCheckoutCone` are both enabled. Defaults to 'false'.
|
||||
|
||||
index.threads::
|
||||
Specifies the number of threads to spawn when loading the index.
|
||||
This is meant to reduce index load time on multiprocessor machines.
|
||||
|
|
|
@ -45,6 +45,20 @@ To avoid interfering with other worktrees, it first enables the
|
|||
When `--cone` is provided, the `core.sparseCheckoutCone` setting is
|
||||
also set, allowing for better performance with a limited set of
|
||||
patterns (see 'CONE PATTERN SET' below).
|
||||
+
|
||||
Use the `--[no-]sparse-index` option to toggle the use of the sparse
|
||||
index format. This reduces the size of the index to be more closely
|
||||
aligned with your sparse-checkout definition. This can have significant
|
||||
performance advantages for commands such as `git status` or `git add`.
|
||||
This feature is still experimental. Some commands might be slower with
|
||||
a sparse index until they are properly integrated with the feature.
|
||||
+
|
||||
**WARNING:** Using a sparse index requires modifying the index in a way
|
||||
that is not completely understood by external tools. If you have trouble
|
||||
with this compatibility, then run `git sparse-checkout init --no-sparse-index`
|
||||
to rewrite your index to not be sparse. Older versions of Git will not
|
||||
understand the sparse directory entries index extension and may fail to
|
||||
interact with your repository until it is disabled.
|
||||
|
||||
'set'::
|
||||
Write a set of patterns to the sparse-checkout file, as given as
|
||||
|
|
|
@ -44,6 +44,13 @@ Git index format
|
|||
localization, no special casing of directory separator '/'). Entries
|
||||
with the same name are sorted by their stage field.
|
||||
|
||||
An index entry typically represents a file. However, if sparse-checkout
|
||||
is enabled in cone mode (`core.sparseCheckoutCone` is enabled) and the
|
||||
`extensions.sparseIndex` extension is enabled, then the index may
|
||||
contain entries for directories outside of the sparse-checkout definition.
|
||||
These entries have mode `040000`, include the `SKIP_WORKTREE` bit, and
|
||||
the path ends in a directory separator.
|
||||
|
||||
32-bit ctime seconds, the last time a file's metadata changed
|
||||
this is stat(2) data
|
||||
|
||||
|
@ -385,3 +392,15 @@ The remaining data of each directory block is grouped by type:
|
|||
in this block of entries.
|
||||
|
||||
- 32-bit count of cache entries in this block
|
||||
|
||||
== Sparse Directory Entries
|
||||
|
||||
When using sparse-checkout in cone mode, some entire directories within
|
||||
the index can be summarized by pointing to a tree object instead of the
|
||||
entire expanded list of paths within that tree. An index containing such
|
||||
entries is a "sparse index". Index format versions 4 and less were not
|
||||
implemented with such entries in mind. Thus, for these versions, an
|
||||
index containing sparse directory entries will include this extension
|
||||
with signature { 's', 'd', 'i', 'r' }. Like the split-index extension,
|
||||
tools should avoid interacting with a sparse index unless they understand
|
||||
this extension.
|
||||
|
|
|
@ -0,0 +1,175 @@
|
|||
Git Sparse-Index Design Document
|
||||
================================
|
||||
|
||||
The sparse-checkout feature allows users to focus a working directory on
|
||||
a subset of the files at HEAD. The cone mode patterns, enabled by
|
||||
`core.sparseCheckoutCone`, allow for very fast pattern matching to
|
||||
discover which files at HEAD belong in the sparse-checkout cone.
|
||||
|
||||
Three important scale dimensions for a Git working directory are:
|
||||
|
||||
* `HEAD`: How many files are present at `HEAD`?
|
||||
|
||||
* Populated: How many files are within the sparse-checkout cone.
|
||||
|
||||
* Modified: How many files has the user modified in the working directory?
|
||||
|
||||
We will use big-O notation -- O(X) -- to denote how expensive certain
|
||||
operations are in terms of these dimensions.
|
||||
|
||||
These dimensions are ordered by their magnitude: users (typically) modify
|
||||
fewer files than are populated, and we can only populate files at `HEAD`.
|
||||
|
||||
Problems occur if there is an extreme imbalance in these dimensions. For
|
||||
example, if `HEAD` contains millions of paths but the populated set has
|
||||
only tens of thousands, then commands like `git status` and `git add` can
|
||||
be dominated by operations that require O(`HEAD`) operations instead of
|
||||
O(Populated). Primarily, the cost is in parsing and rewriting the index,
|
||||
which is filled primarily with files at `HEAD` that are marked with the
|
||||
`SKIP_WORKTREE` bit.
|
||||
|
||||
The sparse-index intends to take these commands that read and modify the
|
||||
index from O(`HEAD`) to O(Populated). To do this, we need to modify the
|
||||
index format in a significant way: add "sparse directory" entries.
|
||||
|
||||
With cone mode patterns, it is possible to detect when an entire
|
||||
directory will have its contents outside of the sparse-checkout definition.
|
||||
Instead of listing all of the files it contains as individual entries, a
|
||||
sparse-index contains an entry with the directory name, referencing the
|
||||
object ID of the tree at `HEAD` and marked with the `SKIP_WORKTREE` bit.
|
||||
If we need to discover the details for paths within that directory, we
|
||||
can parse trees to find that list.
|
||||
|
||||
At time of writing, sparse-directory entries violate expectations about the
|
||||
index format and its in-memory data structure. There are many consumers in
|
||||
the codebase that expect to iterate through all of the index entries and
|
||||
see only files. In fact, these loops expect to see a reference to every
|
||||
staged file. One way to handle this is to parse trees to replace a
|
||||
sparse-directory entry with all of the files within that tree as the index
|
||||
is loaded. However, parsing trees is slower than parsing the index format,
|
||||
so that is a slower operation than if we left the index alone. The plan is
|
||||
to make all of these integrations "sparse aware" so this expansion through
|
||||
tree parsing is unnecessary and they use fewer resources than when using a
|
||||
full index.
|
||||
|
||||
The implementation plan below follows four phases to slowly integrate with
|
||||
the sparse-index. The intention is to incrementally update Git commands to
|
||||
interact safely with the sparse-index without significant slowdowns. This
|
||||
may not always be possible, but the hope is that the primary commands that
|
||||
users need in their daily work are dramatically improved.
|
||||
|
||||
Phase I: Format and initial speedups
|
||||
------------------------------------
|
||||
|
||||
During this phase, Git learns to enable the sparse-index and safely parse
|
||||
one. Protections are put in place so that every consumer of the in-memory
|
||||
data structure can operate with its current assumption of every file at
|
||||
`HEAD`.
|
||||
|
||||
At first, every index parse will call a helper method,
|
||||
`ensure_full_index()`, which scans the index for sparse-directory entries
|
||||
(pointing to trees) and replaces them with the full list of paths (with
|
||||
blob contents) by parsing tree objects. This will be slower in all cases.
|
||||
The only noticeable change in behavior will be that the serialized index
|
||||
file contains sparse-directory entries.
|
||||
|
||||
To start, we use a new required index extension, `sdir`, to allow
|
||||
inserting sparse-directory entries into indexes with file format
|
||||
versions 2, 3, and 4. This prevents Git versions that do not understand
|
||||
the sparse-index from operating on one, while allowing tools that do not
|
||||
understand the sparse-index to operate on repositories as long as they do
|
||||
not interact with the index. A new format, index v5, will be introduced
|
||||
that includes sparse-directory entries by default. It might also
|
||||
introduce other features that have been considered for improving the
|
||||
index, as well.
|
||||
|
||||
Next, consumers of the index will be guarded against operating on a
|
||||
sparse-index by inserting calls to `ensure_full_index()` or
|
||||
`expand_index_to_path()`. After these guards are in place, we can begin
|
||||
leaving sparse-directory entries in the in-memory index structure.
|
||||
|
||||
Even after inserting these guards, we will keep expanding sparse-indexes
|
||||
for most Git commands using the `command_requires_full_index` repository
|
||||
setting. This setting will be on by default and disabled one builtin at a
|
||||
time until we have sufficient confidence that all of the index operations
|
||||
are properly guarded.
|
||||
|
||||
To complete this phase, the commands `git status` and `git add` will be
|
||||
integrated with the sparse-index so that they operate with O(Populated)
|
||||
performance. They will be carefully tested for operations within and
|
||||
outside the sparse-checkout definition.
|
||||
|
||||
Phase II: Careful integrations
|
||||
------------------------------
|
||||
|
||||
This phase focuses on ensuring that all index extensions and APIs work
|
||||
well with a sparse-index. This requires significant increases to our test
|
||||
coverage, especially for operations that interact with the working
|
||||
directory outside of the sparse-checkout definition. Some of these
|
||||
behaviors may not be the desirable ones, such as some tests already
|
||||
marked for failure in `t1092-sparse-checkout-compatibility.sh`.
|
||||
|
||||
The index extensions that may require special integrations are:
|
||||
|
||||
* FS Monitor
|
||||
* Untracked cache
|
||||
|
||||
While integrating with these features, we should look for patterns that
|
||||
might lead to better APIs for interacting with the index. Coalescing
|
||||
common usage patterns into an API call can reduce the number of places
|
||||
where sparse-directories need to be handled carefully.
|
||||
|
||||
Phase III: Important command speedups
|
||||
-------------------------------------
|
||||
|
||||
At this point, the patterns for testing and implementing sparse-directory
|
||||
logic should be relatively stable. This phase focuses on updating some of
|
||||
the most common builtins that use the index to operate as O(Populated).
|
||||
Here is a potential list of commands that could be valuable to integrate
|
||||
at this point:
|
||||
|
||||
* `git commit`
|
||||
* `git checkout`
|
||||
* `git merge`
|
||||
* `git rebase`
|
||||
|
||||
Hopefully, commands such as `git merge` and `git rebase` can benefit
|
||||
instead from merge algorithms that do not use the index as a data
|
||||
structure, such as the merge-ORT strategy. As these topics mature, we
|
||||
may enable the ORT strategy by default for repositories using the
|
||||
sparse-index feature.
|
||||
|
||||
Along with `git status` and `git add`, these commands cover the majority
|
||||
of users' interactions with the working directory. In addition, we can
|
||||
integrate with these commands:
|
||||
|
||||
* `git grep`
|
||||
* `git rm`
|
||||
|
||||
These have been proposed as some whose behavior could change when in a
|
||||
repo with a sparse-checkout definition. It would be good to include this
|
||||
behavior automatically when using a sparse-index. Some clarity is needed
|
||||
to make the behavior switch clear to the user.
|
||||
|
||||
This phase is the first where parallel work might be possible without too
|
||||
much conflicts between topics.
|
||||
|
||||
Phase IV: The long tail
|
||||
-----------------------
|
||||
|
||||
This last phase is less a "phase" and more "the new normal" after all of
|
||||
the previous work.
|
||||
|
||||
To start, the `command_requires_full_index` option could be removed in
|
||||
favor of expanding only when hitting an API guard.
|
||||
|
||||
There are many Git commands that could use special attention to operate as
|
||||
O(Populated), while some might be so rare that it is acceptable to leave
|
||||
them with additional overhead when a sparse-index is present.
|
||||
|
||||
Here are some commands that might be useful to update:
|
||||
|
||||
* `git sparse-checkout set`
|
||||
* `git am`
|
||||
* `git clean`
|
||||
* `git stash`
|
1
Makefile
1
Makefile
|
@ -994,6 +994,7 @@ LIB_OBJS += setup.o
|
|||
LIB_OBJS += shallow.o
|
||||
LIB_OBJS += sideband.o
|
||||
LIB_OBJS += sigchain.o
|
||||
LIB_OBJS += sparse-index.o
|
||||
LIB_OBJS += split-index.o
|
||||
LIB_OBJS += stable-qsort.o
|
||||
LIB_OBJS += strbuf.o
|
||||
|
|
|
@ -14,6 +14,7 @@
|
|||
#include "unpack-trees.h"
|
||||
#include "wt-status.h"
|
||||
#include "quote.h"
|
||||
#include "sparse-index.h"
|
||||
|
||||
static const char *empty_base = "";
|
||||
|
||||
|
@ -110,6 +111,8 @@ static int update_working_directory(struct pattern_list *pl)
|
|||
if (is_index_unborn(r->index))
|
||||
return UPDATE_SPARSITY_SUCCESS;
|
||||
|
||||
r->index->sparse_checkout_patterns = pl;
|
||||
|
||||
memset(&o, 0, sizeof(o));
|
||||
o.verbose_update = isatty(2);
|
||||
o.update = 1;
|
||||
|
@ -138,6 +141,7 @@ static int update_working_directory(struct pattern_list *pl)
|
|||
else
|
||||
rollback_lock_file(&lock_file);
|
||||
|
||||
r->index->sparse_checkout_patterns = NULL;
|
||||
return result;
|
||||
}
|
||||
|
||||
|
@ -276,16 +280,20 @@ static int set_config(enum sparse_checkout_mode mode)
|
|||
"core.sparseCheckoutCone",
|
||||
mode == MODE_CONE_PATTERNS ? "true" : NULL);
|
||||
|
||||
if (mode == MODE_NO_PATTERNS)
|
||||
set_sparse_index_config(the_repository, 0);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static char const * const builtin_sparse_checkout_init_usage[] = {
|
||||
N_("git sparse-checkout init [--cone]"),
|
||||
N_("git sparse-checkout init [--cone] [--[no-]sparse-index]"),
|
||||
NULL
|
||||
};
|
||||
|
||||
static struct sparse_checkout_init_opts {
|
||||
int cone_mode;
|
||||
int sparse_index;
|
||||
} init_opts;
|
||||
|
||||
static int sparse_checkout_init(int argc, const char **argv)
|
||||
|
@ -300,11 +308,15 @@ static int sparse_checkout_init(int argc, const char **argv)
|
|||
static struct option builtin_sparse_checkout_init_options[] = {
|
||||
OPT_BOOL(0, "cone", &init_opts.cone_mode,
|
||||
N_("initialize the sparse-checkout in cone mode")),
|
||||
OPT_BOOL(0, "sparse-index", &init_opts.sparse_index,
|
||||
N_("toggle the use of a sparse index")),
|
||||
OPT_END(),
|
||||
};
|
||||
|
||||
repo_read_index(the_repository);
|
||||
|
||||
init_opts.sparse_index = -1;
|
||||
|
||||
argc = parse_options(argc, argv, NULL,
|
||||
builtin_sparse_checkout_init_options,
|
||||
builtin_sparse_checkout_init_usage, 0);
|
||||
|
@ -323,10 +335,20 @@ static int sparse_checkout_init(int argc, const char **argv)
|
|||
sparse_filename = get_sparse_checkout_filename();
|
||||
res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL, 0);
|
||||
|
||||
if (init_opts.sparse_index >= 0) {
|
||||
if (set_sparse_index_config(the_repository, init_opts.sparse_index) < 0)
|
||||
die(_("failed to modify sparse-index config"));
|
||||
|
||||
/* force an index rewrite */
|
||||
repo_read_index(the_repository);
|
||||
the_repository->index->updated_workdir = 1;
|
||||
}
|
||||
|
||||
core_apply_sparse_checkout = 1;
|
||||
|
||||
/* If we already have a sparse-checkout file, use it. */
|
||||
if (res >= 0) {
|
||||
free(sparse_filename);
|
||||
core_apply_sparse_checkout = 1;
|
||||
return update_working_directory(NULL);
|
||||
}
|
||||
|
||||
|
@ -348,6 +370,7 @@ static int sparse_checkout_init(int argc, const char **argv)
|
|||
add_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0);
|
||||
strbuf_addstr(&pattern, "!/*/");
|
||||
add_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0);
|
||||
pl.use_cone_patterns = init_opts.cone_mode;
|
||||
|
||||
return write_patterns_and_update(&pl);
|
||||
}
|
||||
|
@ -517,19 +540,18 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
|
|||
{
|
||||
int result;
|
||||
int changed_config = 0;
|
||||
struct pattern_list pl;
|
||||
memset(&pl, 0, sizeof(pl));
|
||||
struct pattern_list *pl = xcalloc(1, sizeof(*pl));
|
||||
|
||||
switch (m) {
|
||||
case ADD:
|
||||
if (core_sparse_checkout_cone)
|
||||
add_patterns_cone_mode(argc, argv, &pl);
|
||||
add_patterns_cone_mode(argc, argv, pl);
|
||||
else
|
||||
add_patterns_literal(argc, argv, &pl);
|
||||
add_patterns_literal(argc, argv, pl);
|
||||
break;
|
||||
|
||||
case REPLACE:
|
||||
add_patterns_from_input(&pl, argc, argv);
|
||||
add_patterns_from_input(pl, argc, argv);
|
||||
break;
|
||||
}
|
||||
|
||||
|
@ -539,12 +561,13 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
|
|||
changed_config = 1;
|
||||
}
|
||||
|
||||
result = write_patterns_and_update(&pl);
|
||||
result = write_patterns_and_update(pl);
|
||||
|
||||
if (result && changed_config)
|
||||
set_config(MODE_NO_PATTERNS);
|
||||
|
||||
clear_pattern_list(&pl);
|
||||
clear_pattern_list(pl);
|
||||
free(pl);
|
||||
return result;
|
||||
}
|
||||
|
||||
|
@ -614,6 +637,9 @@ static int sparse_checkout_disable(int argc, const char **argv)
|
|||
strbuf_addstr(&match_all, "/*");
|
||||
add_pattern(strbuf_detach(&match_all, NULL), empty_base, 0, &pl, 0);
|
||||
|
||||
prepare_repo_settings(the_repository);
|
||||
the_repository->settings.sparse_index = 0;
|
||||
|
||||
if (update_working_directory(&pl))
|
||||
die(_("error while refreshing working directory"));
|
||||
|
||||
|
|
40
cache-tree.c
40
cache-tree.c
|
@ -6,6 +6,7 @@
|
|||
#include "object-store.h"
|
||||
#include "replace-object.h"
|
||||
#include "promisor-remote.h"
|
||||
#include "sparse-index.h"
|
||||
|
||||
#ifndef DEBUG_CACHE_TREE
|
||||
#define DEBUG_CACHE_TREE 0
|
||||
|
@ -255,6 +256,24 @@ static int update_one(struct cache_tree *it,
|
|||
|
||||
*skip_count = 0;
|
||||
|
||||
/*
|
||||
* If the first entry of this region is a sparse directory
|
||||
* entry corresponding exactly to 'base', then this cache_tree
|
||||
* struct is a "leaf" in the data structure, pointing to the
|
||||
* tree OID specified in the entry.
|
||||
*/
|
||||
if (entries > 0) {
|
||||
const struct cache_entry *ce = cache[0];
|
||||
|
||||
if (S_ISSPARSEDIR(ce->ce_mode) &&
|
||||
ce->ce_namelen == baselen &&
|
||||
!strncmp(ce->name, base, baselen)) {
|
||||
it->entry_count = 1;
|
||||
oidcpy(&it->oid, &ce->oid);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (0 <= it->entry_count && has_object_file(&it->oid))
|
||||
return it->entry_count;
|
||||
|
||||
|
@ -442,6 +461,8 @@ int cache_tree_update(struct index_state *istate, int flags)
|
|||
if (i)
|
||||
return i;
|
||||
|
||||
ensure_full_index(istate);
|
||||
|
||||
if (!istate->cache_tree)
|
||||
istate->cache_tree = cache_tree();
|
||||
|
||||
|
@ -787,6 +808,19 @@ int cache_tree_matches_traversal(struct cache_tree *root,
|
|||
return 0;
|
||||
}
|
||||
|
||||
static void verify_one_sparse(struct repository *r,
|
||||
struct index_state *istate,
|
||||
struct cache_tree *it,
|
||||
struct strbuf *path,
|
||||
int pos)
|
||||
{
|
||||
struct cache_entry *ce = istate->cache[pos];
|
||||
|
||||
if (!S_ISSPARSEDIR(ce->ce_mode))
|
||||
BUG("directory '%s' is present in index, but not sparse",
|
||||
path->buf);
|
||||
}
|
||||
|
||||
static void verify_one(struct repository *r,
|
||||
struct index_state *istate,
|
||||
struct cache_tree *it,
|
||||
|
@ -809,6 +843,12 @@ static void verify_one(struct repository *r,
|
|||
|
||||
if (path->len) {
|
||||
pos = index_name_pos(istate, path->buf, path->len);
|
||||
|
||||
if (pos >= 0) {
|
||||
verify_one_sparse(r, istate, it, path, pos);
|
||||
return;
|
||||
}
|
||||
|
||||
pos = -pos - 1;
|
||||
} else {
|
||||
pos = 0;
|
||||
|
|
18
cache.h
18
cache.h
|
@ -204,6 +204,8 @@ struct cache_entry {
|
|||
#error "CE_EXTENDED_FLAGS out of range"
|
||||
#endif
|
||||
|
||||
#define S_ISSPARSEDIR(m) ((m) == S_IFDIR)
|
||||
|
||||
/* Forward structure decls */
|
||||
struct pathspec;
|
||||
struct child_process;
|
||||
|
@ -249,6 +251,8 @@ static inline unsigned int create_ce_mode(unsigned int mode)
|
|||
{
|
||||
if (S_ISLNK(mode))
|
||||
return S_IFLNK;
|
||||
if (S_ISSPARSEDIR(mode))
|
||||
return S_IFDIR;
|
||||
if (S_ISDIR(mode) || S_ISGITLINK(mode))
|
||||
return S_IFGITLINK;
|
||||
return S_IFREG | ce_permissions(mode);
|
||||
|
@ -305,6 +309,7 @@ static inline unsigned int canon_mode(unsigned int mode)
|
|||
struct split_index;
|
||||
struct untracked_cache;
|
||||
struct progress;
|
||||
struct pattern_list;
|
||||
|
||||
struct index_state {
|
||||
struct cache_entry **cache;
|
||||
|
@ -319,7 +324,14 @@ struct index_state {
|
|||
drop_cache_tree : 1,
|
||||
updated_workdir : 1,
|
||||
updated_skipworktree : 1,
|
||||
fsmonitor_has_run_once : 1;
|
||||
fsmonitor_has_run_once : 1,
|
||||
|
||||
/*
|
||||
* sparse_index == 1 when sparse-directory
|
||||
* entries exist. Requires sparse-checkout
|
||||
* in cone mode.
|
||||
*/
|
||||
sparse_index : 1;
|
||||
struct hashmap name_hash;
|
||||
struct hashmap dir_hash;
|
||||
struct object_id oid;
|
||||
|
@ -329,6 +341,7 @@ struct index_state {
|
|||
struct mem_pool *ce_mem_pool;
|
||||
struct progress *progress;
|
||||
struct repository *repo;
|
||||
struct pattern_list *sparse_checkout_patterns;
|
||||
};
|
||||
|
||||
/* Name hashing */
|
||||
|
@ -722,6 +735,8 @@ int read_index_from(struct index_state *, const char *path,
|
|||
const char *gitdir);
|
||||
int is_index_unborn(struct index_state *);
|
||||
|
||||
void ensure_full_index(struct index_state *istate);
|
||||
|
||||
/* For use with `write_locked_index()`. */
|
||||
#define COMMIT_LOCK (1 << 0)
|
||||
#define SKIP_IF_UNCHANGED (1 << 1)
|
||||
|
@ -1044,6 +1059,7 @@ struct repository_format {
|
|||
int worktree_config;
|
||||
int is_bare;
|
||||
int hash_algo;
|
||||
int sparse_index;
|
||||
char *work_tree;
|
||||
struct string_list unknown_extensions;
|
||||
struct string_list v1_only_extensions;
|
||||
|
|
44
read-cache.c
44
read-cache.c
|
@ -25,6 +25,7 @@
|
|||
#include "fsmonitor.h"
|
||||
#include "thread-utils.h"
|
||||
#include "progress.h"
|
||||
#include "sparse-index.h"
|
||||
|
||||
/* Mask for the name length in ce_flags in the on-disk index */
|
||||
|
||||
|
@ -47,6 +48,7 @@
|
|||
#define CACHE_EXT_FSMONITOR 0x46534D4E /* "FSMN" */
|
||||
#define CACHE_EXT_ENDOFINDEXENTRIES 0x454F4945 /* "EOIE" */
|
||||
#define CACHE_EXT_INDEXENTRYOFFSETTABLE 0x49454F54 /* "IEOT" */
|
||||
#define CACHE_EXT_SPARSE_DIRECTORIES 0x73646972 /* "sdir" */
|
||||
|
||||
/* changes that can be kept in $GIT_DIR/index (basically all extensions) */
|
||||
#define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \
|
||||
|
@ -101,6 +103,9 @@ static const char *alternate_index_output;
|
|||
|
||||
static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)
|
||||
{
|
||||
if (S_ISSPARSEDIR(ce->ce_mode))
|
||||
istate->sparse_index = 1;
|
||||
|
||||
istate->cache[nr] = ce;
|
||||
add_name_hash(istate, ce);
|
||||
}
|
||||
|
@ -999,8 +1004,14 @@ int verify_path(const char *path, unsigned mode)
|
|||
|
||||
c = *path++;
|
||||
if ((c == '.' && !verify_dotfile(path, mode)) ||
|
||||
is_dir_sep(c) || c == '\0')
|
||||
is_dir_sep(c))
|
||||
return 0;
|
||||
/*
|
||||
* allow terminating directory separators for
|
||||
* sparse directory entries.
|
||||
*/
|
||||
if (c == '\0')
|
||||
return S_ISDIR(mode);
|
||||
} else if (c == '\\' && protect_ntfs) {
|
||||
if (is_ntfs_dotgit(path))
|
||||
return 0;
|
||||
|
@ -1760,6 +1771,10 @@ static int read_index_extension(struct index_state *istate,
|
|||
case CACHE_EXT_INDEXENTRYOFFSETTABLE:
|
||||
/* already handled in do_read_index() */
|
||||
break;
|
||||
case CACHE_EXT_SPARSE_DIRECTORIES:
|
||||
/* no content, only an indicator */
|
||||
istate->sparse_index = 1;
|
||||
break;
|
||||
default:
|
||||
if (*ext < 'A' || 'Z' < *ext)
|
||||
return error(_("index uses %.4s extension, which we do not understand"),
|
||||
|
@ -2273,6 +2288,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)
|
|||
trace2_data_intmax("index", the_repository, "read/cache_nr",
|
||||
istate->cache_nr);
|
||||
|
||||
if (!istate->repo)
|
||||
istate->repo = the_repository;
|
||||
prepare_repo_settings(istate->repo);
|
||||
if (istate->repo->settings.command_requires_full_index)
|
||||
ensure_full_index(istate);
|
||||
|
||||
return istate->cache_nr;
|
||||
|
||||
unmap:
|
||||
|
@ -3012,6 +3033,10 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
|
|||
if (err)
|
||||
return -1;
|
||||
}
|
||||
if (istate->sparse_index) {
|
||||
if (write_index_ext_header(&c, &eoie_c, newfd, CACHE_EXT_SPARSE_DIRECTORIES, 0) < 0)
|
||||
return -1;
|
||||
}
|
||||
|
||||
/*
|
||||
* CACHE_EXT_ENDOFINDEXENTRIES must be written as the last entry before the SHA1
|
||||
|
@ -3071,6 +3096,14 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
|
|||
unsigned flags)
|
||||
{
|
||||
int ret;
|
||||
int was_full = !istate->sparse_index;
|
||||
|
||||
ret = convert_to_sparse(istate);
|
||||
|
||||
if (ret) {
|
||||
warning(_("failed to convert to a sparse-index"));
|
||||
return ret;
|
||||
}
|
||||
|
||||
/*
|
||||
* TODO trace2: replace "the_repository" with the actual repo instance
|
||||
|
@ -3082,6 +3115,9 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
|
|||
trace2_region_leave_printf("index", "do_write_index", the_repository,
|
||||
"%s", get_lock_file_path(lock));
|
||||
|
||||
if (was_full)
|
||||
ensure_full_index(istate);
|
||||
|
||||
if (ret)
|
||||
return ret;
|
||||
if (flags & COMMIT_LOCK)
|
||||
|
@ -3172,9 +3208,10 @@ static int write_shared_index(struct index_state *istate,
|
|||
struct tempfile **temp)
|
||||
{
|
||||
struct split_index *si = istate->split_index;
|
||||
int ret;
|
||||
int ret, was_full = !istate->sparse_index;
|
||||
|
||||
move_cache_to_base_index(istate);
|
||||
convert_to_sparse(istate);
|
||||
|
||||
trace2_region_enter_printf("index", "shared/do_write_index",
|
||||
the_repository, "%s", get_tempfile_path(*temp));
|
||||
|
@ -3182,6 +3219,9 @@ static int write_shared_index(struct index_state *istate,
|
|||
trace2_region_leave_printf("index", "shared/do_write_index",
|
||||
the_repository, "%s", get_tempfile_path(*temp));
|
||||
|
||||
if (was_full)
|
||||
ensure_full_index(istate);
|
||||
|
||||
if (ret)
|
||||
return ret;
|
||||
ret = adjust_shared_perm(get_tempfile_path(*temp));
|
||||
|
|
|
@ -77,4 +77,19 @@ void prepare_repo_settings(struct repository *r)
|
|||
UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_KEEP);
|
||||
|
||||
UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_DEFAULT);
|
||||
|
||||
/*
|
||||
* This setting guards all index reads to require a full index
|
||||
* over a sparse index. After suitable guards are placed in the
|
||||
* codebase around uses of the index, this setting will be
|
||||
* removed.
|
||||
*/
|
||||
r->settings.command_requires_full_index = 1;
|
||||
|
||||
/*
|
||||
* Initialize this as off.
|
||||
*/
|
||||
r->settings.sparse_index = 0;
|
||||
if (!repo_config_get_bool(r, "index.sparse", &value) && value)
|
||||
r->settings.sparse_index = 1;
|
||||
}
|
||||
|
|
11
repository.c
11
repository.c
|
@ -10,6 +10,7 @@
|
|||
#include "object.h"
|
||||
#include "lockfile.h"
|
||||
#include "submodule-config.h"
|
||||
#include "sparse-index.h"
|
||||
|
||||
/* The main repository */
|
||||
static struct repository the_repo;
|
||||
|
@ -261,6 +262,8 @@ void repo_clear(struct repository *repo)
|
|||
|
||||
int repo_read_index(struct repository *repo)
|
||||
{
|
||||
int res;
|
||||
|
||||
if (!repo->index)
|
||||
CALLOC_ARRAY(repo->index, 1);
|
||||
|
||||
|
@ -270,7 +273,13 @@ int repo_read_index(struct repository *repo)
|
|||
else if (repo->index->repo != repo)
|
||||
BUG("repo's index should point back at itself");
|
||||
|
||||
return read_index_from(repo->index, repo->index_file, repo->gitdir);
|
||||
res = read_index_from(repo->index, repo->index_file, repo->gitdir);
|
||||
|
||||
prepare_repo_settings(repo);
|
||||
if (repo->settings.command_requires_full_index)
|
||||
ensure_full_index(repo->index);
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
int repo_hold_locked_index(struct repository *repo,
|
||||
|
|
|
@ -41,6 +41,9 @@ struct repo_settings {
|
|||
enum fetch_negotiation_setting fetch_negotiation_algorithm;
|
||||
|
||||
int core_multi_pack_index;
|
||||
|
||||
unsigned command_requires_full_index:1,
|
||||
sparse_index:1;
|
||||
};
|
||||
|
||||
struct repository {
|
||||
|
|
|
@ -0,0 +1,285 @@
|
|||
#include "cache.h"
|
||||
#include "repository.h"
|
||||
#include "sparse-index.h"
|
||||
#include "tree.h"
|
||||
#include "pathspec.h"
|
||||
#include "trace2.h"
|
||||
#include "cache-tree.h"
|
||||
#include "config.h"
|
||||
#include "dir.h"
|
||||
#include "fsmonitor.h"
|
||||
|
||||
static struct cache_entry *construct_sparse_dir_entry(
|
||||
struct index_state *istate,
|
||||
const char *sparse_dir,
|
||||
struct cache_tree *tree)
|
||||
{
|
||||
struct cache_entry *de;
|
||||
|
||||
de = make_cache_entry(istate, S_IFDIR, &tree->oid, sparse_dir, 0, 0);
|
||||
|
||||
de->ce_flags |= CE_SKIP_WORKTREE;
|
||||
return de;
|
||||
}
|
||||
|
||||
/*
|
||||
* Returns the number of entries "inserted" into the index.
|
||||
*/
|
||||
static int convert_to_sparse_rec(struct index_state *istate,
|
||||
int num_converted,
|
||||
int start, int end,
|
||||
const char *ct_path, size_t ct_pathlen,
|
||||
struct cache_tree *ct)
|
||||
{
|
||||
int i, can_convert = 1;
|
||||
int start_converted = num_converted;
|
||||
enum pattern_match_result match;
|
||||
int dtype;
|
||||
struct strbuf child_path = STRBUF_INIT;
|
||||
struct pattern_list *pl = istate->sparse_checkout_patterns;
|
||||
|
||||
/*
|
||||
* Is the current path outside of the sparse cone?
|
||||
* Then check if the region can be replaced by a sparse
|
||||
* directory entry (everything is sparse and merged).
|
||||
*/
|
||||
match = path_matches_pattern_list(ct_path, ct_pathlen,
|
||||
NULL, &dtype, pl, istate);
|
||||
if (match != NOT_MATCHED)
|
||||
can_convert = 0;
|
||||
|
||||
for (i = start; can_convert && i < end; i++) {
|
||||
struct cache_entry *ce = istate->cache[i];
|
||||
|
||||
if (ce_stage(ce) ||
|
||||
S_ISGITLINK(ce->ce_mode) ||
|
||||
!(ce->ce_flags & CE_SKIP_WORKTREE))
|
||||
can_convert = 0;
|
||||
}
|
||||
|
||||
if (can_convert) {
|
||||
struct cache_entry *se;
|
||||
se = construct_sparse_dir_entry(istate, ct_path, ct);
|
||||
|
||||
istate->cache[num_converted++] = se;
|
||||
return 1;
|
||||
}
|
||||
|
||||
for (i = start; i < end; ) {
|
||||
int count, span, pos = -1;
|
||||
const char *base, *slash;
|
||||
struct cache_entry *ce = istate->cache[i];
|
||||
|
||||
/*
|
||||
* Detect if this is a normal entry outside of any subtree
|
||||
* entry.
|
||||
*/
|
||||
base = ce->name + ct_pathlen;
|
||||
slash = strchr(base, '/');
|
||||
|
||||
if (slash)
|
||||
pos = cache_tree_subtree_pos(ct, base, slash - base);
|
||||
|
||||
if (pos < 0) {
|
||||
istate->cache[num_converted++] = ce;
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
strbuf_setlen(&child_path, 0);
|
||||
strbuf_add(&child_path, ce->name, slash - ce->name + 1);
|
||||
|
||||
span = ct->down[pos]->cache_tree->entry_count;
|
||||
count = convert_to_sparse_rec(istate,
|
||||
num_converted, i, i + span,
|
||||
child_path.buf, child_path.len,
|
||||
ct->down[pos]->cache_tree);
|
||||
num_converted += count;
|
||||
i += span;
|
||||
}
|
||||
|
||||
strbuf_release(&child_path);
|
||||
return num_converted - start_converted;
|
||||
}
|
||||
|
||||
static int set_index_sparse_config(struct repository *repo, int enable)
|
||||
{
|
||||
int res;
|
||||
char *config_path = repo_git_path(repo, "config.worktree");
|
||||
res = git_config_set_in_file_gently(config_path,
|
||||
"index.sparse",
|
||||
enable ? "true" : NULL);
|
||||
free(config_path);
|
||||
|
||||
prepare_repo_settings(repo);
|
||||
repo->settings.sparse_index = 1;
|
||||
return res;
|
||||
}
|
||||
|
||||
int set_sparse_index_config(struct repository *repo, int enable)
|
||||
{
|
||||
int res = set_index_sparse_config(repo, enable);
|
||||
|
||||
prepare_repo_settings(repo);
|
||||
repo->settings.sparse_index = enable;
|
||||
return res;
|
||||
}
|
||||
|
||||
int convert_to_sparse(struct index_state *istate)
|
||||
{
|
||||
int test_env;
|
||||
if (istate->split_index || istate->sparse_index ||
|
||||
!core_apply_sparse_checkout || !core_sparse_checkout_cone)
|
||||
return 0;
|
||||
|
||||
if (!istate->repo)
|
||||
istate->repo = the_repository;
|
||||
|
||||
/*
|
||||
* The GIT_TEST_SPARSE_INDEX environment variable triggers the
|
||||
* index.sparse config variable to be on.
|
||||
*/
|
||||
test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
|
||||
if (test_env >= 0)
|
||||
set_sparse_index_config(istate->repo, test_env);
|
||||
|
||||
/*
|
||||
* Only convert to sparse if index.sparse is set.
|
||||
*/
|
||||
prepare_repo_settings(istate->repo);
|
||||
if (!istate->repo->settings.sparse_index)
|
||||
return 0;
|
||||
|
||||
if (!istate->sparse_checkout_patterns) {
|
||||
istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));
|
||||
if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!istate->sparse_checkout_patterns->use_cone_patterns) {
|
||||
warning(_("attempting to use sparse-index without cone mode"));
|
||||
return -1;
|
||||
}
|
||||
|
||||
if (cache_tree_update(istate, 0)) {
|
||||
warning(_("unable to update cache-tree, staying full"));
|
||||
return -1;
|
||||
}
|
||||
|
||||
remove_fsmonitor(istate);
|
||||
|
||||
trace2_region_enter("index", "convert_to_sparse", istate->repo);
|
||||
istate->cache_nr = convert_to_sparse_rec(istate,
|
||||
0, 0, istate->cache_nr,
|
||||
"", 0, istate->cache_tree);
|
||||
|
||||
/* Clear and recompute the cache-tree */
|
||||
cache_tree_free(&istate->cache_tree);
|
||||
cache_tree_update(istate, 0);
|
||||
|
||||
istate->sparse_index = 1;
|
||||
trace2_region_leave("index", "convert_to_sparse", istate->repo);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)
|
||||
{
|
||||
ALLOC_GROW(istate->cache, nr + 1, istate->cache_alloc);
|
||||
|
||||
istate->cache[nr] = ce;
|
||||
add_name_hash(istate, ce);
|
||||
}
|
||||
|
||||
static int add_path_to_index(const struct object_id *oid,
|
||||
struct strbuf *base, const char *path,
|
||||
unsigned int mode, void *context)
|
||||
{
|
||||
struct index_state *istate = (struct index_state *)context;
|
||||
struct cache_entry *ce;
|
||||
size_t len = base->len;
|
||||
|
||||
if (S_ISDIR(mode))
|
||||
return READ_TREE_RECURSIVE;
|
||||
|
||||
strbuf_addstr(base, path);
|
||||
|
||||
ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);
|
||||
ce->ce_flags |= CE_SKIP_WORKTREE;
|
||||
set_index_entry(istate, istate->cache_nr++, ce);
|
||||
|
||||
strbuf_setlen(base, len);
|
||||
return 0;
|
||||
}
|
||||
|
||||
void ensure_full_index(struct index_state *istate)
|
||||
{
|
||||
int i;
|
||||
struct index_state *full;
|
||||
struct strbuf base = STRBUF_INIT;
|
||||
|
||||
if (!istate || !istate->sparse_index)
|
||||
return;
|
||||
|
||||
if (!istate->repo)
|
||||
istate->repo = the_repository;
|
||||
|
||||
trace2_region_enter("index", "ensure_full_index", istate->repo);
|
||||
|
||||
/* initialize basics of new index */
|
||||
full = xcalloc(1, sizeof(struct index_state));
|
||||
memcpy(full, istate, sizeof(struct index_state));
|
||||
|
||||
/* then change the necessary things */
|
||||
full->sparse_index = 0;
|
||||
full->cache_alloc = (3 * istate->cache_alloc) / 2;
|
||||
full->cache_nr = 0;
|
||||
ALLOC_ARRAY(full->cache, full->cache_alloc);
|
||||
|
||||
for (i = 0; i < istate->cache_nr; i++) {
|
||||
struct cache_entry *ce = istate->cache[i];
|
||||
struct tree *tree;
|
||||
struct pathspec ps;
|
||||
|
||||
if (!S_ISSPARSEDIR(ce->ce_mode)) {
|
||||
set_index_entry(full, full->cache_nr++, ce);
|
||||
continue;
|
||||
}
|
||||
if (!(ce->ce_flags & CE_SKIP_WORKTREE))
|
||||
warning(_("index entry is a directory, but not sparse (%08x)"),
|
||||
ce->ce_flags);
|
||||
|
||||
/* recursively walk into cd->name */
|
||||
tree = lookup_tree(istate->repo, &ce->oid);
|
||||
|
||||
memset(&ps, 0, sizeof(ps));
|
||||
ps.recursive = 1;
|
||||
ps.has_wildcard = 1;
|
||||
ps.max_depth = -1;
|
||||
|
||||
strbuf_setlen(&base, 0);
|
||||
strbuf_add(&base, ce->name, strlen(ce->name));
|
||||
|
||||
read_tree_at(istate->repo, tree, &base, &ps,
|
||||
add_path_to_index, full);
|
||||
|
||||
/* free directory entries. full entries are re-used */
|
||||
discard_cache_entry(ce);
|
||||
}
|
||||
|
||||
/* Copy back into original index. */
|
||||
memcpy(&istate->name_hash, &full->name_hash, sizeof(full->name_hash));
|
||||
istate->sparse_index = 0;
|
||||
free(istate->cache);
|
||||
istate->cache = full->cache;
|
||||
istate->cache_nr = full->cache_nr;
|
||||
istate->cache_alloc = full->cache_alloc;
|
||||
|
||||
strbuf_release(&base);
|
||||
free(full);
|
||||
|
||||
/* Clear and recompute the cache-tree */
|
||||
cache_tree_free(&istate->cache_tree);
|
||||
cache_tree_update(istate, 0);
|
||||
|
||||
trace2_region_leave("index", "ensure_full_index", istate->repo);
|
||||
}
|
|
@ -0,0 +1,11 @@
|
|||
#ifndef SPARSE_INDEX_H__
|
||||
#define SPARSE_INDEX_H__
|
||||
|
||||
struct index_state;
|
||||
void ensure_full_index(struct index_state *istate);
|
||||
int convert_to_sparse(struct index_state *istate);
|
||||
|
||||
struct repository;
|
||||
int set_sparse_index_config(struct repository *repo, int enable);
|
||||
|
||||
#endif
|
3
t/README
3
t/README
|
@ -436,6 +436,9 @@ and "sha256".
|
|||
GIT_TEST_WRITE_REV_INDEX=<boolean>, when true enables the
|
||||
'pack.writeReverseIndex' setting.
|
||||
|
||||
GIT_TEST_SPARSE_INDEX=<boolean>, when true enables index writes to use the
|
||||
sparse-index format by default.
|
||||
|
||||
Naming Tests
|
||||
------------
|
||||
|
||||
|
|
|
@ -1,36 +1,82 @@
|
|||
#include "test-tool.h"
|
||||
#include "cache.h"
|
||||
#include "config.h"
|
||||
#include "blob.h"
|
||||
#include "commit.h"
|
||||
#include "tree.h"
|
||||
#include "sparse-index.h"
|
||||
|
||||
static void print_cache_entry(struct cache_entry *ce)
|
||||
{
|
||||
const char *type;
|
||||
printf("%06o ", ce->ce_mode & 0177777);
|
||||
|
||||
if (S_ISSPARSEDIR(ce->ce_mode))
|
||||
type = tree_type;
|
||||
else if (S_ISGITLINK(ce->ce_mode))
|
||||
type = commit_type;
|
||||
else
|
||||
type = blob_type;
|
||||
|
||||
printf("%s %s\t%s\n",
|
||||
type,
|
||||
oid_to_hex(&ce->oid),
|
||||
ce->name);
|
||||
}
|
||||
|
||||
static void print_cache(struct index_state *istate)
|
||||
{
|
||||
int i;
|
||||
for (i = 0; i < istate->cache_nr; i++)
|
||||
print_cache_entry(istate->cache[i]);
|
||||
}
|
||||
|
||||
int cmd__read_cache(int argc, const char **argv)
|
||||
{
|
||||
struct repository *r = the_repository;
|
||||
int i, cnt = 1;
|
||||
const char *name = NULL;
|
||||
int table = 0, expand = 0;
|
||||
|
||||
if (argc > 1 && skip_prefix(argv[1], "--print-and-refresh=", &name)) {
|
||||
argc--;
|
||||
argv++;
|
||||
initialize_the_repository();
|
||||
prepare_repo_settings(r);
|
||||
r->settings.command_requires_full_index = 0;
|
||||
|
||||
for (++argv, --argc; *argv && starts_with(*argv, "--"); ++argv, --argc) {
|
||||
if (skip_prefix(*argv, "--print-and-refresh=", &name))
|
||||
continue;
|
||||
if (!strcmp(*argv, "--table"))
|
||||
table = 1;
|
||||
else if (!strcmp(*argv, "--expand"))
|
||||
expand = 1;
|
||||
}
|
||||
|
||||
if (argc == 2)
|
||||
cnt = strtol(argv[1], NULL, 0);
|
||||
if (argc == 1)
|
||||
cnt = strtol(argv[0], NULL, 0);
|
||||
setup_git_directory();
|
||||
git_config(git_default_config, NULL);
|
||||
|
||||
for (i = 0; i < cnt; i++) {
|
||||
read_cache();
|
||||
repo_read_index(r);
|
||||
|
||||
if (expand)
|
||||
ensure_full_index(r->index);
|
||||
|
||||
if (name) {
|
||||
int pos;
|
||||
|
||||
refresh_index(&the_index, REFRESH_QUIET,
|
||||
refresh_index(r->index, REFRESH_QUIET,
|
||||
NULL, NULL, NULL);
|
||||
pos = index_name_pos(&the_index, name, strlen(name));
|
||||
pos = index_name_pos(r->index, name, strlen(name));
|
||||
if (pos < 0)
|
||||
die("%s not in index", name);
|
||||
printf("%s is%s up to date\n", name,
|
||||
ce_uptodate(the_index.cache[pos]) ? "" : " not");
|
||||
ce_uptodate(r->index->cache[pos]) ? "" : " not");
|
||||
write_file(name, "%d\n", i);
|
||||
}
|
||||
discard_cache();
|
||||
if (table)
|
||||
print_cache(r->index);
|
||||
discard_index(r->index);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
|
|
@ -0,0 +1,101 @@
|
|||
#!/bin/sh
|
||||
|
||||
test_description="test performance of Git operations using the index"
|
||||
|
||||
. ./perf-lib.sh
|
||||
|
||||
test_perf_default_repo
|
||||
|
||||
SPARSE_CONE=f2/f4/f1
|
||||
|
||||
test_expect_success 'setup repo and indexes' '
|
||||
git reset --hard HEAD &&
|
||||
|
||||
# Remove submodules from the example repo, because our
|
||||
# duplication of the entire repo creates an unlikely data shape.
|
||||
if git config --file .gitmodules --get-regexp "submodule.*.path" >modules
|
||||
then
|
||||
git rm $(awk "{print \$2}" modules) &&
|
||||
git commit -m "remove submodules" || return 1
|
||||
fi &&
|
||||
|
||||
echo bogus >a &&
|
||||
cp a b &&
|
||||
git add a b &&
|
||||
git commit -m "level 0" &&
|
||||
BLOB=$(git rev-parse HEAD:a) &&
|
||||
OLD_COMMIT=$(git rev-parse HEAD) &&
|
||||
OLD_TREE=$(git rev-parse HEAD^{tree}) &&
|
||||
|
||||
for i in $(test_seq 1 4)
|
||||
do
|
||||
cat >in <<-EOF &&
|
||||
100755 blob $BLOB a
|
||||
040000 tree $OLD_TREE f1
|
||||
040000 tree $OLD_TREE f2
|
||||
040000 tree $OLD_TREE f3
|
||||
040000 tree $OLD_TREE f4
|
||||
EOF
|
||||
NEW_TREE=$(git mktree <in) &&
|
||||
NEW_COMMIT=$(git commit-tree $NEW_TREE -p $OLD_COMMIT -m "level $i") &&
|
||||
OLD_TREE=$NEW_TREE &&
|
||||
OLD_COMMIT=$NEW_COMMIT || return 1
|
||||
done &&
|
||||
|
||||
git sparse-checkout init --cone &&
|
||||
git branch -f wide $OLD_COMMIT &&
|
||||
git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . full-index-v3 &&
|
||||
(
|
||||
cd full-index-v3 &&
|
||||
git sparse-checkout init --cone &&
|
||||
git sparse-checkout set $SPARSE_CONE &&
|
||||
git config index.version 3 &&
|
||||
git update-index --index-version=3
|
||||
) &&
|
||||
git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . full-index-v4 &&
|
||||
(
|
||||
cd full-index-v4 &&
|
||||
git sparse-checkout init --cone &&
|
||||
git sparse-checkout set $SPARSE_CONE &&
|
||||
git config index.version 4 &&
|
||||
git update-index --index-version=4
|
||||
) &&
|
||||
git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-index-v3 &&
|
||||
(
|
||||
cd sparse-index-v3 &&
|
||||
git sparse-checkout init --cone --sparse-index &&
|
||||
git sparse-checkout set $SPARSE_CONE &&
|
||||
git config index.version 3 &&
|
||||
git update-index --index-version=3
|
||||
) &&
|
||||
git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-index-v4 &&
|
||||
(
|
||||
cd sparse-index-v4 &&
|
||||
git sparse-checkout init --cone --sparse-index &&
|
||||
git sparse-checkout set $SPARSE_CONE &&
|
||||
git config index.version 4 &&
|
||||
git update-index --index-version=4
|
||||
)
|
||||
'
|
||||
|
||||
test_perf_on_all () {
|
||||
command="$@"
|
||||
for repo in full-index-v3 full-index-v4 \
|
||||
sparse-index-v3 sparse-index-v4
|
||||
do
|
||||
test_perf "$command ($repo)" "
|
||||
(
|
||||
cd $repo &&
|
||||
echo >>$SPARSE_CONE/a &&
|
||||
$command
|
||||
)
|
||||
"
|
||||
done
|
||||
}
|
||||
|
||||
test_perf_on_all git status
|
||||
test_perf_on_all git add -A
|
||||
test_perf_on_all git add .
|
||||
test_perf_on_all git commit -a -m A
|
||||
|
||||
test_done
|
|
@ -205,6 +205,19 @@ test_expect_success 'sparse-checkout disable' '
|
|||
check_files repo a deep folder1 folder2
|
||||
'
|
||||
|
||||
test_expect_success 'sparse-index enabled and disabled' '
|
||||
git -C repo sparse-checkout init --cone --sparse-index &&
|
||||
test_cmp_config -C repo true index.sparse &&
|
||||
test-tool -C repo read-cache --table >cache &&
|
||||
grep " tree " cache &&
|
||||
|
||||
git -C repo sparse-checkout disable &&
|
||||
test-tool -C repo read-cache --table >cache &&
|
||||
! grep " tree " cache &&
|
||||
git -C repo config --list >config &&
|
||||
! grep index.sparse config
|
||||
'
|
||||
|
||||
test_expect_success 'cone mode: init and set' '
|
||||
git -C repo sparse-checkout init --cone &&
|
||||
git -C repo config --list >config &&
|
||||
|
|
|
@ -2,11 +2,15 @@
|
|||
|
||||
test_description='compare full workdir to sparse workdir'
|
||||
|
||||
GIT_TEST_SPLIT_INDEX=0
|
||||
GIT_TEST_SPARSE_INDEX=
|
||||
|
||||
. ./test-lib.sh
|
||||
|
||||
test_expect_success 'setup' '
|
||||
git init initial-repo &&
|
||||
(
|
||||
GIT_TEST_SPARSE_INDEX=0 &&
|
||||
cd initial-repo &&
|
||||
echo a >a &&
|
||||
echo "after deep" >e &&
|
||||
|
@ -87,39 +91,102 @@ init_repos () {
|
|||
|
||||
cp -r initial-repo sparse-checkout &&
|
||||
git -C sparse-checkout reset --hard &&
|
||||
git -C sparse-checkout sparse-checkout init --cone &&
|
||||
|
||||
cp -r initial-repo sparse-index &&
|
||||
git -C sparse-index reset --hard &&
|
||||
|
||||
# initialize sparse-checkout definitions
|
||||
git -C sparse-checkout sparse-checkout set deep
|
||||
git -C sparse-checkout sparse-checkout init --cone &&
|
||||
git -C sparse-checkout sparse-checkout set deep &&
|
||||
git -C sparse-index sparse-checkout init --cone --sparse-index &&
|
||||
test_cmp_config -C sparse-index true index.sparse &&
|
||||
git -C sparse-index sparse-checkout set deep
|
||||
}
|
||||
|
||||
run_on_sparse () {
|
||||
(
|
||||
cd sparse-checkout &&
|
||||
$* >../sparse-checkout-out 2>../sparse-checkout-err
|
||||
"$@" >../sparse-checkout-out 2>../sparse-checkout-err
|
||||
) &&
|
||||
(
|
||||
cd sparse-index &&
|
||||
"$@" >../sparse-index-out 2>../sparse-index-err
|
||||
)
|
||||
}
|
||||
|
||||
run_on_all () {
|
||||
(
|
||||
cd full-checkout &&
|
||||
$* >../full-checkout-out 2>../full-checkout-err
|
||||
"$@" >../full-checkout-out 2>../full-checkout-err
|
||||
) &&
|
||||
run_on_sparse $*
|
||||
run_on_sparse "$@"
|
||||
}
|
||||
|
||||
test_all_match () {
|
||||
run_on_all $* &&
|
||||
run_on_all "$@" &&
|
||||
test_cmp full-checkout-out sparse-checkout-out &&
|
||||
test_cmp full-checkout-err sparse-checkout-err
|
||||
test_cmp full-checkout-out sparse-index-out &&
|
||||
test_cmp full-checkout-err sparse-checkout-err &&
|
||||
test_cmp full-checkout-err sparse-index-err
|
||||
}
|
||||
|
||||
test_sparse_match () {
|
||||
run_on_sparse "$@" &&
|
||||
test_cmp sparse-checkout-out sparse-index-out &&
|
||||
test_cmp sparse-checkout-err sparse-index-err
|
||||
}
|
||||
|
||||
test_expect_success 'sparse-index contents' '
|
||||
init_repos &&
|
||||
|
||||
test-tool -C sparse-index read-cache --table >cache &&
|
||||
for dir in folder1 folder2 x
|
||||
do
|
||||
TREE=$(git -C sparse-index rev-parse HEAD:$dir) &&
|
||||
grep "040000 tree $TREE $dir/" cache \
|
||||
|| return 1
|
||||
done &&
|
||||
|
||||
git -C sparse-index sparse-checkout set folder1 &&
|
||||
|
||||
test-tool -C sparse-index read-cache --table >cache &&
|
||||
for dir in deep folder2 x
|
||||
do
|
||||
TREE=$(git -C sparse-index rev-parse HEAD:$dir) &&
|
||||
grep "040000 tree $TREE $dir/" cache \
|
||||
|| return 1
|
||||
done &&
|
||||
|
||||
git -C sparse-index sparse-checkout set deep/deeper1 &&
|
||||
|
||||
test-tool -C sparse-index read-cache --table >cache &&
|
||||
for dir in deep/deeper2 folder1 folder2 x
|
||||
do
|
||||
TREE=$(git -C sparse-index rev-parse HEAD:$dir) &&
|
||||
grep "040000 tree $TREE $dir/" cache \
|
||||
|| return 1
|
||||
done &&
|
||||
|
||||
# Disabling the sparse-index removes tree entries with full ones
|
||||
git -C sparse-index sparse-checkout init --no-sparse-index &&
|
||||
|
||||
test-tool -C sparse-index read-cache --table >cache &&
|
||||
! grep "040000 tree" cache &&
|
||||
test_sparse_match test-tool read-cache --table
|
||||
'
|
||||
|
||||
test_expect_success 'expanded in-memory index matches full index' '
|
||||
init_repos &&
|
||||
test_sparse_match test-tool read-cache --expand --table
|
||||
'
|
||||
|
||||
test_expect_success 'status with options' '
|
||||
init_repos &&
|
||||
test_sparse_match ls &&
|
||||
test_all_match git status --porcelain=v2 &&
|
||||
test_all_match git status --porcelain=v2 -z -u &&
|
||||
test_all_match git status --porcelain=v2 -uno &&
|
||||
run_on_all "touch README.md" &&
|
||||
run_on_all touch README.md &&
|
||||
test_all_match git status --porcelain=v2 &&
|
||||
test_all_match git status --porcelain=v2 -z -u &&
|
||||
test_all_match git status --porcelain=v2 -uno &&
|
||||
|
@ -135,7 +202,7 @@ test_expect_success 'add, commit, checkout' '
|
|||
write_script edit-contents <<-\EOF &&
|
||||
echo text >>$1
|
||||
EOF
|
||||
run_on_all "../edit-contents README.md" &&
|
||||
run_on_all ../edit-contents README.md &&
|
||||
|
||||
test_all_match git add README.md &&
|
||||
test_all_match git status --porcelain=v2 &&
|
||||
|
@ -144,7 +211,7 @@ test_expect_success 'add, commit, checkout' '
|
|||
test_all_match git checkout HEAD~1 &&
|
||||
test_all_match git checkout - &&
|
||||
|
||||
run_on_all "../edit-contents README.md" &&
|
||||
run_on_all ../edit-contents README.md &&
|
||||
|
||||
test_all_match git add -A &&
|
||||
test_all_match git status --porcelain=v2 &&
|
||||
|
@ -153,7 +220,7 @@ test_expect_success 'add, commit, checkout' '
|
|||
test_all_match git checkout HEAD~1 &&
|
||||
test_all_match git checkout - &&
|
||||
|
||||
run_on_all "../edit-contents deep/newfile" &&
|
||||
run_on_all ../edit-contents deep/newfile &&
|
||||
|
||||
test_all_match git status --porcelain=v2 -uno &&
|
||||
test_all_match git status --porcelain=v2 &&
|
||||
|
@ -186,7 +253,7 @@ test_expect_success 'diff --staged' '
|
|||
write_script edit-contents <<-\EOF &&
|
||||
echo text >>README.md
|
||||
EOF
|
||||
run_on_all "../edit-contents" &&
|
||||
run_on_all ../edit-contents &&
|
||||
|
||||
test_all_match git diff &&
|
||||
test_all_match git diff --staged &&
|
||||
|
@ -252,6 +319,17 @@ test_expect_failure 'checkout and reset (mixed)' '
|
|||
test_all_match git reset update-folder2
|
||||
'
|
||||
|
||||
# Ensure that sparse-index behaves identically to
|
||||
# sparse-checkout with a full index.
|
||||
test_expect_success 'checkout and reset (mixed) [sparse]' '
|
||||
init_repos &&
|
||||
|
||||
test_sparse_match git checkout -b reset-test update-deep &&
|
||||
test_sparse_match git reset deepest &&
|
||||
test_sparse_match git reset update-folder1 &&
|
||||
test_sparse_match git reset update-folder2
|
||||
'
|
||||
|
||||
test_expect_success 'merge' '
|
||||
init_repos &&
|
||||
|
||||
|
@ -280,7 +358,7 @@ test_expect_success 'clean' '
|
|||
echo bogus >>.gitignore &&
|
||||
run_on_all cp ../.gitignore . &&
|
||||
test_all_match git add .gitignore &&
|
||||
test_all_match git commit -m ignore-bogus-files &&
|
||||
test_all_match git commit -m "ignore bogus files" &&
|
||||
|
||||
run_on_sparse mkdir folder1 &&
|
||||
run_on_all touch folder1/bogus &&
|
||||
|
@ -288,14 +366,51 @@ test_expect_success 'clean' '
|
|||
test_all_match git status --porcelain=v2 &&
|
||||
test_all_match git clean -f &&
|
||||
test_all_match git status --porcelain=v2 &&
|
||||
test_sparse_match ls &&
|
||||
test_sparse_match ls folder1 &&
|
||||
|
||||
test_all_match git clean -xf &&
|
||||
test_all_match git status --porcelain=v2 &&
|
||||
test_sparse_match ls &&
|
||||
test_sparse_match ls folder1 &&
|
||||
|
||||
test_all_match git clean -xdf &&
|
||||
test_all_match git status --porcelain=v2 &&
|
||||
test_sparse_match ls &&
|
||||
test_sparse_match ls folder1 &&
|
||||
|
||||
test_path_is_dir sparse-checkout/folder1
|
||||
test_sparse_match test_path_is_dir folder1
|
||||
'
|
||||
|
||||
test_expect_success 'submodule handling' '
|
||||
init_repos &&
|
||||
|
||||
test_all_match mkdir modules &&
|
||||
test_all_match touch modules/a &&
|
||||
test_all_match git add modules &&
|
||||
test_all_match git commit -m "add modules directory" &&
|
||||
|
||||
run_on_all git submodule add "$(pwd)/initial-repo" modules/sub &&
|
||||
test_all_match git commit -m "add submodule" &&
|
||||
|
||||
# having a submodule prevents "modules" from collapse
|
||||
test-tool -C sparse-index read-cache --table >cache &&
|
||||
grep "100644 blob .* modules/a" cache &&
|
||||
grep "160000 commit $(git -C initial-repo rev-parse HEAD) modules/sub" cache
|
||||
'
|
||||
|
||||
test_expect_success 'sparse-index is expanded and converted back' '
|
||||
init_repos &&
|
||||
|
||||
GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
|
||||
git -C sparse-index -c core.fsmonitor="" reset --hard &&
|
||||
test_region index convert_to_sparse trace2.txt &&
|
||||
test_region index ensure_full_index trace2.txt &&
|
||||
|
||||
rm trace2.txt &&
|
||||
GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
|
||||
git -C sparse-index -c core.fsmonitor="" status -uno &&
|
||||
test_region index ensure_full_index trace2.txt
|
||||
'
|
||||
|
||||
test_done
|
||||
|
|
|
@ -750,9 +750,13 @@ static int index_pos_by_traverse_info(struct name_entry *names,
|
|||
strbuf_make_traverse_path(&name, info, names->path, names->pathlen);
|
||||
strbuf_addch(&name, '/');
|
||||
pos = index_name_pos(o->src_index, name.buf, name.len);
|
||||
if (pos >= 0)
|
||||
BUG("This is a directory and should not exist in index");
|
||||
pos = -pos - 1;
|
||||
if (pos >= 0) {
|
||||
if (!o->src_index->sparse_index ||
|
||||
!(o->src_index->cache[pos]->ce_flags & CE_SKIP_WORKTREE))
|
||||
BUG("This is a directory and should not exist in index");
|
||||
} else {
|
||||
pos = -pos - 1;
|
||||
}
|
||||
if (pos >= o->src_index->cache_nr ||
|
||||
!starts_with(o->src_index->cache[pos]->name, name.buf) ||
|
||||
(pos > 0 && starts_with(o->src_index->cache[pos-1]->name, name.buf)))
|
||||
|
@ -1571,6 +1575,7 @@ static int verify_absent(const struct cache_entry *,
|
|||
*/
|
||||
int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)
|
||||
{
|
||||
struct repository *repo = the_repository;
|
||||
int i, ret;
|
||||
static struct cache_entry *dfc;
|
||||
struct pattern_list pl;
|
||||
|
@ -1582,6 +1587,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
|
|||
trace_performance_enter();
|
||||
trace2_region_enter("unpack_trees", "unpack_trees", the_repository);
|
||||
|
||||
prepare_repo_settings(repo);
|
||||
if (repo->settings.command_requires_full_index) {
|
||||
ensure_full_index(o->src_index);
|
||||
ensure_full_index(o->dst_index);
|
||||
}
|
||||
|
||||
if (!core_apply_sparse_checkout || !o->update)
|
||||
o->skip_sparse_checkout = 1;
|
||||
if (!o->skip_sparse_checkout && !o->pl) {
|
||||
|
|
Loading…
Reference in New Issue