mirror/ git
1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-05-19 21:06:09 +02:00
Code
git/builtin
History
Matheus Tavares e9e8adf1a8 parallel-checkout: make it truly parallel 
Use multiple worker processes to distribute the queued entries and call
write_pc_item() in parallel for them. The items are distributed
uniformly in contiguous chunks. This minimizes the chances of two
workers writing to the same directory simultaneously, which could affect
performance due to lock contention in the kernel. Work stealing (or any
other format of re-distribution) is not implemented yet.

The protocol between the main process and the workers is quite simple.
They exchange binary messages packed in pkt-line format, and use
PKT-FLUSH to mark the end of input (from both sides). The main process
starts the communication by sending N pkt-lines, each corresponding to
an item that needs to be written. These packets contain all the
necessary information to load, smudge, and write the blob associated
with each item. Then it waits for the worker to send back N pkt-lines
containing the results for each item. The resulting packet must contain:
the identification number of the item that it refers to, the status of
the operation, and the lstat() data gathered after writing the file (iff
the operation was successful).

For now, checkout always uses a hardcoded value of 2 workers, only to
demonstrate that the parallel checkout framework correctly divides and
writes the queued entries. The next patch will add user configurations
and define a more reasonable default, based on tests with the said
settings.

Co-authored-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Co-authored-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
 		2021-04-19 11:57:05 -07:00
..
add.c add: propagate --chmod errors to exit status 2021-02-24 12:14:51 -08:00
am.c builtin/*: update usage format 2021-01-06 15:10:49 -08:00
annotate.c strvec: rename struct fields 2020-07-30 19:18:06 -07:00
apply.c
…
archive.c
…
bisect--helper.c Merge branch 'jk/bisect-peel-tag-fix' 2021-03-19 15:25:37 -07:00
blame.c Merge branch 'rs/blame-optim' 2021-02-25 16:43:29 -08:00
branch.c Merge branch 'ph/use-delete-refs' 2021-02-05 16:40:45 -08:00
bugreport.c builtin/bugreport.c: use thread-safe localtime_r() 2020-12-01 13:05:37 -08:00
bundle.c Merge branch 'bc/sha-256-part-3' 2020-08-11 18:04:11 -07:00
cat-file.c Merge branch 'cc/cat-file-usage-update' into master 2020-07-09 14:00:41 -07:00
check-attr.c
…
check-ignore.c dir: fix problematic API to avoid memory leaks 2020-08-18 17:17:31 -07:00
check-mailmap.c shortlog: remove unused(?) "repo-abbrev" feature 2021-01-12 14:04:42 -08:00
check-ref-format.c
…
checkout--worker.c parallel-checkout: make it truly parallel 2021-04-19 11:57:05 -07:00
checkout-index.c entry: extract a header file for entry.c functions 2021-03-23 10:34:05 -07:00
checkout.c Merge branch 'mt/parallel-checkout-part-1' 2021-04-02 14:43:14 -07:00
clean.c use CALLOC_ARRAY 2021-03-13 16:00:09 -08:00
clone.c clone: free or UNLEAK further pointers when finished 2021-03-14 15:57:59 -07:00
column.c column, range-diff: downcase option description 2021-03-29 14:06:08 -07:00
commit-graph.c builtin/*: update usage format 2021-01-06 15:10:49 -08:00
commit-tree.c Use OPT_CALLBACK and OPT_CALLBACK_F 2020-04-28 10:47:10 -07:00
commit.c Merge branch 'zh/commit-trailer' 2021-04-07 16:54:08 -07:00
config.c config: implement --fixed-value with --get* 2020-11-25 14:43:48 -08:00
count-objects.c
…
credential-cache--daemon.c unix-socket: add backlog size option to unix_stream_listen() 2021-03-15 14:32:51 -07:00
credential-cache.c unix-socket: disallow chdir() when creating unix domain sockets 2021-03-15 14:32:51 -07:00
credential-store.c crendential-store: use timeout when locking file 2020-11-25 12:30:18 -08:00
credential.c credential: load default config 2020-10-16 12:30:45 -07:00
describe.c refs: switch peel_ref() to peel_iterated_oid() 2021-01-21 15:51:31 -08:00
diff-files.c Merge branch 'jc/diffcore-rotate' 2021-02-25 16:43:30 -08:00
diff-index.c Merge branch 'jc/diffcore-rotate' 2021-02-25 16:43:30 -08:00
diff-tree.c Merge branch 'jc/diffcore-rotate' 2021-02-25 16:43:30 -08:00
diff.c Merge branch 'jc/diffcore-rotate' 2021-02-25 16:43:30 -08:00
difftool.c entry: extract a header file for entry.c functions 2021-03-23 10:34:05 -07:00
env--helper.c assert PARSE_OPT_NONEG in parse-options callbacks 2020-09-30 12:53:47 -07:00
fast-export.c builtin/*: update usage format 2021-01-06 15:10:49 -08:00
fast-import.c use CALLOC_ARRAY 2021-03-13 16:00:09 -08:00
fetch-pack.c connect, transport: encapsulate arg in struct 2021-02-05 13:49:54 -08:00
fetch.c Merge branch 'jt/clone-unborn-head' 2021-02-17 17:21:40 -08:00
fmt-merge-msg.c Lib-ify fmt-merge-msg 2020-03-24 15:04:43 -07:00
for-each-ref.c ref-filter: move ref_sorting flags to a bitfield 2021-01-07 15:13:21 -08:00
for-each-repo.c for-each-repo: do nothing on empty config 2021-01-07 19:12:02 -08:00
fsck.c fsck.c: pass along the fsck_msg_id in the fsck_error callback 2021-03-28 19:03:10 -07:00
gc.c maintenance: fix incorrect `maintenance.repo` path with bare repository 2021-02-23 00:22:45 -08:00
get-tar-commit-id.c
…
grep.c Merge branch 'ab/grep-pcre2-allocfix' 2021-03-22 14:00:23 -07:00
hash-object.c
…
help.c help: drop usage of 'common' and 'useful' for guides 2020-08-04 18:34:01 -07:00
index-pack.c Merge branch 'ab/fsck-api-cleanup' 2021-04-07 16:54:09 -07:00
init-db.c Merge branch 'ah/plugleaks' 2021-04-07 16:54:08 -07:00
interpret-trailers.c Use OPT_CALLBACK and OPT_CALLBACK_F 2020-04-28 10:47:10 -07:00
log.c Merge branch 'zh/format-patch-fractional-reroll-count' 2021-04-02 14:43:14 -07:00
ls-files.c tree.h API: simplify read_tree_recursive() signature 2021-03-20 16:09:26 -07:00
ls-remote.c Merge branch 'ah/plugleaks' 2021-04-07 16:54:08 -07:00
ls-tree.c tree.h API: simplify read_tree_recursive() signature 2021-03-20 16:09:26 -07:00
mailinfo.c
…
mailsplit.c
…
merge-base.c rebase: --fork-point regression fix 2020-02-11 09:59:39 -08:00
merge-file.c
…
merge-index.c
…
merge-ours.c
…
merge-recursive.c
…
merge-tree.c merge-base, xdiff: zero out xpparam_t structures 2020-10-20 12:53:26 -07:00
merge.c use CALLOC_ARRAY 2021-03-13 16:00:09 -08:00
mktag.c fsck.c: add an fsck_set_msg_type() API that takes enums 2021-03-28 19:03:10 -07:00
mktree.c
…
multi-pack-index.c multi-pack-index: add [--[no-]progress] option. 2019-10-23 12:05:06 +09:00
mv.c git mv foo FOO ; git mv foo bar gave an assert 2021-03-03 17:07:12 -08:00
name-rev.c oid_pos(): access table through const pointers 2021-01-28 12:03:26 -08:00
notes.c use CALLOC_ARRAY 2021-03-13 16:00:09 -08:00
pack-objects.c Merge branch 'hx/pack-objects-chunk-comment' 2021-03-30 14:35:37 -07:00
pack-redundant.c use CALLOC_ARRAY 2021-03-13 16:00:09 -08:00
pack-refs.c
…
patch-id.c patch-id: use oid_to_hex() to print multiple object IDs 2019-12-09 12:26:40 -08:00
prune-packed.c Lib-ify prune-packed 2020-03-24 15:04:44 -07:00
prune.c Merge branch 'tb/shallow-cleanup' 2020-05-13 12:19:18 -07:00
pull.c pull: display default warning only when non-ff 2020-12-15 17:39:42 -08:00
push.c Merge branch 'jc/push-delete-nothing' 2021-02-25 16:43:33 -08:00
range-diff.c column, range-diff: downcase option description 2021-03-29 14:06:08 -07:00
read-tree.c Use OPT_CALLBACK and OPT_CALLBACK_F 2020-04-28 10:47:10 -07:00
rebase.c rebase: remove transitory rebase.useBuiltin setting & env 2021-03-23 14:05:58 -07:00
receive-pack.c Merge branch 'rs/calloc-array' 2021-03-19 15:25:38 -07:00
reflog.c reflog expire --stale-fix: be generous about missing objects 2021-02-11 09:21:52 -08:00
remote-ext.c strvec: convert builtin/ callers away from argv_array name 2020-07-28 15:02:18 -07:00
remote-fd.c
…
remote.c Merge branch 'ah/plugleaks' 2021-04-07 16:54:08 -07:00
repack.c builtin/repack.c: reword comment around pack-objects flags 2021-03-05 11:33:52 -08:00
replace.c strvec: rename struct fields 2020-07-30 19:18:06 -07:00
rerere.c
…
reset.c reset: free instead of leaking unneeded ref 2021-03-14 15:57:59 -07:00
rev-list.c rev-list: add --disk-usage option for calculating disk usage 2021-02-11 09:57:55 -08:00
rev-parse.c rev-parse: add option for absolute or relative path formatting 2020-12-12 23:35:51 -08:00
revert.c Merge branch 'en/merge-ort-api-null-impl' 2020-11-18 13:32:53 -08:00
rm.c rm: support the --pathspec-from-file option 2020-02-19 10:56:49 -08:00
send-pack.c push: parse and set flag for "--force-if-includes" 2020-10-03 09:59:19 -07:00
shortlog.c Merge branch 'ab/mailmap' 2021-01-25 14:19:19 -08:00
show-branch.c Merge branch 'jt/interpret-branch-name-fallback' 2020-09-09 13:53:09 -07:00
show-index.c builtin/show-index: provide options to determine hash algo 2020-05-27 10:07:07 -07:00
show-ref.c refs: switch peel_ref() to peel_iterated_oid() 2021-01-21 15:51:31 -08:00
sparse-checkout.c exclude: add flags parameter to add_patterns() 2021-02-16 09:41:33 -08:00
stash.c Merge branch 'mt/parallel-checkout-part-1' 2021-04-02 14:43:14 -07:00
stripspace.c
…
submodule--helper.c Merge branch 'tb/precompose-prefix-too' 2021-02-12 14:21:04 -08:00
symbolic-ref.c symbolic-ref: don't leak shortened refname in check_symref() 2021-03-14 15:57:59 -07:00
tag.c Merge branch 'js/params-vs-args' 2021-02-25 16:43:32 -08:00
unpack-file.c
…
unpack-objects.c Merge branch 'ab/fsck-api-cleanup' 2021-04-07 16:54:09 -07:00
update-index.c Use OPT_CALLBACK and OPT_CALLBACK_F 2020-04-28 10:47:10 -07:00
update-ref.c update-ref: disallow "start" for ongoing transactions 2020-11-16 13:44:01 -08:00
update-server-info.c
…
upload-archive.c strvec: rename struct fields 2020-07-30 19:18:06 -07:00
upload-pack.c
…
var.c
…
verify-commit.c
…
verify-pack.c Merge branch 'bc/sha-256-part-3' 2020-08-11 18:04:11 -07:00
verify-tag.c
…
worktree.c worktree: fix leak in dwim_branch() 2021-03-14 15:57:59 -07:00
write-tree.c
…