Jeff King 6bc0cb5176 http-backend: spool ref negotiation requests to buffer ... When http-backend spawns "upload-pack" to do ref negotiation, it streams the http request body to upload-pack, who then streams the http response back to the client as it reads. In theory, git can go full-duplex; the client can consume our response while it is still sending the request. In practice, however, HTTP is a half-duplex protocol. Even if our client is ready to read and write simultaneously, we may have other HTTP infrastructure in the way, including the webserver that spawns our CGI, or any intermediate proxies. In at least one documented case[1], this leads to deadlock when trying a fetch over http. What happens is basically: 1. Apache proxies the request to the CGI, http-backend. 2. http-backend gzip-inflates the data and sends the result to upload-pack. 3. upload-pack acts on the data and generates output over the pipe back to Apache. Apache isn't reading because it's busy writing (step 1). This works fine most of the time, because the upload-pack output ends up in a system pipe buffer, and Apache reads it as soon as it finishes writing. But if both the request and the response exceed the system pipe buffer size, then we deadlock (Apache blocks writing to http-backend, http-backend blocks writing to upload-pack, and upload-pack blocks writing to Apache). We need to break the deadlock by spooling either the input or the output. In this case, it's ideal to spool the input, because Apache does not start reading either stdout _or_ stderr until we have consumed all of the input. So until we do so, we cannot even get an error message out to the client. The solution is fairly straight-forward: we read the request body into an in-memory buffer in http-backend, freeing up Apache, and then feed the data ourselves to upload-pack. But there are a few important things to note: 1. We limit the in-memory buffer to prevent an obvious denial-of-service attack. This is a new hard limit on requests, but it's unlikely to come into play. The default value is 10MB, which covers even the ridiculous 100,000-ref negotation in the included test (that actually caps out just over 5MB). But it's configurable on the off chance that you don't mind spending some extra memory to make even ridiculous requests work. 2. We must take care only to buffer when we have to. For pushes, the incoming packfile may be of arbitrary size, and we should connect the input directly to receive-pack. There's no deadlock problem here, though, because we do not produce any output until the whole packfile has been read. For upload-pack's initial ref advertisement, we similarly do not need to buffer. Even though we may generate a lot of output, there is no request body at all (i.e., it is a GET, not a POST). [1] http://article.gmane.org/gmane.comp.version-control.git/269020 Test-adapted-from: Dennis Kaarsemaker <dennis@kaarsemaker.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>		2015-05-25 20:43:18 -07:00
..
howto	Documentation: typofixes	2014-11-04 13:14:44 -08:00
RelNotes	Git 2.2.2	2015-01-12 14:06:12 -08:00
technical	Documentation: typofixes	2014-11-04 13:14:44 -08:00
.gitattributes
.gitignore
asciidoc.conf
blame-options.txt
build-docdep.perl
cat-texi.perl
cmd-list.perl
CodingGuidelines	Merge branch 'po/error-message-style'	2014-07-16 11:33:03 -07:00
config.txt	Merge branch 'jk/colors-fix' into maint	2014-12-22 12:16:58 -08:00
date-formats.txt
diff-config.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
diff-format.txt
diff-generate-patch.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
diff-options.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
docbook-xsl.css
docbook.xsl
everyday.txto	doc: add 'everyday' to 'git help'	2014-10-10 16:02:26 -07:00
fetch-options.txt	fetch: allow explicit --refmap to override configuration	2014-06-05 15:13:12 -07:00
fix-texi.perl
git-add.txt	Merge branch 'jl/nor-or-nand-and'	2014-04-08 12:00:28 -07:00
git-am.txt	Merge branch 'rw/apply-does-not-take-ignore-date' into maint	2015-01-12 14:00:16 -08:00
git-annotate.txt
git-apply.txt
git-archimport.txt
git-archive.txt	docs: clarify remote restrictions for git-upload-archive	2014-02-28 09:55:35 -08:00
git-bisect-lk2009.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-bisect.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
git-blame.txt	docs/git-blame: explain more clearly the example pickaxe use	2014-02-11 11:03:07 -08:00
git-branch.txt
git-bundle.txt
git-cat-file.txt	cat-file: provide %(deltabase) batch format	2013-12-26 11:54:26 -08:00
git-check-attr.txt
git-check-ignore.txt
git-check-mailmap.txt
git-check-ref-format.txt
git-checkout-index.txt
git-checkout.txt	Documentation: @{-N} can refer to a commit	2014-01-21 13:50:00 -08:00
git-cherry-pick.txt	parse-options: multi-word argh should use dash to separate words	2014-03-24 10:43:34 -07:00
git-cherry.txt
git-citool.txt
git-clean.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-clone.txt	docs/git-clone: clarify use of --no-hardlinks option	2014-02-11 11:03:07 -08:00
git-column.txt	doc: remote author/documentation sections from more pages	2014-01-27 08:34:34 -08:00
git-commit-tree.txt	commit-tree: add and document --no-gpg-sign	2014-02-24 14:51:35 -08:00
git-commit.txt	Merge branch 'jc/doc-commit-only'	2014-11-18 10:19:42 -08:00
git-config.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
git-count-objects.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
git-credential-cache--daemon.txt	credential-cache: close stderr in daemon process	2014-09-16 11:11:58 -07:00
git-credential-cache.txt
git-credential-store.txt	docs/credential-store: s/--store/--file/	2014-11-06 09:51:08 -08:00
git-credential.txt
git-cvsexportcommit.txt
git-cvsimport.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-cvsserver.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-daemon.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
git-describe.txt
git-diff-files.txt
git-diff-index.txt
git-diff-tree.txt
git-diff.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
git-difftool.txt	difftool: add support for --trust-exit-code	2014-10-28 10:36:57 -07:00
git-fast-export.txt	Merge branch 'mh/doc-remote-helper-xref'	2014-11-19 13:47:56 -08:00
git-fast-import.txt	doc: add some crossrefs between manual pages	2014-11-11 14:47:04 -08:00
git-fetch-pack.txt
git-fetch.txt	docs: Explain the purpose of fetch's and pull's <refspec> parameter.	2014-06-12 09:59:13 -07:00
git-filter-branch.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-fmt-merge-msg.txt
git-for-each-ref.txt	doc: remote author/documentation sections from more pages	2014-01-27 08:34:34 -08:00
git-format-patch.txt	format-patch: add "--signature-file=<file>" option	2014-05-27 12:38:32 -07:00
git-fsck-objects.txt
git-fsck.txt
git-gc.txt	gc --aggressive: make --depth configurable	2014-03-31 10:26:24 -07:00
git-get-tar-commit-id.txt
git-grep.txt	grep: add grep.fullName config variable	2014-03-20 12:38:00 -07:00
git-gui.txt
git-hash-object.txt
git-help.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
git-http-backend.txt	http-backend: spool ref negotiation requests to buffer	2015-05-25 20:43:18 -07:00
git-http-fetch.txt
git-http-push.txt
git-imap-send.txt	Documentation: fix mismatched delimiters in git-imap-send	2014-10-13 13:35:37 -07:00
git-index-pack.txt
git-init-db.txt
git-init.txt	Documentation: git-init: flesh out example	2014-08-08 13:17:42 -07:00
git-instaweb.txt
git-interpret-trailers.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-log.txt
git-ls-files.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
git-ls-remote.txt
git-ls-tree.txt
git-mailinfo.txt
git-mailsplit.txt
git-merge-base.txt
git-merge-file.txt
git-merge-index.txt
git-merge-one-file.txt
git-merge-tree.txt
git-merge.txt	merge: enable defaulttoupstream by default	2014-04-22 12:53:59 -07:00
git-mergetool--lib.txt
git-mergetool.txt	mergetool: document the default for --[no-]prompt	2014-04-24 11:29:05 -07:00
git-mktag.txt
git-mktree.txt
git-mv.txt	mv: better document side effects when moving a submodule	2014-01-07 14:33:04 -08:00
git-name-rev.txt
git-notes.txt	parse-options: multi-word argh should use dash to separate words	2014-03-24 10:43:34 -07:00
git-p4.txt	git p4 doc: use two-line style for options with multiple spellings	2014-01-22 08:06:20 -08:00
git-pack-objects.txt	upload-pack: send shallow info over stdin to pack-objects	2014-03-11 13:32:10 -07:00
git-pack-redundant.txt
git-pack-refs.txt
git-parse-remote.txt
git-patch-id.txt	patch-id: make it stable against hunk reordering	2014-06-10 13:09:24 -07:00
git-prune-packed.txt	Documentation: adjust document title underlining	2014-10-13 13:35:18 -07:00
git-prune.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
git-pull.txt	Merge branch 'jc/maint-pull-docfix' into maint	2014-02-05 14:03:47 -08:00
git-push.txt	Merge branch 'po/everyday-doc' into maint	2014-12-22 12:18:17 -08:00
git-quiltimport.txt	Documentation: adjust document title underlining	2014-10-13 13:35:18 -07:00
git-read-tree.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
git-rebase.txt	Merge branch 'so/rebase-doc-fork-point'	2014-10-14 10:49:07 -07:00
git-receive-pack.txt	signed push: allow stale nonce in stateless mode	2014-09-17 15:19:54 -07:00
git-reflog.txt
git-relink.txt
git-remote-ext.txt	doc: add some crossrefs between manual pages	2014-11-11 14:47:04 -08:00
git-remote-fd.txt	doc: add some crossrefs between manual pages	2014-11-11 14:47:04 -08:00
git-remote-helpers.txto
git-remote-testgit.txt
git-remote.txt	docs/git-remote: capitalize first word of initial blurb	2014-02-11 11:03:07 -08:00
git-repack.txt	Merge branch 'jk/repack-pack-keep-objects'	2014-03-18 13:50:29 -07:00
git-replace.txt	Merge branch 'cc/replace-graft'	2014-07-27 15:14:18 -07:00
git-request-pull.txt	request-pull: documentation updates	2014-03-13 14:22:20 -07:00
git-rerere.txt
git-reset.txt	Merge branch 'jl/nor-or-nand-and'	2014-04-08 12:00:28 -07:00
git-rev-list.txt	pretty: provide a strict ISO 8601 date format	2014-08-29 12:37:02 -07:00
git-rev-parse.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-revert.txt	parse-options: multi-word argh should use dash to separate words	2014-03-24 10:43:34 -07:00
git-rm.txt	rm: better document side effects when removing a submodule	2014-01-07 14:34:06 -08:00
git-send-email.txt	Merge branch 'mt/send-email-cover-to-cc'	2014-06-20 13:12:20 -07:00
git-send-pack.txt	send-pack: take refspecs over stdin	2014-08-26 12:58:02 -07:00
git-sh-i18n--envsubst.txt
git-sh-i18n.txt
git-sh-setup.txt
git-shell.txt	shell doc: remove stray "+" in example	2014-05-08 10:26:26 -07:00
git-shortlog.txt
git-show-branch.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
git-show-index.txt
git-show-ref.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
git-show.txt
git-stage.txt	Documentation: adjust document title underlining	2014-10-13 13:35:18 -07:00
git-stash.txt	stash doc: mention short form -k in save description	2014-02-24 09:13:30 -08:00
git-status.txt	doc: fix 'git status --help' character quoting	2014-10-19 20:45:16 -07:00
git-stripspace.txt
git-submodule.txt	Merge branch 'mc/doc-submodule-sync-recurse' into maint	2014-07-10 11:08:31 -07:00
git-svn.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-symbolic-ref.txt
git-tag.txt	Merge branch 'maint-2.0' into maint	2014-10-07 13:40:51 -07:00
git-tools.txt
git-unpack-file.txt
git-unpack-objects.txt
git-update-index.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
git-update-ref.txt	update-ref --stdin -z: deprecate interpreting the empty string as zeros	2014-04-07 12:09:13 -07:00
git-update-server-info.txt
git-upload-archive.txt	add uploadarchive.allowUnreachable option	2014-02-28 09:55:37 -08:00
git-upload-pack.txt
git-var.txt
git-verify-commit.txt	verify-commit: scriptable commit signature verification	2014-06-23 15:50:31 -07:00
git-verify-pack.txt
git-verify-tag.txt
git-web--browse.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
git-whatchanged.txt
git-write-tree.txt
git.txt	Git 2.2.2	2015-01-12 14:06:12 -08:00
gitattributes.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
gitcli.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
gitcore-tutorial.txt	doc: add 'everyday' to 'git help'	2014-10-10 16:02:26 -07:00
gitcredentials.txt
gitcvs-migration.txt	doc: add 'everyday' to 'git help'	2014-10-10 16:02:26 -07:00
gitdiffcore.txt
giteveryday.txt	doc: add 'everyday' to 'git help'	2014-10-10 16:02:26 -07:00
gitglossary.txt	doc: add 'everyday' to 'git help'	2014-10-10 16:02:26 -07:00
githooks.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
gitignore.txt	Merge branch 'nd/gitignore-trailing-whitespace' into maint	2014-11-12 12:13:12 -08:00
gitk.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
gitmodules.txt	status/commit: show staged submodules regardless of ignore config	2014-04-07 10:32:20 -07:00
gitnamespaces.txt
gitremote-helpers.txt	doc: add some crossrefs between manual pages	2014-11-11 14:47:04 -08:00
gitrepository-layout.txt	read-cache: split-index mode	2014-06-13 11:49:39 -07:00
gitrevisions.txt
gittutorial-2.txt	Merge branch 'sn/tutorial-status-output-example'	2014-11-19 13:47:59 -08:00
gittutorial.txt	Merge branch 'sn/tutorial-status-output-example'	2014-11-19 13:47:59 -08:00
gitweb.conf.txt	Documentation: use "command-line" when used as a compound adjective, and fix other minor grammatical issues	2014-05-21 13:57:10 -07:00
gitweb.txt	Documentation: fix documentation AsciiDoc links for external urls	2014-02-20 14:14:58 -08:00
gitworkflows.txt
glossary-content.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
howto-index.sh	howto-index.sh: use the $( ... ) construct for command substitution	2014-04-17 11:14:57 -07:00
i18n.txt
install-doc-quick.sh
install-webdoc.sh	install-webdoc.sh: use the $( ... ) construct for command substitution	2014-04-17 11:14:58 -07:00
line-range-format.txt
mailmap.txt
Makefile	Merge branch 'bc/asciidoctor'	2014-10-29 10:07:40 -07:00
manpage-1.72.xsl
manpage-base-url.xsl.in
manpage-base.xsl
manpage-bold-literal.xsl
manpage-normal.xsl
manpage-quote-apos.xsl
manpage-suppress-sp.xsl
merge-config.txt
merge-options.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
merge-strategies.txt	Merge branch 'rr/doc-merge-strategies' into maint	2014-04-03 13:39:03 -07:00
pretty-formats.txt	Merge branch 'bc/asciidoc-pretty-formats-fix' into maint	2014-10-29 10:35:10 -07:00
pretty-options.txt	Documentation: fix misuses of "nor"	2014-03-31 15:16:22 -07:00
pull-fetch-param.txt	docs: Explain the purpose of fetch's and pull's <refspec> parameter.	2014-06-12 09:59:13 -07:00
rev-list-options.txt	Documentation: typofixes	2014-11-04 13:14:44 -08:00
revisions.txt	Documentation: mention config sources for @{upstream}	2014-05-13 12:35:00 -07:00
sequencer.txt
SubmittingPatches	SubmittingPatches: final submission is To: maintainer and CC: list	2014-11-13 10:39:24 -08:00
urls-remotes.txt
urls.txt
user-manual.conf
user-manual.txt	Merge branch 'jm/doc-wording-tweaks'	2014-06-16 12:18:39 -07:00