From 26146980f1298e468a49af4aa5b2f1cc5dc8857b Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 10 Sep 2021 10:04:42 -0400 Subject: [PATCH 1/5] t5551: test v2-to-v0 http protocol fallback Since we use the v2 protocol by default, the connection of a v2 client to a v2 server is well covered by the test suite. And with the GIT_TEST_PROTOCOL_VERSION knob, we can easily test a v0 client connecting to a v2-aware server (which will then just speak v0). But we have no regular tests that a v2 client, when encountering a non-v2-aware server, will correctly fall back to using v0. In theory this is a job for the cross-version tests in t/interop, but: - they cover only git:// and file:// clones - they are not part of the usual test suite, so nobody ever runs them anyway Since using v2 over http requires configuring the web server to pass along the Git-Protocol header, we can easily create a situation where the server does not respect the v2 probe, and the conversation falls back to v0. This works just fine. This new test is not about fixing any particular bug, but just making sure that the system works (and continues to work) as expected. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/lib-httpd/apache.conf | 5 +++++ t/t5551-http-fetch-smart.sh | 9 +++++++++ 2 files changed, 14 insertions(+) diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf index afa91e38b0..1321357d8b 100644 --- a/t/lib-httpd/apache.conf +++ b/t/lib-httpd/apache.conf @@ -117,6 +117,11 @@ Alias /auth/dumb/ www/auth/dumb/ SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} SetEnv GIT_HTTP_EXPORT_ALL + + SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} + SetEnv GIT_HTTP_EXPORT_ALL + SetEnv GIT_PROTOCOL + ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/ ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/ ScriptAliasMatch /error_git_upload_pack/(.*)/git-upload-pack error.sh/ diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 4f87d90c5b..cffc47a8e3 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -558,4 +558,13 @@ test_expect_success 'http auth forgets bogus credentials' ' expect_askpass both user@host ' +test_expect_success 'client falls back from v2 to v0 to match server' ' + GIT_TRACE_PACKET=$PWD/trace \ + GIT_TEST_PROTOCOL_VERSION=2 \ + git clone $HTTPD_URL/smart_v0/repo.git repo-v0 && + # check for v0; there the HEAD symref is communicated in the capability + # line; v2 uses a different syntax on each ref advertisement line + grep symref=HEAD:refs/heads/ trace +' + test_done From ff6a37c99e3343633c53f56789afcc8f8165d276 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 10 Sep 2021 10:05:45 -0400 Subject: [PATCH 2/5] http-backend: handle HTTP_GIT_PROTOCOL CGI variable When a client requests the v2 protocol over HTTP, they set the Git-Protocol header. Webservers will generally make that available to our CGI as HTTP_GIT_PROTOCOL in the environment. However, that's not sufficient for upload-pack, etc, to respect it; they look in GIT_PROTOCOL (without the HTTP_ prefix). Either the webserver or the CGI is responsible for relaying that HTTP header into the GIT_PROTOCOL variable. Traditionally, our tests have configured the webserver to do so, but that's a burden on the server admin. We can make this work out of the box by having the http-backend CGI copy the contents of HTTP_GIT_PROTOCOL to GIT_PROTOCOL. There are no new tests here. By removing the SetEnvIf line from our test Apache config, we're now relying on this behavior of http-backend to trigger the v2 protocol there (and there are numerous tests that fail if this doesn't work). There is one subtlety here: we copy HTTP_GIT_PROTOCOL only if there is no existing GIT_PROTOCOL variable. That leaves the webserver admin free to override the client's decision if they choose. This is unlikely to be useful in practice, but is more flexible. And indeed, it allows the v2-to-v0 fallback test added in the previous commit to continue working. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- http-backend.c | 4 ++++ t/lib-httpd/apache.conf | 2 -- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/http-backend.c b/http-backend.c index b329bf63f0..92ceb31f9a 100644 --- a/http-backend.c +++ b/http-backend.c @@ -739,6 +739,7 @@ static int bad_request(struct strbuf *hdr, const struct service_cmd *c) int cmd_main(int argc, const char **argv) { char *method = getenv("REQUEST_METHOD"); + const char *proto_header; char *dir; struct service_cmd *cmd = NULL; char *cmd_arg = NULL; @@ -789,6 +790,9 @@ int cmd_main(int argc, const char **argv) http_config(); max_request_buffer = git_env_ulong("GIT_HTTP_MAX_REQUEST_BUFFER", max_request_buffer); + proto_header = getenv("HTTP_GIT_PROTOCOL"); + if (proto_header) + setenv(GIT_PROTOCOL_ENVIRONMENT, proto_header, 0); cmd->imp(&hdr, cmd_arg); return 0; diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf index 1321357d8b..180a41fe96 100644 --- a/t/lib-httpd/apache.conf +++ b/t/lib-httpd/apache.conf @@ -81,8 +81,6 @@ PassEnv GIT_TRACE PassEnv GIT_CONFIG_NOSYSTEM PassEnv GIT_TEST_SIDEBAND_ALL -SetEnvIf Git-Protocol ".*" GIT_PROTOCOL=$0 - Alias /dumb/ www/ Alias /auth/dumb/ www/auth/dumb/ From 295d81b9e4c91517f4f436a16889027cb86cba57 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 10 Sep 2021 10:09:13 -0400 Subject: [PATCH 3/5] docs/http-backend: mention v2 protocol Historically there was a little bit of configuration needed at the webserver level in order to get the client's v2 protocol probes to Git. But when we introduced the v2 protocol, we never documented these. As of the previous commit, this should mostly work out of the box without any explicit configuration. But it's worth documenting this to make it clear how we expect it to work, especially in the face of webservers which don't provide all headers over the CGI interface. Or anybody who runs across this documentation but has an older version of Git (or _used_ to have an older version, and wonders why they still have a SetEnvIf line in their Apache config and whether it's still necessary). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- Documentation/git-http-backend.txt | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt index 558966aa83..0c5c0dde19 100644 --- a/Documentation/git-http-backend.txt +++ b/Documentation/git-http-backend.txt @@ -16,7 +16,9 @@ A simple CGI program to serve the contents of a Git repository to Git clients accessing the repository over http:// and https:// protocols. The program supports clients fetching using both the smart HTTP protocol and the backwards-compatible dumb HTTP protocol, as well as clients -pushing using the smart HTTP protocol. +pushing using the smart HTTP protocol. It also supports Git's +more-efficient "v2" protocol if properly configured; see the +discussion of `GIT_PROTOCOL` in the ENVIRONMENT section below. It verifies that the directory has the magic file "git-daemon-export-ok", and it will refuse to export any Git directory @@ -77,6 +79,18 @@ Apache 2.x:: SetEnv GIT_PROJECT_ROOT /var/www/git SetEnv GIT_HTTP_EXPORT_ALL ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/ + +# This is not strictly necessary using Apache and a modern version of +# git-http-backend, as the webserver will pass along the header in the +# environment as HTTP_GIT_PROTOCOL, and http-backend will copy that into +# GIT_PROTOCOL. But you may need this line (or something similar if you +# are using a different webserver), or if you want to support older Git +# versions that did not do that copying. +# +# Having the webserver set up GIT_PROTOCOL is perfectly fine even with +# modern versions (and will take precedence over HTTP_GIT_PROTOCOL, +# which means it can be used to override the client's request). +SetEnvIf Git-Protocol ".*" GIT_PROTOCOL=$0 ---------------------------------------------------------------- + To enable anonymous read access but authenticated write access, @@ -264,6 +278,16 @@ a repository with an extremely large number of refs. The value can be specified with a unit (e.g., `100M` for 100 megabytes). The default is 10 megabytes. +Clients may probe for optional protocol capabilities (like the v2 +protocol) using the `Git-Protocol` HTTP header. In order to support +these, the contents of that header must appear in the `GIT_PROTOCOL` +environment variable. Most webservers will pass this header to the CGI +via the `HTTP_GIT_PROTOCOL` variable, and `git-http-backend` will +automatically copy that to `GIT_PROTOCOL`. However, some webservers may +be more selective about which headers they'll pass, in which case they +need to be configured explicitly (see the mention of `Git-Protocol` in +the Apache config from the earlier EXAMPLES section). + The backend process sets GIT_COMMITTER_NAME to '$REMOTE_USER' and GIT_COMMITTER_EMAIL to '$\{REMOTE_USER}@http.$\{REMOTE_ADDR\}', ensuring that any reflogs created by 'git-receive-pack' contain some From 2834a72d5e22fa1138efdeb1f515e894d743cfb4 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 10 Sep 2021 10:09:56 -0400 Subject: [PATCH 4/5] docs/git: discuss server-side config for GIT_PROTOCOL The v2 protocol requires that the GIT_PROTOCOL environment variable gets passed around, but we don't have any documentation describing how this is supposed to work. In particular, we need to note what server admins might need to configure to make things work. The definition of the GIT_PROTOCOL variable is probably the best place for this, since: - we deal with multiple transports (ssh, http, etc). Transport-specific documentation (like the git-http-backend bits added in the previous commit) are helpful for those transports, but this gives a broader overview. Plus we do not have a specific transport endpoint program for ssh, so this is a reasonable place to mention it. - the server side of the protocol involves multiple programs. For now, upload-pack is the only endpoint which uses GIT_PROTOCOL, but that will likely expand in the future. We're better off with a central discussion of what the server admin might need to do. However, for discoverability, this patch adds a pointer from upload-pack's documentation. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- Documentation/git-upload-pack.txt | 8 ++++++++ Documentation/git.txt | 15 +++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt index 9822c1eb1a..070fc78008 100644 --- a/Documentation/git-upload-pack.txt +++ b/Documentation/git-upload-pack.txt @@ -44,6 +44,14 @@ OPTIONS :: The repository to sync from. +ENVIRONMENT +----------- + +`GIT_PROTOCOL`:: + Internal variable used for handshaking the wire protocol. Server + admins may need to configure some transports to allow this + variable to be passed. See the discussion in linkgit:git[1]. + SEE ALSO -------- linkgit:gitnamespaces[7] diff --git a/Documentation/git.txt b/Documentation/git.txt index 6dd241ef83..e4b82599fc 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -894,6 +894,21 @@ for full details. Contains a colon ':' separated list of keys with optional values 'key[=value]'. Presence of unknown keys and values must be ignored. ++ +Note that servers may need to be configured to allow this variable to +pass over some transports. It will be propagated automatically when +accessing local repositories (i.e., `file://` or a filesystem path), as +well as over the `git://` protocol. For git-over-http, it should work +automatically in most configurations, but see the discussion in +linkgit:git-http-backend[1]. For git-over-ssh, the ssh server may need +to be configured to allow clients to pass this variable (e.g., by using +`AcceptEnv GIT_PROTOCOL` with OpenSSH). ++ +This configuration is optional. If the variable is not propagated, then +clients will fall back to the original "v0" protocol (but may miss out +on some performance improvements or features). This variable currently +only affects clones and fetches; it is not yet used for pushes (but may +be in the future). `GIT_OPTIONAL_LOCKS`:: If set to `0`, Git will complete any requested operation without From 1b421e7a5aa6984777029ab799f9a0221875e1dd Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 10 Sep 2021 10:10:19 -0400 Subject: [PATCH 5/5] docs/protocol-v2: point readers transport config discussion We recently added tips for server admins to configure various transports to support v2's GIT_PROTOCOL variable. While the protocol-v2 document is pretty technical and not of interest to most admins, it may be a starting point for them to figure out how to turn on v2. Let's put some pointers from there to the other documentation. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- Documentation/technical/protocol-v2.txt | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 1040d85319..a703d37e08 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -42,7 +42,8 @@ Initial Client Request In general a client can request to speak protocol v2 by sending `version=2` through the respective side-channel for the transport being used which inevitably sets `GIT_PROTOCOL`. More information can be -found in `pack-protocol.txt` and `http-protocol.txt`. In all cases the +found in `pack-protocol.txt` and `http-protocol.txt`, as well as the +`GIT_PROTOCOL` definition in `git.txt`. In all cases the response from the server is the capability advertisement. Git Transport @@ -58,6 +59,8 @@ SSH and File Transport When using either the ssh:// or file:// transport, the GIT_PROTOCOL environment variable must be set explicitly to include "version=2". +The server may need to be configured to allow this environment variable +to pass. HTTP Transport ~~~~~~~~~~~~~~ @@ -81,6 +84,9 @@ A v2 server would reply: Subsequent requests are then made directly to the service `$GIT_URL/git-upload-pack`. (This works the same for git-receive-pack). +The server may need to be configured to pass this header's contents via +the `GIT_PROTOCOL` variable. See the discussion in `git-http-backend.txt`. + Capability Advertisement ------------------------