From 77fb36aa7eba5161c34dbe70ba4dd5449e079c87 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 05:49:47 -0500 Subject: [PATCH 01/16] t5541: run "used receive-pack service" test earlier There's a test in t5541 that confirms that "git push" makes two requests (a GET to /info/refs, and a POST to /git-receive-pack). However, it's a noop unless GIT_TEST_PROTOCOL_VERSION is set to "0", due to 8a1b0978ab (test: request GIT_TEST_PROTOCOL_VERSION=0 when appropriate, 2019-12-23). This means that almost nobody runs it. And indeed, it has been broken since b0c4adcdd7 (remote-curl: send Accept-Language header to server, 2022-07-11). But the fault is not in that commit, but in how brittle the test is. It runs after several operations have been performed, which means that it expects to see the complete set of requests made so far in the script. Commit b0c4adcdd7 added a new test, which means that the "used receive-pack service" test must be updated, too. Let's fix this by making the test less brittle. We'll move it higher in the script, right after the first push has completed. And we'll clear the access log right before doing the push, so we'll see only the requests from that command. This is technically testing less, in that we won't check that all of those other requests also correctly used smart http. But there's no particular reason think that if the first one did, the others wouldn't. After this patch, running: GIT_TEST_PROTOCOL_VERSION=0 ./t5541-http-push-smart.sh passes, whereas it did not before. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5541-http-push-smart.sh | 44 ++++++++++++++++---------------------- 1 file changed, 18 insertions(+), 26 deletions(-) diff --git a/t/t5541-http-push-smart.sh b/t/t5541-http-push-smart.sh index fbad2d5ff5..ef39d14ed2 100755 --- a/t/t5541-http-push-smart.sh +++ b/t/t5541-http-push-smart.sh @@ -41,10 +41,6 @@ GET /smart/test_repo.git/info/refs?service=git-upload-pack HTTP/1.1 200 POST /smart/test_repo.git/git-upload-pack HTTP/1.1 200 EOF test_expect_success 'no empty path components' ' - # Clear the log, so that it does not affect the "used receive-pack - # service" test which reads the log too. - test_when_finished ">\"\$HTTPD_ROOT_PATH\"/access.log" && - # In the URL, add a trailing slash, and see if git appends yet another # slash. cd "$ROOT_PATH" && @@ -67,6 +63,10 @@ test_expect_success 'clone remote repository' ' ' test_expect_success 'push to remote repository (standard)' ' + # Clear the log, so that the "used receive-pack service" test below + # sees just what we did here. + >"$HTTPD_ROOT_PATH"/access.log && + cd "$ROOT_PATH"/test_repo_clone && : >path2 && git add path2 && @@ -80,6 +80,20 @@ test_expect_success 'push to remote repository (standard)' ' test $HEAD = $(git rev-parse --verify HEAD)) ' +test_expect_success 'used receive-pack service' ' + cat >exp <<-\EOF && + GET /smart/test_repo.git/info/refs?service=git-receive-pack HTTP/1.1 200 + POST /smart/test_repo.git/git-receive-pack HTTP/1.1 200 + EOF + + # NEEDSWORK: If the overspecification of the expected result is reduced, we + # might be able to run this test in all protocol versions. + if test "$GIT_TEST_PROTOCOL_VERSION" = 0 + then + check_access_log exp + fi +' + test_expect_success 'push to remote repository (standard) with sending Accept-Language' ' cat >exp <<-\EOF && => Send header: Accept-Language: ko-KR, *;q=0.9 @@ -141,28 +155,6 @@ test_expect_success 'rejected update prints status' ' ' rm -f "$HTTPD_DOCUMENT_ROOT_PATH/test_repo.git/hooks/update" -cat >exp < Date: Thu, 23 Feb 2023 05:50:36 -0500 Subject: [PATCH 02/16] t5541: stop marking "used receive-pack service" test as v0 only We have a test which checks to see if a request to git-receive-pack was made. Originally, it was checking the entire set of requests made in the script so far, including clones, and thus it would break when run with the v2 protocol (since that implies an extra request for fetches). Since the previous commit, though, we are only checking the requests made by a single push. And since there is no v2 push protocol, the test now passes no matter what's in GIT_TEST_PROTOCOL_VERSION. We can just run it all the time. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5541-http-push-smart.sh | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/t/t5541-http-push-smart.sh b/t/t5541-http-push-smart.sh index ef39d14ed2..f8bf533c33 100755 --- a/t/t5541-http-push-smart.sh +++ b/t/t5541-http-push-smart.sh @@ -86,12 +86,7 @@ test_expect_success 'used receive-pack service' ' POST /smart/test_repo.git/git-receive-pack HTTP/1.1 200 EOF - # NEEDSWORK: If the overspecification of the expected result is reduced, we - # might be able to run this test in all protocol versions. - if test "$GIT_TEST_PROTOCOL_VERSION" = 0 - then - check_access_log exp - fi + check_access_log exp ' test_expect_success 'push to remote repository (standard) with sending Accept-Language' ' From f1449a563ff78a2abfc570149a2db39c62398b68 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 05:51:35 -0500 Subject: [PATCH 03/16] t5541: simplify and move "no empty path components" test Commit 9ee6bcd398 (t5541-http-push: add test for URLs with trailing slash, 2010-04-08) added a test that clones a URL with a trailing slash, and confirms that we don't send a doubled slash (like "$url//info/refs") to the server. But this test makes no sense in t5541, which is about pushing. It should have been added in t5551. Let's move it there. But putting it at the end is tricky, since it checks the entire contents of the Apache access log. We could get around this by clearing the log before our test. But there's an even simpler solution: just make sure no doubled slashes appear in the log (fortunately, "http://" does not appear in the log itself). As a bonus, this also lets us drop the check for the v0 protocol (which is otherwise necessary since v2 makes multiple requests, and check_access_log insists on exactly matching the number of requests, even though we don't care about that here). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5541-http-push-smart.sh | 18 ------------------ t/t5551-http-fetch-smart.sh | 9 +++++++++ 2 files changed, 9 insertions(+), 18 deletions(-) diff --git a/t/t5541-http-push-smart.sh b/t/t5541-http-push-smart.sh index f8bf533c33..d0211cd8be 100755 --- a/t/t5541-http-push-smart.sh +++ b/t/t5541-http-push-smart.sh @@ -36,24 +36,6 @@ test_expect_success 'setup remote repository' ' setup_askpass_helper -cat >exp <log && + ! grep "//" log +' + test_done From a58f4d6328b71dbcaab517bf5b0b487be4029cf0 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 05:52:10 -0500 Subject: [PATCH 04/16] t5551: drop redundant grep for Accept-Language Commit b0c4adcdd7 (remote-curl: send Accept-Language header to server, 2022-07-11) added tests to make sure the header is sent via HTTP. However, it checks in two places: 1. In the expected trace output, we check verbatim for the header and its value. 2. Afterwards, we grep for the header again in the trace file. This (2) is probably cargo-culted from the earlier grep for Accept-Encoding. It is needed for the encoding because we smudge the value of that header when doing the verbatim check; see 1a53e692af (remote-curl: accept all encodings supported by curl, 2018-05-22). But we don't do so for the language header, so any problem that the "grep" would catch in (2) would already have been caught by (1). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 10b7e7cda2..29d489768e 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -103,10 +103,7 @@ test_expect_success 'clone http repository' ' test_cmp exp actual.smudged && grep "Accept-Encoding:.*gzip" actual >actual.gzip && - test_line_count = 2 actual.gzip && - - grep "Accept-Language: ko-KR, *" actual >actual.language && - test_line_count = 2 actual.language + test_line_count = 2 actual.gzip fi ' From 4a21230ab0d8b6aac4b00e9f948a34d377d2a6af Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 05:54:02 -0500 Subject: [PATCH 05/16] t5551: lower-case headers in expected curl trace There's a test in t5551 which checks the curl trace (after simplifying it a bit). It doesn't work with HTTP/2, because in that case curl outputs all of the headers in lower-case. Even though this test is run with HTTP/2 by t5559, nobody has noticed because checking the trace only happens if GIT_TEST_PROTOCOL_VERSION is manually set to "0". Let's fix this by lower-casing all of the header names in the trace, and then checking for those in our expected code (this is easier than making HTTP/2 traces look like HTTP/1.1, since HTTP/1.1 uses title-casing). Sadly, we can't quite do this in our existing sed script. This works if you have GNU sed: s/^\\([><]\\) \\([A-Za-z0-9-]*:\\)/\1 \L\2\E/ but \L is a GNU-ism, and I don't think there's a portable solution. We could just "tr A-Z a-z" on the way in, of course, but that makes the non-header parts harder to read (e.g., lowercase "post" requests). But to paraphrase Baron Munchausen, I have learned from experience that a modicum of Perl can be most efficacious. Note that this doesn't quite get the test passing with t5559; there are more fixes needed on top. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 55 ++++++++++++++++++++----------------- 1 file changed, 30 insertions(+), 25 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 29d489768e..a81f852cbf 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -35,30 +35,35 @@ setup_askpass_helper test_expect_success 'clone http repository' ' cat >exp <<-\EOF && > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1 - > Accept: */* - > Accept-Encoding: ENCODINGS - > Accept-Language: ko-KR, *;q=0.9 - > Pragma: no-cache + > accept: */* + > accept-encoding: ENCODINGS + > accept-language: ko-KR, *;q=0.9 + > pragma: no-cache < HTTP/1.1 200 OK - < Pragma: no-cache - < Cache-Control: no-cache, max-age=0, must-revalidate - < Content-Type: application/x-git-upload-pack-advertisement + < pragma: no-cache + < cache-control: no-cache, max-age=0, must-revalidate + < content-type: application/x-git-upload-pack-advertisement > POST /smart/repo.git/git-upload-pack HTTP/1.1 - > Accept-Encoding: ENCODINGS - > Content-Type: application/x-git-upload-pack-request - > Accept: application/x-git-upload-pack-result - > Accept-Language: ko-KR, *;q=0.9 - > Content-Length: xxx + > accept-encoding: ENCODINGS + > content-type: application/x-git-upload-pack-request + > accept: application/x-git-upload-pack-result + > accept-language: ko-KR, *;q=0.9 + > content-length: xxx < HTTP/1.1 200 OK - < Pragma: no-cache - < Cache-Control: no-cache, max-age=0, must-revalidate - < Content-Type: application/x-git-upload-pack-result + < pragma: no-cache + < cache-control: no-cache, max-age=0, must-revalidate + < content-type: application/x-git-upload-pack-result EOF GIT_TRACE_CURL=true GIT_TEST_PROTOCOL_VERSION=0 LANGUAGE="ko_KR.UTF-8" \ git clone --quiet $HTTPD_URL/smart/repo.git clone 2>err && test_cmp file clone/file && tr '\''\015'\'' Q / } - /^> User-Agent: /d - /^> Host: /d + /^> user-agent: /d + /^> host: /d /^> POST /,$ { /^> Accept: [*]\\/[*]/d } - s/^> Content-Length: .*/> Content-Length: xxx/ + s/^> content-length: .*/> content-length: xxx/ /^> 00..want /d /^> 00.*done/d - /^< Server: /d - /^< Expires: /d - /^< Date: /d - /^< Content-Length: /d - /^< Transfer-Encoding: /d + /^< server: /d + /^< expires: /d + /^< date: /d + /^< content-length: /d + /^< transfer-encoding: /d " >actual && # NEEDSWORK: If the overspecification of the expected result is reduced, we # might be able to run this test in all protocol versions. if test "$GIT_TEST_PROTOCOL_VERSION" = 0 then - sed -e "s/^> Accept-Encoding: .*/> Accept-Encoding: ENCODINGS/" \ + sed -e "s/^> accept-encoding: .*/> accept-encoding: ENCODINGS/" \ actual >actual.smudged && test_cmp exp actual.smudged && - grep "Accept-Encoding:.*gzip" actual >actual.gzip && + grep "accept-encoding:.*gzip" actual >actual.gzip && test_line_count = 2 actual.gzip fi ' From 8dfe36b0075b99cdd5386e85419bc3edfd2fbcd5 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 05:56:17 -0500 Subject: [PATCH 06/16] t5551: handle HTTP/2 when checking curl trace We check that the curl trace of a clone has the lines we expect, but this won't work when we run the test under t5559, because a few details are different under HTTP/2 (but nobody noticed because it only happens when you manually set GIT_TEST_PROTOCOL_VERSION to "0"). We can handle both HTTP protocols with a few tweaks: - we'll drop the HTTP "101 Switching Protocols" response, as well as various protocol upgrade headers. These details aren't interesting to us. We just want to make sure the correct protocol was used (and we do in the main request/response lines). - successful HTTP/2 responses just say "200" and not "200 OK"; we can normalize these - replace HTTP/1.1 with a variable in the request/response lines. We can use the existing $HTTP_PROTO for this, as it's already set to "HTTP/2" when appropriate. We do need to tweak the fallback value to "HTTP/1.1" to match what curl will write (prior to this patch, the fallback value didn't matter at all; we only checked if it was the literal string "HTTP/2"). Note that several lines still expect HTTP/1.1 unconditionally. The first request does so because the client requests an upgrade during the request. The POST request and response do so because you can't do an upgrade if there is a request body. (This will all be different if we trigger HTTP/2 via ALPN, but the tests aren't yet capable of that). This is enough to let: GIT_TEST_PROTOCOL_VERSION=0 ./t5559-http-fetch-smart-http2.sh pass the "clone http repository" test (but there are some other failures later on). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index a81f852cbf..716c9dbb69 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -1,6 +1,6 @@ #!/bin/sh -: ${HTTP_PROTO:=HTTP} +: ${HTTP_PROTO:=HTTP/1.1} test_description="test smart fetching over http via http-backend ($HTTP_PROTO)" GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME @@ -33,13 +33,13 @@ test_expect_success 'create http-accessible bare repository' ' setup_askpass_helper test_expect_success 'clone http repository' ' - cat >exp <<-\EOF && + cat >exp <<-EOF && > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1 > accept: */* > accept-encoding: ENCODINGS > accept-language: ko-KR, *;q=0.9 > pragma: no-cache - < HTTP/1.1 200 OK + < $HTTP_PROTO 200 OK < pragma: no-cache < cache-control: no-cache, max-age=0, must-revalidate < content-type: application/x-git-upload-pack-advertisement @@ -83,6 +83,14 @@ test_expect_success 'clone http repository' ' s/^/> / } + /^< HTTP/ { + s/200$/200 OK/ + } + /^< HTTP\\/1.1 101/d + /^[><] connection: /d + /^[><] upgrade: /d + /^> http2-settings: /d + /^> user-agent: /d /^> host: /d /^> POST /,$ { From 2f87277dfad49215ee151a787f2025d1dd74f19b Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 05:57:27 -0500 Subject: [PATCH 07/16] t5551: stop forcing clone to run with v0 protocol In the "clone http repository" test, we check the curl trace to make sure the expected requests were made. This whole script was marked to handle only the v0 protocol in d790ee1707 (tests: fix protocol version for overspecifications, 2019-02-25). That makes sense, since v2 requires an extra request, so tests as specific as this would fail unless modified. Later, in preparation for v2 becoming the default, this was tweaked by 8a1b0978ab (test: request GIT_TEST_PROTOCOL_VERSION=0 when appropriate, 2019-12-23). There we run the trace check only if the user has explicitly asked to test protocol version 0. But it also forced the clone itself to run with the v0 protocol. This makes the check for "can we expect a v0 trace" silly; it will always be v0. But much worse, it means that the clone we are testing is not like the one that normal users would run. They would use the defaults, which are now v2. And since this is supposed to be a basic check of clone-over-http, we should do the same. Let's fix this by dropping the extra v0 override. The test still passes because the trace checking only kicks in if we asked to use v0 explicitly (this is the same as before; even though we were running a v0 clone, unless you specifically set GIT_TEST_PROTOCOL_VERSION=0, the trace check was always skipped). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 716c9dbb69..4191174584 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -55,7 +55,7 @@ test_expect_success 'clone http repository' ' < content-type: application/x-git-upload-pack-result EOF - GIT_TRACE_CURL=true GIT_TEST_PROTOCOL_VERSION=0 LANGUAGE="ko_KR.UTF-8" \ + GIT_TRACE_CURL=true LANGUAGE="ko_KR.UTF-8" \ git clone --quiet $HTTPD_URL/smart/repo.git clone 2>err && test_cmp file clone/file && tr '\''\015'\'' Q Date: Thu, 23 Feb 2023 05:59:00 -0500 Subject: [PATCH 08/16] t5551: handle v2 protocol when checking curl trace After cloning an http repository, we check the curl trace to make sure the expected requests were made. But since the expected trace was never updated to handle v2, it is only run when you ask the test suite to run in v0 mode (which hardly anybody does). Let's update it to handle both protocols. This isn't too hard since v2 just sends an extra header and an extra request. So we can just annotate those extra lines and strip them out for v0 (and drop the annotations for v2). I didn't bother handling v1 here, as it's not really of practical interest (it would drop the extra v2 request, but still have the "git-protocol" lines). There's a similar tweak needed at the end. Since we check the "accept-encoding" value loosely, we grep for it rather than finding it in the verbatim trace. This grep insists that there are exactly 2 matches, but of course in v2 with the extra request there are 3. We could tweak the number, but it's simpler still to just check that we saw at least one match. The verbatim check already confirmed how many instances of the header we have; we're really just checking here that "gzip" is in the value (it's possible, of course, that the headers could have different values, but that seems like an unlikely bug). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 4191174584..9d99cefb92 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -33,12 +33,13 @@ test_expect_success 'create http-accessible bare repository' ' setup_askpass_helper test_expect_success 'clone http repository' ' - cat >exp <<-EOF && + cat >exp.raw <<-EOF && > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1 > accept: */* > accept-encoding: ENCODINGS > accept-language: ko-KR, *;q=0.9 > pragma: no-cache + {V2} > git-protocol: version=2 < $HTTP_PROTO 200 OK < pragma: no-cache < cache-control: no-cache, max-age=0, must-revalidate @@ -48,13 +49,32 @@ test_expect_success 'clone http repository' ' > content-type: application/x-git-upload-pack-request > accept: application/x-git-upload-pack-result > accept-language: ko-KR, *;q=0.9 + {V2} > git-protocol: version=2 > content-length: xxx < HTTP/1.1 200 OK < pragma: no-cache < cache-control: no-cache, max-age=0, must-revalidate < content-type: application/x-git-upload-pack-result + {V2} > POST /smart/repo.git/git-upload-pack HTTP/1.1 + {V2} > accept-encoding: ENCODINGS + {V2} > content-type: application/x-git-upload-pack-request + {V2} > accept: application/x-git-upload-pack-result + {V2} > accept-language: ko-KR, *;q=0.9 + {V2} > git-protocol: version=2 + {V2} > content-length: xxx + {V2} < HTTP/1.1 200 OK + {V2} < pragma: no-cache + {V2} < cache-control: no-cache, max-age=0, must-revalidate + {V2} < content-type: application/x-git-upload-pack-result EOF + if test "$GIT_TEST_PROTOCOL_VERSION" = 0 + then + sed "/^{V2}/d" exp + else + sed "s/^{V2} //" exp + fi && + GIT_TRACE_CURL=true LANGUAGE="ko_KR.UTF-8" \ git clone --quiet $HTTPD_URL/smart/repo.git clone 2>err && test_cmp file clone/file && @@ -107,17 +127,11 @@ test_expect_success 'clone http repository' ' /^< transfer-encoding: /d " >actual && - # NEEDSWORK: If the overspecification of the expected result is reduced, we - # might be able to run this test in all protocol versions. - if test "$GIT_TEST_PROTOCOL_VERSION" = 0 - then - sed -e "s/^> accept-encoding: .*/> accept-encoding: ENCODINGS/" \ - actual >actual.smudged && - test_cmp exp actual.smudged && + sed -e "s/^> accept-encoding: .*/> accept-encoding: ENCODINGS/" \ + actual >actual.smudged && + test_cmp exp actual.smudged && - grep "accept-encoding:.*gzip" actual >actual.gzip && - test_line_count = 2 actual.gzip - fi + grep "accept-encoding:.*gzip" actual >actual.gzip ' test_expect_success 'fetch changes via http' ' From 795d713e2c3f5da18f35958a7f1c81cc5b36e312 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 06:00:38 -0500 Subject: [PATCH 09/16] t5551: handle v2 protocol in upload-pack service test We perform a clone and a fetch, and then check that we saw the expected requests in Apache's access log. In the v2 protocol, there will be one extra request to /git-upload-pack for each operation (since the initial /info/refs probe is just used to upgrade the protocol). As a result, this test is a noop unless the use of the v0 protocol is forced. Which means that hardly anybody runs it, since you have to do so manually. Let's update it to handle v2 and run it always. We could do this by just conditionally adding in the extra POST lines. But if we look at the origin of the test in 7da4e2280c (test smart http fetch and push, 2009-10-30), the point is really just to make sure that the smart git-upload-pack service was used at all. So rather than counting up the individual requests, let's just make sure we saw each of the expected types. This is a bit looser, but makes maintenance easier. Since we're now matching with grep, we can also loosen the HTTP/1.1 match, which allows this test to pass when run with HTTP/2 via t5559. That lets: GIT_TEST_PROTOCOL_VERSION=0 ./t5559-http-fetch-smart-http2.sh run to completion, which previously failed (and of course it works if you use v2, as well). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 9d99cefb92..b912958518 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -143,19 +143,9 @@ test_expect_success 'fetch changes via http' ' ' test_expect_success 'used upload-pack service' ' - cat >exp <<-\EOF && - GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1 200 - POST /smart/repo.git/git-upload-pack HTTP/1.1 200 - GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1 200 - POST /smart/repo.git/git-upload-pack HTTP/1.1 200 - EOF - - # NEEDSWORK: If the overspecification of the expected result is reduced, we - # might be able to run this test in all protocol versions. - if test "$GIT_TEST_PROTOCOL_VERSION" = 0 - then - check_access_log exp - fi + strip_access_log >log && + grep "GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/[0-9.]* 200" log && + grep "POST /smart/repo.git/git-upload-pack HTTP/[0-9.]* 200" log ' test_expect_success 'follow redirects (301)' ' From 87d38afa0d6900fa2c3ddeba51528af9d26cd161 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 06:01:31 -0500 Subject: [PATCH 10/16] t5551: simplify expected cookie file After making an HTTP request that should store cookies, we check that the expected values are in the cookie file. We don't want to look at the whole file, because it has noisy comments at the top that we shouldn't depend on. But we strip out the interesting bits using "tail -3", which is brittle. It requires us to put an extra blank line in our expected output, and it would fail to notice any reordering or extra content in the cookie file. Instead, let's just grep for non-blank lines that are not comments, which more directly describes what we're interested in. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index b912958518..2f15a707d4 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -294,7 +294,6 @@ test_expect_success 'cookies stored in http.cookiefile when http.savecookies set 127.0.0.1 FALSE /smart_cookies/ FALSE 0 othername othervalue EOF sort >expect_cookies.txt <<-\EOF && - 127.0.0.1 FALSE /smart_cookies/ FALSE 0 othername othervalue 127.0.0.1 FALSE /smart_cookies/repo.git/info/ FALSE 0 name value EOF @@ -306,8 +305,8 @@ test_expect_success 'cookies stored in http.cookiefile when http.savecookies set # might be able to run this test in all protocol versions. if test "$GIT_TEST_PROTOCOL_VERSION" = 0 then - tail -3 cookies.txt | sort >cookies_tail.txt && - test_cmp expect_cookies.txt cookies_tail.txt + grep "^[^#]" cookies.txt | sort >cookies_stripped.txt && + test_cmp expect_cookies.txt cookies_stripped.txt fi ' From 93ea5bf3a888381cb5c2e75436c8c48fd9577f01 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 06:02:31 -0500 Subject: [PATCH 11/16] t5551: handle v2 protocol in cookie test After making a request, we check that it stored the expected cookies. This depends on the protocol version, because the cookies we store depend on the exact requests we made (and for ls-remote, v2 will always hit /git-upload-pack to get the refs, whereas v0 is happy with the initial ref advertisement). As a result, hardly anybody runs this test, as you'd have to manually set GIT_TEST_PROTOCOL_VERSION=0 to do so. Let's teach it to handle both protocol versions. One way to do this would be to make the expectation conditional on the protocol used. But there's a simpler solution. The reason that v0 doesn't hit /git-upload-pack is that ls-remote doesn't fetch any objects. If we instead do a fetch (making sure there's an actual object to grab), then both v0 and v2 will hit the same endpoints and set the same cookies. Note that we do have to clean up our new tag here; otherwise it confuses the later "clone 2,000 tags" test. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 2f15a707d4..2e42271cb8 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -295,19 +295,22 @@ test_expect_success 'cookies stored in http.cookiefile when http.savecookies set EOF sort >expect_cookies.txt <<-\EOF && 127.0.0.1 FALSE /smart_cookies/ FALSE 0 othername othervalue + 127.0.0.1 FALSE /smart_cookies/repo.git/ FALSE 0 name value 127.0.0.1 FALSE /smart_cookies/repo.git/info/ FALSE 0 name value EOF git config http.cookiefile cookies.txt && git config http.savecookies true && - git ls-remote $HTTPD_URL/smart_cookies/repo.git main && - # NEEDSWORK: If the overspecification of the expected result is reduced, we - # might be able to run this test in all protocol versions. - if test "$GIT_TEST_PROTOCOL_VERSION" = 0 - then - grep "^[^#]" cookies.txt | sort >cookies_stripped.txt && - test_cmp expect_cookies.txt cookies_stripped.txt - fi + test_when_finished " + git --git-dir=\"\$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" \ + tag -d cookie-tag + " && + git --git-dir="$HTTPD_DOCUMENT_ROOT_PATH/repo.git" \ + tag -m "foo" cookie-tag && + git fetch $HTTPD_URL/smart_cookies/repo.git cookie-tag && + + grep "^[^#]" cookies.txt | sort >cookies_stripped.txt && + test_cmp expect_cookies.txt cookies_stripped.txt ' test_expect_success 'transfer.hiderefs works over smart-http' ' From b71a2bf11fa7b2b146a88b9af5b0ae05e5796b24 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 06:05:07 -0500 Subject: [PATCH 12/16] t5551: drop curl trace lines without headers We pick apart a curl trace, looking for "=> Send header:" and so on, and matching against an expected set of requests and responses. We remove "== Info" lines entirely. However, our parser is fooled when running the test with LIB_HTTPD_SSL on Ubuntu 20.04 (as found in our linux-gcc CI job), as curl hands us an "Info" buffer with a newline, and we get: == Info: successfully set certificate verify locations: == Info: CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs => Send SSL data[...] which results in the "CApath" line ending up in the cleaned-up output, causing the test to fail. Arguably the tracing code should detect this and put it on two separate "== Info" lines. But this is actually a curl bug, fixed by their 80d73bcca (tls: provide the CApath verbose log on its own line, 2020-08-18). It's simpler to just work around it here. Since we are using GIT_TRACE_CURL, every line should just start with one of "<=", "==", or "=>", and we can throw away anything else. In fact, we can just replace the pattern for deleting "*" lines. Those were from the old GIT_CURL_VERBOSE output, but we switched over in 14e24114d9 (t5551-http-fetch-smart.sh: use the GIT_TRACE_CURL environment var, 2016-09-05). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 2e42271cb8..13b38c7ef5 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -86,7 +86,7 @@ test_expect_success 'clone http repository' ' '\'' | sed -e " s/Q\$// - /^[*] /d + /^[^<=]/d /^== Info:/d /^=> Send header, /d /^=> Send header:$/d From 9d15b1e5df56dd4409766fe8ef55752436ca9f3d Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 06:05:55 -0500 Subject: [PATCH 13/16] t/lib-httpd: respect $HTTPD_PROTO in expect_askpass() When the HTTP tests are run with LIB_HTTPD_SSL in the environment, then we access the test server as https://. This causes expect_askpass to complain, because it tries to blindly match "http://" in the prompt shown to the user. We can adjust this to use $HTTPD_PROTO, which is set during the setup phase. Note that this is enough for t5551 and t5559 to pass when run with https, but there are similar problems in other scripts that will need to be fixed before the whole suite can run with LIB_HTTPD_SSL. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/lib-httpd.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh index 608949ea80..c8fd4fc713 100644 --- a/t/lib-httpd.sh +++ b/t/lib-httpd.sh @@ -280,11 +280,11 @@ expect_askpass() { none) ;; pass) - echo "askpass: Password for 'http://$2@$dest': " + echo "askpass: Password for '$HTTPD_PROTO://$2@$dest': " ;; both) - echo "askpass: Username for 'http://$dest': " - echo "askpass: Password for 'http://$2@$dest': " + echo "askpass: Username for '$HTTPD_PROTO://$dest': " + echo "askpass: Password for '$HTTPD_PROTO://$2@$dest': " ;; *) false From 3c14419c6b2747fa0c60d8d63e5ff59b58327de3 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 06:06:44 -0500 Subject: [PATCH 14/16] t/lib-httpd: enable HTTP/2 "h2" protocol, not just h2c Commit 73c49a4474 (t: run t5551 tests with both HTTP and HTTP/2, 2022-11-11) added Apache config to enable HTTP/2. However, it only enabled the "h2c" protocol, which allows cleartext HTTP/2 (generally based on an upgrade header during an HTTP/1.1 request). This is what t5559 is generally testing, since by default we don't set up SSL/TLS. However, it should be possible to run t5559 with LIB_HTTPD_SSL set. In that case, Apache will advertise support for HTTP/2 via ALPN during the TLS handshake. But we need to tell it support "h2" (the non-cleartext version) to do so. Without that, then curl does not even try to do the HTTP/1.1 upgrade (presumably because after seeing that we did TLS but didn't get the ALPN indicator, it assumes it would be fruitless). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/lib-httpd/apache.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf index 0294739a77..6d5d66caf8 100644 --- a/t/lib-httpd/apache.conf +++ b/t/lib-httpd/apache.conf @@ -31,7 +31,7 @@ ErrorLog error.log LoadModule http2_module modules/mod_http2.so -Protocols h2c +Protocols h2 h2c From 86190028a813786bb8f92a93ab07b44ac5f005a1 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 06:07:29 -0500 Subject: [PATCH 15/16] t5559: fix test failures with LIB_HTTPD_SSL One test needs to be tweaked in order for t5559 to pass with SSL/TLS set up. When we make our initial clone, we check that the curl trace of requests is what we expected. But we need to fix two things: - along with ignoring "data" lines from the trace, we need to ignore "SSL data" lines - when TLS is used, the server is able to tell the client (via ALPN) that it supports HTTP/2 before the first HTTP request is made. So rather than request an upgrade using an HTTP header, it can just speak HTTP/2 immediately With this patch, running: LIB_HTTPD_SSL=1 ./t5559-http-fetch-smart-http2.sh works, whereas it did not before. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5551-http-fetch-smart.sh | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 13b38c7ef5..0908534f25 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -33,8 +33,19 @@ test_expect_success 'create http-accessible bare repository' ' setup_askpass_helper test_expect_success 'clone http repository' ' + if test_have_prereq HTTP2 && test "$HTTPD_PROTO" = "https" + then + # ALPN lets us immediately use HTTP/2; likewise, POSTs with + # bodies can use it because they do not need to upgrade + INITIAL_PROTO=HTTP/2 + else + # either we are not using HTTP/2, or the initial + # request is sent via HTTP/1.1 and asks for upgrade + INITIAL_PROTO=HTTP/1.1 + fi && + cat >exp.raw <<-EOF && - > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1 + > GET /smart/repo.git/info/refs?service=git-upload-pack $INITIAL_PROTO > accept: */* > accept-encoding: ENCODINGS > accept-language: ko-KR, *;q=0.9 @@ -44,25 +55,25 @@ test_expect_success 'clone http repository' ' < pragma: no-cache < cache-control: no-cache, max-age=0, must-revalidate < content-type: application/x-git-upload-pack-advertisement - > POST /smart/repo.git/git-upload-pack HTTP/1.1 + > POST /smart/repo.git/git-upload-pack $INITIAL_PROTO > accept-encoding: ENCODINGS > content-type: application/x-git-upload-pack-request > accept: application/x-git-upload-pack-result > accept-language: ko-KR, *;q=0.9 {V2} > git-protocol: version=2 > content-length: xxx - < HTTP/1.1 200 OK + < $INITIAL_PROTO 200 OK < pragma: no-cache < cache-control: no-cache, max-age=0, must-revalidate < content-type: application/x-git-upload-pack-result - {V2} > POST /smart/repo.git/git-upload-pack HTTP/1.1 + {V2} > POST /smart/repo.git/git-upload-pack $INITIAL_PROTO {V2} > accept-encoding: ENCODINGS {V2} > content-type: application/x-git-upload-pack-request {V2} > accept: application/x-git-upload-pack-result {V2} > accept-language: ko-KR, *;q=0.9 {V2} > git-protocol: version=2 {V2} > content-length: xxx - {V2} < HTTP/1.1 200 OK + {V2} < $INITIAL_PROTO 200 OK {V2} < pragma: no-cache {V2} < cache-control: no-cache, max-age=0, must-revalidate {V2} < content-type: application/x-git-upload-pack-result @@ -96,6 +107,8 @@ test_expect_success 'clone http repository' ' s/= Recv header:// /^<= Recv data/d /^=> Send data/d + /^<= Recv SSL data/d + /^=> Send SSL data/d /^$/d /^< $/d From 8f2146dbf15566fa60787a3261a048e4d5116d6a Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 23 Feb 2023 06:08:41 -0500 Subject: [PATCH 16/16] t5559: make SSL/TLS the default The point of t5559 is run the regular t5551 tests with HTTP/2. But it does so with the "h2c" protocol, which uses cleartext upgrades from HTTP/1.1 to HTTP/2 (rather than learning about HTTP/2 support during the TLS negotiation). This has a few problems: - it's not very indicative of the real world. In practice, most servers that support HTTP/2 will also support TLS. - support for upgrading does not seem as robust. In particular, we've run into bugs in some versions of Apache's mod_http2 that trigger only with the upgrade mode. See: https://lore.kernel.org/git/Y8ztIqYgVCPILJlO@coredump.intra.peff.net/ So the upside is that this change makes our HTTP/2 tests more robust and more realistic. The downside is that if we can't set up SSL for any reason, we'll skip the tests (even though you _might_ have been able to run the HTTP/2 tests the old way). We could probably have a conditional fallback, but it would be complicated for little gain, and it's not even clear it would help (i.e., would any test environment even have HTTP/2 but not SSL support?). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/t5559-http-fetch-smart-http2.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/t/t5559-http-fetch-smart-http2.sh b/t/t5559-http-fetch-smart-http2.sh index 9eece71c2c..54aa9d3bff 100755 --- a/t/t5559-http-fetch-smart-http2.sh +++ b/t/t5559-http-fetch-smart-http2.sh @@ -1,4 +1,5 @@ #!/bin/sh HTTP_PROTO=HTTP/2 +LIB_HTTPD_SSL=1 . ./t5551-http-fetch-smart.sh