Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

embedded ssh won't clone on 1.19.3 #24981

Closed
g-pechorin opened this issue May 29, 2023 · 8 comments · Fixed by #25330
Closed

embedded ssh won't clone on 1.19.3 #24981

g-pechorin opened this issue May 29, 2023 · 8 comments · Fixed by #25330
Labels
issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail

Comments

@g-pechorin
Copy link

Description

When using docker+alpine+arm6, and the embedded SSHD server, everything works up until trying to cone repository.

Running git clone from the embedded ssh server just stalls "forever"

C:\Users\peter\Desktop\fff
λ git clone ssh://bulloch@raspberrypi:5190/peter/asda.git
Cloning into 'asda'...

Gitea Version

1.19.3-0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

docker+alpine atop rasbian

How are you running Gitea?

docker+alpine atop rasbian

Database

SQLite

@wxiaoguang
Copy link
Contributor

wxiaoguang commented May 29, 2023

Could you help to run ssh for your Gitea server and share the full logs?

Like this:

ssh -vvv GitUser@GiteaServer -p GiteaSshPort git-receive-pack TheOwner/TheRepo.git

# for github (demo):
ssh -vvv git@github.com -p 22 git-receive-pack wxiaoguang/playground.git

Normally you could see some verbose logs with git response:

OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/user/.ssh/config
debug: ...
debug: ...
debug: ...
debug: ...
00f18cb091cec4b877236cd5cfa608180e34e55e5c6c refs/heads/mainreport-status report-status-v2 delete-refs side-band-64k ofs-delta atomic push-options object-format=sha1 agent=github/spokes-receive-pack-14212a8c32379761bfb7c8707291458d5acdd11e
004773cb36c2467b0933ec9fec49b626fc65709e647d refs/heads/test-highlight
004b9d6c9b556a98e306c32270ecd113f4ca93e3e992 refs/heads/wxiaoguang-patch-1
004bbc66ce44b58da391065c864905f5cb97f801c8b6 refs/heads/wxiaoguang-patch-2
0000
(then Ctrl+C to stop the test)

@g-pechorin
Copy link
Author

that gives a lot of interesting output!

... I'm not sure what it means ... here's the file though

gites-vvv-log.txt

@g-pechorin
Copy link
Author

and here it is as "not an attachement" sorry

λ ssh -vvv bulloch@raspberrypi -p 5191 git-receive-pack peter/asda.git
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug3: Failed to open file:C:/Users/peter/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolving "raspberrypi" port 5191
debug2: ssh_connect_direct
debug1: Connecting to raspberrypi [192.168.0.3] port 5191.
debug1: Connection established.
debug1: identity file C:\\Users\\peter/.ssh/id_rsa type 0
debug3: Failed to open file:C:/Users/peter/.ssh/id_rsa-cert error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_rsa-cert.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_rsa-cert type -1
debug3: Failed to open file:C:/Users/peter/.ssh/id_dsa error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_dsa.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_dsa type -1
debug3: Failed to open file:C:/Users/peter/.ssh/id_dsa-cert error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_dsa-cert.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_dsa-cert type -1
debug3: Failed to open file:C:/Users/peter/.ssh/id_ecdsa error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_ecdsa.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_ecdsa type -1
debug3: Failed to open file:C:/Users/peter/.ssh/id_ecdsa-cert error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_ecdsa-cert.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_ecdsa-cert type -1
debug3: Failed to open file:C:/Users/peter/.ssh/id_ed25519 error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_ed25519.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_ed25519 type -1
debug3: Failed to open file:C:/Users/peter/.ssh/id_ed25519-cert error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_ed25519-cert.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_ed25519-cert type -1
debug3: Failed to open file:C:/Users/peter/.ssh/id_xmss error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_xmss.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_xmss type -1
debug3: Failed to open file:C:/Users/peter/.ssh/id_xmss-cert error:2
debug3: Failed to open file:C:/Users/peter/.ssh/id_xmss-cert.pub error:2
debug1: identity file C:\\Users\\peter/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to raspberrypi:5191 as 'bulloch'
debug3: put_host_port: [raspberrypi]:5191
debug3: hostkeys_foreach: reading file "C:\\Users\\peter/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file C:\\Users\\peter/.ssh/known_hosts:15
debug3: load_hostkeys: loaded 1 keys from [raspberrypi]:5191
debug3: Failed to open file:C:/Users/peter/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-256,rsa-sha2-512,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:UQcQm5a2QKkdfwc+UpSQlgsbfc18qb5ttFwllC/yRzQ
debug3: put_host_port: [192.168.0.3]:5191
debug3: put_host_port: [raspberrypi]:5191
debug3: hostkeys_foreach: reading file "C:\\Users\\peter/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file C:\\Users\\peter/.ssh/known_hosts:15
debug3: load_hostkeys: loaded 1 keys from [raspberrypi]:5191
debug3: Failed to open file:C:/Users/peter/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: hostkeys_foreach: reading file "C:\\Users\\peter/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file C:\\Users\\peter/.ssh/known_hosts:14
debug3: load_hostkeys: loaded 1 keys from [192.168.0.3]:5191
debug3: Failed to open file:C:/Users/peter/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug1: Host '[raspberrypi]:5191' is known and matches the RSA host key.
debug1: Found key in C:\\Users\\peter/.ssh/known_hosts:15
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug3: unable to connect to pipe \\\\.\\pipe\\openssh-ssh-agent, error: 2
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\peter/.ssh/id_rsa RSA SHA256:zQB6Pf3IfravQ1CYpTVgmT8zHCIrYggs4wREHwVHXCA
debug1: Will attempt key: C:\\Users\\peter/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\peter/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\peter/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\peter/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: C:\\Users\\peter/.ssh/id_rsa RSA SHA256:zQB6Pf3IfravQ1CYpTVgmT8zHCIrYggs4wREHwVHXCA
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: C:\\Users\\peter/.ssh/id_rsa RSA SHA256:zQB6Pf3IfravQ1CYpTVgmT8zHCIrYggs4wREHwVHXCA
debug3: sign_and_send_pubkey: RSA SHA256:zQB6Pf3IfravQ1CYpTVgmT8zHCIrYggs4wREHwVHXCA
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to raspberrypi ([192.168.0.3]:5191).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug1: Sending command: git-receive-pack peter/asda.git
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
2023/05/29 10:46:46 ...s/setting/setting.go:99:getWorkPath() [I] Provided work path  is not absolute - will be made absolute against the current working directory
00be48aca939a24131c606cafe64abb6c66d524d8cd5 refs/heads/main

@wxiaoguang
Copy link
Contributor

wxiaoguang commented May 29, 2023

Your Gitea & SSH server just works.

For the problem itself, I can see 2 suspicious points here:

  1. There is no 0000 line in your log (see my demo output above, there is one), the 0000 means that the first round-trip finishes. Did you miss that line in your log but that line really existed? Or that line really didn't exist in your output?

  2. The log: "2023/05/29 10:46:46 ...s/setting/setting.go:99:getWorkPath() [I] Provided work path is not absolute - will be made absolute against the current working directory". It seems that you didn't set your GITEA_WORK_DIR environment variable correctly. How do you run Gitea? If you could provide a minimal reproducible setup (with detailed steps), I can try to reproduce it on my side. If I can reproduce, then I can try to figure out.

If you build your own Docker image, you need to follow the Gitea's docker mechisam strictly:

@wxiaoguang wxiaoguang added issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail and removed type/bug labels May 29, 2023
@wxiaoguang
Copy link
Contributor

Hi g-pechorin, has the problem been solved?

@g-pechorin
Copy link
Author

g-pechorin commented Jun 2, 2023 via email

@wxiaoguang
Copy link
Contributor

wxiaoguang commented Jun 18, 2023

Reproduced the bug.

It might be caused by #19815

You can try to set [server].APP_DATA_PATH to an absolute path and set correct GITEA_WORK_DIR in your environment.


Any log message printed during the git protocol would cause the protocol broken.

I think there are still more similar cases, while they need to be cleaned one by one.

@lunny lunny closed this as completed in 2cdf260 Jun 21, 2023
wxiaoguang added a commit to wxiaoguang/gitea that referenced this issue Jun 21, 2023
# The problem

There were many "path tricks":

* By default, Gitea uses its program directory as its work path
* Gitea tries to use the "work path" to guess its "custom path" and
"custom conf (app.ini)"
* Users might want to use other directories as work path
* The non-default work path should be passed to Gitea by GITEA_WORK_DIR
or "--work-path"
* But some Gitea processes are started without these values
    * The "serv" process started by OpenSSH server
    * The CLI sub-commands started by site admin
* The paths are guessed by SetCustomPathAndConf again and again
* The default values of "work path / custom path / custom conf" can be
changed when compiling

# The solution

* Use `InitWorkPathAndCommonConfig` to handle these path tricks, and use
test code to cover its behaviors.
* When Gitea's web server runs, write the WORK_PATH to "app.ini", this
value must be the most correct one, because if this value is not right,
users would find that the web UI doesn't work and then they should be
able to fix it.
* Then all other sub-commands can use the WORK_PATH in app.ini to
initialize their paths.
* By the way, when Gitea starts for git protocol, it shouldn't output
any log, otherwise the git protocol gets broken and client blocks
forever.

The "work path" priority is: WORK_PATH in app.ini > cmd arg --work-path
> env var GITEA_WORK_DIR > builtin default

The "app.ini" searching order is: cmd arg --config > cmd arg "work path
/ custom path" > env var "work path / custom path" > builtin default

## ⚠️ BREAKING

If your instance's "work path / custom path / custom conf" doesn't meet
the requirements (eg: work path must be absolute), Gitea will report a
fatal error and exit. You need to set these values according to the
error log.

----

Close go-gitea#24818
Close go-gitea#24222
Close go-gitea#21606
Close go-gitea#21498
Close go-gitea#25107
Close go-gitea#24981
Maybe close go-gitea#24503

Replace go-gitea#23301
Replace go-gitea#22754

And maybe more
# Conflicts:
#	cmd/web.go
@g-pechorin
Copy link
Author

I need to try this (had an unrelated hardware failure)

silverwind pushed a commit that referenced this issue Jun 28, 2023
More fix for #24981

* #24981


Close #22361

* #22361

There were many patches for Gitea's sub-commands to satisfy the facts:

* Some sub-commands shouldn't output any log, otherwise the git protocol
would be broken
* Sometimes the users want to see "verbose" or "quiet" outputs

That's a longstanding problem, and very fragile. This PR is only a quick
patch for the problem.

In the future, the sub-command system should be refactored to a clear
solution.

----

Other changes:

* Use `ReplaceAllWriters` to replace
`RemoveAllWriters().AddWriters(writer)`, then it's an atomic operation.
* Remove unnecessary `syncLevelInternal` calls, because
`AddWriters/addWritersInternal` already calls it.

Co-authored-by: Giteabot <teabot@gitea.io>
wxiaoguang added a commit to wxiaoguang/gitea that referenced this issue Jun 28, 2023
More fix for go-gitea#24981

* go-gitea#24981


Close go-gitea#22361

* go-gitea#22361

There were many patches for Gitea's sub-commands to satisfy the facts:

* Some sub-commands shouldn't output any log, otherwise the git protocol
would be broken
* Sometimes the users want to see "verbose" or "quiet" outputs

That's a longstanding problem, and very fragile. This PR is only a quick
patch for the problem.

In the future, the sub-command system should be refactored to a clear
solution.

----

Other changes:

* Use `ReplaceAllWriters` to replace
`RemoveAllWriters().AddWriters(writer)`, then it's an atomic operation.
* Remove unnecessary `syncLevelInternal` calls, because
`AddWriters/addWritersInternal` already calls it.

Co-authored-by: Giteabot <teabot@gitea.io>
lunny pushed a commit that referenced this issue Jun 28, 2023
Backport #25537

More fix for #24981

* #24981

Close #22361, #25552

* #22361
* #25552

There were many patches for Gitea's sub-commands to satisfy the facts:

* Some sub-commands shouldn't output any log, otherwise the git protocol
would be broken
* Sometimes the users want to see "verbose" or "quiet" outputs

That's a longstanding problem, and very fragile. This PR is only a quick
patch for the problem.

In the future, the sub-command system should be refactored to a clear
solution.

----

Other changes:

* Use `ReplaceAllWriters` to replace
`RemoveAllWriters().AddWriters(writer)`, then it's an atomic operation.
* Remove unnecessary `syncLevelInternal` calls, because
`AddWriters/addWritersInternal` already calls it.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants