Data race when terminating the progress #27643

Lantern-r · 2023-10-16T12:14:20Z

Description

lantern@SaaS:/Users/lantern/Documents/GitHub/gitea$ ./gitea web
2023/10/16 20:05:22 cmd/web.go:242:runWeb() [I] Starting Gitea on PID: 211
2023/10/16 20:05:22 cmd/web.go:111:showWebStartupMessage() [I] Gitea version: development built with go1.21.3
2023/10/16 20:05:22 cmd/web.go:112:showWebStartupMessage() [I] * RunMode: prod
2023/10/16 20:05:22 cmd/web.go:113:showWebStartupMessage() [I] * AppPath: /Users/lantern/Documents/GitHub/gitea/gitea_race
2023/10/16 20:05:22 cmd/web.go:114:showWebStartupMessage() [I] * WorkPath: /Users/lantern/Documents/GitHub/gitea
2023/10/16 20:05:22 cmd/web.go:115:showWebStartupMessage() [I] * CustomPath: /Users/lantern/Documents/GitHub/gitea/custom
2023/10/16 20:05:22 cmd/web.go:116:showWebStartupMessage() [I] * ConfigFile: /Users/lantern/Documents/GitHub/gitea/custom/conf/app.ini
2023/10/16 20:05:22 cmd/web.go:117:showWebStartupMessage() [I] Prepare to run install page
2023/10/16 20:05:23 cmd/web.go:304:listen() [I] Listen: http://0.0.0.0:3000
2023/10/16 20:05:23 cmd/web.go:308:listen() [I] AppURL(ROOT_URL): http://localhost:3000/
2023/10/16 20:05:23 ...s/graceful/server.go:70:NewServer() [I] Starting new Web server: tcp:0.0.0.0:3000 on PID: 211
^C2023/10/16 20:05:25 ...eful/manager_unix.go:195:handleSignals() [W] PID 211. Received SIGINT. Shutting down...
2023/10/16 20:05:25 ...eful/server_hooks.go:46:doShutdown() [I] PID: 211 Listener ([::]:3000) closed.
==================
WARNING: DATA RACE
Write at 0x00c0000b0be0 by goroutine 38:
  runtime.racewrite()
      <autogenerated>:1 +0x1e
  code.gitea.io/gitea/modules/graceful.(*Manager).WaitForServers()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/manager.go:232 +0x56
  code.gitea.io/gitea/modules/graceful.(*Manager).doShutdown.func1()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/manager.go:150 +0x4e

Previous read at 0x00c0000b0be0 by main goroutine:
  runtime.raceread()
      <autogenerated>:1 +0x1e
  code.gitea.io/gitea/modules/graceful.(*Manager).RegisterServer()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/manager_unix.go:282 +0x54
  code.gitea.io/gitea/modules/graceful.(*Server).Serve()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/server.go:174 +0xe9
  code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/server.go:115 +0x4d8
  code.gitea.io/gitea/modules/graceful.HTTPListenAndServe()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/server_http.go:32 +0x9d
  code.gitea.io/gitea/cmd.runHTTP()
      /Users/lantern/Documents/GitHub/gitea/cmd/web_graceful.go:18 +0xa33
  code.gitea.io/gitea/cmd.listen()
      /Users/lantern/Documents/GitHub/gitea/cmd/web.go:320 +0x9e5
  code.gitea.io/gitea/cmd.serveInstall()
      /Users/lantern/Documents/GitHub/gitea/cmd/web.go:137 +0x10c
  code.gitea.io/gitea/cmd.runWeb()
      /Users/lantern/Documents/GitHub/gitea/cmd/web.go:251 +0x313
  code.gitea.io/gitea/cmd.prepareSubcommandWithConfig.prepareWorkPathAndCustomConf.func1()
      /Users/lantern/Documents/GitHub/gitea/cmd/main.go:117 +0x28d
  github.com/urfave/cli/v2.(*Command).Run()
      /home/lantern/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:274 +0x1334
  github.com/urfave/cli/v2.(*Command).Run()
      /home/lantern/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0x120b
  github.com/urfave/cli/v2.(*App).RunContext()
      /home/lantern/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:332 +0x1249
  github.com/urfave/cli/v2.(*App).Run()
      /home/lantern/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:309 +0x64
  code.gitea.io/gitea/cmd.RunMainApp()
      /Users/lantern/Documents/GitHub/gitea/cmd/main.go:176 +0x36
  main.main()
      /Users/lantern/Documents/GitHub/gitea/main.go:46 +0xb5

Goroutine 38 (running) created at:
  code.gitea.io/gitea/modules/graceful.(*Manager).doShutdown()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/manager.go:149 +0x4f3
  code.gitea.io/gitea/modules/graceful.(*Manager).DoGracefulShutdown()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/manager_unix.go:273 +0xf2
  code.gitea.io/gitea/modules/graceful.(*Manager).handleSignals()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/manager_unix.go:196 +0x6a4
  code.gitea.io/gitea/modules/graceful.(*Manager).start.func4()
      /Users/lantern/Documents/GitHub/gitea/modules/graceful/manager_unix.go:113 +0x4f
==================
2023/10/16 20:05:25 cmd/web.go:355:listen() [I] HTTP Listener: 0.0.0.0:3000 Closed
2023/10/16 20:05:25 .../graceful/manager.go:168:doHammerTime() [W] Setting Hammer condition
2023/10/16 20:05:26 .../graceful/manager.go:184:doTerminate() [W] Terminating
2023/10/16 20:05:26 ...eful/manager_unix.go:208:handleSignals() [W] PID: 211. Background context for manager closed - context canceled - Shutting down...
2023/10/16 20:05:26 cmd/web.go:145:serveInstall() [I] PID: 211 Gitea Web Finished

git brach: main

lantern@SaaS:/Users/lantern/Documents/GitHub/gitea$ git branch
* main

possible fix:

gitea/modules/graceful/manager.go:232

func (g *Manager) WaitForServers() {
	g.lock.Lock() // add lock and unlock
	defer g.lock.Unlock()
	g.runningServerWaitGroup.Wait()
}

gitea/modules/graceful/manager_unix.go:280

func (g *Manager) RegisterServer() {
	KillParent()
	g.lock.Lock() // add lock and lock
	defer g.lock.Unlock()
	g.runningServerWaitGroup.Add(1)
}

Gitea Version

Gitea version development built with go1.21.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

ubuntu 22.04

./gitea web

Database

None

The text was updated successfully, but these errors were encountered:

The graceful manager has some bugs (#27643, #28062). This is a preparation for further fixes.

The graceful manager has some bugs (go-gitea#27643, go-gitea#28062). This is a preparation for further fixes.

Shea690901 · 2024-02-24T07:03:35Z

I don't know if it is the same or 'just' a related Problem, but I can't even start the server without getting this message.
It is a fresh install using Postgres as backend and crashes before it generates any table:

gitea[1319990]: 2024/02/23 03:59:43 cmd/web.go:111:showWebStartupMessage() [I] Gitea version: 1.21.1 built with GNU Make 4.4.1, go1.21.4 : bindata,pam,sqlite,sqlite_unlock_notify
gitea[1319990]: 2024/02/23 03:59:43 cmd/web.go:112:showWebStartupMessage() [I] * RunMode: prod                                                                               
gitea[1319990]: 2024/02/23 03:59:43 cmd/web.go:113:showWebStartupMessage() [I] * AppPath: /usr/bin/gitea                                                                     
gitea[1319990]: 2024/02/23 03:59:43 cmd/web.go:114:showWebStartupMessage() [I] * WorkPath: /var/lib/gitea                                                                    
gitea[1319990]: 2024/02/23 03:59:43 cmd/web.go:115:showWebStartupMessage() [I] * CustomPath: /var/lib/gitea/custom                                                           
gitea[1319990]: 2024/02/23 03:59:43 cmd/web.go:116:showWebStartupMessage() [I] * ConfigFile: /etc/gitea/app.ini                                                              
gitea[1319990]: 2024/02/23 03:59:43 cmd/web.go:117:showWebStartupMessage() [I] Prepare to run install page                                                                   
gitea[1319990]: panic: sync: WaitGroup is reused before previous Wait has returned                                                                                           
systemd[1]: gitea.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

This happens both with 1.21.1 (latest package of my distro [gentoo]) and 1.21.6....

wxiaoguang · 2024-02-24T07:08:40Z

Could you share your steps? Including the environment variables & config files, etc.

ps: would you like try docker? Usually I recommend to use docker if it is possible.

Shea690901 · 2024-02-24T22:22:56Z

install package
edit /etc/gitea/app.ini
create database
systemctl start gitea

app.ini

wxiaoguang · 2024-02-25T01:55:52Z

edit /etc/gitea/app.ini

I think it's caused by the incorrect config. Please refer to your logs, there should be some error logs. ps: in some cause, if the panic occurs too quickly, maybe some logs could be missing.

I would suggest to empty the app.ini and try to start a fresh installation.

For example, on my side, when error occurs:

2024/02/25 09:52:36 cmd/web.go:304:listen() [I] Listen: https://[::]:3001
2024/02/25 09:52:36 cmd/web.go:308:listen() [I] AppURL(ROOT_URL): https://gitea.soleil.ddnsfree.com:3001/
2024/02/25 09:52:36 cmd/web_https.go:92:toTLSCiphers() [W] Unknown cipher: ALL:!DH:!kRSA:!SRP:!kDHd:!DSS:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK:!RC4:!ADH:!LOW@STRENGTH
2024/02/25 09:52:36 cmd/web_https.go:170:runHTTPS() [E] Failed to load https cert file /etc/letsencrypt/live/gitea.soleil.ddnsfree.com/fullchain.pem for tcp:[::]:3001: open /etc/letsencrypt/live/gitea.soleil.ddnsfree.com/fullchain.pem: no such file or directory
2024/02/25 09:52:36 cmd/web.go:353:listen() [E] Failed to start server: open /etc/letsencrypt/live/gitea.soleil.ddnsfree.com/fullchain.pem: no such file or directory
2024/02/25 09:52:36 cmd/web.go:355:listen() [I] HTTP Listener: [::]:3001 Closed
2024/02/25 09:52:36 cmd/web.go:139:serveInstall() [E] Unable to open listener for installer. Is Gitea already running?
panic: sync: WaitGroup is reused before previous Wait has returned

Shea690901 · 2024-03-05T18:23:22Z

The warning concerning "Unknown cipher" looks interesting, would be nice if somewhere within the docs I could read how I have to put this....
But since it's 'only' a warning it shouldn't trigger any problems concerning WaitGroup...

And the error concerning the certificates doesn't happen for me, because obviously I have those files with the correct access rights at the specified locations....

wxiaoguang · 2024-03-06T00:10:38Z

Yup, it might not be related to "certificates", nor "Unknown cipher" , but I think it could be related some "incorrect config" (#27643 (comment)). Actually I think there is a bug when running the "install" page when there are errors in the config file.

I would still suggest you to try to empty the app.ini and try to start a fresh installation, to see whether the problem still exists.

wxiaoguang · 2024-03-06T10:19:28Z

And a temp fix (for the "panic"): -> Avoid unexpected panic in graceful manager #29629

There is a fundamental design problem of the "manager" and the "wait group". If nothing has started, the "Wait" just panics: sync: WaitGroup is reused before previous Wait has returned There is no clear solution besides a complete rewriting of the "manager" If there are some mistakes in the app.ini, end users would just see the "panic", but not the real error messages. A real case: #27643 This PR is just a quick fix for the annoying panic problem.

Shea690901 · 2024-03-10T12:33:27Z

I'll try the fix as soon as I have hoghspeedvolume avaliable again....
If I have truly any errors in my config it will hopefully give me some hints, since now I don't get any warning/error concerning my config!!!
Also IF there are any config errors (aside of typos) it is clearly the fault of the documentation, since I edited my app.ini according what I found there....

wxiaoguang · 2024-05-07T13:13:19Z

It should have been fixed by : Refactor graceful manager, fix misused WaitGroup (#29738)

Feel free to reopen if there is still any problem.

Lantern-r added the type/bug label Oct 16, 2023

Lantern-r changed the title ~~Data Race When Terminate Progress~~ Data race when terminating the progress Oct 16, 2023

This was referenced Nov 15, 2023

Force shutting down panic #28062

Closed

Improve graceful manager code/comment #28063

Merged

wxiaoguang added a commit that referenced this issue Nov 15, 2023

Improve graceful manager code/comment (#28063)

79394b3

The graceful manager has some bugs (#27643, #28062). This is a preparation for further fixes.

fuxiaohei pushed a commit to fuxiaohei/gitea that referenced this issue Jan 17, 2024

Improve graceful manager code/comment (go-gitea#28063)

674788b

The graceful manager has some bugs (go-gitea#27643, go-gitea#28062). This is a preparation for further fixes.

silverwind pushed a commit to silverwind/gitea that referenced this issue Feb 20, 2024

Improve graceful manager code/comment (go-gitea#28063)

b4eec51

The graceful manager has some bugs (go-gitea#27643, go-gitea#28062). This is a preparation for further fixes.

lunny added the issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail label Feb 25, 2024

wxiaoguang removed the issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail label Mar 6, 2024

wxiaoguang mentioned this issue Mar 6, 2024

Avoid unexpected panic in graceful manager #29629

Merged

wxiaoguang closed this as completed May 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data race when terminating the progress #27643

Data race when terminating the progress #27643

Lantern-r commented Oct 16, 2023

Shea690901 commented Feb 24, 2024

wxiaoguang commented Feb 24, 2024 •

edited

Shea690901 commented Feb 24, 2024

wxiaoguang commented Feb 25, 2024 •

edited

Shea690901 commented Mar 5, 2024

wxiaoguang commented Mar 6, 2024

wxiaoguang commented Mar 6, 2024 •

edited

Shea690901 commented Mar 10, 2024

wxiaoguang commented May 7, 2024

Data race when terminating the progress #27643

Data race when terminating the progress #27643

Comments

Lantern-r commented Oct 16, 2023

Description

Gitea Version

Can you reproduce the bug on the Gitea demo site?

Log Gist

Screenshots

Git Version

Operating System

How are you running Gitea?

Database

Shea690901 commented Feb 24, 2024

wxiaoguang commented Feb 24, 2024 • edited

Shea690901 commented Feb 24, 2024

wxiaoguang commented Feb 25, 2024 • edited

Shea690901 commented Mar 5, 2024

wxiaoguang commented Mar 6, 2024

wxiaoguang commented Mar 6, 2024 • edited

Shea690901 commented Mar 10, 2024

wxiaoguang commented May 7, 2024

wxiaoguang commented Feb 24, 2024 •

edited

wxiaoguang commented Feb 25, 2024 •

edited

wxiaoguang commented Mar 6, 2024 •

edited