Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pause queues #15928

Merged
merged 27 commits into from Jan 22, 2022
Merged

Pause queues #15928

merged 27 commits into from Jan 22, 2022

Conversation

zeripath
Copy link
Contributor

@zeripath zeripath commented May 19, 2021

Add functionality to pause and resume queues with the ability to hand back unhandled data.

This also now alerts if a queue configuration will result in no workers and no worker scalling (as in #18189) - and will allow no-worker configurations on redis to not result in loss from the queue.

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label May 19, 2021
@zeripath zeripath added this to the 1.15.0 milestone May 19, 2021
@zeripath
Copy link
Contributor Author

FAO @lafriks

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label May 19, 2021
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
…eues

Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath and others added 2 commits May 20, 2021 21:32
@lafriks
Copy link
Member

lafriks commented May 20, 2021

From code point of view looks good to me, will test it on weekend together with my WIP PR

@codecov-commenter

This comment has been minimized.

@zeripath zeripath modified the milestones: 1.15.0, 1.16.0 Jun 23, 2021
@zeripath zeripath mentioned this pull request Jul 14, 2021
@lafriks
Copy link
Member

lafriks commented Jul 23, 2021

Build seems to be failing now

@zeripath
Copy link
Contributor Author

yeah I must have made a broken merge when I fixed the last conflict.

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath
Copy link
Contributor Author

fixed

zeripath and others added 3 commits July 23, 2021 16:55
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jan 21, 2022
@zeripath
Copy link
Contributor Author

As an additional thing we can now add warnings for no workers and finally safely handle the case of no worker configurations for redis.

…redis/levelqueue

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath mentioned this pull request Jan 21, 2022
Copy link
Contributor

@wxiaoguang wxiaoguang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, while maybe util.StopTimer(timer) can be used to reduce some code.

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Jan 22, 2022
@lafriks
Copy link
Member

lafriks commented Jan 22, 2022

🚀

@zeripath
Copy link
Contributor Author

Make lgtm work

@zeripath zeripath merged commit a82fd98 into go-gitea:main Jan 22, 2022
@zeripath zeripath deleted the pause-queues branch January 22, 2022 22:45
zjjhot pushed a commit to zjjhot/gitea that referenced this pull request Jan 23, 2022
* 'main' of https://github.com/go-gitea/gitea:
  Add packagist webhook (go-gitea#18224)
  Fix mime-type detection for HTTP server (go-gitea#18370)
  Always use git command but not os.Command (go-gitea#18363)
  Add deprecated for LFS_CONTENT_PATH on zh-cn docs (go-gitea#18362)
  [skip ci] Updated translations via Crowdin
  Make gitea, gitea-vet future-proof (go-gitea#18361)
  Pause queues (go-gitea#15928)
  Disable content sniffing on `PlainTextBytes` (go-gitea#18359)
  Update github.com/duo-labs/webauthn (go-gitea#18357)
zeripath added a commit that referenced this pull request Jan 24, 2022
Implements the Pausable interface on WrappedQueues and PersistableChannelUniqueQueues

Reference #15928

Signed-off-by: Andrew Thornton art27@cantab.net
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
* Start adding mechanism to return unhandled data

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Create pushback interface

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add Pausable interface to WorkerPool and Manager

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Implement Pausable and PushBack for the bytefifos

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Wire in UI for pausing

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add testcases and fix a few issues

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix build

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prevent "race" in the test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix jsoniter mismerge

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix conflicts

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix format

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add warnings for no worker configurations and prevent data-loss with redis/levelqueue

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Use StopTimer

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
…tea#18393)

Implements the Pausable interface on WrappedQueues and PersistableChannelUniqueQueues

Reference go-gitea#15928

Signed-off-by: Andrew Thornton art27@cantab.net
@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/feature Completely new functionality. Can only be merged if feature freeze is not active.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants