Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add configuration to support free concurrent tasks and optimize git library update performance issues #2424

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

gaoyue1989
Copy link
Contributor

In the process of frequently creating tasks in large batches, several problems were discovered:

  1. Concurrent tasks of the same template are not supported
  2. Taskpool is slow to deliver tasks every 5 seconds.
  3. The git library will need to be updated every time a task is executed, but in most cases this is not necessary.

This is controlled through the incremental scheme, and is switched by configuring GitCacheTime, TaskFreeConcurrencyMode, and TaskPoolTickerSec. When not configured, it remains the same as the previous code.

Consider that inventory is created through taskid when using git. It is actually created every time. Currently, GitCacheTime does not control this.

warlocgao and others added 3 commits October 15, 2024 15:42
feat:Add free concurrency mode configuration.

feat:Add cache configuration to task git clone pull operation to reduce git pressure.
@fiftin
Copy link
Collaborator

fiftin commented Oct 15, 2024

Hi @gaoyue1989

Did you test the PR in real tasks?

@fiftin
Copy link
Collaborator

fiftin commented Oct 15, 2024

Caching doesn't resolve the conflict issue and introduces unexpected behavior.
I think we need to check for new commits (without pulling) on each task run.
In free concurrency mode, new tasks fail if a new commit appears. Or ignore new commits if active tasks exist. We can add numeric option for this: max_skip_attempts_for_new_commits (name must be changed :D).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants