Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Add e2e tests for inconsistency between worker group's replicas and the number of Pods #2575

Open
1 of 2 tasks
Tracked by #2173
kevin85421 opened this issue Nov 26, 2024 · 1 comment
Open
1 of 2 tasks
Tracked by #2173

[Feature] Add e2e tests for inconsistency between worker group's replicas and the number of Pods #2575

kevin85421 opened this issue Nov 26, 2024 · 1 comment
Assignees
@dentiny
Labels
1.3.0 autoscaler ci enhancement New feature or request good first issue Good for newcomers

Comments

@kevin85421
Copy link
Member

kevin85421 commented Nov 26, 2024 •
edited
Loading

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

ray-project/ray#48909 fixes an issue in Ray Autoscaler V2. The bug occurs when submitting many Ray tasks simultaneously, triggering the Autoscaler to create multiple Ray nodes. According to the PR description, it should create 10 Ray Pods. However, due to the delay between the replicas in the RayCluster CR spec and the actual number of Pods, only 5 Pods (or fewer than 10) are created.

Add an e2e tests:

  • Submit many Ray tasks at the same time.
  • Make sure the cluster can scale up to maxReplicas and all tasks can finish successfully.

ray-project/ray#48909 will be included in Ray 2.41. If the release has not been made yet, we can test V1 first and then add a follow-up PR for V2 later.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!
@kevin85421 kevin85421 added enhancement New feature or request triage autoscaler ci 1.3.0 good first issue Good for newcomers and removed triage labels Nov 26, 2024
@kevin85421 kevin85421 mentioned this issue Nov 26, 2024
Open
15 tasks
@simotw
Copy link

simotw commented Nov 27, 2024

Hi @kevin85421 , I would like to work on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dentiny dentiny

Labels
1.3.0 autoscaler ci enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kevin85421 @dentiny @simotw