[Feature] RayService HA test - GCS fault tolerance + kill GCS process

[CheyuWu]

Why are these changes needed?

Description

• Create a RayService with GCS FT enabled. No Ray Serve replica should be deployed on the head Pod.
• Kill the GCS process on the head Pod pkill gcs_server.
• Wait until the head Pod is removed from the K8s serve service.
• Use locust to submit requests until the new Ray head is running and ready for 30 seconds.
• No request should be dropped.

Related issue number

Closes #2577

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[CheyuWu]

@kevin85421 @MortalHappiness PTAL

The request is currently being failed. I'm not sure if it's a problem with the testing code or a bug.

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
        --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
        POST     /                                                                               1421    73(5.14%) |     58       0    3369     14 |    9.70        0.00
        --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
                 Aggregated                                                                      1421    73(5.14%) |     58       0    3369     14 |    9.70        0.00

[MortalHappiness]

Possibly related to #2593