Inconsistent behavior of E2E testing on same MacOS image

[KangxuanYe]

Description

Hi, our team is implementing iOS SDK and we need to run E2E testing for some reasons (simulators need to be launched). We are doing E2E testing in parallel on 2 simulators each time.

Recently, after we migrated our VM OS image from internal-macos12 to macOS-latest-internal/macOS-14-arm64, we are facing an issue that the same piece of code can have different behavior. Sometimes, every run/test can be completed successfully but sometimes, lots of E2E test cases fail randomly.

Those are the same thing on internal-macros12 and we can tell it is pretty stable when we are running E2E test on internal-macos12 image. However, we have no idea why it becomes so flaky now and the only differences we can observe is OS image version differences. On some versions we have table experience and on some versions, we don't.

Could you please explain or help investigate why same code will randomly have different behavior on CI pipeline/VM?

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

So far, by our testing, we can say

image version: 20241108.422 is good
image version: 20241022.361 is terrible
image version: 20241119.505 is good to us.

Is it regression?

No

Expected behavior

Same piece of code should have the same behavior. Either a PR change should fail consistently or it should pass consistently. It can never be a situation that it sometimes pass and sometimes not.

Actual behavior

Any piece of code change or main branch will fail on E2E testing frequently and randomly on some OS image versions of macOS-14-arm64.

Repro steps

For some reasons, I may be not able to share our CI pipeline with you.

[erik-bershel]

Hey @KangxuanYe!

I'm very sorry that you have to deal with such instability. 😞 But the information you provided is not enough to conduct an investigation and I can only guess. 🤷‍♂️ If you can't provide links to specific launches or the code of the pipelines themselves, then let's try to start from the error output - this information should not be closed, I believe.

If you're talking about problems with simulators, then perhaps researching the changelog of releases can help you or give you some ideas: 20241108.422, 20241022.361, 20241119.505

[KangxuanYe]

Hi @erik-bershel , thanks for responding quickly. I don't think we can disclose our pipeline or a specific launches for you for privacy reasons. But let me run something and to see if I can provide you some of data.

[erik-bershel]

Hey @KangxuanYe!

New runners are deployed. 🎉 Could you check how the situation is on them? 🙇 I saw that you disabled iOS E2E tests in code.