Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tool] Fix failing UI tests on GitHub Action runners #305

Open
pylapp opened this issue Nov 25, 2024 · 1 comment
Open

[Tool] Fix failing UI tests on GitHub Action runners #305

pylapp opened this issue Nov 25, 2024 · 1 comment
Labels
🙏 help wanted Extra attention is needed 🔨 tool Related to tooling

Comments

@pylapp
Copy link
Member

pylapp commented Nov 25, 2024

Description

Today we use swift-snapshot-testing to make assertions on UI rendering of views.
We have Fastlane running these tests and it work as expected.
We have our internal GitLab CI runner, running the same Fastlane lanes, and it works like expected.
However in GitHub Actions some tests do not pass, even if they pass with the other environment, without further details.

It seems the provided environment with GitHub runners is unstable making some test fail (even if they pass on our GitLab CI and local computers), e.g. testAllTypographies*, testAllSizes*, testAllElevations* don't pass in GitHub Actions level but pass in our internal levels.

We do not know if there are hardware resources issues, shadow tomeouts or other configurations trobules witht he Mac runners or the in-use simulators.

Thus we will have checks failed for UI tests and must disable requirement son them to preven pull request to be merged.

This is a bit sad and we should find a workaround or a fix for this issue.
This is not mandatory as we plan to move the demo app, with the UI tests, in out internal GitLab software forge in the future.

But could be nice to have a fix befoire.

Details

Below are the outputs of our last job:

 [16:40:24]: ▸ Test Suite OUDSTokensElevationUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:40:39]: ▸ ::error ::    testAllElevationsInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (15.131 seconds)
[16:40:40]: ▸ ::error ::    testAllElevationsInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.325 seconds)
[16:40:54]: ▸ ::error ::    testAllElevationsOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.344 seconds)
[16:41:28]: ▸ ::error ::    testAllElevationsOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (13.717 seconds)
[16:41:28]: ▸ Test Suite OUDSTokensOpacityUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:41:33]: ▸     ✔ [OUDSTokensOpacityUITests] testAllOpacitiesInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.511 seconds)
[16:41:33]: ▸     ✔ [OUDSTokensOpacityUITests] testAllOpacitiesInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.119 seconds)
[16:41:33]: ▸     ✔ [OUDSTokensOpacityUITests] testAllOpacitiesOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.139 seconds)
[16:41:34]: ▸     ✔ [OUDSTokensOpacityUITests] testAllOpacitiesOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.135 seconds)
[16:41:34]: ▸ Test Suite OUDSTokensBorderUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:41:34]: ▸     ✔ [OUDSTokensBorderUITests] testAllBordersInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.386 seconds)
[16:41:35]: ▸     ✔ [OUDSTokensBorderUITests] testAllBordersInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.291 seconds)
[16:41:38]: ▸     ✔ [OUDSTokensBorderUITests] testAllBordersOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.370 seconds)
[16:41:38]: ▸     ✔ [OUDSTokensBorderUITests] testAllBordersOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.253 seconds)
[16:41:39]: ▸ Test Suite OUDSTokensSpaceUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:41:43]: ▸     ✔ [OUDSTokensSpaceUITests] testAllSpaceInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.751 seconds)
[16:41:43]: ▸     ✔ [OUDSTokensSpaceUITests] testAllSpaceInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.348 seconds)
[16:41:44]: ▸     ✔ [OUDSTokensSpaceUITests] testAllSpacesOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.428 seconds)
[16:42:42]: ▸     ✔ [OUDSTokensSpaceUITests] testAllSpacesOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.564 seconds)
[16:42:43]: ▸ Test Suite OUDSTokensTypographyUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:44:09]: ▸ ::error ::    testAllTypographiesInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (104.090 seconds)
[16:44:18]: ▸ ::error ::    testAllTypographiesInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (42.247 seconds)
[16:44:23]: ▸ ::error ::    testAllTypographiesOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (8.834 seconds)
[16:44:41]: ▸ ::error ::    testAllTypographiesOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.303 seconds)
[16:44:42]: ▸ Test Suite OUDSTokensColorUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:44:43]: ▸     ✔ [OUDSTokensColorUITests] testAllColorsInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.323 seconds)
[16:44:43]: ▸     ✔ [OUDSTokensColorUITests] testAllColorsInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.951 seconds)
[16:44:44]: ▸     ✔ [OUDSTokensColorUITests] testAllColorsOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.311 seconds)
[16:44:44]: ▸     ✔ [OUDSTokensColorUITests] testAllColorsOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.065 seconds)
[16:44:44]: ▸ Test Suite OUDSTokensSizeUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:44:47]: ▸ ::error ::    testAllSizeInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (2.872 seconds)
[16:44:50]: ▸ ::error ::    testAllSizesInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (3.435 seconds)
[16:44:52]: ▸ ::error ::    testAllSizesOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (2.221 seconds)
[16:44:54]: ▸ ::error ::    testAllSizesOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.749 seconds)

In attachments the logs.

logs_31306604165.zip
job-logs.txt

@pylapp pylapp added 🔨 tool Related to tooling 🙏 help wanted Extra attention is needed labels Nov 25, 2024
pylapp added a commit that referenced this issue Nov 25, 2024
Signed-off-by: Pierre-Yves Lapersonne <[email protected]>
@pylapp
Copy link
Member Author

pylapp commented Nov 26, 2024

Idea of @ludovic35: check the l10n of the views in the used simulators. Maybe the simulator is not in the same locale as our, and the rendring of the view with the displayed token text are not the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🙏 help wanted Extra attention is needed 🔨 tool Related to tooling
Projects
Status: Triage
Development

No branches or pull requests

1 participant