False errors with testRepetitionMode

[jwelton]

When using the new testRepetitionMode in Xcode a test can be automatically re-run if it fails. I've noticed that when a test is run multiple times, previous iteration errors are still reported even if the later iteration passes. In this case, the test suite run passes (since the later iteration passed), but danger still reports errors.

[diogot]

Hi, I didn't know about testRepetitionMode!
We read all errors in the test report, but it might be possible to check if the test re-run and passed successfully.
Would you be interested in investigate this?

[blyscuit]

On Xcode it will look like this.
The report generated from Xcode will contain these on fail test:

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 1 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.108 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 2 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.066 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 3 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.067 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 4 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.067 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 5 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.148 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 6 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.065 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 7 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.068 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 8 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.065 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 9 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.066 seconds).
Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' started (Iteration 10 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyTests/Sources/Specs/Presentation/Modules/SurveySelection/SurveySelectionViewDataSourceSpec.swift:80: error: -[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date] : expected to equal <Thursday, January 1>, got <วันพฤหัสบดี, มกราคม 1>

Test Case '-[SurveyTests.SurveySelectionViewDataSourceSpec a_Survey_Selection_View_Data_Source__its_fetch__sets_surveyHeader_with_correct_date]' failed (0.068 seconds).

and Success test:

Test Suite 'ZZ_AccountSpec' started at 2023-01-13 13:49:17.589
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' started (Iteration 1 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyKIFUITests/Sources/Specs/Presentation/Module/ZZ_AccountSpec.swift:45: error: -[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components] : expected to equal <0>, got <3>

Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' failed (1.702 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' started (Iteration 2 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyKIFUITests/Sources/Specs/Presentation/Module/ZZ_AccountSpec.swift:45: error: -[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components] : expected to equal <0>, got <4>

Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' failed (1.639 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' started (Iteration 3 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyKIFUITests/Sources/Specs/Presentation/Module/ZZ_AccountSpec.swift:45: error: -[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components] : expected to equal <0>, got <3>

Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' failed (1.632 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' started (Iteration 4 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyKIFUITests/Sources/Specs/Presentation/Module/ZZ_AccountSpec.swift:45: error: -[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components] : expected to equal <0>, got <4>

Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' failed (1.664 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' started (Iteration 5 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyKIFUITests/Sources/Specs/Presentation/Module/ZZ_AccountSpec.swift:45: error: -[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components] : expected to equal <0>, got <2>

Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' failed (1.628 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' started (Iteration 6 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyKIFUITests/Sources/Specs/Presentation/Module/ZZ_AccountSpec.swift:45: error: -[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components] : expected to equal <0>, got <5>

Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' failed (1.638 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' started (Iteration 7 of 10).
/Users/blys/AndroidStudioProjects/Survey/iosApp/SurveyKIFUITests/Sources/Specs/Presentation/Module/ZZ_AccountSpec.swift:45: error: -[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components] : expected to equal <0>, got <2>

Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' failed (1.639 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' started (Iteration 8 of 10).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__it_shows_its_ui_components]' passed (1.627 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__its_dismiss_area__when_tapped__dismisses_account_screen]' started (Iteration 1 of 10).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__its_dismiss_area__when_tapped__dismisses_account_screen]' passed (0.722 seconds).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__its_logout_button__when_tapped__shows_login_screen]' started (Iteration 1 of 10).
Test Case '-[SurveyKIFUITests.ZZ_AccountSpec an_Account_screen__its_open__its_logout_button__when_tapped__shows_login_screen]' passed (1.920 seconds).
Test Suite 'ZZ_AccountSpec' failed at 2023-01-13 13:49:33.404.
	 Executed 10 tests, with 7 failures (0 unexpected) in 15.810 (15.816) seconds