Skip to content

[Bug]: Fix iOS flakiness #1032

Description

@Dor-bl

Summary

The ios-tests job in Functional Tests fails intermittently on main, with no corresponding code change to explain the failure — i.e. it's CI/infra flakiness, not a product regression.

Data

Sampled the last 8 ios-tests job runs triggered by a push to main (2026-08-28 → 2026-09-11):

Run Date Conclusion
#391 09-11 failure
#390 09-10 success
#385 09-09 failure
#386 09-09 success
#375/#374 09-04 failure
#368 09-02 failure
#367 09-02 success
#362 08-28 success

4 of 8 (50%) failed, alternating pass/fail across consecutive pushes with no relevant diff in between — a clear flake signature rather than a regression.

Failure signatures observed

All failures happen in test setup (OneTimeSetUp/SetUp, i.e. session creation), never in an actual assertion — confirming this is a session-start/simulator-boot race, not application logic:

  1. WDA launch timeout (run #391) — Session.Logs.LogTests (CanCaptureCrashlogTest, CanCaptureServerTest, CanCaptureSyslogTest, CanRetrieveLogTypesTest — all 4 in the fixture) fail with:

    OpenQA.Selenium.UnknownErrorException : Unable to start WebDriverAgent session. Original error: A new session could not be created. Details: Error Domain=XCTDaemonErrorDomain Code=5 "Timed out attempting to launch app."

  2. WDA app not registered with SpringBoard (same run) — LockDeviceTest.LockTestWithSeconds fails with:

    OpenQA.Selenium.UnknownErrorException : Unable to launch WebDriverAgent. ... (domain=FBSOpenApplicationServiceErrorDomain, code=1): ... "Application "com.facebook.WebDriverAgentRunner.xctrunner" is unknown to FrontBoard"

  3. Simulator boot timeout (run #385) — AlertTests.BeforeAll fails with:

    OpenQA.Selenium.UnknownErrorException : The simulator <udid> has failed to finish booting after 120s.

  4. Element-wait timeout downstream of the same slowness (run fix: IHidesKeyboard interface does not include 'key' and 'strategy' overloads - (#390) and (#389) #391) — AppTests.CanActivateAppFromBackgroundTest: FindElement by accessibility ID times out waiting for the app to return to foreground.

Likely root cause

.github/workflows/functional-test.yml's ios-tests job has no retry or readiness gate around the two flakiest steps:

  • Setup iOS Simulator (futureware-tech/simulator-action@v5, wait_for_boot: true) waits for the simulator to report booted, but doesn't confirm the prebuilt WDA app downloaded in the previous step is actually installed/registered with SpringBoard before tests start.
  • Run iOS functional tests invokes dotnet test exactly once — no [Retry(n)] on session-dependent fixtures, and no step-level retry/back-off if the whole run fails.

Given prebuilt WDA is downloaded and referenced via LOCAL_PREBUILT_WDA rather than built by Xcode locally, there's a plausible race between the simulator reporting "booted" and it having finished registering the just-installed WDA app with FrontBoard/SpringBoard — matching signatures 1 and 2 above.

Suggested next steps

  • Add a WDA-readiness check (e.g. poll xcrun simctl / attempt a throwaway session) after Setup iOS Simulator and before Run iOS functional tests, instead of relying solely on wait_for_boot.
  • Increase Appium's wdaLaunchTimeout/wdaConnectionTimeout capabilities used by the iOS test setup.
  • Add [Retry(2)] (NUnit) to the most session-fragile fixtures, or a CI-level retry of the whole dotnet test invocation on failure.
  • Consider explicitly polling xcrun simctl bootstatus <udid> -b rather than trusting wait_for_boot: true alone.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions