Skip to content

System.Transactions.Tests.OleTxTests.Recovery timeout #122621

Closed

Description

@AndyAyersMS

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1234474
Build error leg or test failing: System.Transactions.Local.Tests.WorkItemExecution
Pull request: #122425

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "'System.Transactions.Local.Tests' END OF WORK ITEM LOG: Command timed out, and was killed",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1234474
Error message validated: ['System.Transactions.Local.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 12/17/2025 9:40:16 PM UTC

Report

Build Definition Test Pull Request
1342845 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #124544
1339732 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #125670
1337967 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution
1337501 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #125288
1336493 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution
2927142 dotnet-runtime System.Transactions.Local.Tests.WorkItemExecution
1335407 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #125554
1334955 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #125468
1333328 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution
1326106 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution
1324436 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #125282
1324022 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #125276
1323172 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #125258
1318672 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #124114
1314458 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #124915
1312829 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #120326
1312559 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution
1310148 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #124815
2910060 dotnet-runtime System.Transactions.Local.Tests.WorkItemExecution
1303381 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #124670
1302129 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #123848
1300146 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #123939
1299229 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution
1298591 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #124544
1298222 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #122526
1298219 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution #124536

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 9 21

Activity

  1. added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    on Dec 17, 2025
  2. added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Dec 17, 2025
  3. changed the title [-]System.Transactions.Local.Tests timeout[/-] [+]System.Transactions.Tests.OleTxTests.Recovery timeout[/+] on Dec 17, 2025
  4. added and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Dec 17, 2025
  5. dotnet-policy-service commented on Dec 17, 2025

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @roji, @SamMonoRT
    See info in area-owners.md if you want to be subscribed.

  6. JulieLeeMSFT commented on Mar 5, 2026

    @JulieLeeMSFT
    Member

    Failed in:

    Failed tests:

    - net11.0-windows-Release-x64-jitstress2_jitstressregs0x1000-Windows.10.Amd64.Open
      - System.Transactions.Local.Tests
    
  7. danmoseley commented on Mar 20, 2026

    @danmoseley
    Contributor

    The Recovery() test in OleTxTests has a structural issue that likely causes the work item timeout.

    The test spawns a remote process via RemoteExecutor.Invoke with a CrashingEnlistment. That remote process does Thread.Sleep(Timeout) where Timeout = TimeSpan.FromMinutes(1) — it unconditionally burns a full minute staying alive. If MSDTC is slow or unresponsive on the CI machine, the main process also blocks on WaitOne(Timeout) calls (up to 1 minute each, and there are several). These add up and can push the entire test work item past its time limit.

    The core problem is that nothing bounds the wall-clock time of a hung MSDTC interaction — if MSDTC never delivers the commit notification, both the remote process and the main test thread block until their respective 1-minute timeouts expire, and RemoteInvokeHandle.Dispose() also blocks waiting for the remote process to exit.

    Suggested fix: Add a TimeOut to RemoteInvokeOptions so that a hung remote process gets killed rather than blocking indefinitely:

    new RemoteInvokeOptions { ExpectedExitCode = 42, TimeOut = 120_000 }

    Optionally also add [Fact(Timeout = 180_000)] to the test method so that if the main thread is stuck on a WaitOne, xunit marks the test as failed and moves on rather than letting it burn the entire work item budget. A leaked threadpool thread is a small price compared to killing the whole work item.

  8. jakobbotsch commented on Mar 30, 2026

    @jakobbotsch
    Member

    @danmoseley This failed again in #126169. I opened a new issue #126304 since I didn't realize this one existed.

  9. locked and limited conversation to collaborators on Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Known Build ErrorUse this to report build issues in the .NET Helix tabarea-System.Transactionsblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions