Skip to content

Conversation

@NachoEchevarria
Copy link
Collaborator

@NachoEchevarria NachoEchevarria commented Nov 4, 2025

Summary of changes

We have detected the following error in the CI in some debugger unit tests:

2025-11-04T09:35:52.6197579Z 09:35:52 [DBG]   Failed Datadog.Trace.Tests.Debugger.DebuggerSnapshotCreatorTests.ObjectStructure_EmptyArray [2 s]
2025-11-04T09:35:52.6199535Z 09:35:52 [DBG]   Error Message:
2025-11-04T09:35:52.6199996Z 09:35:52 [DBG]    VerifyException : Results do not match.
2025-11-04T09:35:52.6200382Z 09:35:52 [DBG] Differences:
2025-11-04T09:35:52.6200868Z 09:35:52 [DBG] Received: DebuggerSnapshotCreatorTests.ObjectStructure_EmptyArray.received.txt
2025-11-04T09:35:52.6201403Z 09:35:52 [DBG] Verified: DebuggerSnapshotCreatorTests.ObjectStructure_EmptyArray.verified.txt
2025-11-04T09:35:52.6202014Z 09:35:52 [DBG] Received Content:
2025-11-04T09:35:52.6202362Z 09:35:52 [DBG] {
2025-11-04T09:35:52.6202703Z 09:35:52 [DBG]   local0: {
2025-11-04T09:35:52.6203061Z 09:35:52 [DBG]     elements: [],
2025-11-04T09:35:52.6203435Z 09:35:52 [DBG]     notCapturedReason: timeout,
2025-11-04T09:35:52.6203982Z 09:35:52 [DBG]     size: 0,
2025-11-04T09:35:52.6204341Z 09:35:52 [DBG]     type: Int32[]
2025-11-04T09:35:52.6204689Z 09:35:52 [DBG]   }
2025-11-04T09:35:52.6205018Z 09:35:52 [DBG] }
2025-11-04T09:35:52.6205359Z 09:35:52 [DBG] Verified Content:
2025-11-04T09:35:52.6205718Z 09:35:52 [DBG] {
2025-11-04T09:35:52.6206057Z 09:35:52 [DBG]   local0: {
2025-11-04T09:35:52.6206398Z 09:35:52 [DBG]     elements: [],
2025-11-04T09:35:52.6206754Z 09:35:52 [DBG]     size: 0,
2025-11-04T09:35:52.6207116Z 09:35:52 [DBG]     type: Int32[]
2025-11-04T09:35:52.6207459Z 09:35:52 [DBG]   }
2025-11-04T09:35:52.6207780Z 09:35:52 [DBG] }

The proposed solution is to increase the timeout:
ConfigurationKeys.Debugger.MaxTimeToSerialize, "1000"

The error could be reproduced locally by setting this config value to 1, which confirms that this solution should work.

In other tests (debugger integration tests), we are already setting this value to 1000.

Reason for change

Implementation details

Test coverage

Other details

@github-actions github-actions bot added the area:tests unit tests, integration tests label Nov 4, 2025
@NachoEchevarria NachoEchevarria changed the title Increase debugger serialize timeout Increase debugger serialize timeout for unit tests Nov 4, 2025
@NachoEchevarria NachoEchevarria marked this pull request as ready for review November 4, 2025 11:26
@NachoEchevarria NachoEchevarria requested a review from a team as a code owner November 4, 2025 11:26
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Nov 4, 2025

⚠️ Tests

⚠️ Warnings

❄️ 1 New flaky test detected

IpcClientTest from Datadog.Trace.Tests.Ci.Ipc.IpcTests (Datadog)
Expected finalValue.ClientValue to be 20, but found 19 (difference of -1).

ℹ️ Info

🧪 All tests passed

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 15bedb6 | Docs | Datadog PR Page | Was this helpful? Give us feedback!

Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@dd-trace-dotnet-ci-bot
Copy link

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (7769) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration67.98 ± (67.95 - 68.15) ms68.15 ± (68.17 - 68.41) ms+0.2%✅⬆️
.NET Framework 4.8 - Bailout
duration71.84 ± (71.66 - 71.94) ms71.96 ± (71.86 - 72.11) ms+0.2%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1044.59 ± (1047.15 - 1054.69) ms1046.55 ± (1056.59 - 1067.50) ms+0.2%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms22.05 ± (22.01 - 22.08) ms22.18 ± (22.14 - 22.22) ms+0.6%✅⬆️
process.time_to_main_ms78.86 ± (78.68 - 79.03) ms79.15 ± (79.01 - 79.29) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.88 ± (10.88 - 10.89) MB10.87 ± (10.87 - 10.87) MB-0.1%
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%
.NET Core 3.1 - Bailout
process.internal_duration_ms21.95 ± (21.93 - 21.97) ms21.98 ± (21.95 - 22.01) ms+0.1%✅⬆️
process.time_to_main_ms79.59 ± (79.49 - 79.68) ms80.18 ± (80.11 - 80.26) ms+0.7%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.90 ± (10.90 - 10.91) MB10.92 ± (10.91 - 10.92) MB+0.1%✅⬆️
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms208.96 ± (207.59 - 210.33) ms210.70 ± (209.36 - 212.04) ms+0.8%✅⬆️
process.time_to_main_ms513.98 ± (513.31 - 514.65) ms513.61 ± (513.04 - 514.18) ms-0.1%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed52.63 ± (52.61 - 52.65) MB52.44 ± (52.42 - 52.46) MB-0.4%
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.0%
.NET 6 - Baseline
process.internal_duration_ms20.80 ± (20.77 - 20.83) ms20.85 ± (20.83 - 20.88) ms+0.3%✅⬆️
process.time_to_main_ms68.12 ± (67.99 - 68.24) ms68.39 ± (68.29 - 68.50) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.59 ± (10.58 - 10.59) MB10.60 ± (10.60 - 10.61) MB+0.1%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 6 - Bailout
process.internal_duration_ms20.72 ± (20.69 - 20.74) ms20.75 ± (20.73 - 20.78) ms+0.2%✅⬆️
process.time_to_main_ms68.95 ± (68.89 - 69.02) ms69.24 ± (69.16 - 69.32) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.62 ± (10.61 - 10.62) MB10.65 ± (10.65 - 10.65) MB+0.3%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms201.09 ± (198.44 - 203.75) ms196.44 ± (195.31 - 197.56) ms-2.3%
process.time_to_main_ms483.64 ± (483.07 - 484.21) ms480.51 ± (479.88 - 481.13) ms-0.6%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed51.70 ± (51.64 - 51.77) MB51.31 ± (51.24 - 51.38) MB-0.8%
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.1%
.NET 8 - Baseline
process.internal_duration_ms18.87 ± (18.84 - 18.91) ms19.03 ± (19.00 - 19.06) ms+0.8%✅⬆️
process.time_to_main_ms67.32 ± (67.19 - 67.44) ms67.63 ± (67.52 - 67.74) ms+0.5%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.66 ± (7.65 - 7.66) MB7.64 ± (7.63 - 7.64) MB-0.2%
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 8 - Bailout
process.internal_duration_ms18.85 ± (18.83 - 18.87) ms18.99 ± (18.96 - 19.01) ms+0.7%✅⬆️
process.time_to_main_ms68.36 ± (68.29 - 68.43) ms68.82 ± (68.74 - 68.89) ms+0.7%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.68 ± (7.67 - 7.69) MB7.69 ± (7.68 - 7.70) MB+0.1%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms178.45 ± (177.55 - 179.35) ms177.76 ± (176.95 - 178.57) ms-0.4%
process.time_to_main_ms460.83 ± (460.26 - 461.41) ms454.27 ± (453.77 - 454.77) ms-1.4%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed38.85 ± (38.82 - 38.88) MB38.63 ± (38.61 - 38.65) MB-0.6%
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)+0.2%✅⬆️

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration191.78 ± (191.77 - 192.71) ms192.31 ± (192.23 - 193.16) ms+0.3%✅⬆️
.NET Framework 4.8 - Bailout
duration195.03 ± (194.90 - 195.58) ms195.58 ± (195.38 - 195.88) ms+0.3%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1165.21 ± (1166.16 - 1175.84) ms1154.29 ± (1162.34 - 1172.41) ms-0.9%
.NET Core 3.1 - Baseline
process.internal_duration_ms185.80 ± (185.44 - 186.16) ms186.92 ± (186.64 - 187.20) ms+0.6%✅⬆️
process.time_to_main_ms80.44 ± (80.22 - 80.65) ms80.64 ± (80.43 - 80.84) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.09 ± (16.04 - 16.15) MB16.07 ± (16.05 - 16.10) MB-0.1%
runtime.dotnet.threads.count19 ± (19 - 20)20 ± (20 - 20)+0.8%✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms186.21 ± (185.90 - 186.53) ms186.12 ± (185.80 - 186.45) ms-0.0%
process.time_to_main_ms81.92 ± (81.74 - 82.09) ms81.63 ± (81.45 - 81.82) ms-0.3%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.17 ± (16.15 - 16.19) MB16.11 ± (16.08 - 16.13) MB-0.4%
runtime.dotnet.threads.count21 ± (20 - 21)21 ± (21 - 21)+0.2%✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms393.59 ± (390.87 - 396.31) ms394.66 ± (391.90 - 397.41) ms+0.3%✅⬆️
process.time_to_main_ms514.99 ± (514.20 - 515.79) ms514.89 ± (514.32 - 515.45) ms-0.0%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed62.81 ± (62.66 - 62.96) MB62.73 ± (62.57 - 62.89) MB-0.1%
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)-0.0%
.NET 6 - Baseline
process.internal_duration_ms190.81 ± (190.47 - 191.16) ms191.49 ± (191.00 - 191.98) ms+0.4%✅⬆️
process.time_to_main_ms69.83 ± (69.62 - 70.05) ms69.34 ± (69.17 - 69.52) ms-0.7%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed15.87 ± (15.70 - 16.03) MB16.21 ± (16.11 - 16.32) MB+2.2%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 19)+2.0%✅⬆️
.NET 6 - Bailout
process.internal_duration_ms190.36 ± (190.06 - 190.65) ms190.73 ± (190.28 - 191.17) ms+0.2%✅⬆️
process.time_to_main_ms70.54 ± (70.42 - 70.66) ms70.59 ± (70.42 - 70.76) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed15.86 ± (15.69 - 16.04) MB15.99 ± (15.86 - 16.13) MB+0.8%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)+0.9%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms411.01 ± (407.54 - 414.48) ms404.01 ± (401.31 - 406.71) ms-1.7%
process.time_to_main_ms486.78 ± (486.16 - 487.39) ms480.94 ± (480.36 - 481.52) ms-1.2%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed61.87 ± (61.72 - 62.02) MB61.81 ± (61.65 - 61.96) MB-0.1%
runtime.dotnet.threads.count30 ± (29 - 30)30 ± (29 - 30)+0.0%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms188.61 ± (188.31 - 188.92) ms189.03 ± (188.77 - 189.29) ms+0.2%✅⬆️
process.time_to_main_ms68.91 ± (68.76 - 69.06) ms69.06 ± (68.93 - 69.20) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.76 ± (11.74 - 11.78) MB11.73 ± (11.68 - 11.78) MB-0.3%
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 18)-0.4%
.NET 8 - Bailout
process.internal_duration_ms188.04 ± (187.84 - 188.24) ms188.28 ± (188.07 - 188.49) ms+0.1%✅⬆️
process.time_to_main_ms69.84 ± (69.76 - 69.92) ms69.99 ± (69.90 - 70.08) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.80 ± (11.77 - 11.82) MB11.70 ± (11.62 - 11.79) MB-0.8%
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)-0.8%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms357.16 ± (355.73 - 358.60) ms358.27 ± (356.81 - 359.73) ms+0.3%✅⬆️
process.time_to_main_ms463.51 ± (462.79 - 464.24) ms456.97 ± (456.29 - 457.65) ms-1.4%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed50.47 ± (50.43 - 50.52) MB50.22 ± (50.18 - 50.26) MB-0.5%
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)+0.4%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7769) - mean (68ms)  : 67, 70
    master - mean (68ms)  : 67, 69

    section Bailout
    This PR (7769) - mean (72ms)  : 71, 73
    master - mean (72ms)  : 70, 73

    section CallTarget+Inlining+NGEN
    This PR (7769) - mean (1,062ms)  : 980, 1145
    master - mean (1,051ms)  : 1002, 1099

Loading
FakeDbCommand (.NET Core 3.1)
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7769) - mean (107ms)  : 104, 109
    master - mean (106ms)  : 104, 108

    section Bailout
    This PR (7769) - mean (107ms)  : 106, 108
    master - mean (107ms)  : 105, 108

    section CallTarget+Inlining+NGEN
    This PR (7769) - mean (758ms)  : 725, 791
    master - mean (749ms)  : 727, 772

Loading
FakeDbCommand (.NET 6)
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7769) - mean (94ms)  : 93, 96
    master - mean (94ms)  : 92, 96

    section Bailout
    This PR (7769) - mean (95ms)  : 94, 96
    master - mean (94ms)  : 93, 95

    section CallTarget+Inlining+NGEN
    This PR (7769) - mean (709ms)  : 668, 750
    master - mean (718ms)  : 665, 771

Loading
FakeDbCommand (.NET 8)
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7769) - mean (93ms)  : 91, 95
    master - mean (92ms)  : 90, 95

    section Bailout
    This PR (7769) - mean (94ms)  : 92, 95
    master - mean (93ms)  : 92, 94

    section CallTarget+Inlining+NGEN
    This PR (7769) - mean (673ms)  : 653, 692
    master - mean (670ms)  : 651, 689

Loading
HttpMessageHandler (.NET Framework 4.8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7769) - mean (193ms)  : 188, 197
    master - mean (192ms)  : 188, 197

    section Bailout
    This PR (7769) - mean (196ms)  : 193, 198
    master - mean (195ms)  : 192, 199

    section CallTarget+Inlining+NGEN
    This PR (7769) - mean (1,167ms)  : 1092, 1243
    master - mean (1,171ms)  : 1101, 1241

Loading
HttpMessageHandler (.NET Core 3.1)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7769) - mean (276ms)  : 271, 281
    master - mean (275ms)  : 270, 280

    section Bailout
    This PR (7769) - mean (276ms)  : 271, 280
    master - mean (276ms)  : 272, 280

    section CallTarget+Inlining+NGEN
    This PR (7769) - mean (942ms)  : 896, 987
    master - mean (951ms)  : 904, 999

Loading
HttpMessageHandler (.NET 6)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7769) - mean (269ms)  : 263, 276
    master - mean (269ms)  : 263, 274

    section Bailout
    This PR (7769) - mean (270ms)  : 264, 276
    master - mean (269ms)  : 265, 272

    section CallTarget+Inlining+NGEN
    This PR (7769) - mean (916ms)  : 869, 963
    master - mean (934ms)  : 870, 998

Loading
HttpMessageHandler (.NET 8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7769) - mean (268ms)  : 264, 272
    master - mean (268ms)  : 264, 272

    section Bailout
    This PR (7769) - mean (268ms)  : 265, 271
    master - mean (268ms)  : 265, 271

    section CallTarget+Inlining+NGEN
    This PR (7769) - mean (848ms)  : 826, 870
    master - mean (852ms)  : 837, 867

Loading

@NachoEchevarria NachoEchevarria merged commit 24c865f into master Nov 6, 2025
157 checks passed
@NachoEchevarria NachoEchevarria deleted the nacho/DebuggerUnitTestsTimeout branch November 6, 2025 13:31
@github-actions github-actions bot added this to the vNext-v3 milestone Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tests unit tests, integration tests type:flake-fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants