Tweak CI variables #7784

andrewlock · 2025-11-05T17:49:36Z

Summary of changes

Don't clobber the global filter in linux integration tests
Rename the "force" variables to not imply they're specific to "exploration" tests

Reason for change

The filter gets clobbered - if you explicitly set the Filter variable it gets ignored in linux integration tests
It's confusing having exploration in the name

Implementation details

Fix it

Test coverage

Meh

bouwkast

Thanks!

dd-trace-dotnet-ci-bot · 2025-11-05T18:50:10Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (7784) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	68.65 ± (68.67 - 68.96) ms	68.74 ± (68.75 - 68.98) ms	+0.1%	✅⬆️
.NET Framework 4.8 - Bailout
duration	72.55 ± (72.48 - 72.86) ms	72.80 ± (72.62 - 72.87) ms	+0.3%	✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1049.31 ± (1049.36 - 1054.86) ms	1120.54 ± (1103.23 - 1120.72) ms	+6.8%	✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms	22.12 ± (22.09 - 22.15) ms	22.60 ± (22.51 - 22.68) ms	+2.2%	✅⬆️
process.time_to_main_ms	79.83 ± (79.67 - 79.99) ms	82.09 ± (81.75 - 82.42) ms	+2.8%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.91 ± (10.90 - 10.91) MB	10.89 ± (10.89 - 10.90) MB	-0.1%	✅
runtime.dotnet.threads.count	12 ± (12 - 12)	12 ± (12 - 12)	+0.0%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	22.05 ± (22.03 - 22.08) ms	22.01 ± (21.98 - 22.03) ms	-0.2%	✅
process.time_to_main_ms	81.34 ± (81.21 - 81.48) ms	81.40 ± (81.26 - 81.54) ms	+0.1%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.94 ± (10.94 - 10.95) MB	10.93 ± (10.93 - 10.94) MB	-0.1%	✅
runtime.dotnet.threads.count	13 ± (13 - 13)	13 ± (13 - 13)	+0.0%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	209.82 ± (208.48 - 211.16) ms	210.05 ± (208.83 - 211.27) ms	+0.1%	✅⬆️
process.time_to_main_ms	519.79 ± (519.21 - 520.38) ms	516.96 ± (516.37 - 517.55) ms	-0.5%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	52.67 ± (52.65 - 52.69) MB	52.53 ± (52.50 - 52.55) MB	-0.3%	✅
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.0%	✅
.NET 6 - Baseline
process.internal_duration_ms	20.80 ± (20.77 - 20.83) ms	20.95 ± (20.92 - 20.98) ms	+0.7%	✅⬆️
process.time_to_main_ms	69.18 ± (69.05 - 69.31) ms	69.90 ± (69.79 - 70.02) ms	+1.1%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.61 ± (10.61 - 10.61) MB	10.60 ± (10.60 - 10.60) MB	-0.1%	✅
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 6 - Bailout
process.internal_duration_ms	20.75 ± (20.72 - 20.79) ms	20.84 ± (20.81 - 20.87) ms	+0.4%	✅⬆️
process.time_to_main_ms	69.97 ± (69.83 - 70.11) ms	71.02 ± (70.91 - 71.14) ms	+1.5%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.63 ± (10.62 - 10.63) MB	10.65 ± (10.65 - 10.66) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	201.92 ± (199.26 - 204.57) ms	197.51 ± (196.25 - 198.76) ms	-2.2%	✅
process.time_to_main_ms	486.02 ± (485.46 - 486.58) ms	480.66 ± (480.08 - 481.25) ms	-1.1%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	51.66 ± (51.60 - 51.72) MB	51.37 ± (51.30 - 51.45) MB	-0.6%	✅
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	-0.0%	✅
.NET 8 - Baseline
process.internal_duration_ms	19.02 ± (19.00 - 19.05) ms	18.96 ± (18.93 - 18.99) ms	-0.3%	✅
process.time_to_main_ms	68.27 ± (68.14 - 68.40) ms	68.49 ± (68.35 - 68.64) ms	+0.3%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.65 ± (7.64 - 7.65) MB	7.66 ± (7.65 - 7.66) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 8 - Bailout
process.internal_duration_ms	18.98 ± (18.96 - 19.01) ms	18.99 ± (18.95 - 19.02) ms	+0.0%	✅⬆️
process.time_to_main_ms	69.46 ± (69.34 - 69.58) ms	69.35 ± (69.23 - 69.48) ms	-0.2%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.69 ± (7.68 - 7.70) MB	7.68 ± (7.67 - 7.68) MB	-0.1%	✅
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	179.32 ± (178.51 - 180.14) ms	179.67 ± (178.98 - 180.36) ms	+0.2%	✅⬆️
process.time_to_main_ms	461.73 ± (461.26 - 462.20) ms	457.79 ± (457.38 - 458.21) ms	-0.9%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	38.80 ± (38.77 - 38.83) MB	38.64 ± (38.62 - 38.67) MB	-0.4%	✅
runtime.dotnet.threads.count	27 ± (27 - 27)	27 ± (27 - 27)	+0.2%	✅⬆️

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	194.30 ± (194.74 - 195.78) ms	193.21 ± (193.16 - 194.07) ms	-0.6%	✅
.NET Framework 4.8 - Bailout
duration	197.19 ± (197.22 - 197.90) ms	197.85 ± (197.72 - 198.65) ms	+0.3%	✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1168.58 ± (1167.61 - 1175.32) ms	1162.54 ± (1166.86 - 1175.27) ms	-0.5%	✅
.NET Core 3.1 - Baseline
process.internal_duration_ms	190.35 ± (189.91 - 190.79) ms	187.89 ± (187.50 - 188.29) ms	-1.3%	✅
process.time_to_main_ms	81.51 ± (81.31 - 81.71) ms	80.94 ± (80.70 - 81.17) ms	-0.7%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.02 ± (16.00 - 16.05) MB	16.11 ± (16.09 - 16.13) MB	+0.5%	✅⬆️
runtime.dotnet.threads.count	20 ± (20 - 20)	20 ± (20 - 20)	+0.2%	✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms	190.25 ± (189.78 - 190.72) ms	187.54 ± (187.19 - 187.89) ms	-1.4%	✅
process.time_to_main_ms	83.39 ± (83.18 - 83.60) ms	82.22 ± (82.08 - 82.36) ms	-1.4%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.10 ± (16.08 - 16.13) MB	16.17 ± (16.14 - 16.19) MB	+0.4%	✅⬆️
runtime.dotnet.threads.count	21 ± (21 - 21)	21 ± (21 - 21)	+0.1%	✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	399.26 ± (396.67 - 401.85) ms	393.81 ± (391.36 - 396.26) ms	-1.4%	✅
process.time_to_main_ms	521.83 ± (520.92 - 522.75) ms	517.15 ± (516.51 - 517.79) ms	-0.9%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	63.25 ± (63.11 - 63.39) MB	63.20 ± (63.06 - 63.34) MB	-0.1%	✅
runtime.dotnet.threads.count	29 ± (29 - 29)	29 ± (29 - 30)	+0.1%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	191.99 ± (191.65 - 192.33) ms	192.04 ± (191.63 - 192.46) ms	+0.0%	✅⬆️
process.time_to_main_ms	70.09 ± (69.90 - 70.28) ms	69.61 ± (69.46 - 69.77) ms	-0.7%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.17 ± (16.05 - 16.29) MB	16.13 ± (16.00 - 16.26) MB	-0.2%	✅
runtime.dotnet.threads.count	19 ± (19 - 19)	19 ± (19 - 19)	+0.4%	✅⬆️
.NET 6 - Bailout
process.internal_duration_ms	196.65 ± (195.95 - 197.34) ms	190.74 ± (190.46 - 191.01) ms	-3.0%	✅
process.time_to_main_ms	72.63 ± (72.33 - 72.93) ms	70.49 ± (70.39 - 70.58) ms	-2.9%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.35 ± (16.32 - 16.37) MB	16.09 ± (15.94 - 16.24) MB	-1.6%	✅
runtime.dotnet.threads.count	20 ± (20 - 20)	19 ± (19 - 19)	-5.0%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	413.95 ± (410.95 - 416.94) ms	415.49 ± (412.13 - 418.84) ms	+0.4%	✅⬆️
process.time_to_main_ms	489.80 ± (488.85 - 490.74) ms	480.16 ± (479.62 - 480.70) ms	-2.0%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	62.24 ± (62.09 - 62.39) MB	62.07 ± (61.94 - 62.21) MB	-0.3%	✅
runtime.dotnet.threads.count	30 ± (29 - 30)	29 ± (29 - 29)	-0.2%	✅
.NET 8 - Baseline
process.internal_duration_ms	190.91 ± (190.52 - 191.30) ms	190.47 ± (190.14 - 190.81) ms	-0.2%	✅
process.time_to_main_ms	69.69 ± (69.51 - 69.87) ms	69.64 ± (69.47 - 69.81) ms	-0.1%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.76 ± (11.73 - 11.78) MB	11.70 ± (11.68 - 11.73) MB	-0.5%	✅
runtime.dotnet.threads.count	18 ± (18 - 18)	18 ± (18 - 18)	+0.2%	✅⬆️
.NET 8 - Bailout
process.internal_duration_ms	191.05 ± (190.62 - 191.48) ms	189.70 ± (189.38 - 190.02) ms	-0.7%	✅
process.time_to_main_ms	70.98 ± (70.83 - 71.13) ms	70.38 ± (70.28 - 70.49) ms	-0.8%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.77 ± (11.75 - 11.80) MB	11.78 ± (11.75 - 11.80) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	19 ± (19 - 19)	+0.4%	✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	365.50 ± (364.16 - 366.83) ms	361.68 ± (360.24 - 363.12) ms	-1.0%	✅
process.time_to_main_ms	470.76 ± (470.10 - 471.41) ms	459.29 ± (458.72 - 459.87) ms	-2.4%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	50.49 ± (50.46 - 50.52) MB	50.24 ± (50.21 - 50.27) MB	-0.5%	✅
runtime.dotnet.threads.count	29 ± (29 - 29)	29 ± (29 - 29)	+0.1%	✅⬆️

Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts

FakeDbCommand (.NET Framework 4.8)

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7784) - mean (69ms)  : 67, 71
    master - mean (69ms)  : 67, 71

    section Bailout
    This PR (7784) - mean (73ms)  : 71, 74
    master - mean (73ms)  : 71, 75

    section CallTarget+Inlining+NGEN
    This PR (7784) - mean (1,112ms)  : 985, 1239
    master - mean (1,052ms)  : 1013, 1091

FakeDbCommand (.NET Core 3.1)

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7784) - mean (111ms)  : 106, 117
    master - mean (108ms)  : 105, 110

    section Bailout
    This PR (7784) - mean (109ms)  : 107, 111
    master - mean (109ms)  : 107, 111

    section CallTarget+Inlining+NGEN
    This PR (7784) - mean (762ms)  : 735, 790
    master - mean (763ms)  : 723, 804

FakeDbCommand (.NET 6)

gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7784) - mean (96ms)  : 94, 98
    master - mean (95ms)  : 93, 97

    section Bailout
    This PR (7784) - mean (97ms)  : 95, 99
    master - mean (96ms)  : 93, 98

    section CallTarget+Inlining+NGEN
    This PR (7784) - mean (707ms)  : 678, 737
    master - mean (717ms)  : 676, 758

FakeDbCommand (.NET 8)

gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7784) - mean (94ms)  : 91, 96
    master - mean (94ms)  : 91, 97

    section Bailout
    This PR (7784) - mean (95ms)  : 93, 97
    master - mean (95ms)  : 92, 97

    section CallTarget+Inlining+NGEN
    This PR (7784) - mean (670ms)  : 648, 692
    master - mean (670ms)  : 652, 687

HttpMessageHandler (.NET Framework 4.8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7784) - mean (194ms)  : 189, 198
    master - mean (195ms)  : 188, 203

    section Bailout
    This PR (7784) - mean (198ms)  : 193, 204
    master - mean (198ms)  : 194, 201

    section CallTarget+Inlining+NGEN
    This PR (7784) - mean (1,171ms)  : 1109, 1233
    master - mean (1,171ms)  : 1117, 1226

HttpMessageHandler (.NET Core 3.1)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7784) - mean (278ms)  : 272, 283
    master - mean (281ms)  : 269, 292

    section Bailout
    This PR (7784) - mean (278ms)  : 274, 283
    master - mean (282ms)  : 272, 292

    section CallTarget+Inlining+NGEN
    This PR (7784) - mean (953ms)  : 903, 1003
    master - mean (961ms)  : 916, 1007

HttpMessageHandler (.NET 6)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7784) - mean (270ms)  : 265, 274
    master - mean (270ms)  : 266, 274

    section Bailout
    This PR (7784) - mean (269ms)  : 266, 273
    master - mean (278ms)  : 262, 294

    section CallTarget+Inlining+NGEN
    This PR (7784) - mean (929ms)  : 868, 989
    master - mean (938ms)  : 881, 996

HttpMessageHandler (.NET 8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7784) - mean (270ms)  : 265, 274
    master - mean (270ms)  : 264, 276

    section Bailout
    This PR (7784) - mean (270ms)  : 266, 274
    master - mean (272ms)  : 265, 278

    section CallTarget+Inlining+NGEN
    This PR (7784) - mean (853ms)  : 834, 872
    master - mean (868ms)  : 844, 892

andrewlock added 2 commits November 5, 2025 17:44

Don't clobber the filter with DockerGroup

bd2f68b

Rename "force_exploration_tests_with" => "force_run_tests_with"

2768d12

andrewlock requested a review from a team as a code owner November 5, 2025 17:49

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Nov 5, 2025

bouwkast approved these changes Nov 5, 2025

View reviewed changes

lucaspimentel approved these changes Nov 5, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

andrewlock merged commit 068cdd0 into master Nov 5, 2025
152 checks passed

andrewlock deleted the andrew/ci/tweak-variables branch November 5, 2025 19:29

github-actions bot added this to the vNext-v3 milestone Nov 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Tweak CI variables #7784

Tweak CI variables #7784

Uh oh!

andrewlock commented Nov 5, 2025

Uh oh!

bouwkast left a comment

Uh oh!

This comment has been minimized.

dd-trace-dotnet-ci-bot bot commented Nov 5, 2025

FakeDbCommand

HttpMessageHandler

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Tweak CI variables #7784

Tweak CI variables #7784

Uh oh!

Conversation

andrewlock commented Nov 5, 2025

Summary of changes

Reason for change

Implementation details

Test coverage

Uh oh!

bouwkast left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

dd-trace-dotnet-ci-bot bot commented Nov 5, 2025

Execution-Time Benchmarks Report ⏱️

FakeDbCommand

HttpMessageHandler

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants