Skip to content

Cosmos DB partial integration testing #7752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Nov 14, 2025
Merged

Cosmos DB partial integration testing #7752

merged 11 commits into from
Nov 14, 2025

Conversation

@pablomartinezbernardo
Copy link
Contributor

@pablomartinezbernardo pablomartinezbernardo commented Nov 3, 2025 •
edited
Loading

Summary of changes

Integration test Cosmos DB instrumentation with mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview.

Due to incomplete support in the emulator, only item queries can be currently tested.

Reason for change

No integration tests exist for Cosmos DB. This PR changes it using the under development Linux-based emulator. Compatibility is limited, but it supports items operations, which are arguably used the most. The emulator will also help development of future Cosmos DB support.

Implementation details

Keep current test to keep supporting full manual testing, while doing specific testing for CI.

@pr-commenter
Copy link

pr-commenter bot commented Nov 3, 2025 •
edited
Loading

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7752 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.161
  • 1 benchmarks are slower, with geometric mean 2.064
  • 6 benchmarks have fewer allocations
  • 5 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472 6.01 KB 6.08 KB 72 B 1.20%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1 5.63 KB 5.7 KB 63 B 1.12%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.6μs 58.4ns 340ns 0 0 0 5.53 KB
master StartStopWithChild netcoreapp3.1 13.8μs 73.1ns 373ns 0 0 0 5.63 KB
master StartStopWithChild net472 23.9μs 131ns 777ns 1.04 0.313 0.104 6.01 KB
#7752 StartStopWithChild net6.0 10.8μs 59.4ns 371ns 0 0 0 5.51 KB
#7752 StartStopWithChild netcoreapp3.1 14.8μs 56.1ns 202ns 0 0 0 5.7 KB
#7752 StartStopWithChild net472 21.8μs 123ns 813ns 0.931 0.233 0 6.08 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 929μs 172ns 644ns 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.04ms 190ns 735ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.23ms 915ns 3.54μs 0 0 0 3.31 KB
#7752 WriteAndFlushEnrichedTraces net6.0 937μs 308ns 1.19μs 0 0 0 2.71 KB
#7752 WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 88.7ns 332ns 0 0 0 2.7 KB
#7752 WriteAndFlushEnrichedTraces net472 1.22ms 693ns 2.69μs 0 0 0 3.31 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 1.04μs 6.32ns 63.2ns 0 0 0 1.22 KB
master AllCycleSimpleBody netcoreapp3.1 1.39μs 7.58ns 42.2ns 0 0 0 1.2 KB
master AllCycleSimpleBody net472 1.08μs 0.104ns 0.39ns 0.194 0 0 1.23 KB
master AllCycleMoreComplexBody net6.0 7.19μs 9.86ns 38.2ns 0 0 0 4.72 KB
master AllCycleMoreComplexBody netcoreapp3.1 8.87μs 42.5ns 170ns 0 0 0 4.62 KB
master AllCycleMoreComplexBody net472 7.67μs 3.37ns 13.1ns 0.728 0 0 4.74 KB
master ObjectExtractorSimpleBody net6.0 318ns 0.0601ns 0.225ns 0 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 407ns 2.09ns 9.78ns 0 0 0 272 B
master ObjectExtractorSimpleBody net472 300ns 0.0354ns 0.133ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 6.21μs 32ns 153ns 0 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 7.92μs 7.77ns 30.1ns 0 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 6.69μs 3.6ns 13.9ns 0.6 0 0 3.8 KB
#7752 AllCycleSimpleBody net6.0 1.06μs 6.11ns 46.1ns 0 0 0 1.22 KB
#7752 AllCycleSimpleBody netcoreapp3.1 1.4μs 7.8ns 47.5ns 0 0 0 1.2 KB
#7752 AllCycleSimpleBody net472 1.03μs 0.319ns 1.23ns 0.195 0 0 1.23 KB
#7752 AllCycleMoreComplexBody net6.0 6.99μs 36.9ns 188ns 0 0 0 4.72 KB
#7752 AllCycleMoreComplexBody netcoreapp3.1 9.17μs 3.32ns 12.9ns 0 0 0 4.62 KB
#7752 AllCycleMoreComplexBody net472 7.63μs 3.81ns 14.8ns 0.721 0 0 4.74 KB
#7752 ObjectExtractorSimpleBody net6.0 317ns 1.64ns 7.85ns 0 0 0 280 B
#7752 ObjectExtractorSimpleBody netcoreapp3.1 394ns 2.21ns 14.7ns 0 0 0 272 B
#7752 ObjectExtractorSimpleBody net472 298ns 0.0317ns 0.119ns 0.0446 0 0 281 B
#7752 ObjectExtractorMoreComplexBody net6.0 6.35μs 31.1ns 135ns 0 0 0 3.78 KB
#7752 ObjectExtractorMoreComplexBody netcoreapp3.1 7.79μs 36.2ns 136ns 0 0 0 3.69 KB
#7752 ObjectExtractorMoreComplexBody net472 6.69μs 1.22ns 4.72ns 0.603 0 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 76.8μs 337ns 1.22μs 0 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 96.7μs 307ns 1.11μs 0 0 0 32.4 KB
master EncodeArgs net472 115μs 16.7ns 64.8ns 4.6 0 0 32.51 KB
master EncodeLegacyArgs net6.0 143μs 22.5ns 78.1ns 0 0 0 2.15 KB
master EncodeLegacyArgs netcoreapp3.1 202μs 77ns 298ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 265μs 116ns 450ns 0 0 0 2.16 KB
#7752 EncodeArgs net6.0 76.9μs 103ns 398ns 0 0 0 32.4 KB
#7752 EncodeArgs netcoreapp3.1 96μs 282ns 1.09μs 0 0 0 32.4 KB
#7752 EncodeArgs net472 110μs 7.05ns 27.3ns 4.97 0 0 32.51 KB
#7752 EncodeLegacyArgs net6.0 146μs 77.7ns 301ns 0 0 0 2.15 KB
#7752 EncodeLegacyArgs netcoreapp3.1 197μs 154ns 597ns 0 0 0 2.14 KB
#7752 EncodeLegacyArgs net472 263μs 35.3ns 137ns 0 0 0 2.17 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #7752

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑netcoreapp3.1 2.064 414,401.04 855,378.54

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 393μs 92ns 332ns 0 0 0 4.55 KB
master RunWafRealisticBenchmark netcoreapp3.1 415μs 562ns 2.02μs 0 0 0 4.48 KB
master RunWafRealisticBenchmark net472 427μs 70.6ns 274ns 0 0 0 4.66 KB
master RunWafRealisticBenchmarkWithAttack net6.0 282μs 29.3ns 113ns 0 0 0 2.24 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 696μs 10.6μs 102μs 0 0 0 2.22 KB
master RunWafRealisticBenchmarkWithAttack net472 309μs 40.4ns 156ns 0 0 0 2.29 KB
#7752 RunWafRealisticBenchmark net6.0 396μs 117ns 455ns 0 0 0 4.55 KB
#7752 RunWafRealisticBenchmark netcoreapp3.1 801μs 16.1μs 146μs 0 0 0 4.48 KB
#7752 RunWafRealisticBenchmark net472 426μs 38.4ns 144ns 0 0 0 4.66 KB
#7752 RunWafRealisticBenchmarkWithAttack net6.0 284μs 35.3ns 132ns 0 0 0 2.24 KB
#7752 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 721μs 2.42μs 9.36μs 0 0 0 2.22 KB
#7752 RunWafRealisticBenchmarkWithAttack net472 311μs 40ns 155ns 0 0 0 2.29 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 59.6μs 48.3ns 181ns 0 0 0 14.52 KB
master SendRequest netcoreapp3.1 71.6μs 136ns 489ns 0 0 0 17.42 KB
master SendRequest net472 0.00482ns 0.00196ns 0.00758ns 0 0 0 0 b
#7752 SendRequest net6.0 61.3μs 39ns 151ns 0 0 0 14.52 KB
#7752 SendRequest netcoreapp3.1 72.7μs 116ns 434ns 0 0 0 17.42 KB
#7752 SendRequest net472 0.00602ns 0.00231ns 0.00893ns 0 0 0 0 b
Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0 7 B 2 B -5 B -71.43%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net472 73 B 0 b -73 B -100.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net472 47 B 0 b -47 B -100.00%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master OriginalCharSlice net6.0 1.87ms 3.8μs 14.2μs 0 0 0 640.01 KB
master OriginalCharSlice netcoreapp3.1 2.14ms 9.79μs 36.6μs 0 0 0 640 KB
master OriginalCharSlice net472 2.66ms 137ns 493ns 100 0 0 641.95 KB
master OptimizedCharSlice net6.0 1.37ms 435ns 1.68μs 0 0 0 7 B
master OptimizedCharSlice netcoreapp3.1 1.67ms 631ns 2.45μs 0 0 0 1 B
master OptimizedCharSlice net472 1.9ms 484ns 1.87μs 0 0 0 73 B
master OptimizedCharSliceWithPool net6.0 800μs 84.8ns 317ns 0 0 0 2 B
master OptimizedCharSliceWithPool netcoreapp3.1 842μs 101ns 390ns 0 0 0 0 b
master OptimizedCharSliceWithPool net472 1.18ms 88.2ns 342ns 0 0 0 47 B
#7752 OriginalCharSlice net6.0 1.88ms 4.29μs 16.6μs 0 0 0 640.01 KB
#7752 OriginalCharSlice netcoreapp3.1 2.12ms 7.61μs 29.5μs 0 0 0 640 KB
#7752 OriginalCharSlice net472 2.68ms 163ns 564ns 100 0 0 641.95 KB
#7752 OptimizedCharSlice net6.0 1.41ms 86.5ns 335ns 0 0 0 2 B
#7752 OptimizedCharSlice netcoreapp3.1 1.66ms 286ns 1.03μs 0 0 0 1 B
#7752 OptimizedCharSlice net472 2ms 438ns 1.7μs 0 0 0 0 b
#7752 OptimizedCharSliceWithPool net6.0 821μs 32.5ns 126ns 0 0 0 2 B
#7752 OptimizedCharSliceWithPool netcoreapp3.1 899μs 101ns 392ns 0 0 0 0 b
#7752 OptimizedCharSliceWithPool net472 1.13ms 86.1ns 322ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Faster 🎉 Fewer allocations 🎉

Faster 🎉 in #7752

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 1.161 766,946.25 660,710.94

Fewer allocations 🎉 in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 56.02 KB 55.73 KB -297 B -0.53%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 656μs 2.73μs 9.83μs 0 0 0 41.72 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 756μs 3.85μs 18μs 0 0 0 41.86 KB
master WriteAndFlushEnrichedTraces net472 923μs 3.76μs 14.6μs 4.46 0 0 56.02 KB
#7752 WriteAndFlushEnrichedTraces net6.0 705μs 3.81μs 21.6μs 0 0 0 41.74 KB
#7752 WriteAndFlushEnrichedTraces netcoreapp3.1 660μs 1.65μs 6.38μs 0 0 0 41.82 KB
#7752 WriteAndFlushEnrichedTraces net472 847μs 2.49μs 9.3μs 4.46 0 0 55.73 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.9μs 9.37ns 39.8ns 0 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 2.5μs 7.47ns 28.9ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.8μs 3.11ns 12.1ns 0.152 0.0138 0 987 B
#7752 ExecuteNonQuery net6.0 1.95μs 1.88ns 7.29ns 0 0 0 1.02 KB
#7752 ExecuteNonQuery netcoreapp3.1 2.63μs 7.5ns 28.1ns 0 0 0 1.02 KB
#7752 ExecuteNonQuery net472 2.8μs 3.3ns 12.8ns 0.144 0.0144 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.73μs 7.96ns 30.8ns 0 0 0 1.03 KB
master CallElasticsearch netcoreapp3.1 2.24μs 7.2ns 27.9ns 0 0 0 1.03 KB
master CallElasticsearch net472 3.48μs 1.88ns 7.28ns 0.156 0 0 1.04 KB
master CallElasticsearchAsync net6.0 1.84μs 1.42ns 4.93ns 0 0 0 1.01 KB
master CallElasticsearchAsync netcoreapp3.1 2.41μs 10.7ns 41.6ns 0 0 0 1.08 KB
master CallElasticsearchAsync net472 3.64μs 3.03ns 11.7ns 0.164 0 0 1.1 KB
#7752 CallElasticsearch net6.0 1.68μs 8.66ns 40.6ns 0 0 0 1.03 KB
#7752 CallElasticsearch netcoreapp3.1 2.2μs 7.33ns 28.4ns 0 0 0 1.03 KB
#7752 CallElasticsearch net472 3.64μs 7.42ns 28.7ns 0.163 0 0 1.04 KB
#7752 CallElasticsearchAsync net6.0 1.86μs 2.64ns 10.2ns 0 0 0 1.01 KB
#7752 CallElasticsearchAsync netcoreapp3.1 2.41μs 9.17ns 35.5ns 0 0 0 1.08 KB
#7752 CallElasticsearchAsync net472 3.6μs 4.37ns 16.9ns 0.163 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.83μs 6.94ns 26ns 0 0 0 952 B
master ExecuteAsync netcoreapp3.1 2.54μs 2.88ns 11.2ns 0 0 0 952 B
master ExecuteAsync net472 2.66μs 3.5ns 13.5ns 0.133 0 0 915 B
#7752 ExecuteAsync net6.0 1.92μs 9.02ns 37.2ns 0 0 0 952 B
#7752 ExecuteAsync netcoreapp3.1 2.38μs 11.4ns 44.2ns 0 0 0 952 B
#7752 ExecuteAsync net472 2.59μs 1.59ns 6.16ns 0.142 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 7.02μs 7.86ns 30.4ns 0 0 0 2.36 KB
master SendAsync netcoreapp3.1 8.33μs 8.82ns 34.2ns 0 0 0 2.9 KB
master SendAsync net472 12.3μs 9.24ns 35.8ns 0.487 0 0 3.18 KB
#7752 SendAsync net6.0 6.98μs 4.99ns 18.7ns 0 0 0 2.36 KB
#7752 SendAsync netcoreapp3.1 8.7μs 15.9ns 61.8ns 0 0 0 2.9 KB
#7752 SendAsync net472 12.1μs 8.92ns 34.5ns 0.483 0 0 3.18 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 256.97 KB 278.18 KB 21.22 KB 8.26%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472 274.5 KB 278.53 KB 4.02 KB 1.47%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0 43.98 KB 44.53 KB 552 B 1.26%

Fewer allocations 🎉 in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 275.83 KB 259.07 KB -16.76 KB -6.08%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 65.54 KB 57.34 KB -8.19 KB -12.50%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 45.2μs 259ns 2.15μs 0 0 0 43.98 KB
master StringConcatBenchmark netcoreapp3.1 49.9μs 257ns 1.23μs 0 0 0 43.34 KB
master StringConcatBenchmark net472 55.9μs 272ns 1.22μs 0 0 0 65.54 KB
master StringConcatAspectBenchmark net6.0 445μs 2.26μs 11.1μs 0 0 0 256.97 KB
master StringConcatAspectBenchmark netcoreapp3.1 525μs 1.79μs 6.2μs 0 0 0 275.83 KB
master StringConcatAspectBenchmark net472 404μs 2.13μs 10.4μs 0 0 0 274.5 KB
#7752 StringConcatBenchmark net6.0 41.7μs 217ns 1.11μs 0 0 0 44.53 KB
#7752 StringConcatBenchmark netcoreapp3.1 49.7μs 294ns 2.88μs 0 0 0 43.15 KB
#7752 StringConcatBenchmark net472 56.4μs 241ns 963ns 0 0 0 57.34 KB
#7752 StringConcatAspectBenchmark net6.0 491μs 2.25μs 8.1μs 0 0 0 278.18 KB
#7752 StringConcatAspectBenchmark netcoreapp3.1 506μs 2.5μs 10.6μs 0 0 0 259.07 KB
#7752 StringConcatAspectBenchmark net472 403μs 2.31μs 17μs 0 0 0 278.53 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.64μs 13.9ns 72.3ns 0 0 0 1.7 KB
master EnrichedLog netcoreapp3.1 3.57μs 8.07ns 31.2ns 0 0 0 1.7 KB
master EnrichedLog net472 3.84μs 4.18ns 16.2ns 0.251 0 0 1.64 KB
#7752 EnrichedLog net6.0 2.73μs 0.754ns 2.72ns 0 0 0 1.7 KB
#7752 EnrichedLog netcoreapp3.1 3.63μs 17.2ns 73ns 0 0 0 1.7 KB
#7752 EnrichedLog net472 3.88μs 2.6ns 9.73ns 0.25 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 123μs 38ns 137ns 0 0 0 4.31 KB
master EnrichedLog netcoreapp3.1 127μs 111ns 401ns 0 0 0 4.31 KB
master EnrichedLog net472 169μs 53.8ns 208ns 0 0 0 4.52 KB
#7752 EnrichedLog net6.0 122μs 69.2ns 259ns 0 0 0 4.31 KB
#7752 EnrichedLog netcoreapp3.1 126μs 99.5ns 345ns 0 0 0 4.31 KB
#7752 EnrichedLog net472 166μs 36.9ns 143ns 0 0 0 4.52 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 5.27μs 15.9ns 59.6ns 0 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 6.62μs 24.1ns 90.1ns 0 0 0 2.26 KB
master EnrichedLog net472 7.64μs 4.03ns 15.1ns 0.306 0 0 2.08 KB
#7752 EnrichedLog net6.0 4.88μs 7.52ns 29.1ns 0 0 0 2.26 KB
#7752 EnrichedLog netcoreapp3.1 7.13μs 15.3ns 59.2ns 0 0 0 2.26 KB
#7752 EnrichedLog net472 7.52μs 11.2ns 43.3ns 0.302 0 0 2.08 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2μs 10.1ns 44ns 0 0 0 1.2 KB
master SendReceive netcoreapp3.1 2.59μs 11.8ns 45.7ns 0 0 0 1.2 KB
master SendReceive net472 3.12μs 2.8ns 10.5ns 0.186 0 0 1.2 KB
#7752 SendReceive net6.0 1.98μs 7.29ns 27.3ns 0 0 0 1.2 KB
#7752 SendReceive netcoreapp3.1 2.6μs 12.5ns 53.1ns 0 0 0 1.2 KB
#7752 SendReceive net472 3.08μs 3.82ns 14.8ns 0.186 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.35μs 13.7ns 52.9ns 0 0 0 1.58 KB
master EnrichedLog netcoreapp3.1 5.56μs 10.1ns 39.2ns 0 0 0 1.63 KB
master EnrichedLog net472 6.98μs 6.73ns 25.2ns 0.314 0 0 2.03 KB
#7752 EnrichedLog net6.0 4.36μs 8.01ns 30ns 0 0 0 1.58 KB
#7752 EnrichedLog netcoreapp3.1 5.61μs 17.8ns 69.1ns 0 0 0 1.63 KB
#7752 EnrichedLog net472 6.68μs 5.86ns 22.7ns 0.3 0 0 2.03 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 765ns 4.08ns 23.1ns 0 0 0 576 B
master StartFinishSpan netcoreapp3.1 953ns 4.91ns 24.6ns 0 0 0 576 B
master StartFinishSpan net472 950ns 0.151ns 0.545ns 0.09 0 0 578 B
master StartFinishScope net6.0 926ns 0.157ns 0.566ns 0 0 0 696 B
master StartFinishScope netcoreapp3.1 1.19μs 5.65ns 22.6ns 0 0 0 696 B
master StartFinishScope net472 1.14μs 0.698ns 2.7ns 0.103 0 0 658 B
#7752 StartFinishSpan net6.0 776ns 3.54ns 14.1ns 0 0 0 576 B
#7752 StartFinishSpan netcoreapp3.1 957ns 4.83ns 21.6ns 0 0 0 576 B
#7752 StartFinishSpan net472 957ns 0.0867ns 0.325ns 0.0917 0 0 578 B
#7752 StartFinishScope net6.0 926ns 0.337ns 1.31ns 0 0 0 696 B
#7752 StartFinishScope netcoreapp3.1 1.2μs 5.95ns 26.6ns 0 0 0 696 B
#7752 StartFinishScope net472 1.15μs 0.749ns 2.9ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.08μs 1.08ns 4.19ns 0 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 1.4μs 6.58ns 24.6ns 0 0 0 696 B
master RunOnMethodBegin net472 1.45μs 1.06ns 4.09ns 0.101 0 0 658 B
#7752 RunOnMethodBegin net6.0 1.09μs 5.39ns 22.2ns 0 0 0 696 B
#7752 RunOnMethodBegin netcoreapp3.1 1.42μs 6.85ns 28.3ns 0 0 0 696 B
#7752 RunOnMethodBegin net472 1.45μs 0.991ns 3.84ns 0.102 0 0 658 B

@dd-trace-dotnet-ci-bot
Copy link

dd-trace-dotnet-ci-bot bot commented Nov 3, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (7752) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration74.35 ± (74.28 - 75.12) ms74.63 ± (74.42 - 75.18) ms+0.4%✅⬆️
.NET Framework 4.8 - Bailout
duration80.22 ± (79.93 - 80.66) ms78.50 ± (78.51 - 79.06) ms-2.1%✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1117.66 ± (1119.26 - 1129.74) ms1112.12 ± (1115.62 - 1126.17) ms-0.5%✅
.NET Core 3.1 - Baseline
process.internal_duration_ms22.89 ± (22.82 - 22.97) ms22.86 ± (22.77 - 22.94) ms-0.2%✅
process.time_to_main_ms86.97 ± (86.66 - 87.28) ms86.85 ± (86.51 - 87.18) ms-0.1%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.91 ± (10.91 - 10.91) MB10.90 ± (10.90 - 10.91) MB-0.1%✅
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms22.61 ± (22.56 - 22.67) ms22.62 ± (22.56 - 22.67) ms+0.0%✅⬆️
process.time_to_main_ms87.33 ± (87.02 - 87.64) ms87.34 ± (87.01 - 87.66) ms+0.0%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.95 ± (10.95 - 10.95) MB10.94 ± (10.94 - 10.95) MB-0.0%✅
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms218.42 ± (216.98 - 219.87) ms218.22 ± (216.80 - 219.65) ms-0.1%✅
process.time_to_main_ms548.34 ± (547.21 - 549.46) ms543.81 ± (542.69 - 544.94) ms-0.8%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed52.48 ± (52.46 - 52.50) MB52.43 ± (52.41 - 52.45) MB-0.1%✅
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.4%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms21.65 ± (21.59 - 21.72) ms21.68 ± (21.62 - 21.74) ms+0.1%✅⬆️
process.time_to_main_ms75.20 ± (74.86 - 75.53) ms75.79 ± (75.42 - 76.15) ms+0.8%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.60 ± (10.60 - 10.60) MB10.63 ± (10.63 - 10.64) MB+0.3%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 6 - Bailout
process.internal_duration_ms21.70 ± (21.62 - 21.77) ms21.77 ± (21.70 - 21.84) ms+0.3%✅⬆️
process.time_to_main_ms77.97 ± (77.53 - 78.40) ms77.58 ± (77.21 - 77.94) ms-0.5%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.68 ± (10.67 - 10.68) MB10.66 ± (10.66 - 10.67) MB-0.1%✅
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms208.08 ± (206.52 - 209.65) ms206.57 ± (205.21 - 207.92) ms-0.7%✅
process.time_to_main_ms517.01 ± (515.90 - 518.11) ms506.87 ± (505.86 - 507.87) ms-2.0%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed51.60 ± (51.57 - 51.63) MB51.58 ± (51.56 - 51.61) MB-0.0%✅
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.0%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms19.66 ± (19.60 - 19.72) ms19.75 ± (19.68 - 19.82) ms+0.5%✅⬆️
process.time_to_main_ms74.09 ± (73.76 - 74.42) ms74.28 ± (73.90 - 74.66) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.65 ± (7.65 - 7.66) MB7.65 ± (7.64 - 7.65) MB-0.1%✅
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 8 - Bailout
process.internal_duration_ms19.74 ± (19.68 - 19.80) ms19.87 ± (19.81 - 19.93) ms+0.7%✅⬆️
process.time_to_main_ms76.38 ± (76.06 - 76.69) ms76.38 ± (76.08 - 76.67) ms-0.0%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.71 ± (7.71 - 7.72) MB7.71 ± (7.70 - 7.71) MB-0.1%✅
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms190.34 ± (189.38 - 191.29) ms190.99 ± (190.07 - 191.91) ms+0.3%✅⬆️
process.time_to_main_ms489.56 ± (488.57 - 490.54) ms484.28 ± (483.17 - 485.39) ms-1.1%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed38.85 ± (38.81 - 38.88) MB38.76 ± (38.72 - 38.80) MB-0.2%✅
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)+0.0%✅⬆️

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration192.93 ± (193.00 - 193.85) ms191.88 ± (191.71 - 192.36) ms-0.5%✅
.NET Framework 4.8 - Bailout
duration195.89 ± (195.70 - 196.15) ms196.03 ± (195.74 - 196.38) ms+0.1%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1167.84 ± (1170.49 - 1180.26) ms1162.87 ± (1166.24 - 1174.82) ms-0.4%✅
.NET Core 3.1 - Baseline
process.internal_duration_ms187.65 ± (187.23 - 188.07) ms188.22 ± (187.73 - 188.70) ms+0.3%✅⬆️
process.time_to_main_ms80.39 ± (80.20 - 80.59) ms80.60 ± (80.41 - 80.80) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.07 ± (16.04 - 16.10) MB16.19 ± (16.16 - 16.22) MB+0.8%✅⬆️
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (19 - 20)-0.3%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms187.30 ± (186.99 - 187.61) ms187.78 ± (187.42 - 188.13) ms+0.3%✅⬆️
process.time_to_main_ms81.56 ± (81.44 - 81.68) ms81.70 ± (81.55 - 81.85) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.15 ± (16.12 - 16.17) MB16.20 ± (16.17 - 16.24) MB+0.4%✅⬆️
runtime.dotnet.threads.count21 ± (20 - 21)21 ± (20 - 21)+0.1%✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms391.93 ± (389.73 - 394.14) ms394.38 ± (391.71 - 397.06) ms+0.6%✅⬆️
process.time_to_main_ms514.76 ± (514.10 - 515.42) ms516.25 ± (515.67 - 516.84) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed62.85 ± (62.70 - 63.01) MB62.86 ± (62.70 - 63.02) MB+0.0%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)-0.0%✅
.NET 6 - Baseline
process.internal_duration_ms191.69 ± (191.29 - 192.08) ms192.01 ± (191.65 - 192.37) ms+0.2%✅⬆️
process.time_to_main_ms69.58 ± (69.41 - 69.75) ms69.91 ± (69.74 - 70.07) ms+0.5%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.08 ± (15.94 - 16.22) MB16.29 ± (16.19 - 16.38) MB+1.3%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)19 ± (18 - 19)+2.5%✅⬆️
.NET 6 - Bailout
process.internal_duration_ms190.48 ± (190.15 - 190.81) ms191.22 ± (190.93 - 191.52) ms+0.4%✅⬆️
process.time_to_main_ms70.42 ± (70.31 - 70.53) ms70.68 ± (70.59 - 70.77) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed15.98 ± (15.82 - 16.13) MB16.24 ± (16.10 - 16.37) MB+1.6%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 20)+1.4%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms413.82 ± (410.59 - 417.04) ms407.35 ± (404.56 - 410.14) ms-1.6%✅
process.time_to_main_ms485.74 ± (485.15 - 486.32) ms481.43 ± (480.88 - 481.98) ms-0.9%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed62.09 ± (61.95 - 62.23) MB61.89 ± (61.74 - 62.04) MB-0.3%✅
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 30)+0.2%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms189.57 ± (189.19 - 189.95) ms189.42 ± (189.15 - 189.68) ms-0.1%✅
process.time_to_main_ms69.10 ± (68.89 - 69.31) ms68.91 ± (68.76 - 69.06) ms-0.3%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.77 ± (11.73 - 11.82) MB11.79 ± (11.76 - 11.82) MB+0.2%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 18)+0.0%✅⬆️
.NET 8 - Bailout
process.internal_duration_ms189.00 ± (188.71 - 189.29) ms189.71 ± (189.35 - 190.06) ms+0.4%✅⬆️
process.time_to_main_ms69.85 ± (69.75 - 69.95) ms70.23 ± (70.09 - 70.37) ms+0.5%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.85 ± (11.81 - 11.88) MB11.82 ± (11.79 - 11.84) MB-0.3%✅
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)+0.3%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms361.34 ± (359.79 - 362.89) ms360.78 ± (359.36 - 362.21) ms-0.2%✅
process.time_to_main_ms463.68 ± (463.03 - 464.32) ms457.01 ± (456.39 - 457.64) ms-1.4%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed50.34 ± (50.30 - 50.38) MB50.29 ± (50.25 - 50.33) MB-0.1%✅
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)+0.0%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (75ms)  : 69, 81
    master - mean (75ms)  : 69, 80

    section Bailout
    This PR (7752) - mean (79ms)  : 75, 83
    master - mean (80ms)  : 75, 86

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (1,121ms)  : 1044, 1197
    master - mean (1,125ms)  : 1046, 1203
Loading
FakeDbCommand (.NET Core 3.1) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (117ms)  : 111, 123
    master - mean (117ms)  : 112, 123

    section Bailout
    This PR (7752) - mean (117ms)  : 111, 123
    master - mean (117ms)  : 111, 124

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (796ms)  : 752, 839
    master - mean (799ms)  : 749, 848
Loading
FakeDbCommand (.NET 6) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (105ms)  : 97, 112
    master - mean (104ms)  : 97, 111

    section Bailout
    This PR (7752) - mean (106ms)  : 99, 114
    master - mean (106ms)  : 98, 115

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (745ms)  : 707, 782
    master - mean (757ms)  : 715, 800
Loading
FakeDbCommand (.NET 8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (103ms)  : 96, 110
    master - mean (102ms)  : 95, 110

    section Bailout
    This PR (7752) - mean (105ms)  : 98, 112
    master - mean (105ms)  : 98, 112

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (707ms)  : 676, 739
    master - mean (709ms)  : 681, 737
Loading
HttpMessageHandler (.NET Framework 4.8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (192ms)  : 189, 195
    master - mean (193ms)  : 188, 199

    section Bailout
    This PR (7752) - mean (196ms)  : 193, 199
    master - mean (196ms)  : 194, 198

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (1,171ms)  : 1108, 1233
    master - mean (1,175ms)  : 1101, 1250
Loading
HttpMessageHandler (.NET Core 3.1) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (277ms)  : 271, 283
    master - mean (277ms)  : 272, 282

    section Bailout
    This PR (7752) - mean (277ms)  : 273, 282
    master - mean (277ms)  : 273, 281

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (942ms)  : 896, 988
    master - mean (939ms)  : 900, 978
Loading
HttpMessageHandler (.NET 6) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (270ms)  : 265, 275
    master - mean (270ms)  : 264, 275

    section Bailout
    This PR (7752) - mean (270ms)  : 267, 273
    master - mean (269ms)  : 265, 273

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (919ms)  : 875, 962
    master - mean (929ms)  : 873, 986
Loading
HttpMessageHandler (.NET 8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (268ms)  : 264, 272
    master - mean (269ms)  : 262, 276

    section Bailout
    This PR (7752) - mean (269ms)  : 264, 274
    master - mean (268ms)  : 264, 272

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (849ms)  : 831, 867
    master - mean (858ms)  : 837, 879
Loading

@datadog-official

This comment has been minimized.

Sign in to view
pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/build/_build/Build.Steps.cs
var filter = (string.IsNullOrWhiteSpace(Filter), IsWin) switch
{
(false, _) => $"({Filter}){dockerFilter}{armFilter}",
(false, _) => $"({Filter})&(SkipInCI!=True){dockerFilter}{armFilter}",
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After adding IntegrationTestFilter: DockerGroup=$(dockerGroup)
 to the jobs, we started falling through this case, which ignored SkipInCI

pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/build/PackageVersionsGeneratorDefinitions.json
"SampleProjectName": "Samples.CosmosDb",
"NugetPackageSearchName": "Microsoft.Azure.Cosmos",
"MinVersion": "3.6.0",
"MinVersion": "3.12.0",
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to use HttpClientFactory when constructing the SDK client for use with the emulator, which has a self-signed certificate that .net fails to validate by default. 3.12.0 is the first version that supports HttpClientFactory

pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/test/test-applications/integrations/Samples.CosmosDb/Program.cs
Comment on lines 107 to 109

var user = await database.CreateUserAsync("user");
Console.WriteLine($"{DateTime.Now:o}: Created user: {user.Resource.Id}\n");
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Emulator doesn't support creating users

pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/test/test-applications/integrations/Samples.CosmosDb/Program.cs Outdated
ApplicationName = "CosmosDBDotnetQuickstart",
RequestTimeout = TimeSpan.FromMinutes(10),
OpenTcpConnectionTimeout = TimeSpan.FromMinutes(1),
ConnectionMode = ConnectionMode.Gateway,
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Emulator only supports gateway mode

pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/test/test-applications/integrations/Samples.CosmosDb/Program.cs Outdated
RequestTimeout = TimeSpan.FromMinutes(10),
OpenTcpConnectionTimeout = TimeSpan.FromMinutes(1),
ConnectionMode = ConnectionMode.Gateway,
LimitToEndpoint = true,
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.cosmosclientoptions.limittoendpoint?view=azure-dotnet

Otherwise the SDK will attempt to connect to 127.0.0.1 after the first request

@pablomartinezbernardo pablomartinezbernardo changed the title Run cosmos tests on emulator Cosmos DB partial integration testing Nov 4, 2025
@pablomartinezbernardo pablomartinezbernardo marked this pull request as ready for review November 4, 2025 13:40
@pablomartinezbernardo pablomartinezbernardo requested review from a team as code owners November 4, 2025 13:40
bouwkast
bouwkast reviewed Nov 4, 2025
View reviewed changes
Copy link
Collaborator

@bouwkast bouwkast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure how I feel about this, one reason why I closed #6683 was because the docker container couldn't run all of the necessary commands.

I think this is running into the same / similar issue where only a small subset of the functionality of CosmosDb works within the docker container.

I think we should maybe consider to go back to running the Cosmos Emulator (Windows) in CI as opposed to this

@pablomartinezbernardo
Copy link
Contributor Author

pablomartinezbernardo commented Nov 4, 2025

I think this is running into the same / similar issue where only a small subset of the functionality of CosmosDb works within the docker container.

It is a subset, but I would argue against it being small. The only opeartions that we are missing are queries over databases, containers and users. Operations on items, which are the most common scenarios, are supported. Some complex operations on items are not supported, but that becomes irrelevant, as testing the simpler cases completely tests our instrumentation.

I think we should maybe consider to go back to running the Cosmos Emulator (Windows) in CI as opposed to this

To defend your case... it is not only a Windows emulator, it can be run on Linux. Just not on arm64. It also looks like it will be heavier than the linux-based one:

docker images mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator                                                                                                                                                                                           
REPOSITORY                                               TAG             IMAGE ID       CREATED        SIZE
mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator   latest          bbf28177ceb0   2 months ago   3.06GB
mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator   vnext-preview   efe42b6f3e32   3 months ago   900MB

It's a valid option as well, the con is that there's no way to run it on arm machines (Not even with QEMU), which means

  • No local testing for mac users
  • No arm CI, as we are running dependencies on arm as well

Let's hear more opinions about this!

@bouwkast
Copy link
Collaborator

bouwkast commented Nov 4, 2025

This is the emulator that I was mentioning (non-Docker) https://learn.microsoft.com/en-us/azure/cosmos-db/emulator-release-notes - but yeah your limitations still apply.

We used to run this in CI, but it proved to be too flakey.

Could we do something where we split between the old way of testing and the new way of testing ->

{
    "IntegrationName": "CosmosDb",
    "SampleProjectName": "Samples.CosmosDb",
    "NugetPackageSearchName": "Microsoft.Azure.Cosmos",
    "MinVersion": "3.6.0",
    "MaxVersionExclusive": "4.0.0",
    "SpecificVersions": [
      "3.6.*",
      "3.*.*"
    ],
    "VersionConditions": [{
      "MinVersion": "3.29.0",
      "ExcludeTargetFrameworks": ["netcoreapp2.1", "netcoreapp3.0"]
    }]
  },
  {
    "IntegrationName": "CosmosDb_Emulator",
    "SampleProjectName": "Samples.CosmosDb",
    "NugetPackageSearchName": "Microsoft.Azure.Cosmos",
    "MinVersion": "3.12.0",
    "MaxVersionExclusive": "4.0.0",
    "SpecificVersions": [
      "3.12.*",
      "3.*.*"
    ],
    "VersionConditions": [{
      "MinVersion": "3.29.0",
      "ExcludeTargetFrameworks": ["netcoreapp2.1", "netcoreapp3.0"]
    }]

bouwkast
bouwkast reviewed Nov 4, 2025
View reviewed changes
docker-compose.yml
Comment on lines +68 to +71
cosmosdb-emulator_arm64:
image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
command: ["--protocol", "https"]

Copy link
Collaborator

@bouwkast bouwkast Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will require a VM update after merging to not slow down CI for too long - I'd think that we could probably do so around .NET 10 GA release

Copy link
Collaborator

@bouwkast bouwkast Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😢
We will have to update the VMs for this, we can merge it and not update the VMs but there will be some additional pipeline time hit.

I guess fine, I'll make another task to update them again after this gets merged.

@pablomartinezbernardo pablomartinezbernardo marked this pull request as draft November 7, 2025 12:58
@pablomartinezbernardo pablomartinezbernardo marked this pull request as ready for review November 7, 2025 14:21
bouwkast
bouwkast approved these changes Nov 13, 2025
View reviewed changes
Copy link
Collaborator

@bouwkast bouwkast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice thanks!

Will need the merge conflicts fixed and the VMs will need to be updated again at some point to pre pull the new docker image.

@pablomartinezbernardo
Run cosmos tests on emulator
4ac946b 
# Conflicts:
#	tracer/build/PackageVersionsLatestMajors.g.props
#	tracer/build/PackageVersionsLatestMinors.g.props
#	tracer/build/PackageVersionsLatestSpecific.g.props
#	tracer/build/supported_versions.json
#	tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/PackageVersionsLatestMajors.g.cs
#	tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/PackageVersionsLatestMinors.g.cs
#	tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/PackageVersionsLatestSpecific.g.cs
@pablomartinezbernardo
.net fx compat
fa2633d
@pablomartinezbernardo
Use the correct image, protocol https, specific container for arm, an…
430984c 
…d change name
@pablomartinezbernardo
Hostname scrubbers
6cab3f2
@pablomartinezbernardo
3.12.0 should also be tested
a1420e1
@pablomartinezbernardo
Avoid unnecessary diff changes
d0c60f4
@pablomartinezbernardo
LimitToEndpoint = true && SkipInCI
695e33f
@pablomartinezbernardo
Separate sample app for vnext CI tests
508ee5e
@pablomartinezbernardo pablomartinezbernardo merged commit 936f949 into master Nov 14, 2025
153 checks passed
@pablomartinezbernardo pablomartinezbernardo deleted the pmartinez/cosmos-tests branch November 14, 2025 11:38
@github-actions github-actions bot added this to the vNext-v3 milestone Nov 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lucaspimentel lucaspimentel lucaspimentel approved these changes

@bouwkast bouwkast bouwkast approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

vNext-v3

Development

Successfully merging this pull request may close these issues.

4 participants

@pablomartinezbernardo @bouwkast @lucaspimentel