Skip to content

Commit 4aa6f7d

Browse files
Fix GRPC instrumentation for BuildHttpErrorResponse. Enable integration tests. (#7457)
## Summary of changes [This error](https://app.datadoghq.com/error-tracking?query=source%3Adotnet%20-%2AStackOverflowException%2A%20-%2Adebugger%2A%20-%2Aappsec%2A&et-side=data&et-viz=events&order=total_count&refresh_mode=sliding&source=all&sp=%5B%7B%22p%22%3A%7B%22issueId%22%3A%225b4a9f2a-3ae6-11f0-9077-da7ad0900002%22%7D%2C%22i%22%3A%22error-tracking-issue%22%7D%5D&view=spans&from_ts=1756801478434&to_ts=1756887878434&live=true) was logged: ``` Error : Exception occurred when calling the CallTarget integration continuation. System.TypeInitializationException ---> Datadog.Trace.ClrProfiler.CallTarget.CallTargetInvokerException ---> System.ArgumentException at Datadog.Trace.Util.ThrowHelper.ThrowArgumentException(String message) at Datadog.Trace.ClrProfiler.CallTarget.Handlers.IntegrationMapper.CreateBeginMethodDelegate(Type integrationType, Type targetType, Type[] argumentsTypes) at Datadog.Trace.ClrProfiler.CallTarget.Handlers.BeginMethodHandler`6..cctor() ``` This error would only happen when method Grpc.AspNetCore.Server.Internal.GrpcProtocolHelpers.BuildHttpErrorResponse is called, which happens when there is an error during the Grpc call. In order to fix it, the signature of OnMethodBegin<TTarget, TGrpcStatusCode> in GrpcProtocolHelpersBuildHttpErrorResponseIntegration.cs had to be changed by adding generic types. To test this change, a new case has been added to the GRPC sample application. By adding this header to the request (context.Request.Headers["content-type"] = "application/json";), we can test this instrumentation by causing an error during the GRPC call: Content-Type 'application/json' is not supported Additionally, some fixes have been done in the test app. In the following code, we were always throwing the exception because the delay set in theVerySlow method was 300 msecs, lower than 600, so the method would finish on time, which is not the expected outcome : ``` try { _logger.LogInformation("Sending very slow request to self"); await client.VerySlowAsync(new HelloRequest { Name = "GreeterClient" }, deadline: DateTime.UtcNow.AddMilliseconds(600)); throw new Exception("Received reply, when should have exceeded deadline"); } ``` By setting a lower deadline, we make sure that the deadline is exceeded. We were not detecting this unexpected behavior because in grpctests.cs, we were capturing the exception ExitCodeException: ``` catch (ExitCodeException) { // There is a race condition in GRPC version < v2.43.0 that can cause ObjectDisposedException // when a deadline is exceeded. Skip the test if we hit it: grpc/grpc-dotnet#1550 if (!string.IsNullOrEmpty(packageVersion) && new Version(packageVersion) < new Version("2.43.0") && processResult is not null && processResult.StandardError.Contains("ObjectDisposedException")) { throw new SkipException("Hit race condition in GRPC deadline exceeded"); } } ``` After capturing the exception, we would not launch an SkipException but we would end the test without errors. A new throw statement has been added to control that we only skip the tests in the case of ObjectDisposedException. This change in the GRPC tests has also effectively enabled the GRPC legacy tests, that have a different instrumentation but were suffering the same issue (throw exception and skip). The snapshots of the legacy tests were not matching exactly the result obtained after enabling these tests. The instrumentation of the legacy methods or the legacy sample app have not changed significantly, so we would probably need to evaluate if the results that we are getting are the expected ones. Anyway, this work would probably be out of the scope of this PR. ## Reason for change ## Implementation details ## Test coverage ## Other details <!-- Fixes #{issue} --> <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. -->
1 parent 7f14b71 commit 4aa6f7d

File tree

14 files changed

+762
-320
lines changed

14 files changed

+762
-320
lines changed

tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Grpc/GrpcDotNet/GrpcAspNetCoreServer/GrpcProtocolHelpersBuildHttpErrorResponseIntegration.cs

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,14 @@
1-
// <copyright file="GrpcProtocolHelpersBuildHttpErrorResponseIntegration.cs" company="Datadog">
1+
// <copyright file="GrpcProtocolHelpersBuildHttpErrorResponseIntegration.cs" company="Datadog">
22
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
33
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
44
// </copyright>
55
#if !NET461
66

77
#nullable enable
88

9+
using System;
910
using System.ComponentModel;
11+
using System.Runtime.CompilerServices;
1012
using Datadog.Trace.ClrProfiler.CallTarget;
1113
using Datadog.Trace.Configuration;
1214
using Datadog.Trace.ExtensionMethods;
@@ -32,6 +34,8 @@ namespace Datadog.Trace.ClrProfiler.AutoInstrumentation.Grpc.GrpcDotNet.GrpcAspN
3234
[EditorBrowsable(EditorBrowsableState.Never)]
3335
public class GrpcProtocolHelpersBuildHttpErrorResponseIntegration
3436
{
37+
internal enum GrpcStatusCode;
38+
3539
/// <summary>
3640
/// OnMethodBegin callback
3741
/// </summary>
@@ -40,7 +44,7 @@ public class GrpcProtocolHelpersBuildHttpErrorResponseIntegration
4044
/// <param name="grpcStatusCode">The GRPC status code</param>
4145
/// <param name="message">The error message to set</param>
4246
/// <returns>Calltarget state value</returns>
43-
internal static CallTargetState OnMethodBegin(HttpResponse response, int httpStatusCode, int grpcStatusCode, string message)
47+
internal static CallTargetState OnMethodBegin<TTarget>(HttpResponse response, int httpStatusCode, GrpcStatusCode grpcStatusCode, string message)
4448
{
4549
var tracer = Tracer.Instance;
4650
if (GrpcCoreApiVersionHelper.IsSupported
@@ -49,7 +53,7 @@ internal static CallTargetState OnMethodBegin(HttpResponse response, int httpSta
4953
{
5054
// This code path is only called when there's a fundamental failure that isn't even processed
5155
// (e.g. wrong Http protocol, invalid content-type etc)
52-
GrpcCommon.RecordFinalStatus(span, grpcStatusCode, message, ex: null);
56+
GrpcCommon.RecordFinalStatus(span, (int)grpcStatusCode, message, ex: null);
5357

5458
// There won't be any response metadata, as interceptors haven't executed, but we can grab
5559
// the request metadata directly from the HttpRequest

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GrpcTests.cs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -132,8 +132,6 @@ public async Task IntegrationDisabled()
132132

133133
public class GrpcHttpsTests : GrpcTestsBase
134134
{
135-
private const string ServiceName = "Samples.GrpcDotNet";
136-
137135
public GrpcHttpsTests(ITestOutputHelper output)
138136
: base("GrpcDotNet", output, usesAspNetCore: true)
139137
{
@@ -426,6 +424,8 @@ static void FixVerySlowClientSpans(IImmutableList<MockSpan> spans)
426424
{
427425
throw new SkipException("Hit race condition in GRPC deadline exceeded");
428426
}
427+
428+
throw;
429429
}
430430

431431
bool IsGrpcClientSpan(MockSpan span)

0 commit comments

Comments
 (0)