feat: trace self-observability - otlptracegrpc exporter metrics #7142

Mojachieee · 2025-08-07T14:24:55Z

exporters/otlp/otlptrace/otlptracegrpc/client.go

codecov · 2025-08-07T14:26:26Z

Codecov Report

❌ Patch coverage is 90.55118% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.9%. Comparing base (eda888f) to head (980b74b).
⚠️ Report is 140 commits behind head on main.

Files with missing lines	Patch %	Lines
exporters/otlp/otlptrace/otlptracegrpc/client.go	88.2%	8 Missing and 4 partials ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##            main   #7142    +/-   ##
======================================
  Coverage   82.9%   82.9%            
======================================
  Files        264     265     +1     
  Lines      24628   24754   +126     
======================================
+ Hits       20423   20535   +112     
- Misses      3822    3832    +10     
- Partials     383     387     +4

Files with missing lines	Coverage Δ
...rters/otlp/otlptrace/otlptracegrpc/internal/x/x.go	`100.0% <100.0%> (ø)`
exporters/otlp/otlptrace/otlptracegrpc/client.go	`89.9% <88.2%> (-1.2%)`	⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

exporters/otlp/otlptrace/otlptracegrpc/internal/x/x.go

exporters/otlp/otlptrace/otlptracegrpc/client.go

flc1125 · 2025-08-07T14:46:54Z

We are missing a description similar to this: https://github.com/open-telemetry/opentelemetry-go/pull/7027/files#diff-f49e6c5c367e6599dcbde1bd811bc47159e72e190936055e27f5d65739eb12d8R9-R11

flc1125 · 2025-08-07T15:00:13Z

The following indicators are missing some attributes (including unit tests):

https://github.com/open-telemetry/semantic-conventions/blob/v1.36.0/docs/otel/sdk-metrics.md#metric-otelsdkexporterspaninflight

otel.sdk.exporter.span.inflight
- server.address
- server.port
otel.sdk.exporter.span.exported
- server.address
- server.port
otel.sdk.exporter.operation.duration
- server.address
- server.port
- rpc.grpc.status_code

exporters/otlp/otlptrace/otlptracegrpc/client_test.go

CHANGELOG.md

Co-authored-by: Flc゛ <[email protected]>

Copilot

Pull Request Overview

This PR adds experimental self-observability metrics to the OTLP trace gRPC exporter. When enabled via the OTEL_GO_X_SELF_OBSERVABILITY environment variable, the exporter will emit metrics tracking span export operations including in-flight spans, exported spans count, and operation duration.

Introduces an experimental feature flag system for enabling self-observability
Adds metric instrumentation to track span export operations in the gRPC exporter
Updates module dependencies to include required metric packages

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`exporters/otlp/otlptrace/otlptracegrpc/internal/x/x.go`	Implements experimental feature flag system for self-observability
`exporters/otlp/otlptrace/otlptracegrpc/internal/x/x_test.go`	Tests for the experimental feature flag functionality
`exporters/otlp/otlptrace/otlptracegrpc/client.go`	Adds metric instrumentation to the gRPC client for tracking export operations
`exporters/otlp/otlptrace/otlptracegrpc/client_test.go`	Comprehensive tests for self-observability metrics functionality
`exporters/otlp/otlptrace/otlptracegrpc/go.mod`	Updates dependencies to include metric packages
`CHANGELOG.md`	Documents the new experimental feature

exporters/otlp/otlptrace/otlptracegrpc/client_test.go

Mojachieee · 2025-08-08T10:34:30Z

The following indicators are missing some attributes (including unit tests):

https://github.com/open-telemetry/semantic-conventions/blob/v1.36.0/docs/otel/sdk-metrics.md#metric-otelsdkexporterspaninflight

otel.sdk.exporter.span.inflight

server.address

server.port

otel.sdk.exporter.span.exported

server.address

server.port

otel.sdk.exporter.operation.duration

server.address

server.port

rpc.grpc.status_code

These are all added now

exporters/otlp/otlptrace/otlptracegrpc/client.go

CHANGELOG.md

…vability

Co-authored-by: Flc゛ <[email protected]>

flc1125

One last remaining issue.

Also, please resolve the conflicts.

Finally: Thank you for your contribution.

exporters/otlp/otlptrace/otlptracegrpc/client_unit_test.go

Co-authored-by: Flc゛ <[email protected]>

MrAlias · 2025-08-13T20:54:10Z

exporters/otlp/otlptrace/otlptracegrpc/client.go

+	m := mp.Meter("go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc",
+		metric.WithInstrumentationVersion(sdk.Version()),
+		metric.WithSchemaURL(semconv.SchemaURL))


Suggested change

m := mp.Meter("go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc",

metric.WithInstrumentationVersion(sdk.Version()),

metric.WithSchemaURL(semconv.SchemaURL))

m := mp.Meter(

"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc",

metric.WithInstrumentationVersion(sdk.Version()),

metric.WithSchemaURL(semconv.SchemaURL),

)

MrAlias · 2025-08-13T20:57:08Z

exporters/otlp/otlptrace/otlptracegrpc/client.go

+	c.initSelfObservability()
+


Flattening initSelfObservability seems appropriate. This is the only call site and this function is scoped to setup a new client, which includes telemetry.

MrAlias · 2025-08-13T20:58:28Z

exporters/otlp/otlptrace/otlptracegrpc/client.go

+
+		defer func() {
+			duration := time.Since(start)
+			durationAttrs := make([]attribute.KeyValue, 0, len(c.selfObservabilityAttrs)+2)


This is allocated every call. A pool should be used to amortize the slice allocation.

MrAlias · 2025-08-13T21:04:47Z

exporters/otlp/otlptrace/otlptracegrpc/client.go

+			for _, ss := range ps.ScopeSpans {
+				spanCount += len(ss.Spans)


Handle nil values.

Suggested change

for _, ss := range ps.ScopeSpans {

spanCount += len(ss.Spans)

for _, ss := range ps.GetScopeSpans() {

spanCount += len(ss.GetSpans())

MrAlias · 2025-08-13T21:20:44Z

exporters/otlp/otlptrace/otlptracegrpc/client.go

+			durationAttrs := make([]attribute.KeyValue, 0, len(c.selfObservabilityAttrs)+2)
+			durationAttrs = append(durationAttrs, c.selfObservabilityAttrs...)
+			durationAttrs = append(durationAttrs,
+				c.operationDurationMetric.AttrRPCGRPCStatusCode(otelconv.RPCGRPCStatusCodeAttr(status.Code(err))))
+
+			exportedAttrs := make([]attribute.KeyValue, 0, len(c.selfObservabilityAttrs)+1)
+			exportedAttrs = append(exportedAttrs, c.selfObservabilityAttrs...)
+
+			if err != nil {
+				// Try to extract the underlying gRPC status error, if there is one
+				rootErr := err
+				if s, ok := status.FromError(err); ok {
+					rootErr = s.Err()
+				}


Exact allocations can be made here for the cost of a few more branches which is worth it.

Suggested change

durationAttrs := make([]attribute.KeyValue, 0, len(c.selfObservabilityAttrs)+2)

durationAttrs = append(durationAttrs, c.selfObservabilityAttrs...)

durationAttrs = append(durationAttrs,

c.operationDurationMetric.AttrRPCGRPCStatusCode(otelconv.RPCGRPCStatusCodeAttr(status.Code(err))))

exportedAttrs := make([]attribute.KeyValue, 0, len(c.selfObservabilityAttrs)+1)

exportedAttrs = append(exportedAttrs, c.selfObservabilityAttrs...)

if err != nil {

// Try to extract the underlying gRPC status error, if there is one

rootErr := err

if s, ok := status.FromError(err); ok {

rootErr = s.Err()

}

rootErr := err

// Extract the underlying gRPC status error, if there is one.

if s, ok := status.FromError(err); ok {

rootErr = s.Err()

}

n := len(c.selfObservabilityAttrs)

var durationAttrs, exportedAttrs []attribute.KeyValue

if rootErr != nil {

durationAttrs = make([]attribute.KeyValue, n, n+2)

exportedAttrs = make([]attribute.KeyValue, n, n+1)

} else {

durationAttrs = make([]attribute.KeyValue, n, n+1)

exportedAttrs = make([]attribute.KeyValue, n, n)

}

_ = copy(durationAttrs, c.selfObservabilityAttrs)

scAttr := c.operationDurationMetric.AttrRPCGRPCStatusCode(otelconv.RPCGRPCStatusCodeAttr(status.Code(err)))

durationAttrs = append(durationAttrs, scAttr)

_ = copy(exportedAttrs, c.selfObservabilityAttrs)

if err != nil {

MrAlias · 2025-08-13T21:39:44Z

exporters/otlp/otlptrace/otlptracegrpc/client.go

+// nextExporterID returns a new unique ID for an exporter.
+// the starting value is 0, and it increments by 1 for each call.


nit

Suggested change

// nextExporterID returns a new unique ID for an exporter.

// the starting value is 0, and it increments by 1 for each call.

// nextExporterID returns a monotonically increasing int64 starting at 0

MrAlias · 2025-08-13T21:41:13Z

exporters/otlp/otlptrace/otlptracegrpc/client_test.go

+						DataPoints: []metricdata.HistogramDataPoint[float64]{
+							{
+								Attributes: attribute.NewSet(
+									semconv.OTelComponentName("otlp_grpc_span_exporter/1"),


This relies on test execution order. It is brittle and will break when test are run in parallel or new cases are added. The generator needs to be reset per test case or this needs to not be evaluated as strictly.

MrAlias · 2025-08-13T21:42:28Z

exporters/otlp/otlptrace/otlptracegrpc/client_test.go

+			if tt.enabled {
+				t.Setenv("OTEL_GO_X_SELF_OBSERVABILITY", "true")
+			}


There are two test cases and this conditional splits them. They should be made into their own tests to just remove the complexity being added to accommodate everything here.

MrAlias · 2025-08-13T21:42:46Z

exporters/otlp/otlptrace/otlptracegrpc/client_test.go

+			}
+
+			original := otel.GetMeterProvider()
+			defer otel.SetMeterProvider(original)


Suggested change

defer otel.SetMeterProvider(original)

t.Cleanup(func() { otel.SetMeterProvider(original) })

MrAlias · 2025-08-13T21:43:13Z

exporters/otlp/otlptrace/otlptracegrpc/client_unit_test.go

 	}
 }
+
+func Test_getServerAttrs(t *testing.T) {


Suggested change

func Test_getServerAttrs(t *testing.T) {

func TestGetServerAttrs(t *testing.T) {

pellared · 2025-09-01T19:55:12Z

@Mojachieee PTAL #7272

flc1125 · 2025-09-11T01:24:04Z

Hi, since this process involves specification adjustments and historical review records (which may contain invalid review suggestions), it’s impossible to tell which items need attention amid the large volume of information.

Could we create a new PR based on the current branch before preparing for the review, so that subsequent reviews can proceed more smoothly?

Thanks~

MrAlias · 2025-09-15T20:50:47Z

@Mojachieee please take a look at #7007. I have updated the issue with a check list of items from our project Observability guidelines that need to be completed in this PR.

As @flc1125 pointed out, you may find it easier to open a new PR. But feel free to update this PR as well. If you plan to update this PR, please mark it as a draft while you are working to address the items listed in #7007 and the Observability guidelines.

Mojachieee added 3 commits August 7, 2025 10:53

first stab at self-observability for grpctrace

855af09

record inflight, duration and exported metrics

d00f6d5

add test + fix attributes

e43d525

Mojachieee requested review from MrAlias, XSAM, dashpole, dmathieu, flc1125 and pellared as code owners August 7, 2025 14:24

Mojachieee commented Aug 7, 2025

View reviewed changes

exporters/otlp/otlptrace/otlptracegrpc/client.go Outdated Show resolved Hide resolved

update changelog

b56985d

flc1125 reviewed Aug 7, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Apply suggestions from code review

fc8906e

Co-authored-by: Flc゛ <[email protected]>

flc1125 requested a review from Copilot August 7, 2025 15:17

Copilot AI reviewed Aug 7, 2025

View reviewed changes

exporters/otlp/otlptrace/otlptracegrpc/client_test.go Outdated Show resolved Hide resolved

exporters/otlp/otlptrace/otlptracegrpc/client_test.go Outdated Show resolved Hide resolved

exporters/otlp/otlptrace/otlptracegrpc/client_test.go Outdated Show resolved Hide resolved

Mojachieee added 3 commits August 7, 2025 16:33

slighty move when metrics are recorded

f43e478

update docs

10ddefa

add server.address and server.port attributes

8f95619

Mojachieee and others added 4 commits August 8, 2025 11:34

Merge branch 'main' into grpc-trace-self-observability

ff89a0b

lint

6078876

fix godoc link

9d4986f

ignore links used in unit tests

0ad22d0

Mojachieee requested a review from flc1125 August 8, 2025 12:26

flc1125 reviewed Aug 11, 2025

View reviewed changes

Mojachieee and others added 4 commits August 11, 2025 10:26

address comments, cleanup flow

16ba326

Merge remote-tracking branch 'origin/main' into grpc-trace-self-obser…

2452347

…vability

lint

6e11c8a

Update CHANGELOG.md

5437bf0

Co-authored-by: Flc゛ <[email protected]>

Mojachieee requested a review from flc1125 August 11, 2025 09:35

lint

23a6a65

flc1125 approved these changes Aug 13, 2025

View reviewed changes

exporters/otlp/otlptrace/otlptracegrpc/client_unit_test.go Outdated Show resolved Hide resolved

Mojachieee and others added 2 commits August 13, 2025 10:06

Merge branch 'main' into grpc-trace-self-observability

73b7e7b

Update exporters/otlp/otlptrace/otlptracegrpc/client_unit_test.go

980b74b

Co-authored-by: Flc゛ <[email protected]>

MrAlias reviewed Aug 13, 2025

View reviewed changes

MrAlias added this to the v1.39.0 milestone Sep 15, 2025

Mojachieee closed this Sep 16, 2025

		for _, ss := range ps.ScopeSpans {
		spanCount += len(ss.Spans)

		// nextExporterID returns a new unique ID for an exporter.
		// the starting value is 0, and it increments by 1 for each call.

	// nextExporterID returns a new unique ID for an exporter.
	// the starting value is 0, and it increments by 1 for each call.
	// nextExporterID returns a monotonically increasing int64 starting at 0

	defer otel.SetMeterProvider(original)
	t.Cleanup(func() { otel.SetMeterProvider(original) })

	func Test_getServerAttrs(t *testing.T) {
	func TestGetServerAttrs(t *testing.T) {

feat: trace self-observability - otlptracegrpc exporter metrics #7142

feat: trace self-observability - otlptracegrpc exporter metrics #7142

Uh oh!

Conversation

Mojachieee commented Aug 7, 2025 • edited by flc1125 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flc1125 commented Aug 7, 2025

Uh oh!

flc1125 commented Aug 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mojachieee commented Aug 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flc1125 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pellared commented Sep 1, 2025

Uh oh!

flc1125 commented Sep 11, 2025

Uh oh!

MrAlias commented Sep 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Mojachieee commented Aug 7, 2025 •

edited by flc1125

Loading

codecov bot commented Aug 7, 2025 •

edited

Loading