feat: trace watchable messages #7403

shreealt · 2025-11-01T07:33:43Z

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #7324

Release Notes: Yes/No

Signed-off-by: Shreemaan Abhishek <[email protected]>

zirain · 2025-11-01T12:39:53Z

api/v1alpha1/envoygateway_metrics_types.go

 }

+// EnvoyGatewayTraces defines control plane tracing configurations.
+type EnvoyGatewayTraces struct {


In EG, we use tracing in other place, better to keep consistency.

gateway/api/v1alpha1/envoygateway_types.go

Line 524 in 4c36666

type RateLimitTracing struct {

I didn't understand your point.

the name should be EnvoyGatewayTracing instead of EnvoyGatewayTraces.

api/v1alpha1/envoygateway_metrics_types.go

api/v1alpha1/envoyproxy_metric_types.go

Signed-off-by: Shreemaan Abhishek <[email protected]>

codecov · 2025-11-02T12:05:36Z

Codecov Report

❌ Patch coverage is 45.85366% with 111 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.11%. Comparing base (c29a629) to head (9836161).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/gatewayapi/runner/runner.go	2.63%	36 Missing and 1 partial ⚠️
internal/provider/kubernetes/controller.go	43.47%	26 Missing ⚠️
internal/traces/register.go	20.00%	19 Missing and 1 partial ⚠️
internal/gatewayapi/resource/resource.go	50.00%	4 Missing and 4 partials ⚠️
internal/message/types.go	55.55%	4 Missing and 4 partials ⚠️
internal/xds/runner/runner.go	79.16%	3 Missing and 2 partials ⚠️
internal/admin/console/api.go	25.00%	2 Missing and 1 partial ⚠️
internal/cmd/server.go	0.00%	2 Missing ⚠️
internal/logging/log.go	81.81%	1 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (45.85%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7403      +/-   ##
==========================================
- Coverage   72.26%   71.11%   -1.16%     
==========================================
  Files         231      275      +44     
  Lines       34084    34940     +856     
==========================================
+ Hits        24632    24848     +216     
- Misses       7676     8296     +620     
- Partials     1776     1796      +20

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

arkodg · 2025-11-03T23:03:26Z

thanks for working on this @shreealt

is it possible to break this feature up into 2

the trace/span plumbing that can be implemented and trace ids be logged with current logger/stdout, that can be muted using log levels ( INFO/ERROR), this can be part of v1.6
in v1.7 we can add the API to export the traces and sample it

internal/cmd/server.go

arkodg · 2025-11-04T01:42:30Z

internal/xds/runner/runner.go

+				parentCtx = update.Value.Context
+			}
+
+			_, span := tracer.Start(parentCtx, "XdsRunner.subscribeAndTranslate")


can we add another child span to r.cache.GenerateNewSnapshot and use the trace id as the version inside it, so this id trickles into envoy proxy

done, but I hope this doesn't come off as a surprise to users?

Signed-off-by: Shreemaan Abhishek <[email protected]>

…feat/tracing

Signed-off-by: Shreemaan Abhishek <[email protected]>

…feat/tracing

arkodg · 2025-11-18T00:30:22Z

internal/gatewayapi/runner/runner.go

-			r.Logger.Info("received an update", "key", update.Key)
-			val := update.Value
+		func(update message.Update[string, *resource.ControllerResourcesContext], errChan chan error) {
+			parentCtx := context.Background()


is this defensive code needed ?

why not? 🤔

is it guaranteed that these values will not be nil?

arkodg · 2025-11-18T00:39:16Z

internal/gatewayapi/runner/runner.go

 			// so when a delete is triggered, delete all keys
-			if update.Delete || val == nil {
+			if update.Delete || valWrapper == nil || valWrapper.Resources == nil {
+				span.AddEvent("delete_all_keys")


this is not a useful span
instead can we create two child spans

one to track translate time

one to track statusUpdate

arkodg · 2025-11-18T00:43:33Z

internal/globalratelimit/runner/runner.go


 	message.HandleSubscription(message.Metadata{Runner: r.Name(), Message: message.XDSIRMessageName}, c,
-		func(update message.Update[string, *ir.Xds], errChan chan error) {
+		func(update message.Update[string, *message.XdsIRWithContext], errChan chan error) {


similar to the gateway-api runner can we add a parent span and 2 child span for translate & updateSnapshot

arkodg · 2025-11-18T00:46:13Z

internal/xds/cache/snapshotcache.go

-var Hash = cachev3.IDHash{}
+var (
+	Hash   = cachev3.IDHash{}
+	tracer = otel.Tracer("envoy-gateway/gateway-api")


Suggested change

tracer = otel.Tracer("envoy-gateway/gateway-api")

tracer = otel.Tracer("envoy-gateway/xds")

also does the envoy-gateway prefix exist since thats the idiomatic way to do things ?

yes, pretty much that.

arkodg · 2025-11-18T00:48:30Z

internal/xds/cache/snapshotcache.go

 	defer s.mu.Unlock()

-	version := s.newSnapshotVersion()
+	_, span := tracer.Start(ctx, "SnapshotCache.GenerateNewSnapshot")


can we keep things consistent and add spans in runner instead of the within the func ?

I think it's cleaner this way? 🤔

there are two usages of this function in runner.go, so the current way is better than wrapping GenerateNewSnapshot in two places?

arkodg · 2025-11-18T00:49:22Z

internal/xds/runner/runner.go

 	defaultMaxConnectionAgeGrace = 2 * time.Minute
 )

+var tracer = otel.Tracer("envoy-gateway/gateway-api")


Suggested change

var tracer = otel.Tracer("envoy-gateway/gateway-api")

var tracer = otel.Tracer("envoy-gateway/xds")

arkodg · 2025-11-18T00:53:19Z

internal/xds/runner/runner.go

+			traceLogger := r.Logger.WithTrace(parentCtx)
+			traceLogger.Info("received an update")
+
+			_, span := tracer.Start(parentCtx, "XdsRunner.subscribeAndTranslate")


can we add 2 child spans, one for translate and one for update snapshot

arkodg · 2025-11-18T00:55:51Z

test/helm/gateway-addons-helm/e2e.in.yaml

  enabled: false
 opentelemetry-collector:
  enabled: true
+  mode: deployment


can you undo the changes in test

arkodg · 2025-11-18T00:59:57Z

hey @shreealt left some comments, once those are addressed, can you also share before/after logs, tia !

arkodg · 2025-11-20T00:22:01Z

@shreealt we'll also need Equal functions for the new message structure, to avoid comparing Context and only compare the Data

Signed-off-by: Shreemaan Abhishek <[email protected]>

shreealt · 2025-11-21T07:03:07Z

/retest

Signed-off-by: Shreemaan Abhishek <[email protected]>

arkodg · 2025-11-21T23:22:23Z

internal/xds/translator/translator.go

 	emptyClusterName = "EmptyCluster"
 )

+var tracer = otel.Tracer("envoy-gateway/xds/translator")


lets not add any Otel things to the lib, but in the runner please, in case these libs are made public and used purely for translation

Signed-off-by: Shreemaan Abhishek <[email protected]>

shreealt added 8 commits October 31, 2025 07:35

feat: trace watchable messages

11155f0

Signed-off-by: Shreemaan Abhishek <[email protected]>

license header

f66879c

Signed-off-by: Shreemaan Abhishek <[email protected]>

lint

2c0d0a1

Signed-off-by: Shreemaan Abhishek <[email protected]>

default traces

4777f57

Signed-off-by: Shreemaan Abhishek <[email protected]>

disability

0af47d1

Signed-off-by: Shreemaan Abhishek <[email protected]>

deepcopy errors

c620c6e

Signed-off-by: Shreemaan Abhishek <[email protected]>

lint

aa6b91b

Signed-off-by: Shreemaan Abhishek <[email protected]>

e2e

2ad7399

Signed-off-by: Shreemaan Abhishek <[email protected]>

shreealt requested a review from a team as a code owner November 1, 2025 07:33

shreealt added 2 commits November 1, 2025 13:22

Merge branch 'main' of github.com:envoyproxy/gateway into feat/tracing

f8d9f42

Signed-off-by: Shreemaan Abhishek <[email protected]>

doc

ff03fd5

Signed-off-by: Shreemaan Abhishek <[email protected]>

zirain reviewed Nov 1, 2025

View reviewed changes

api/v1alpha1/envoygateway_metrics_types.go Outdated Show resolved Hide resolved

zirain reviewed Nov 1, 2025

View reviewed changes

api/v1alpha1/envoygateway_metrics_types.go Outdated Show resolved Hide resolved

zirain reviewed Nov 1, 2025

View reviewed changes

api/v1alpha1/envoyproxy_metric_types.go Outdated Show resolved Hide resolved

shreealt added 6 commits November 2, 2025 16:30

minor fixes

bf58e55

Signed-off-by: Shreemaan Abhishek <[email protected]>

rc

c0846eb

Signed-off-by: Shreemaan Abhishek <[email protected]>

generate

1c5e52e

Signed-off-by: Shreemaan Abhishek <[email protected]>

fix coverage tests

623406e

Signed-off-by: Shreemaan Abhishek <[email protected]>

fix: e2e config

fd41925

Signed-off-by: Shreemaan Abhishek <[email protected]>

runner test fix

8392b41

Signed-off-by: Shreemaan Abhishek <[email protected]>

Merge branch 'main' into feat/tracing

934dca2

shreealt mentioned this pull request Nov 3, 2025

Add support for watchable message tracing #7324

Open

arkodg reviewed Nov 4, 2025

View reviewed changes

internal/cmd/server.go Show resolved Hide resolved

arkodg reviewed Nov 4, 2025

View reviewed changes

shreealt added 3 commits November 4, 2025 10:38

logger

fcadab2

Signed-off-by: Shreemaan Abhishek <[email protected]>

mod

afc0812

Signed-off-by: Shreemaan Abhishek <[email protected]>

mod

0480ec5

Signed-off-by: Shreemaan Abhishek <[email protected]>

shreealt added 2 commits November 7, 2025 11:36

fix

3728b82

Signed-off-by: Shreemaan Abhishek <[email protected]>

Merge branch 'main' into feat/tracing

6c9d222

shreealt requested review from arkodg and zirain November 15, 2025 08:51

shreealt added 7 commits November 15, 2025 14:22

Merge branch 'main' into feat/tracing

3d3340b

revert api

d4b7eb4

Signed-off-by: Shreemaan Abhishek <[email protected]>

Merge branch 'feat/tracing' of github.com-work:shreealt/gateway into …

b59ff0d

…feat/tracing

rm type

c4b00c1

Signed-off-by: Shreemaan Abhishek <[email protected]>

Merge branch 'main' into feat/tracing

58ec2e2

Merge branch 'main' of github.com:envoyproxy/gateway into feat/tracing

3edaaf4

Signed-off-by: Shreemaan Abhishek <[email protected]>

Merge branch 'feat/tracing' of github.com-work:shreealt/gateway into …

520a840

…feat/tracing

arkodg reviewed Nov 18, 2025

View reviewed changes

more spans and cleanups

93f2ace

Signed-off-by: Shreemaan Abhishek <[email protected]>

lint

3bf3309

Signed-off-by: Shreemaan Abhishek <[email protected]>

arkodg reviewed Nov 21, 2025

View reviewed changes

shreealt added 2 commits November 23, 2025 11:11

equal and tests

1e1322b

Signed-off-by: Shreemaan Abhishek <[email protected]>

equal and tests

cd7fdd6

Signed-off-by: Shreemaan Abhishek <[email protected]>

shreealt force-pushed the feat/tracing branch from ab45164 to cd7fdd6 Compare November 23, 2025 05:33

no otel things in translator

9836161

Signed-off-by: Shreemaan Abhishek <[email protected]>

shreealt requested a review from arkodg November 23, 2025 08:42

	tracer = otel.Tracer("envoy-gateway/gateway-api")
	tracer = otel.Tracer("envoy-gateway/xds")

	var tracer = otel.Tracer("envoy-gateway/gateway-api")
	var tracer = otel.Tracer("envoy-gateway/xds")

feat: trace watchable messages #7403

Are you sure you want to change the base?

feat: trace watchable messages #7403

Conversation

shreealt commented Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zirain Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

arkodg commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arkodg Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arkodg commented Nov 18, 2025

Uh oh!

arkodg commented Nov 20, 2025

Uh oh!

shreealt commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shreealt commented Nov 1, 2025 •

edited

Loading

zirain Nov 1, 2025 •

edited

Loading

codecov bot commented Nov 2, 2025 •

edited

Loading

arkodg commented Nov 3, 2025 •

edited

Loading

arkodg Nov 18, 2025 •

edited

Loading