feat(telemetry): shutdown with a deadline by iunanua · Pull Request #276 · DataDog/dd-trace-rs

iunanua · 2026-07-09T13:33:32Z

What does this PR do?

Stop telemetry (It had been completely forgotten)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7bf16fe885

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db8024b5c7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

datadog-datadog-prod-us1 · 2026-07-10T08:41:28Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 16088ff | Docs | Datadog PR Page | Give us feedback!}

- Replace the bool/AtomicBool registration handshake with a type-safe TelemetryUser token so the global user count can no longer underflow; the decrement is consumed exactly once via TelemetryUser::trigger_stop. - Floor the telemetry shutdown wait to at least 1s and make it non-fatal: since telemetry is signalled to stop up front and drains in parallel, an already-drained worker is now observed as stopped instead of being force-cancelled and misreported as a timeout when the deadline passed. - Document that the wait is only bounded while libdatadog honors the cancel deadline (the real fix belongs upstream in the condvar wait). - Add coverage for the disabled-config path and the shutdown-timeout path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

paullegranddc

What happens if I create a tracer, shut it down then create a new one latter?
Will the telemetry worker restart, and still allow sending telemetry

iunanua · 2026-07-10T15:28:38Z

What happens if I create a tracer, shut it down then create a new one latter? Will the telemetry worker restart, and still allow sending telemetry

Nope, that's one suggestion made by codex. But I ruled it out because it's an unlikely scenario. Is it worth it?

paullegranddc · 2026-07-10T15:39:25Z

I ruled it out because it's an unlikely scenario. Is it worth it?

I'm not sure, but you could imagine a situation where the tracer is not used as a global, but a scoped object.

We can tolerate not receiving telemetry if a user restarts a tracer, but I think we should at least make sure that using a shutdown telemetry handle does not panic or cause memory leaks.
Also all the test you added test shut down against the TestHandle, not against the actual telemetry runtime. I am not sure that enough.
I'd like to see integration tests

Build a tracer, shut it down (stopping the process-global telemetry worker as the last user), then build and use another tracer and shut it down too. Telemetry is a non-restartable singleton, so this pins the graceful-degradation contract: the reuse must not panic, underflow the user count, or hang, and both shutdowns must return cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…fety Loop build->use->shutdown many times: after the first iteration stops the process-global worker, each later cycle pushes span activity through the stopped handle and stops it again. Asserts no panic/error/hang across cycles; comment documents running under a leak sanitizer to also cover leaks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A scoped tracer provider that is dropped (not shutdown() explicitly) must still release its telemetry-user registration: the SDK triggers shutdown from Drop on the last reference. Drop the provider off a blocking thread (its shutdown blocks) and assert it completes cleanly, then build/use/drop a second scoped provider to confirm the SDK-Drop -> token-release path stays safe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add three integration tests exercising distinct shutdown paths: - telemetry disabled: no user registered, stop/wait is skipped, shutdown clean - concurrent shutdown of several providers: races the shared atomic refcount - repeated shutdown on the same handle: idempotent, no refcount underflow Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

iunanua · 2026-07-29T15:03:16Z

I ruled it out because it's an unlikely scenario. Is it worth it?

I'm not sure, but you could imagine a situation where the tracer is not used as a global, but a scoped object.

Tracers (both global and scope-specific) are registered as telemetry users, so telemetry keeps track of active users and only shuts down when the last active user requests a shutdown.

We can tolerate not receiving telemetry if a user restarts a tracer, but I think we should at least make sure that using a shutdown telemetry handle does not panic or cause memory leaks. Also all the test you added test shut down against the TestHandle, not against the actual telemetry runtime. I am not sure that enough. I'd like to see integration tests

I have added integration test to prove that:

different combinations of scoped and global tracers shutdown correctly
calling shutdown twice or multiple times (from different tracers) does not panic
dropping a scoped tracer calls shutdown and there is no problem when dropping again another scoped tracer
shutdown has no problems with telemetry off
concurrent shutdown has no problems
calling shutdown twice from the same tracer does not panic

@paullegranddc let me know if you can think of any other interesting tests.

Shutdown telemetry with a deadline

7bf16fe

iunanua marked this pull request as ready for review July 9, 2026 13:37

iunanua requested a review from a team as a code owner July 9, 2026 13:37

chatgpt-codex-connector Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread datadog-opentelemetry/src/span_processor.rs Outdated

Comment thread datadog-opentelemetry/src/span_processor.rs Outdated

Comment thread datadog-opentelemetry/src/core/telemetry.rs Outdated

Comment thread datadog-opentelemetry/src/core/telemetry.rs Outdated

iunanua marked this pull request as draft July 9, 2026 14:03

Propagate an Error if timeout and handle multiple processors

769db9a

iunanua force-pushed the igor/telemetry-shutdown branch 2 times, most recently from 421d0ab to cfcebe6 Compare July 9, 2026 15:25

Trigger stop telemetry

db8024b

iunanua force-pushed the igor/telemetry-shutdown branch from cfcebe6 to db8024b Compare July 9, 2026 15:30

iunanua marked this pull request as ready for review July 9, 2026 15:31

chatgpt-codex-connector Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread datadog-opentelemetry/src/span_processor.rs Outdated

Comment thread datadog-opentelemetry/src/core/telemetry.rs

Comment thread datadog-opentelemetry/src/span_processor.rs Outdated

iunanua force-pushed the igor/telemetry-shutdown branch from 6c83089 to b8e18c9 Compare July 10, 2026 08:40

iunanua force-pushed the igor/telemetry-shutdown branch from b8e18c9 to f1df8df Compare July 10, 2026 09:15

iunanua force-pushed the igor/telemetry-shutdown branch from f1df8df to c876f77 Compare July 10, 2026 09:15

paullegranddc reviewed Jul 10, 2026

View reviewed changes

iunanua and others added 4 commits July 10, 2026 17:59

integration tests for global and local tracers

d11f97e

iunanua force-pushed the igor/telemetry-shutdown branch from dfb3c56 to 488f237 Compare July 29, 2026 14:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(telemetry): shutdown with a deadline - #276

feat(telemetry): shutdown with a deadline#276
iunanua wants to merge 9 commits into
mainfrom
igor/telemetry-shutdown

iunanua commented Jul 9, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 10, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

paullegranddc left a comment

Uh oh!

iunanua commented Jul 10, 2026

Uh oh!

paullegranddc commented Jul 10, 2026

Uh oh!

iunanua commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

iunanua commented Jul 9, 2026

What does this PR do?

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 10, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paullegranddc left a comment

Choose a reason for hiding this comment

Uh oh!

iunanua commented Jul 10, 2026

Uh oh!

paullegranddc commented Jul 10, 2026

Uh oh!

iunanua commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-datadog-prod-us1 Bot commented Jul 10, 2026 •

edited by datadog-official Bot

Loading