Overriding Decisions of the Built-in Trace Samplers via a "sample.priority" Like Span Attribute

[stoty]

I propose adding a feature to the built in Sampler classes (AlwaysOn, AlwaysOff, TraceIdRatioBased) which would allow the sampling decision can be overriden by setting a Span attribute.

This would let users force tracing on or off when starting a Span, whitout writing and registering a custom Sampler.

span = tracer.spanBuilder("configurable").setAtttribute("sample.priority", 1).startSpan().

Background:

I am trying to migrate Apache Phoenix from HTrace to OpenTelemetry tracing.
Apache Phoenix is an SQL Layer over HBase. HBase has built-in support for OpenTelemetry.
It depends on opentelemetry-api, and by default expects the implementation to be provided by the OpenTelemetry java agent, and configured via its environment variables.

One of my goals is making the tracing configurable from the JDBC API and the statement text.

Either on the connection level by setting a propery, or adding a JDBC URL parameter to
https://docs.oracle.com/javase/8/docs/api/java/sql/DriverManager.html#getConnection-java.lang.String-java.util.Properties-

Or on a query by query basis with a query hint like
SELECT /*+ TRACE=ON */ * FROM TABLE
Doing this is harder than I expected.

The Trace API doesn't seem to allow setting sampling explicitly, only via the Sampler object, which is not available in opentelemetry-api, only in opentelemetry-sdk.

I have looked at using the "parentbased_always_on" built-in sampler, and simply not starting a Span from Phoenix if don't want to enable tracing. However, this wouldn't work, because the underlaying HBase client library would start sampled spans for each call into it anyway, which is not what I need.

So I need a way to start either sampling on non-sampling Spans based on information in Phoenix.
The way to do it seems to be writing a custom sampler, setting some attribute on the Span, and then checking this on the Sampler to make the decision.

I believe that this a viable soultion, but I need add a compile-time dependency on opentelemetry-sdk for this, and I am afraid that adding my custom Sampler to an agent based setup may cause unforeseen complications.

Searching for solutions, I came across the sample.priority semantic attribute for Opentracing, which seems to do what I want, but I could not find a similar feature in OpenTelemetry

Looking at TraceIdRatioBasedSampler, it would be super easy to add an override based on a Span attribute, all it takes is a hash lookup and a comparison.

[trask]

hi @stoty!

the specification repository is probably the best place to raise this

(and check out open-telemetry/opentelemetry-specification#530 in case it's similar to your request/need)

[stoty]

Thanks a lot, @trask !

[stoty]

Fore the record,
open-telemetry/opentelemetry-specification#3725
is the new one.