How would you model spans for long-running operation (up to several hours)?

[Radrik5]

Suppose I have a service, that sometimes makes heavy SQL queries, that can take an hour to complete. Also, I can have a customer call asking why their request takes so long.

If I model spans traditionally: 1 server span for client request and 1 client span for DB request, both spans will be not available for analysis until DB request ends.

Is there another model for this use case?

Some ideas:

1. Export incomplete spans (e.g., after a minute of inactivity) and then overwrite them later with complete spans. I think it's currently not possible but can it be added later?
2. Split server span - move time from receiving client request to sending DB request into separate span - it will end early and will be available for analysis.
3. Split DB span - e.g. after 5 minutes end span as "not finished" and start another one for the rest of the operation.

[austinlparker]

Probably the most idiomatic way to model this would be to split the operation into several distinct traces, then use links to join them together. The server would write a complete trace when the request was made, then whatever service is long-polling the DB to check for completion would create traces for it's work that would link back to the initial trace.

[dyladan]

Another simple solution would be to emit log events when the request starts and when important events happen, or even just check-ins that there is no update yet. Is this an operation that is expected to take a long time or is this an unexpected anomaly? Your wording seems to imply that the operation is expected to take this long at least sometimes.

Incomplete span export would be simple to implement with our SDK implementation (onStart event in span processor), but you wouldn't get any check-ins until the span ends. Also, I'm not aware of any tracing backends which are able to display partial spans which obviously limits the value here. I'm also not sure what value it would provide. If your user is asking why a request is taking so long, having a span in your backend that is started but not ended doesn't tell you anything other than that the request has started and not ended, something your user already knows. You would have to enrich it with more interesting events along the way in order to see how far along the process is.

[Radrik5]

@austinlparker thanks for sharing the idea but I think it would be not convenient to use in Jaeger UI: when I find the trace with client request (using corresponding attributes), how would I find the trace with DB request? It's easy to go from DB request to client request but I need the opposite.

@dyladan thanks for your reply!

Is this an operation that is expected to take a long time or is this an unexpected anomaly? Your wording seems to imply that the operation is expected to take this long at least sometimes.

We're trying to make it an unexpected anomaly :) It should take several seconds but sometimes, when DB is very busy, it takes an hour or so.

I'm not aware of any tracing backends which are able to display partial spans which obviously limits the value here. I'm also not sure what value it would provide. If your user is asking why a request is taking so long, having a span in your backend that is started but not ended doesn't tell you anything other than that the request has started and not ended, something your user already knows. You would have to enrich it with more interesting events along the way in order to see how far along the process is.

It's true that I know that request has started and not ended but I don't know if it is still executing and where it is still executing in several downstream services. Besides long-running DB requests there are other reasons why the response may be delayed or not delivered at all.

Do you know tracing backends which are able to override existing spans? We could export incomplete spans with artificial duration and attribute "incomplete=true" to be able to view them in UI.

[dyladan]

I'm not aware of any sorry

[adriangb]

I posted open-telemetry/opentelemetry-specification#3732 which seems related.

The way I've seen this solved is to send a child span as soon as a span starts within a custom Tracer. It's messy and the backend had to be specifically coded to handle it but it works.

I would much rather OTEL (1) allowed sending out span events immediately and (2) had a semantic convention for a "span started" event.