Filter traces based on the prompt variant

[AhmedHawater2003]

Currently, there is no straightforward way to group all traces generated from the same prompt variant. The closest workaround is to create a test set that contains all my experiments as test cases and run the variant through an evaluation. However, this approach doesn’t fully leverage the capabilities of the tracing feature, since it requires rerunning the test cases.

Additionally, I often experiment in the playground and want to revisit previous results, but once they’re cleared from the playground environment, the existing trace filtering options make it difficult to retrieve them efficiently.

It would also be helpful to have an option to filter traces by a specific version of a prompt variant, or to include traces from all versions.

Thank you so much for the amazing work!

[mmabrouk]

Hello @AhmedHawater2003,

Thanks for creating this issue.

Let me make sure I understand what you're looking for. You want to filter traces by variant, right?

I'm curious where you're running these prompt variants?

If I understand your workflow correctly, it would be:

1. Create a new variant in the playground or deploy it to production
2. Run your prompt on this variant/environment outside of Agenta
3. Go to observability and filter to see the traces created for that variant

Is that correct?

You mentioned experimenting with prompts in the playground and wanting to find them later. Here's the challenge with that: playground prompts are stateless. You might edit a config, run the prompt, and never save it. So we can't really attach a variant to those traces. What we could do instead is let you filter traces by whether they came from an application and were created in the playground. Would that solve the problem?

For production variants though, this makes total sense. We should be able to filter traces by variant and even by specific versions of a variant.

Does this match what you're thinking? I want to make sure we're solving the right problem here.

[AhmedHawater2003]

While being unfortunate to realize that, but your explanation the stateless nature of the playground makes total sense.

For our specific workflow, at lest for now, we don’t utilize the observability features of Agenta for prompts running outside of Agent, so we wouldn’t directly benefit from the feature. However, i believe this would be very handy for other users who go through the workflow you described.