Stamp observability metadata/tags on preset automation conversations

[juanmichelini]

Why

Automation-run sessions and regular UI conversations are indistinguishable in Laminar today. Traces only carry session_id (the conversation UUID) — no origin signal in metadata or tags — so there's no way to filter automation traffic out of (or into) trace views, cost analysis, or signals.

Preset scripts create their RemoteConversation directly against the sandbox agent server, bypassing the app-server start path, so any origin stamping has to happen here. The SDK plumbing already exists (Conversation(observability_metadata=..., observability_tags=...) → Laminar.set_trace_metadata / root span tags) — the presets just weren't using it.

What

Both preset scripts (presets/prompt/sdk_main.py, presets/plugin/sdk_main.py) now pass observability context to Conversation():

Field	Value	Source
metadata trigger	automation	constant — matches the app-server convention (see companion PR OpenHands/sandbox-server#9) so automation vs. UI traffic splits on one key
tag trigger:automation	—	same
metadata automation_trigger	cron / event	AUTOMATION_EVENT_PAYLOAD.trigger
tag automation_trigger:<type>	—	same
metadata automation_id / automation_name	—	AUTOMATION_EVENT_PAYLOAD
metadata automation_run_id	—	AUTOMATION_RUN_ID env
metadata experiment_id / experiment_variant	—	plugin preset A/B config, when present

The kwargs are gated on an inspect.signature check (same pattern as the existing _conversation_supports_user_id() gate), so sandboxes running older SDK versions silently skip them instead of raising.

Example Laminar query after this lands:

SELECT simpleJSONExtractString(metadata, 'trigger') AS trigger, count()
FROM traces
WHERE start_time > now() - INTERVAL 7 DAY
GROUP BY trigger

or filter by the trigger:automation tag, or drill into a single automation via metadata.automation_name.

Notes

• Preset scripts are packaged into the automation tarball at creation time, so existing automations pick this up only when updated/re-created (or whenever the preset script is refreshed).
• Companion PR: Include conversation trigger in observability metadata and tags sandbox-server#9 threads ConversationTrigger into the app-server observability context, covering conversations started through POST /api/v1/app-conversations (GUI, Slack, Jira, resolver, and API-triggered automation conversations).

Testing

• Extracted _build_observability_context from both scripts and unit-tested it directly (full payload, empty payload, cron-only) — all pass
• pytest tests/test_preset_router.py tests/test_ab_testing_integration.py — 114 passed
• Full suite: 940 passed; the 338 errors are pre-existing docker-socket-dependent tests that fail identically on the base commit in this environment
• ruff check / ruff format --check clean (presets dir is excluded from lint per pyproject.toml; verified the scripts are unchanged-lint-parity with base)

This PR was created by an AI agent (OpenHands) on behalf of the user.

@juanmichelini can click here to continue refining the PR

[github-actions]