feat(observability): include the structured OCSF event in sandbox log lines served by the gateway API #4080

User Story

As a developer building automated tooling on the OpenShell gateway API (for example, an evaluation harness that grades what an agent did inside a sandbox), I want to read each OCSF event for a sandbox as a structured OCSF record through the same API I already use for sandbox logs (GetSandboxLogs, WatchSandbox, and the SDKs' log methods), so that my tooling consumes OpenShell's audit events as data, works the same on every compute driver, and doesn't break when the human-oriented shorthand format changes.

Problem Statement

The supervisor builds a full OCSF v1.8.0 event for every network, HTTP, process, finding and config event, but the gateway API exposes only its shorthand text rendering.

When the supervisor pushes an OCSF event to the gateway, it sends format_shorthand() as the message and deliberately leaves SandboxLogLine.fields empty (log_push.rs, asserted by ocsf_events_push_shorthand_with_ocsf_level_and_no_fields). Every read path is built on that message, so all of them return the shorthand string: GetSandboxLogs, WatchSandbox, openshell logs, Rust watch_logs, and the Go SDK.

The structured record exists only in the opt-in JSONL file (ocsf_json_enabled), and that file is written inside the supervisor's own filesystem (/var/log/openshell-ocsf.*.log). No gateway API serves it.

Impact / Why This Matters

Without this, an API client that needs event details must parse the shorthand text, for example:

NET:OPEN [MED] DENIED /usr/bin/curl(0) -> audit.ext-log.com:443 [reason:transparent_tcp_policy_denied]
NET:REFUSE [MED] DENIED audit.ext-log.com [reason:policy_dns_ineligible]

• Fragile. The shorthand is a human-readable rendering, not a documented contract. Its shape varies by event (the caller and port are optional, and HTTP events have no caller), and it changes between releases (OCSF shorthand renders Unknown and Other severities as [INFO] #3886 changes severity rendering). A client's parser can silently drop events after an upgrade. For security tooling, a dropped "denied" event can turn a real finding into a false negative.
• Lossy. The shorthand omits fields the record carries: the matching rule (firewall_rule), the parent process, disposition versus action, status_detail, the resolved IP, and the container UID and image. Confirmed this on 0.1.2 by comparing the shorthand line and the JSONL record for the same blocked connection.
• The file isn't a usable substitute for API clients. Reading the JSONL requires driver-specific access to the supervisor's container (podman cp, kubectl cp, a log-shipping sidecar), plus the right permissions. On some drivers the file isn't produced at all (ocsf_json_enabled silently writes nothing with the MicroVM driver (supervisor cannot open /var/log) #3855 MicroVM, OCSF JSONL enabled and OCSF events emitted, but /var/log/openshell-ocsf.* is not created on Docker Desktop / WSL2 #3895 Docker Desktop/WSL2).

Proposed Design

When a client reads sandbox logs through the gateway API, each OCSF log line can also carry the complete OCSF JSON record for that event, alongside the existing shorthand message.

• The shorthand message, the level (OCSF) and all current behavior stay unchanged, so existing clients are unaffected.
• The structured record is the same OCSF record the JSONL sink writes, respecting the configured ocsf_schema_version.
• It's opt-in, to keep the default log stream small, either through the existing ocsf_json_enabled setting (per sandbox or global) or through a request option on GetSandboxLogs / WatchSandbox. Maintainers can choose which.
• It's available on every compute driver, because it travels the existing supervisor-to-gateway log path rather than a file.
• The SDKs and openshell logs can surface it (for example, a JSON output mode in the CLI).

This is not a request for export to external destinations (#2762). It makes the existing log API a machine-readable source that other tools can consume.

Acceptance Criteria

• With the feature enabled, every OCSF line returned by GetSandboxLogs and WatchSandbox for a sandbox includes its complete OCSF record as JSON.
• The record matches what the JSONL sink would write for the same event, including class_uid, action / disposition, status_detail, dst_endpoint, actor.process (with the parent process), firewall_rule and container.
• With the feature disabled (the default), log responses are unchanged.
• It works on the Docker, Podman, Kubernetes and VM drivers.
• The Python, Rust and Go SDKs, and openshell logs, expose the structured record.
• The documentation states which representation is the stable machine-readable contract.

Alternatives Considered

1. Parse the shorthand text (what we do today). It works, but it's fragile and lossy, as described above, and every consumer re-implements the parser.
2. Read the JSONL file from the supervisor's filesystem. It's structured, but it requires driver-specific access outside the OpenShell API, and it doesn't work on every driver (ocsf_json_enabled silently writes nothing with the MicroVM driver (supervisor cannot open /var/log) #3855, OCSF JSONL enabled and OCSF events emitted, but /var/log/openshell-ocsf.* is not created on Docker Desktop / WSL2 #3895).
3. Flatten selected fields into SandboxLogLine.fields. This is better than text, but nested OCSF objects (actor, parent process, endpoints) don't fit a flat string map well. The full record is simpler and loses nothing.
4. Gateway export to SIEM destinations (feat(observability): OCSF event export from gateway to external SIEM destinations #2762). Declined as out of scope. This proposal is much narrower and fits the stated direction of OpenShell producing a source of OCSF events that other tools consume.

Agent Investigation

Findings from reading v0.1.2 branch, confirmed against a live 0.1.2 gateway with the Podman driver:

• crates/openshell-supervisor-process/src/log_push.rs (LogPushLayer::on_event, around line 57): for OCSF_TARGET events, clone_current_event() already returns the full event. The layer sends format_shorthand() with an empty fields map. to_json() (crates/openshell-ocsf/src/forma t/jsonl.rs) is available on the same value.
• proto/openshell.proto SandboxLogLine has a fields map documented as "structured key-value fields from the tracing event", which is empty for OCSF events.
• crates/openshell-supervisor/src/main.rs (around lines 345–360): the JSONL sink (OcsfJsonlLayer) writes to /var/log/openshell-ocsf.* in the supervisor's filesystem. In 0.1.x that's the supervisor container, not the workload, so ExecSandbox can't read it. We read it with podman cp.
• crates/openshell-server/src/tracing_setup.rs: the gateway-side JSONL sink exists only for Windows/MXC. A code comment notes that "a cross-platform sink needs an explicit storage and configuration contract."
• Live check (0.1.2, Podman): with ocsf_json_enabled=true on one sandbox, a blocked curl produced this JSONL record. The API returned only the corresponding shorthand line.

{"class_name":"Network Activity","activity_name":"Open","action":"Denied","disposition":"Blocked",
 "status_detail":"transparent_tcp_policy_denied",
 "dst_endpoint":{"domain":"audit.ext-log.com","ip":"198.18.0.42","port":443},
 "actor":{"process":{"name":"/usr/bin/curl","pid":0,"parent_process":{"name":"/usr/bin/bash","pid":0}}},
 "container":{"name":"eval-72e144c9d8","uid":"1d25f83b-…","image":{"name":"localhost/document-assistant-local:latest"}}}

Checklist

• I've reviewed existing issues and the published docs
• This is a design proposal, not a "please build this" request