fix(ocsf): emit schema-conformant OCSF events and honest downgrades

[zanetworker]

Summary

OCSF events from OpenShell failed the official OCSF validator, natively at 1.8.0 and after the 1.1 or 1.3 downgrade. The vendored-schema tests did not notice because they checked only top-level, non-profile attributes, and the downgrade stripped three fields by name before relabelling metadata.version.

This makes native events from every builder pass https://schema.ocsf.io/api/1.8.0/validate (apart from top-level container, open on #4283), and makes the downgrade schema-driven: an event is labelled 1.1 or 1.3 only when it validates against that version, otherwise it stays unchanged at 1.8.0 and is counted. No placeholder data is ever inserted.

Related Issue

Closes #4283
Closes #4284

Changes

Native OCSF 1.8 (#4283):

• device.os carries the required type_id and type (Linux 200, Windows 100, macOS 300) through a new OsTypeId enum.
• HTTP Activity no longer emits is_src_dst_assignment_known, which no OCSF version defines on that class.
• ProcessActivityBuilder requires an acting process before build() (type-state, like the HTTP and Network builders). Sandbox process events name the openshell-sandbox launcher with its real pid.
• connection_info carries the required direction_id and direction, defaulting to Unknown.
• SshActivityBuilder requires a source or destination endpoint before build(). All eight SSH events had none; they now name the supervisor's SSH server as dst_endpoint (svc_name: "ssh" and the listen socket path in name), using new Endpoint name and svc_name fields.
• Endpoint::from_domain and the policy DNS endpoint also set hostname, the endpoint name OCSF 1.1 and 1.3 accept.
• validate_required_fields recurses into nested objects, rejects undefined attributes, enforces profile attributes only when declared, and uses the event's metadata.version. New test every_builder_emits_schema_conformant_events. The HTTP and Network compile_fail doctests now fail only for the reason they claim.

Keeping it fixed:

• New e2e test ocsf_jsonl_conformance (Podman CI set) enables JSONL with a 1.1 target, generates denied traffic, copies the supervisor's JSONL with podman cp, and validates every record against the version it claims. It covers production call sites rather than builders, and found the SSH endpoint gap. The e2e crate takes openshell-ocsf (with test-support) as a dev-dependency so both suites share one validator.
• With test-support, the fallback EventContext carries a sandbox identity, so producer tests emit container and device fields as production does. Top-level container is the single named exception in the validator, linked to bug(ocsf): native OCSF 1.8 events fail the official schema validator #4283.
• schemas/ocsf/README.md documents every vendored version, the update and regeneration steps, a native version bump checklist, and validation with the official OCSF server. vendored_schemas_cover_the_native_version_and_every_target fails when a version is not vendored.

Schema downgrade (#4284):

• format/downgrade.rs compares each event against the target version's class and object definitions, recursively. Undefined attributes (for example launch_type, is_alert, state, is_src_dst_assignment_known, ai_model) move under unmapped.downgraded_attributes; unsupported profiles are dropped; requirements and at_least_one constraints are checked. Returns DowngradeOutcome instead of bool. metadata.version is written in full (1.1.0, not 1.1).
• format/downgrade_defs.rs is generated from vendored official 1.1.0 and 1.3.0 schemas; downgrade_definitions_match_vendored_schemas fails on drift and UPDATE_OCSF_DOWNGRADE_DEFS=1 regenerates it.
• Gateway metric openshell_ocsf_log_kept_native_total{class_uid,target}. The supervisor tallies kept-native records and its settings poll loop logs a warning with the count and latest reason when the tally grows.
• Docs: SIEM schema version compatibility rewritten (the claim that the structure is identical across 1.1 to 1.8 is gone), device.os.type_id, the new metric, and the gateway schema_version row.

User-visible: downgraded files can contain both target-version and 1.8.0 records (route by metadata.version); metadata.version reads 1.1.0 rather than 1.1; HTTP events drop is_src_dst_assignment_known; endpoints gain hostname.

Deferred: top-level container is undefined in every OCSF version but identifies the affected sandbox on gateway events (#4283). Many Detection Finding, Network and SSH events lack action_id or src_endpoint and so stay native for 1.1 targets; adding that data needs a per-call-site decision. The SIGTERM-failure Process Activity event still uses the Process::new("unknown", 0) fallback.

Testing

• Real run on Podman (e2e/with-podman-gateway.sh, supervisor and sandbox images built from this branch) with ocsf_schema_version=1.1 and denied DNS, transparent TCP and SSH activity. Of 21 supervisor JSONL records, all 9 labelled 1.1.0 return no errors from the official 1.1.0 validator; the other 12 stayed at 1.8.0 for missing src_endpoint (9), a Unix-socket SSH session (2) or missing action_id (1). The supervisor logged the kept-native warning on unchanged polls.
• OPENSHELL_E2E_PODMAN_TEST=ocsf_jsonl_conformance mise run e2e:podman passes. It failed before the SSH fix with "SSH Activity" 1.8.0 "SSH connection accepted on supervisor Unix socket": Missing at_least_one field from [dst_endpoint, src_endpoint].
• The official OCSF server (ocsf/ocsf-server built from source, schemas compiled with ocsf-schema-compiler 1.1.1, one instance each for 1.1.0, 1.3.0 and 1.8.0) returns no errors for the real supervisor records, every builder's native and downgraded events, and the new SSH endpoint shape. It does not enforce at_least_one constraints, which is why the e2e test uses the vendored-schema validator.
• One event per builder: native output passes the official 1.8.0 validator apart from top-level container (every class failed before on device.os.type_id), and downgraded to 1.1.0 and 1.3.0 every event validates against the version it claims (18/18).
• cargo test -p openshell-ocsf (172 unit, integration and doc tests); cargo test --lib for openshell-server (2028), openshell-supervisor (154), openshell-supervisor-network (1471), openshell-supervisor-process (105), openshell-sandbox; cargo check -p openshell-driver-mxc --tests
• cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check
• Unit tests added/updated

proxy::tests::mediated_connect_keeps_workload_bytes_read_with_the_synthesized_header is load-sensitive: under full CPU saturation it fails 10/10 on both main and this branch, and passes in normal runs on both.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[johntmyers]

/ok to test 78621e8

[github-actions]

Label test:e2e applied for 78621e8. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

PR Review Status

The independent review found no blocking defects in the OCSF conformance and schema downgrade changes. @zanetworker, I checked your note about top-level container: it is present in the base, remains an acknowledged design question, and the downgrade preserves its data under unmapped.

Current-head testing has started with test:e2e and test:windows. Trivy and Helm are green; Branch Checks, E2E, and both Windows lanes are pending. The E2E Label Help bot requests Re-run all jobs, but the identified run still has an active first attempt.

Action required: rerun Branch E2E Checks after the active attempt finishes. Gator has operator authorization and the narrow sandbox permission to perform that rerun in a subsequent cycle.

Blocking findings: None.

Carried findings: None.

Gator metadata

• Validation: Focused OCSF correctness fixes for linked issues #4283 and #4284; operator requested review and relevant tests.
• Docs: Fern documentation updates describe OS identity, mixed-version JSONL, downgrade preservation, gateway configuration, and the kept-native metric.
• Checks: DCO, OpenShell / Helm Lint, and OpenShell / Trivy Changes pass; Branch Checks and both Windows MSVC PR lint and test lanes are pending on the reviewed head.
• E2E: test:e2e applied; current-head mirror verified; Branch E2E Checks run 37657018590 attempt 1 is active. Bot-required rerun remains outstanding. Relevant E2E coverage includes the new Podman JSONL conformance test.
• Head SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Base SHA: 3fc93e28273fe304db634e6d5e9a52d57200389e
• Merge base SHA: 834b79a8c2351370e7c6c363423f5be18d440256
• Patch ID: 33dfd9e90bc20c4d4b8493efc716d9a8d3c32edb
• Gator payload: 11
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[johntmyers]

gator-agent

Author Follow-Up Nudge

This PR has been in gator:in-review for more than 48 business hours and required CI still needs attention.

@zanetworker, please make the vendored OCSF schemas available to the relocated Podman E2E test and push an update. The rootless Podman run fails in supervisor_jsonl_conforms_to_the_version_each_record_claims because the loader uses the build machine's CARGO_MANIFEST_DIR and cannot find schemas/ocsf/v1.1.0/classes/base_event.json on the remote test machine. Verify that the archived test can load its schemas and validate the JSONL records there.

@NVIDIA/openshell-maintainers, both Windows lanes also need a gate resolution: Clippy reports unnecessary_wraps and needless_pass_by_ref_mut in crates/openshell-cli/src/ssh.rs at lines 369, 376, and 423. That file is unchanged by this PR.

Gator metadata

• Head SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Gator payload: 11
• Next state: gator:in-review
• Nudge scope: Existing CI follow-up; no new code-review disposition.

[zanetworker]

Fixed in aae174742: the vendored schemas are now compiled into the test binary (crates/openshell-ocsf/src/validation/embedded.rs, generated with a drift test), so archived binaries need no files from the build machine. Verified by running the ocsf_jsonl_conformance nextest archive from an unrelated directory with --workspace-remap and the repo's schemas/ hidden, against a Podman gateway: 1 passed. The Windows Clippy findings are in crates/openshell-cli/src/ssh.rs, which this PR doesn't change.

@johntmyers could you take a look, and /ok to test aae17474233af3ddd272f43557a01a7cf0c7f677 when you get a chance?

[johntmyers]

/ok to test aae1747

[johntmyers]

gator-agent

PR Review Status

Thanks @zanetworker (Adel Zaalouk), I checked your update about relocating the archived test: the loader now reads compiled-in schemas, and all 112 vendored schemas are included. That addresses the build-machine path dependency. Your note about the earlier Windows Clippy failures also holds: ssh.rs is unchanged by this PR.

The independent follow-up review found one new Windows regression in the drift test. It generates platform-native paths while the committed table uses forward slashes. The inline finding covers both the failing comparison and regeneration.

Action required: normalize the generated schema paths to forward slashes and verify the focused drift test on Windows.

Blocking findings:

• GATOR-aae17474-01: Windows path separators break the new embedded-schema drift check and regeneration.

Carried findings: None. The prior schema-relocation obligation is addressed by the loader change; current-head E2E will verify it in CI.

I posted /ok to test for the full current head. Branch Checks, Helm, E2E, and Windows workflows are queued or running. The existing test:e2e and test:windows labels remain appropriate. Gator stays in review while the author fixes the path normalization.

Gator metadata

• Validation: Focused OCSF correctness changes for linked issues #4283 and #4284; continuing the previously validated PR.
• Docs: Existing Fern updates cover the user-visible changes; schema README documents embedding and regeneration.
• Checks: DCO passes; current-head Branch Checks and Helm workflows are active, Windows is queued. Trivy is not yet confirmed green.
• E2E: Current-head Branch E2E Checks run 38081852521 is active. The older bot-required rerun instruction referred to the previous head; no current-head rerun instruction was present when checked.
• Head SHA: aae17474233af3ddd272f43557a01a7cf0c7f677
• Base SHA: eeba0e7954c0fb4d8e9e2e29d1bfa68e290eb8b3
• Merge base SHA: 834b79a8c2351370e7c6c363423f5be18d440256
• Patch ID: 892df22a6665f52060e783e6810464d88dae8f16
• Gator payload: 11
• Review mode: follow_up
• Previous reviewed SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[johntmyers]

gator-agent

Warning — GATOR-aae17474-01 · Normalize embedded schema paths on Windows

Summary: Both Windows workspace test lanes run this new drift check. Native relative paths contain backslashes, so the generated table differs from the committed forward-slash table and the check fails. Regenerating on Windows also inserts unescaped backslashes into Rust string literals.

Fix: Normalize relative schema paths to forward slashes before storing and interpolating them. Keep the drift check enabled on Windows.

Verify: On native Windows, run cargo test -p openshell-ocsf embedded_schemas_match_vendored_files. It should pass without changing embedded.rs; regeneration should produce the same table on Windows and Unix.

Agent context

• Location: crates/openshell-ocsf/src/validation/schema.rs:266
• This follow-up introduces the test and generator; the previous head had neither. The comparison and regeneration share this path conversion.
• Reviewed statically; native Windows execution was not available in this sandbox.