Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/skills/orchestration/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ For COMPOSE or AUTO compose segments, also apply `frontend-design` before writin

## 2.6 Narration voice

When the piece has voiceover, run `ovs speech-capabilities` and copy its executable route/model/voice/format into the Gate B plan together with the BCP-47 video language and a natural speed. Do not invent a voice id. Before Gate B, run `ovs narration fit --text ... --target ...`; revise over/under text internally before any paid synthesis. After `ovs speak`, probe the produced audio and run the same fit with `--measured`; retime scenes from the measured duration without silently shortening the approved target. If no TTS provider is configured, tell the user and explicitly choose silent delivery or wait for configuration.
When the piece has voiceover, run `ovs speech-capabilities` and copy its executable route/model/voice/format into the Gate B plan together with the BCP-47 video language and a natural speed. Do not invent a voice id. Before Gate B, run `ovs narration fit --text ... --target ...`; revise OVER-budget text internally before any paid synthesis — a read that finishes early is an accepted delivery, so pad or extend only when the trailing silence is not the ending you intended. After `ovs speak`, probe the produced audio and run the same fit with `--measured`; retime scenes from the measured duration without silently shortening the approved target. If no TTS provider is configured, tell the user and explicitly choose silent delivery or wait for configuration.

TALKING-HEAD note: if a GENERATE clip already returned lip-synced built-in speech, THAT is the voice — do NOT synthesize a narration over it (a fresh TTS track desyncs from the mouth). Use `ovs speak` only for a silent clip, or for COMPOSE / EDIT / off-screen voiceover.

Expand Down Expand Up @@ -92,7 +92,7 @@ Ingest every supplied clip from evidence (probe + transcribe/OCR-or-frame-readin

## plan.json as the editable record (all lines) — keep follow-up edits cheap

Once a draft exists, keep `project/plan.json` faithful so a later tweak only re-touches one piece (never the whole video): (1) every produced segment carries its real output under `produced_path` + `status:"done"`; (2) narration is `tracks.narration` whose lines each carry their own `produced_path`, so one line can be re-voiced alone; (3) captions are DATA in `tracks.captions.lines` ({text, start_sec, target_sec}) — NOT burned into the picture — so a typo is a one-line edit re-burned at assemble; (4) set top-level `"draft": "render/draft.mp4"`.
Once a draft exists, keep `project/plan.json` faithful so a later tweak only re-touches one piece (never the whole video): (1) every produced segment carries its real output under `produced_path` + `status:"done"`; (2) narration is `tracks.narration` whose lines each carry their own `produced_path`, so one line can be re-voiced alone; (3) captions are DATA in `tracks.captions.lines` ({text, start_sec, target_sec}) — NOT burned into the picture — so a typo is a one-line edit re-burned at assemble; (4) record delivery outputs under the reserved `_runtime` envelope (`"_runtime": {"render": {"draft": "render/draft.mp4"}}`), never as a new top-level key — unknown top-level plan fields are approval-bearing, so a root `draft` key would invalidate the recorded Gate B approval.

**Local follow-up edits — make the minimal targeted change; never redo the whole video.** Once a `plan.json` is present and the user asks to change ONE local thing (a segment's narration / caption / text, a trim, volume / speed, a single shot swap), edit ONLY the matching entry in `plan.json` and re-produce ONLY what it touched (`ovs speak` for that one line, `ovs draft` for that one compose segment, or `ovs edit` for that one cut), then re-assemble. Do NOT re-author the whole EDL and DO NOT regenerate a segment whose `status` is `done` that the user did not touch. Fall back to a full re-plan only when the request genuinely restructures the timeline.

Expand Down
14 changes: 8 additions & 6 deletions packages/skills/stage-assemble/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,22 +11,24 @@ How to execute a validated `project/plan.json` into one finished file. By the ti

Iterate segments in `order`. For each, produce its `produced_path` according to `source`, then write that path + `status:"done"` back into the segment so a resume never re-produces it:

- **edit** → `stage-edit`: `ovs edit trim` the `input_id` to `[in_sec, out_sec]` → `project/cuts/<id>.mp4`.
- **edit** → `stage-edit`: `ovs edit trim` the `input_id` to `[in_sec, out_sec]` → `project/cuts/<id>.mp4`. Those are the EDL's field names, not flags — the command is `ovs edit trim <input> --start <in_sec> --end <out_sec> --out project/cuts/<id>.mp4`.
- **compose** → `stage-compose`: build a small visual-only manifest-owned composition for `spec.kind` (title card, lower-third, stat card, captions) under `project/compositions/<id>/` → run `ovs draft project/compositions/<id> --out project/parts/<id>.mp4 --quality draft --report project/reports/<id>-compose-report.json`. This keeps compose segments on the same manifest/source/check/video-QA path as standalone COMPOSE while still letting the assembler own narration and loudness.
- **generate** → `stage-generate` (+ `stage-consistency` for recurring characters): only AFTER gate C. `ovs video`/`ovs image` → `project/assets/<id>.mp4`. For `operation:"edit"`, pass the exact original reference video and obey top-level `references` plus `edit_strategy`; never widen it into regeneration. A failed/unknown paid attempt is not an automatic retry. Preserve completed siblings and require a new output path for any later authorized attempt.
- **provided** → use `spec.asset_id` as-is (probe it first; conform aspect/fps if needed).

Billable `generate` segments must not run before gate C has confirmed the count from `cost_estimate`. Produce cheap/free segments (edit, compose, provided) freely.

Only `compose` segments need snapshot/frame QA evidence, because their artifact is HTML. An `edit`/`generate`/`provided` segment is captured by its own `produced_path` file the moment that file exists — never run a composition QA phase on one, and never treat it as unfinished work because it has no frame evidence.

## Step 2 — Assemble in ffmpeg tiers (the default path)

Assemble deterministically, bottom-up. This tiered order is the default; it is predictable and cheap, and keeps each clip's real audio intact:

1. **Primary track** — `ovs edit concat` the primary-layer `produced_path`s in `order` → `project/render/primary.mp4`. Conform aspect/fps on the way in if sources differ.
2. **Overlays / bg** — for each overlay/bg segment, `ovs edit overlay` its part onto the primary over the window of the segment named in `over` (title cards, lower-thirds, logos). Composed layers are VISUAL-ONLY — they must not carry their own narration audio. **This includes a compose segment that IS the primary track (a full-video composition): render it SILENT — do not put a narration `<audio>` in its `index.html`. The assembler owns narration (tier 3), so a composition that bakes it in would mean narration is added TWICE (the "two voices" defect).**
3. **Narration — added EXACTLY ONCE, here.** If active, require the Gate-B-signed `tracks.narration.synthesis` profile. Run `ovs narration fit` for each timed line before `ovs speak`; shorten over-budget text in the plan without changing the approved meaning. Synthesize with the exact signed voice/model/format/speed, probe the result, rerun measured fit, and write each line's `produced_path`. If measured timing misses, revise once using the suggested unit budget rather than repeatedly billing or forcing speed. Add all produced lines in ONE `ovs edit mix` call at their `start_sec`. The default existing-audio rejection catches compose segments that accidentally baked narration; re-render those SILENT. Check coverage, and disclose any intentional silent tail at Gate D. Preserve a generated talking head's built-in lip-synced audio instead of adding a second voice.
2. **Overlays / bg** — for each overlay/bg segment, `ovs edit overlay` its part onto the primary over the window of the segment named in `over` (lower-thirds, logo boxes). An overlay must be SMALLER than the frame: composed parts render as opaque video with no alpha channel, and `ovs edit overlay` refuses a full-frame opaque layer because it would replace the footage under it, not composite over it. That refusal is a plan-shape problem, not a retry — never work around it with re-encoding, resizing tricks, or a hand-written ffmpeg graph; re-plan the beat as a composed `primary` segment with no footage underneath. Composed layers are VISUAL-ONLY — they must not carry their own narration audio. **This includes a compose segment that IS the primary track (a full-video composition): render it SILENT — do not put a narration `<audio>` in its `index.html`. The assembler owns narration (tier 3), so a composition that bakes it in would mean narration is added TWICE (the "two voices" defect).**
3. **Narration — added EXACTLY ONCE, here.** If active, require the Gate-B-signed `tracks.narration.synthesis` profile. Run `ovs narration fit` for each timed line before `ovs speak`; shorten over-budget text in the plan without changing the approved meaning. Synthesize with the exact signed voice/model/format/speed, probe the result, rerun measured fit, and write each line's `produced_path`. If measured timing misses, revise once using the suggested unit budget rather than repeatedly billing or forcing speed. Add all produced lines in ONE `ovs edit mix` call at their `start_sec`. The default existing-audio rejection catches compose segments that accidentally baked narration; re-render those SILENT. READ the coverage report as QA, not decoration: `coverageRatio` only says how far the LAST line reached — a half-silent track can still score 0.95 on it. Judge `voicedRatio` and `interiorGaps` for dead air, and treat `overlapped` as broken audio (two lines speaking at once). Never raise the speech speed to fit and never pad the script with filler words; re-time the lines inside their windows, shorten over-long beats, or add the planned music bed. Disclose any intentional silent tail at Gate D. Preserve a generated talking head's built-in lip-synced audio instead of adding a second voice.
4. **Music** — add `tracks.music` ducked under narration by the planned amount.
5. **Captions** — turn `tracks.captions.lines` (`{text, start_sec, target_sec}`) into a `.srt`, then `ovs edit burnsubs`. Captions are DATA in the plan — burned ONLY here at assemble — so a later typo fix is a one-line edit re-burned, never a re-render of the picture. If `burnsubs` fails because the runtime ffmpeg lacks subtitle filter support, stop and report that blocker; do not hand-write a fallback ffmpeg graph, PNG subtitle overlay, or drawtext pipeline.
5. **Captions** — turn `tracks.captions.lines` (`{text, start_sec, target_sec}`) into a `.srt`, then `ovs edit burnsubs`. Captions are DATA in the plan — burned ONLY here at assemble — so a later typo fix is a one-line edit re-burned, never a re-render of the picture. Skip a caption line whose text merely repeats that scene's visible on-screen copy — burning the same sentence twice on one frame reads as a defect. If `burnsubs` fails because the runtime ffmpeg lacks subtitle filter support, stop and report that blocker; do not hand-write a fallback ffmpeg graph, PNG subtitle overlay, or drawtext pipeline.
6. **Loudness** — run `ovs edit normalize-loudness project/render/draft.mp4 --out project/render/video.mp4`. It normalizes to the `video-craft` §7 targets (~−14 LUFS integrated, true-peak ≤ ~−1 dBTP) and returns measured loudness; use `ovs edit loudness` only for diagnosis without writing an output.

Apply the plan's `style_kit` for cohesion: composed layers (titles/captions/cards) use its `palette` + `fonts`. A single `lut` graded across all clips is what unifies tonally mixed sources — until a grade op is available, keep mixed sources close at capture/trim and lean on the shared palette + consistent captions for cohesion rather than promising a uniform grade.
Expand All @@ -52,7 +54,7 @@ For a partial child failure or later revision, reset only the affected child and

## Step 4 — QA report, then gate D

Before showing the draft, run the QA pass and write `project/render_report.json` with these sections:
Before showing the draft, run the QA pass and write `project/render_report.json`. It judges the delivered ARTIFACT against the approved plan — length, canvas, narration placement measured from real speech, loudness, declared captions — so every route to the file meets the same bar, including a hand-written ffmpeg assembly. Sections:

- **technical_probe** — `ovs edit probe` the draft/final (real duration / resolution / fps / audio present); confirm it matches the plan's aspect + total.
- **promise_preservation** — `ovs plan promise-check project/plan.json --probe-produced`. At gate D this probes each primary segment's `produced_path` and computes the REAL primary-track motion ratio vs. `motion_min_ratio` plus the `source_required` invariant; missing/unreadable produced media or a fail means **"slideshow / promise broken" — do not deliver**. Send it back (below). Do not eyeball this; let the numbers decide.
Expand All @@ -77,7 +79,7 @@ Bound repetition, not recovery: allow at most **2** send-back rounds for the sam
## Rules

- Walk the approved plan; if assembly reveals the plan is wrong, surface it and re-gate — do not silently re-plan.
- Write `produced_path` + `status` back per segment as you go (resumability + the QA pass depend on it).
- Write `produced_path` + `status` back per segment as you go (resumability + the QA pass depend on it), and record delivery outputs under the reserved `_runtime` envelope (`_runtime.render.draft` / `_runtime.render.final_path`), never as new top-level plan keys — unknown top-level fields are approval-bearing and would invalidate the Gate B signature. Skipping the write-back is what makes an already-delivered video look never-started on resume.
- One output file is the deliverable; `cuts/` and `parts/` are intermediates.
- **Narration is added exactly ONCE — in the mix tier, never baked into a compose render.** Compose segments (including a full-video composition used as the primary track) render SILENT (no narration `<audio>`); the assembler mixes narration via `ovs edit mix` with `segments` placed per line. The mix's default `--on-existing-audio reject` enforces this — a "base already has audio" mix rejection is the signal a segment wrongly baked audio in; re-render it silent, then re-mix.
- **No ad-hoc ffmpeg fallbacks for captions.** Caption burn-in is a low-freedom operation owned by `ovs edit burnsubs`; a failed burnsubs call is a tool/runtime blocker, not permission to invent a custom subtitles/drawtext/PNG-overlay command.
Expand Down
4 changes: 2 additions & 2 deletions packages/skills/stage-edit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ How to intelligently edit **real user-supplied footage** while keeping source an

## Intelligent Edit Contract

Write `project/plan.json#edit_strategy` whenever OVS decides what to change rather than merely executing user-provided timecodes. Declare its `mode` (`deterministic`, `semantic`, or `mixed`), exact objectives, only the evidence signals actually used, and non-empty, non-overlapping `preserve`/`may_change`.
Write `project/plan.json#edit_strategy` whenever OVS decides what to change rather than merely executing user-provided timecodes. Declare its `mode` (`deterministic`, `semantic`, or `mixed`), exact objectives, only the evidence signals actually used, and non-empty, non-overlapping `preserve`/`may_change`. All four list fields are non-empty ARRAYS of strings — one entry per item, never a single sentence; `objectives` is the one most often written as prose, and the validator rejects a bare string with `E_EDIT_STRATEGY_BOUNDARY`. `may_change` must name every class of change the user authorized: omitting one silently narrows the plan's declared authority below what they asked for.

Declare every source/reference in top-level `references` with media type, reproduce/edit/guide intent and basis, roles, required state, preservation boundary, target segments, and video temporal anchors. A semantic video edit is `source:"generate"`, `media_kind:"video"`, `operation:"edit"` with its original in `reference_video_paths` or `reference_video_urls`; it remains owned by EDIT and counts as billable.

Expand Down Expand Up @@ -65,7 +65,7 @@ When the user wants highlights / clips "about X" or a localized version, transcr
- `delivery_promise:{ type:"source_led", source_required:true }`; set `aspect` from the SOURCE's real probed dimensions (a landscape source is `16:9`, not the portrait default).
Each narration line stays its own entry, so a later edit can re-voice ONE line without touching the rest.
3. **Generate each beat FROM the plan, then record its `produced_path`.** Run `ovs narration fit --text ... --target ...` before TTS and shorten until it fits naturally. Run `ovs speak` per line, save the audio under `project/assets/narration/line-XX.*`, probe the measured duration, then run `ovs narration fit --measured ...` and retime/shorten any miss before writing `produced_path`. Never speed up past the approved natural profile or let a line run long/short. Coverage must span ~0→clip-end, not stop at the halfway mark.
4. **Assemble with `ovs edit` — keep the picture untouched.** Place the narration lines at their `start_sec` in ONE `ovs edit mix` call via `segments` JSON (one entry per line — that is HOW per-line `start_sec` alignment happens), then use `ovs edit normalize-loudness` for the deliverable. Burn captions from `tracks.captions.lines` (.srt → `ovs edit burnsubs`) if present. The source footage usually already HAS audio, so `mix` rejects by default — choose `--on-existing-audio mix` to keep the original sound under the voiceover, or `--on-existing-audio replace` to drop it. Write each line's `produced_path` + `status` and the top-level `draft` / `video` paths back to plan.json so the record matches the result. Never pre-bake one big narration file — that destroys per-line separability.
4. **Assemble with `ovs edit` — keep the picture untouched.** Place the narration lines at their `start_sec` in ONE `ovs edit mix` call via `segments` JSON (one entry per line — that is HOW per-line `start_sec` alignment happens), then use `ovs edit normalize-loudness` for the deliverable. Burn captions from `tracks.captions.lines` (.srt → `ovs edit burnsubs`) if present. The source footage usually already HAS audio, so `mix` rejects by default — choose `--on-existing-audio mix` to keep the original sound under the voiceover, or `--on-existing-audio replace` to drop it. Write each line's `produced_path` + `status` back to plan.json, and record the delivery under the reserved `_runtime` envelope (`"_runtime": {"render": {"draft": "render/draft.mp4", "final_path": "render/video.mp4"}}`) — never as new TOP-LEVEL keys: unknown top-level fields are approval-bearing, so a root `draft`/`video` key invalidates the Gate B signature the moment the render lands. `_runtime` and `produced_path` are runtime locators excluded from the signed plan identity. Never pre-bake one big narration file — that destroys per-line separability.
5. **Self-check before presenting:** `project/plan.json` validates (`ovs plan validate project/plan.json`); every narration line has a `produced_path` and a window matching its OCR/transcript/frame-read text; mix returned a coverage report with no surprising uncovered tail or overshoot; `project/render/video.mp4` exists. Then tell the user the draft is ready and they can ask for follow-up tweaks (re-voice a line, fix a caption) and you'll change only that.

When the clip has NO spoken audio, or its meaning lives in ON-SCREEN TEXT (a screen-recording, a slideshow, a captioned montage), transcription returns nothing — the content is in the pixels, not the audio. An empty audio track does NOT mean an empty screen. Read what is on screen instead of guessing, in this strict order (cost-first):
Expand Down
Loading
Loading