[Schema Consistency] Consistency Check 2026-05-16 — Stale engine list in schema descriptions, unenforced engine.agent restriction

[github-actions[bot]]

Overview

Found 2 new actionable inconsistencies and confirmed 1 previously-reported issue is fixed. The recurring pattern is around the engine_config $def: the schema's free-text descriptions list 5 built-in engines while the engine registry registers 7 (opencode and pi are missing from the schema descriptions). A second finding shows that the engine.agent field's documented restriction ("copilot engine only") is not actually enforced by the compiler — the field is silently ignored when set on other engines.

Summary

• Inconsistencies found: 2 actionable + 1 fixed (verified)
• Categories analyzed: schema ↔ parser ↔ docs ↔ workflows
• Strategies used: engine-registry-vs-schema (proven, 2nd consecutive hit) and schema-restriction-vs-runtime-enforcement (new)
• Pre-computed data: /tmp/gh-aw/agent/schema-diff.json (consumed in step 0)

Critical Issues

1. Schema engine-id descriptions are stale — opencode and pi are missing in 4 places

Severity: High (user-facing schema confusion; recurring — flagged 2026-05-15, still present)

The engine registry in pkg/workflow/agentic_engine.go:440-446 registers 7 engines:

NewClaudeEngine(), NewCodexEngine(), NewCopilotEngine(),
NewGeminiEngine(), NewOpenCodeEngine(), NewCrushEngine(), NewPiEngine(),

But four description strings in pkg/parser/schemas/main_workflow_schema.json still list only 5:

Line	Field	Current text
~10192	engine_config.oneOf[0] (string form)	"built-in ('claude', 'codex', 'copilot', 'gemini', 'crush') or a named catalog entry"
~10200	engine_config.oneOf[1].properties.id	"built-in ('claude', 'codex', 'copilot', 'gemini', 'crush') or a named catalog entry"
~10422	engine_config.runtime.id	"Runtime adapter identifier (e.g. 'codex', 'claude', 'copilot', 'gemini', 'crush')"
~10544	engine_config.provider.id (or related)	"Unique engine identifier (e.g. 'copilot', 'claude', 'codex', 'gemini', 'crush')"

docs/src/content/docs/reference/engines.md (the engines reference page) correctly lists all 7 — only the schema descriptions are stale. So the schema description is the canonical inconsistency.

Fix: update all 4 descriptions to 'claude', 'codex', 'copilot', 'gemini', 'opencode', 'crush', 'pi' (or a shorter "see docs" pointer that won't drift again).

2. engine.agent claims "copilot engine only" but no validation enforces it

Severity: Medium (silent misconfiguration)

Schema definition at pkg/parser/schemas/main_workflow_schema.json ($defs.engine_config.oneOf[1].properties.agent):

{
  "type": "string",
  "description": "Agent identifier to pass to copilot --agent flag (copilot engine only). Specifies which custom agent to use for the workflow."
}

The field is only read in pkg/workflow/copilot_engine_execution.go:85 (Copilot's executor). No code path in pkg/workflow/engine_validation.go or pkg/workflow/agent_validation.go rejects (or warns about) engine.agent when engine.id is anything other than copilot. Compare with the similar pattern in validateUniversalLLMConsumerModel at pkg/workflow/agent_validation.go:162, which does enforce engine-conditional requirements for opencode/crush.

Impact: A user setting engine: { id: claude, agent: my-agent } gets no error and the agent value is silently ignored. The schema and docs assert a restriction the compiler doesn't enforce.

Fix options: (a) add a validator that errors when engine.agent is set on a non-copilot engine, mirroring validateUniversalLLMConsumerModel; or (b) relax the description to "used by the copilot engine; ignored by other engines" if silent-ignore is the intended behavior.

Confirmed Fixed Since Last Run

tracker-id maxLength is now validated ✅

The 2026-05-15 run reported that extractTrackerID only enforced minLength despite the schema declaring maxLength: 128. Verified today at pkg/workflow/frontmatter_extraction_metadata.go:106-114:

if len(trackerID) < 8 { return "", fmt.Errorf("tracker-id must be at least 8 characters long...") }
if len(trackerID) > 128 { return "", fmt.Errorf("tracker-id exceeds maximum length of 128 characters...") }

Both bounds and the pattern are now enforced. No action needed.

Pre-computed gaps — interpretation notes

The schema-diff generator's field_gaps block needs context to read correctly:

• in_schema_not_parser (57 fields): All top-level schema fields are listed here because pkg/parser/*.go only declares yaml-tagged structs for input parameter schemas (type, required, default, options) — main frontmatter is processed in pkg/workflow/*.go via map[string]any access, not via yaml struct tags. This category is structural noise, not a real gap.
• in_schema_not_workflow (40 fields): Same root cause — fields like experiments, features, metadata are read by frontmatter["key"] map access (see pkg/workflow/compiler_experiments.go:60, frontmatter_extraction_metadata.go:19), not via yaml struct tags. Real gap detection requires also grepping for frontmatter["<field>"] patterns.
• in_used_not_schema (34 entries): As noted in the 2026-05-15 cache strategy used-not-schema-false-positive-detection, these are mostly content inside markdown code-fences within workflow .md files (e.g., the daily-subagent-optimizer.md template). The diff generator is matching YAML-looking prose. Worth fixing in the diff generator itself.

A more accurate field-gap analysis would need to either (1) parse workflow frontmatter blocks only (stop at ---), or (2) build a hybrid index of yaml tags plus literal frontmatter["key"] lookups in pkg/workflow/*.go.

Recommendations

1. Fix the 4 stale engine-list descriptions in pkg/parser/schemas/main_workflow_schema.json (lines ~10192, 10200, 10422, 10544). Consider replacing the inline list with a stable phrase like "a built-in engine ID (see docs/reference/engines.md) or a named catalog entry" so future engine additions don't reintroduce this drift.
2. Decide engine.agent semantics: add a validator that rejects engine.agent on non-copilot engines, or relax the schema description to match the silent-ignore behavior. Mirror the pattern in pkg/workflow/agent_validation.go:155-175 (validateUniversalLLMConsumerModel).
3. Improve the schema-diff generator to (a) restrict workflow scanning to frontmatter blocks only, and (b) include frontmatter["key"] lookups when building the parser/workflow index. This will eliminate the bulk of the false-positives in in_schema_not_* and in_used_not_schema.
4. Reuse the new strategy schema-restriction-vs-runtime-enforcement next run on other engine_config fields containing "only"/"requires"/"must use" phrases.

Strategy Performance

• Used: engine-registry-vs-schema (proven, success_count → 2) and schema-restriction-vs-runtime-enforcement (new)
• Findings: 2 actionable, 1 verified-fixed
• Effectiveness: HIGH — engine-registry-vs-schema has now produced findings 2 consecutive runs; the new restriction-enforcement strategy yielded a clean finding on first use
• Should reuse: YES for both

Next Steps

• Update 4 stale engine-list descriptions in main_workflow_schema.json
• Choose direction for engine.agent: validator vs. description relax
• Tighten the schema-diff generator (frontmatter-only scanning, map-lookup index)
• Re-run schema-constraints-vs-extraction on other $defs with maxLength/pattern constraints

References:

• §25954685922

Generated by ✅ Schema Consistency Checker · ● 7.3M · ◷

• expires on May 17, 2026, 6:18 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Schema Consistency Checker.

A newer discussion is available at Discussion #32772.