[Schema Consistency] Schema Consistency Check - Nov 15: Engine-Specific Feature Documentation Gaps

[github-actions[bot]]

🔍 Schema Consistency Check - November 15, 2025

Executive Summary

This analysis discovered 5 actionable inconsistencies using a novel comment-driven approach that starts from code comments and developer documentation rather than schema definitions. The primary finding is a pattern of engine-specific feature support that is correctly implemented but poorly documented in schema descriptions.

Key Highlight: While implementation quality is high (with proper validation logic), schema documentation lacks critical engine compatibility information, potentially causing user confusion.

Strategy: NEW approach - Strategy-016 "Comment-Driven Documentation Gap Analysis" using bottom-up analysis from code comments to schema validation.

Quick Stats

• 🔴 Critical Issues: 3 (HTTP transport engine docs, timeout-minutes default, max-turns engine support)
• 🟡 Moderate Issues: 2 (tools meta-fields, PR max parameter)
• ✅ Positive Findings: 5 (perfect consistency in campaign, max-patch-size, roles, safety-prompt, HTTP validation)
• 📊 Total Findings: 10

Critical Issues Requiring Action

Critical Issues

1. 🔴 HTTP Transport Engine-Specific Support Not Documented

Severity: HIGH
Location: pkg/parser/schemas/mcp_config_schema.json

Issue:
Schema defines HTTP as valid MCP transport type but doesn't document it's only supported by specific engines (claude, copilot, codex). Custom engines only support stdio transport.

Evidence:

// Schema allows HTTP
{
  "type": "string",
  "enum": ["stdio", "http", "local"],
  "description": "MCP connection type (local is an alias for stdio)"
}

// But validation rejects for unsupported engines
func (c *Compiler) validateHTTPTransportSupport(...) error {
    if !engine.SupportsHTTPTransport() {
        return fmt.Errorf("tool '%s' uses HTTP transport which is not supported by engine '%s' (only stdio transport is supported)", ...)
    }
}

Impact: Users configuring HTTP transport for custom engines will get compile-time errors without understanding the engine limitation.

Recommendation:
Update MCP schema type description:

"description": "MCP connection type (local is an alias for stdio). Note: HTTP transport only supported by claude, copilot, and codex engines. Custom engines support stdio only."

---

2. 🟡 timeout-minutes Default Value Claim Unverified

Severity: HIGH (Documentation Accuracy)
Location: pkg/parser/schemas/main_workflow_schema.json

Issue:
Schema claims "Defaults to 15 minutes for agentic workflows" but no code implements this default. Field extracted from frontmatter but no default value assigned.

Evidence:

• Schema: "Defaults to 15 minutes for agentic workflows"
• Code search: No = 15 or timeout = 15 found in workflow package
• Implementation: TimeoutMinutes string field extracted but no default set

Impact: Misleading documentation. Users expect 15-minute timeout but likely get GitHub Actions default of 360 minutes (6 hours).

Recommendation:
Either:

1. Implement the 15-minute default in compiler if this is desired behavior
2. Update schema to reflect actual behavior: "Uses GitHub Actions default (360 minutes) unless specified"

---

3. 🟡 max-turns Engine-Specific Support Not in Schema

Severity: MEDIUM
Location: Engine configuration

Issue:
Schema documents max-turns feature but doesn't mention engine compatibility. Custom engines don't support this feature.

Evidence:

func (c *Compiler) validateMaxTurnsSupport(...) error {
    if !engine.SupportsMaxTurns() {
        return fmt.Errorf("max-turns not supported: engine '%s' does not support the max-turns feature", ...)
    }
}

Impact: Users may configure max-turns for custom engines and get compile errors.

Recommendation:
Add to max-turns schema description: "Note: Not all engines support max-turns. Supported by claude, copilot, and codex engines."

Moderate Issues

Moderate Issues

4. tools Meta-Fields Mixed with Actual Tools

Severity: LOW (Design/Organization)

Issue: Schema lists safety-prompt, timeout, and startup-timeout as tool properties alongside actual tools (github, bash, etc), but these are configuration fields that get removed before passing to engine.

Evidence:

// Code explicitly removes these as "not actual tools"
delete(tools, "safety-prompt")
delete(tools, "timeout")
delete(tools, "startup-timeout")

Recommendation: Add comment in schema noting these are "meta configuration fields" rather than actual tools.

---

5. Pull Request max Parameter Limitation Undocumented

Severity: LOW

Issue: Code comment indicates max parameter not supported for PRs (always 1), but schema doesn't document this.

Evidence:

// Note: max parameter is not supported for pull requests (always limited to 1)

Recommendation: Add to create-pull-request schema description: "Note: max parameter not applicable (always 1)"

Positive Findings

Positive Findings ✅

Perfect Consistency Across Schema, Comments, and Implementation

1. campaign Validation - 8 character minimum perfectly aligned
2. max-patch-size Default - 1024 KB default consistent everywhere
3. roles Default - ['admin', 'maintainer', 'write'] perfectly matched
4. safety-prompt Default - true (enabled) consistent
5. HTTP Transport Validation - Excellent implementation with proper interface methods

Key Insight: Code quality is exceptional. Validation logic is solid with proper error messages. Only issue is schema documentation needs to catch up with implementation.

Methodology & Strategy Details

Methodology

Strategy 016: Comment-Driven Documentation Gap Analysis (NEW)

Novel Approach: Bottom-up analysis starting from code comments and inline documentation, working backwards to schema validation.

Why This Strategy:

• Day of year 319 % 10 = 9 → Try new, radically different approach
• Previous 13 strategies covered top-down analysis (schema → code)
• New strategy inverts the direction (code comments → schema)

Analysis Steps:

1. Search for TODOs/FIXMEs in parser and workflow code related to schema/validation
2. Extract code comments describing fields, validation, defaults
3. Cross-reference comments with schema descriptions
4. Validate claimed defaults against actual implementation
5. Check inline code examples for schema compliance
6. Compare comment-described behavior vs schema documentation

Key Discovery:
Code comments revealed engine-specific feature support patterns that were correctly implemented but poorly documented in schema. This pattern would be hard to detect with traditional field enumeration strategies.

Strategy Performance:

• Findings: 10 total (5 issues + 5 positive confirmations)
• Unique Issues: 3 (engine documentation gaps)
• Effectiveness: VERY HIGH
• Should Reuse: YES - every 5-6 analyses

Complete Analysis Details

Key Insights

1. Engine-Specific Documentation is the Primary Gap

Pattern Identified: Features correctly implemented and validated, but schema doesn't document engine compatibility.

Examples:

• HTTP transport (works for claude/copilot/codex, not custom)
• max-turns (supported by major engines, not custom)
• web-search (engine-dependent)

Solution Pattern: Add engine compatibility notes to schema descriptions for all engine-dependent features.

---

2. Code Comment Accuracy is Exceptional

Finding: Developer comments in code closely match actual implementation.

Examples:

• campaign validation (8 chars) - perfect match
• max-patch-size default (1024) - perfect match
• safety-prompt default (true) - perfect match

Impact: High comment quality makes bottom-up analysis reliable and effective.

---

3. Implementation Quality Exceeds Documentation

Finding: Validation logic is solid with proper interface methods and error messages, but schema docs lag behind.

HTTP Transport Example:

• ✅ Clean interface method SupportsHTTPTransport()
• ✅ Proper validation with helpful error messages
• ✅ Correct engine support implementation
• ❌ No schema warning about engine limitations

---

4. Bottom-Up Analysis Complements Top-Down Strategies

This Strategy Found:

• Engine compatibility documentation gaps (new)
• Default value claim verification (different angle)
• Comment-to-reality consistency (new dimension)

Previous Strategies Found:

• Field enumeration mismatches (strategy-001)
• Schema-parser struct misalignment (strategy-011)
• Required field enforcement gaps (strategy-003)

Conclusion: Comment-driven analysis catches different issues than field-level validation.

Recommendations

Immediate Actions (High Priority)

1. ✅ Verify timeout-minutes Default Behavior

   • Check if 15-minute default is intended or GitHub Actions default (360 min) is acceptable
   • Update schema description or implement 15-minute default
   • Effort: 15 minutes

2. ✅ Add Engine Compatibility Documentation

   • Update HTTP transport schema description with engine requirements
   • Add max-turns engine support note
   • Document other engine-specific features
   • Effort: 1-2 hours

Short-Term Improvements (Medium Priority)

3. 📋 Create Engine Support Matrix Documentation

   • Document which engines support which features
   • Add to main documentation
   • Reference in schema descriptions
   • Effort: 2-3 hours

4. 📋 Document Safe-Outputs Limitations

   • Add PR max parameter note
   • Document other safe-output configuration limits
   • Effort: 30 minutes

Long-Term Considerations (Low Priority)

5. 🔧 Restructure tools Schema Organization
   • Consider separating meta-configuration from actual tools
   • Improve clarity of what constitutes a "tool"
   • Effort: 4-6 hours (includes migration planning)

---

Files Referenced

Schema Files:

• pkg/parser/schemas/main_workflow_schema.json
• pkg/parser/schemas/mcp_config_schema.json

Implementation Files:

• pkg/workflow/agent_validation.go - HTTP transport and max-turns validation
• pkg/workflow/compiler.go - Field extraction and processing
• pkg/workflow/frontmatter_extraction.go - Campaign validation
• pkg/workflow/safe_outputs.go - max-patch-size default
• pkg/workflow/role_checks.go - roles default
• pkg/workflow/agentic_engine.go - Engine capability interface
• pkg/workflow/claude_engine.go, copilot_engine.go, codex_engine.go - Engine support

---

Next Steps

• URGENT: Verify timeout-minutes default (15 vs 360 minutes)
• HIGH: Add engine compatibility notes to schema descriptions
• MEDIUM: Create engine feature support matrix documentation
• MEDIUM: Document safe-outputs parameter limitations
• LOW: Consider tools schema restructuring for clarity

---

Analysis Completed: 2025-11-15
Strategy Used: Strategy-016 (NEW) - Comment-Driven Documentation Gap Analysis
Total Findings: 10 (5 issues, 5 positive)
Strategy Cache: Updated with strategy-016 (total: 14 strategies)
Next Analysis: 2025-11-16 (automatic)

---

Strategy Cache Update

Strategy-016 has been successfully added to the cache with:

• Success Count: 1
• Effectiveness: very-high
• Recommendation: Use every 5-6 analyses for schema-reality validation

This bottom-up approach complements existing top-down strategies and successfully identified a pattern of engine-specific feature documentation gaps that previous strategies missed.

AI generated by Schema Consistency Checker

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.