[Schema Consistency] Schema Consistency Analysis: Conditional Logic & Feature Flags (2025-11-16)

[github-actions[bot]]

🔍 Schema Consistency Check - 2025-11-16

This analysis focuses on conditional logic and feature flag patterns across the schema, parser implementation, compiler validation, and documentation. The strategy examines how oneOf/anyOf patterns are validated and whether conditional requirements (like engine-specific features) are properly documented and enforced.

Summary

Inconsistencies Found: 7 findings (1 critical, 2 moderate, 4 informational)

Key Discovery: Engine-specific feature requirements are validated at compile-time but completely absent from the JSON schema, preventing IDE/editor tools from warning users during workflow authoring.

Full Analysis Report

Analysis Metadata

• Strategy Used: Strategy-005 (Conditional Logic & Feature Flag Analysis)
• Last Used: 2025-10-30 (17 days ago)
• New Strategy: No (proven strategy, 70% selection probability)
• Day of Year: 320 (320 mod 10 = 0, use cached strategy)
• Effectiveness: Very High (previous success: 6 findings, very-high rating)

Categories Analyzed

✅ Schema → Parser: oneOf pattern handling, conditional validation
✅ Schema → Documentation: oneOf examples, feature descriptions
✅ Parser → Compiler: Feature interaction validation, engine requirements
✅ Schema → Workflows: Real-world usage patterns, conditional configurations

Critical Issues

1. Engine-Specific Feature Requirements Missing from Schema

Severity: 🔴 Critical
Impact: High - Users discover incompatibilities only at compile time, not during authoring

Three engine-specific features are validated in the compiler but have zero schema documentation:

a. HTTP Transport Engine Requirement

Location: pkg/workflow/agent_validation.go:118-129
Validation: HTTP MCP tools require engine support (e.g., copilot)
Schema Status: ❌ No compatibility information in schema
Error Example: "tool 'my-tool' uses HTTP transport which is not
               supported by engine 'codex'. Only stdio transport
               is supported."

Problem: IDE validation cannot warn users that their engine choice doesn't support HTTP MCP tools.

b. max-turns Engine Requirement

Location: pkg/workflow/agent_validation.go:137-152
Validation: max-turns only works with specific engines
Schema Status: ❌ No engine compatibility marker
Error Example: "max-turns not supported: engine 'codex' does not
               support the max-turns feature. Use engine: copilot"

Evidence from workflows:

• .github/workflows/cloclo.md: uses max-turns: 100 with copilot engine ✅
• .github/workflows/commit-changes-analyzer.md: uses max-turns: 100 ✅
• Users with other engines would only discover incompatibility at compile time

c. web-search Tool Engine Requirement

Location: pkg/workflow/agent_validation.go:161-173
Validation: Warning if engine doesn't support web-search
Schema Status: ❌ No compatibility information
Impact: Runtime warning instead of design-time guidance

Recommendation: Add engine compatibility markers to schema descriptions:

{
  "max-turns": {
    "type": "string",
    "pattern": "^[0-9]+$",
    "description": "Maximum conversation turns. ⚠️ Requires engine with max-turns support (e.g., copilot). Not supported by: codex, custom engines."
  },
  "tools": {
    "properties": {
      "(tool-name)": {
        "properties": {
          "type": {
            "enum": ["stdio", "http"],
            "description": "MCP server transport. ⚠️ HTTP transport requires engine support (e.g., copilot). stdio is universally supported."
          }
        }
      }
    }
  }
}

Files Affected:

• pkg/parser/schemas/main_workflow_schema.json - needs compatibility warnings
• pkg/parser/schemas/mcp_config_schema.json - needs transport compatibility docs
• docs/src/content/docs/reference/frontmatter.md - should document requirements

---

Moderate Issues

2. Conditional Default Values Not Documented in Schema

Severity: 🟡 Moderate
Impact: Medium - Users rely on trial-and-error to discover resource-specific defaults

Finding: Safe-outputs max parameter has different defaults depending on resource type:

// From pkg/workflow/safe_outputs.go
Issues: max default = 1
Discussions: max default = 1
Agent tasks: max default = 1
Comments: max default = 1
Labels: max default = 3  // Different!

Schema Status: ❌ Schema defines max as optional integer but doesn't document conditional defaults

Current schema:

{
  "max": {
    "type": "integer",
    "description": "Maximum number to create",
    "minimum": 1
  }
}

Recommended schema:

{
  "max": {
    "type": "integer",
    "description": "Maximum number to create (default: 1 for issues/discussions/comments/agent-tasks, 3 for labels)",
    "minimum": 1,
    "default": 1
  }
}

Code Reference: pkg/workflow/safe_outputs.go (implementation is consistent, just needs schema documentation)

---

3. Network Configuration Usage Guidance Missing

Severity: 🟡 Moderate
Impact: Medium - Users don't know when to use string vs object format

Finding: Network field supports oneOf pattern (string OR object):

# String format (ecosystem identifier)
network: defaults
network: containers

# Object format (explicit domains)
network:
  allowed-domains:
    - api.github.com
    - example.com

Usage Statistics (from 78 workflows):

• String format: 5 workflows
• Object format: 26 workflows
• Total network configs: 31

Problem: Documentation shows both formats but doesn't explain when to use which:

• ✅ Schema correctly defines oneOf pattern
• ✅ Compiler handles both formats correctly
• ❌ Docs lack usage guidance for choosing format

Recommendation: Add to docs/src/content/docs/reference/frontmatter.md:

### Choosing Network Format

**Use string format** for common scenarios:
- `network: defaults` - Basic infrastructure (package managers, GitHub APIs, cert services)
- `network: containers` - Container registries (Docker Hub, GHCR, etc.)

**Use object format** for custom restrictions:
- Fine-grained control over allowed domains
- Additional ecosystem identifiers + custom domains
- Security-sensitive workflows requiring explicit allowlists

---

Informational Findings (Working Correctly)

4. oneOf Patterns: Excellent Implementation ✅

Finding: 13 fields use oneOf patterns (string OR object) - all working correctly:

Schema fields with oneOf:
✅ on (string trigger OR object with config)
✅ permissions (string shorthand OR granular object)
✅ runs-on (string runner OR advanced object)
✅ concurrency (string group OR object with cancel-in-progress)
✅ env, environment, container, network, steps, post-steps, cache, roles

Evidence:

• Schema: Correctly defines all oneOf patterns
• Parser: Uses resolveSchemaWithOneOf() to provide good autocomplete (defaults to object variant)
• Docs: frontmatter-full.md explicitly shows both formats with examples
• Compiler: Handles both variants correctly

Code Reference: pkg/parser/schema.go:638-639 (oneOf resolution for autocomplete)

No action needed - this is exemplary implementation.

---

5. Command Event Restrictions: Working as Designed ✅

Finding: Command trigger correctly restricts events to prevent conflicts

Test Coverage: pkg/workflow/compiler_test.go:591-639

• "command with conflicting issues event - should error"
• "command with conflicting issue_comment event - should error"
• "command with conflicting pull_request event - should error"
• "command with conflicting pull_request_review_comment event - should error"

Real-World Usage: Workflows correctly use event restrictions:

# .github/workflows/plan.md
on:
  command:
    name: plan
    events: [issue_comment, discussion_comment]  # Explicit restriction ✅

# .github/workflows/brave.md
on:
  command:
    name: brave
    events: [issue_comment]  # Single event ✅

Schema: Correctly defines on.command.events as oneOf (string OR array) with enum validation

No issues found - conditional validation working perfectly.

---

6. Safe-Outputs Conditional Defaults: Consistent Implementation ✅

Finding: Default max values are consistently implemented per resource type

Implementation: pkg/workflow/safe_outputs.go

• All code paths use consistent defaults
• Issues/discussions/comments: max=1
• Labels: max=3
• Agent tasks: max=1

Testing: Real workflows rely on these defaults:

# Many workflows omit 'max' parameter, relying on defaults
safe-outputs:
  create-issue:
    title-prefix: "[task] "
    labels: [task]
    # max omitted, defaults to 1 ✅

Issue: Defaults are in code but not schema (see Moderate Issue #2)

No compiler bugs - implementation is solid, just needs schema documentation.

---

7. Permissions oneOf: String Format Defined but Unused ✅

Finding: Schema defines string shorthand for permissions, but no workflows use it

Schema Definition:

{
  "permissions": {
    "oneOf": [
      {
        "type": "string",
        "enum": ["read-all", "write-all", "read", "write"]
      },
      {
        "type": "object",
        "properties": { /* granular permissions */ }
      }
    ]
  }
}

Usage Statistics (from 78 workflows):

• String shorthand: 0 workflows
• Object format: 243 usages
• Observation: Feature works but isn't adopted

Possible Reasons:

• Users prefer explicit granular control
• String shorthand may be too coarse-grained
• Documentation examples favor object format

No action needed - valid feature that users choose not to use. Consider deprecation if never adopted.

---

Documentation Gaps

Schema Documentation Needed

1. Engine Compatibility Markers (Critical)

   • Add to: pkg/parser/schemas/main_workflow_schema.json
   • Fields: max-turns, tools.(name).type (HTTP transport), web-search
   • Format: Add "⚠️ Requires engine support: ..." to descriptions

2. Conditional Defaults (Moderate)

   • Add to: pkg/parser/schemas/main_workflow_schema.json
   • Field: safe-outputs.(resource).max
   • Format: "default: 1 for issues/discussions/comments, 3 for labels"

User Documentation Needed

3. Network Format Selection Guide (Moderate)

   • Add to: docs/src/content/docs/reference/frontmatter.md
   • Section: Network configuration
   • Content: When to use string vs object format

4. Engine Feature Matrix (High Priority)

   • Add to: docs/src/content/docs/reference/engines.md (or create new page)
   • Content: Table showing which features each engine supports
   • Columns: Engine ID, HTTP Transport, max-turns, web-search, other features

Example Engine Matrix:

| Engine | HTTP Transport | max-turns | web-search | Notes |
|--------|---------------|-----------|------------|-------|
| copilot | ✅ | ✅ | ✅ | Full feature support |
| codex | ❌ | ❌ | ❌ | stdio only |
| custom | Varies | ❌ | ❌ | Check engine docs |

---

Schema Improvements Needed

1. Add Engine Compatibility Annotations

File: pkg/parser/schemas/main_workflow_schema.json

Changes:

{
  "properties": {
    "engine": {
      "oneOf": [
        {
          "type": "string"
        },
        {
          "type": "object",
          "properties": {
            "max-turns": {
              "type": "string",
              "pattern": "^[0-9]+$",
              "description": "Maximum conversation turns (limits iteration depth). ⚠️ Requires engine with max-turns support. Supported by: copilot. Not supported by: codex, custom engines."
            }
          }
        }
      ]
    },
    "tools": {
      "patternProperties": {
        "^[a-zA-Z0-9_-]+$": {
          "properties": {
            "type": {
              "enum": ["stdio", "http"],
              "description": "MCP server transport type. ⚠️ HTTP transport requires engine support (e.g., copilot). stdio is universally supported by all engines."
            }
          }
        }
      }
    }
  }
}

2. Document Conditional Defaults

File: pkg/parser/schemas/main_workflow_schema.json

Changes (for each safe-output resource):

{
  "$defs": {
    "issue_config": {
      "properties": {
        "max": {
          "type": "integer",
          "minimum": 1,
          "default": 1,
          "description": "Maximum number of issues to create per workflow run (default: 1)"
        }
      }
    },
    "label_config": {
      "properties": {
        "max": {
          "type": "integer",
          "minimum": 1,
          "default": 3,
          "description": "Maximum number of labels to add per workflow run (default: 3)"
        }
      }
    }
  }
}

---

Parser Updates Required

None needed - parser correctly handles all conditional patterns.

Evidence:

• ✅ resolveSchemaWithOneOf() provides good autocomplete for oneOf fields
• ✅ Parser doesn't enforce engine compatibility (correct - that's compiler's job)
• ✅ All frontmatter fields parsed correctly regardless of format

---

Workflow Violations

None found - all workflows comply with conditional requirements:

Checked:

• ✅ Command workflows correctly use events array to restrict triggers
• ✅ max-turns only used with compatible engines (copilot)
• ✅ Network configs use valid formats (string OR object)
• ✅ Permissions follow schema patterns
• ✅ No workflows violate engine compatibility constraints

Sample Validation:

# Workflows with max-turns
.github/workflows/cloclo.md: engine: copilot, max-turns: 100 ✅
.github/workflows/commit-changes-analyzer.md: engine: copilot, max-turns: 100 ✅
.github/workflows/daily-multi-device-docs-tester.md: engine: copilot, max-turns: 30 ✅

# All use copilot engine - no violations

---

Recommendations

Immediate Actions (Critical)

1. Add Engine Compatibility to Schema 🔴

   • File: pkg/parser/schemas/main_workflow_schema.json
   • Fields: max-turns, tools.type (HTTP)
   • Add warning markers in descriptions
   • Priority: High - enables IDE validation

2. Create Engine Feature Matrix Documentation 🔴

   • File: docs/src/content/docs/reference/engines.md (new)
   • Table of engine capabilities
   • Link from frontmatter.md references to engines
   • Priority: High - user discovery issue

Short-Term Actions (Moderate)

3. Document Conditional Defaults in Schema 🟡

   • File: pkg/parser/schemas/main_workflow_schema.json
   • Add "default: X" to schema
   • Update descriptions to explain conditional behavior
   • Priority: Medium - improves user experience

4. Add Network Format Usage Guide 🟡

   • File: docs/src/content/docs/reference/frontmatter.md
   • Section on when to use string vs object format
   • Priority: Medium - reduces confusion

Long-Term Actions (Low Priority)

5. Consider Deprecating Unused String Formats ⚪
   • Observation: permissions string shorthand never used
   • Action: Evaluate if simplification possible
   • Priority: Low - not causing issues

---

Strategy Performance

• Strategy Used: Strategy-005 (Conditional Logic & Feature Flag Analysis)
• Findings: 7 (1 critical, 2 moderate, 4 info)
• Effectiveness: Very High
• Should Reuse: ✅ Yes - excellent for finding schema-validation gaps
• Best For: Discovering runtime validation that lacks design-time schema support

Why This Strategy Succeeded:

• Focused on relationships between fields rather than individual field validation
• Examined validation logic in compiler to find schema gaps
• Cross-referenced real workflow usage to validate findings
• Identified pattern: implementation is correct, schema documentation lags

Comparison to Previous Runs:

• 2025-10-30 run: 6 findings, focused on conditional requirements
• 2025-11-16 run: 7 findings, discovered engine compatibility gap (new)
• Evolution: Strategy now checks for runtime validation not in schema

---

Next Steps

Immediate:

• Add engine compatibility warnings to schema descriptions (max-turns, HTTP transport)
• Create engine feature matrix documentation page

Follow-Up:

• Document conditional defaults in schema
• Add network configuration usage guide
• Review permissions string shorthand usage (consider deprecation)

Validation:

• Test IDE schema validation with updated compatibility warnings
• Verify documentation improvements help users discover engine limitations earlier

---

References:

• Analysis strategy details: /tmp/gh-aw/cache-memory/strategy-selected-2025-11-16.txt
• Detailed findings: /tmp/gh-aw/cache-memory/strategy-005-findings-2025-11-16.md
• Strategy database: /tmp/gh-aw/cache-memory/strategies.json

AI generated by Schema Consistency Checker

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.