🎯 Repository Quality Improvement Report - Workflow Compilation Feedback Quality

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Compilation Feedback Quality

Analysis Date: 2025-11-15
Focus Area: Workflow Compilation Feedback Quality
Strategy Type: Custom
Custom Area: Yes - This focus area addresses the unique needs of gh-aw's core function: compiling markdown workflows to YAML. The quality of error messages, validation feedback, and user guidance directly impacts developer productivity and adoption.

Executive Summary

The gh-aw compilation system demonstrates strong fundamentals in error reporting, with sophisticated schema validation, precise error location tracking, and user-friendly message formatting. The system processes 336 error creation sites across 55 files using console formatting, with 952 lines dedicated to error quality infrastructure (schema validation, location detection, typo suggestions).

However, analysis reveals opportunities to enhance the developer experience through more comprehensive troubleshooting guidance, improved actionability of error messages, and better progressive disclosure of validation details. Currently, only 13% of error messages (45/336) include actionable suggestions, and there's no centralized troubleshooting documentation despite 22 docs mentioning errors/debugging.

Full Analysis Report

Focus Area: Workflow Compilation Feedback Quality

Current State Assessment

The compilation feedback system consists of several well-architected components:

Metrics Collected:

Metric	Value	Status
Files with console formatting	55	✅ Good coverage
Error creation sites	336	⚠️ High surface area
Error messages with suggestions	45 (13%)	⚠️ Low suggestion rate
Error messages with doc links	16 (5%)	❌ Needs improvement
Lines in error quality infrastructure	952	✅ Strong investment
Workflow markdown files in repo	114	✅ Good test coverage
Docs mentioning errors/troubleshooting	22	⚠️ Scattered
Dedicated troubleshooting guide	0	❌ Missing

Findings

Strengths

1. Sophisticated Error Location System: The pkg/parser/schema.go (44 lines) and pkg/parser/location.go provide precise line/column tracking with context
2. Schema-Based Validation: JSON schema validation with cleanJSONSchemaErrorMessage and generateSchemaBasedSuggestions
3. Typo Detection: Excellent typo suggestions (tested: "engien" → "Did you mean 'engine'?")
4. Console Formatting: Consistent use of console.FormatErrorMessage for user-facing output
5. Context-Rich Errors: Error messages include code snippets with line markers (^^^)
6. Security-First Messaging: Strict mode errors include actionable alternatives with doc links

Example of excellent error quality:

✗ ../../../../../tmp/test-typo.md:3:1: error: Unknown property: engien. Did you mean 'engine'?
1 | on: push
2 | engien: copilot
3 | ---
    ^^^

Areas for Improvement

1. Low Documentation Link Coverage (5%): Only 16/336 error messages include documentation links

   • Severity: Medium - Users need guidance beyond error text
   • Impact: Increases support burden, slows learning curve

2. Limited Actionable Suggestions (13%): Only 45/336 errors include "Did you mean" or similar guidance

   • Severity: Medium - Users may struggle to resolve issues independently
   • Impact: Reduces self-service capability

3. No Centralized Troubleshooting Guide: 22 docs mention errors but no unified troubleshooting resource

   • Severity: High - Common patterns not documented
   • Impact: Users repeatedly encounter same issues

4. Validation Details Not Progressive: All validation errors shown at once, no summary mode

   • Severity: Low - Can overwhelm new users
   • Impact: Cognitive load for complex workflows

5. Error Recovery Paths Unclear: Errors identify problems but don't always suggest next steps

   • Severity: Medium - Users know what's wrong but not what to do
   • Impact: Frustration, trial-and-error fixes

Detailed Analysis

Error Message Quality Patterns

Excellent Examples Found:

• Strict mode permissions: Provides alternatives, explains why, includes doc link
• Typo detection: Suggests correct spelling with Levenshtein distance
• Type mismatches: Shows expected vs actual types with file context

Improvement Opportunities:

• Network validation errors: Could suggest common allowed domains
• MCP configuration errors: Could link to MCP server setup examples
• Tool permission mismatches: Could show example valid configurations
• Import resolution failures: Could suggest checking file paths

Documentation Gap Analysis

Current state:

• 22 documentation pages mention errors/troubleshooting
• Content is scattered across reference, guides, concepts
• No single "Common Errors and Solutions" guide
• No error code/category system for quick lookup

Ideal state:

• Centralized troubleshooting guide with categories
• Error messages reference specific doc sections
• Common pitfalls documented with solutions
• Searchable error message index

Progressive Disclosure Opportunity

Current compilation output shows all errors at once:

✗ Compiled 1 workflow(s): 1 error(s), 0 warning(s)
✗ Failed workflows:
  - test-workflow.md
✗ [full error details]

Could implement:

• Summary mode: Just counts and workflow names
• Detail mode: Current behavior (default)
• Verbose mode: Include validation steps, checks performed
• JSON mode: Already exists, good for tooling

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

Task 1: Create Comprehensive Troubleshooting Guide

Priority: High
Estimated Effort: Large
Focus Area: Documentation

Description:
Create a centralized troubleshooting guide that documents common compilation errors, validation failures, and their solutions. This should become the go-to resource for users encountering issues.

Acceptance Criteria:

• New file created at docs/src/content/docs/guides/troubleshooting.md
• Organized by error category (Schema Validation, Strict Mode, Tools/MCP, Network, Permissions)
• Each section includes example error, explanation, and solution
• References to related documentation sections
• Searchable format with clear headings
• Includes "Quick Fixes" section for most common issues

Code Region: docs/src/content/docs/guides/

Create a comprehensive troubleshooting guide for gh-aw workflow compilation errors.

Structure the guide with these main sections:

1. **Quick Fixes** - Top 5-10 most common errors with one-line solutions
2. **Schema Validation Errors** - Unknown properties, type mismatches, required fields
3. **Strict Mode Restrictions** - Permissions, network, tool limitations
4. **MCP and Tool Configuration** - Server setup, tool permissions, allowed domains
5. **Network and Security** - Firewall configuration, domain allowlists
6. **Workflow Syntax** - YAML formatting, frontmatter structure, expressions
7. **Getting More Help** - Verbose mode, debug logs, reporting issues

For each error category:
- Show the actual error message users see
- Explain what causes it in plain language
- Provide step-by-step solution
- Include working example configuration
- Link to relevant reference docs

Use real examples from the codebase in .github/workflows/ directory.

Follow the Diátaxis "How-to Guide" format: goal-oriented, practical, focused on solving specific problems.

---

Task 2: Enhance Error Messages with Documentation Links

Priority: High
Estimated Effort: Medium
Focus Area: Code Quality

Description:
Systematically add documentation links to error messages where appropriate, focusing on the most common validation failures and configuration issues.

Acceptance Criteria:

• Network validation errors link to network configuration docs
• MCP configuration errors link to MCP server setup guide
• Permission errors link to permissions reference (already done for strict mode)
• Tool configuration errors link to tools reference
• Safe outputs errors link to safe outputs documentation
• At least 50% of validation errors include doc links (up from current 5%)

Code Region: pkg/workflow/validation.go, pkg/parser/schema.go, pkg/workflow/mcp_*.go

Add helpful documentation links to error messages in the workflow validation and parser code.

Focus on these high-impact error scenarios:

1. **Network validation errors** (pkg/workflow/validation.go)
   - Add link to (redacted)
   - Example: "Network configuration required. See: [docs link]"

2. **MCP configuration errors** (pkg/workflow/mcp_*.go)
   - Add link to (redacted)
   - Example: "Invalid MCP server configuration. See: [docs link]"

3. **Tool configuration errors** (pkg/workflow/tools.go)
   - Add link to (redacted)
   - Example: "Unknown tool configuration. See: [docs link]"

4. **Schema validation errors** (pkg/parser/schema.go)
   - Enhance generateSchemaBasedSuggestions to include relevant doc links
   - Map error types to appropriate documentation sections

Pattern to follow (from existing strict mode errors):
```go
return fmt.Errorf("error explanation. Use 'alternative' to solve. See: (redacted)")

Ensure links are:

• Specific to the error (not just homepage)
• Stable (use versioned docs where applicable)
• Helpful (direct to solution, not just general reference)

---

#### Task 3: Add Actionable Suggestions to Common Errors

**Priority**: Medium  
**Estimated Effort**: Medium  
**Focus Area**: Code Quality

**Description:**
Enhance error messages to include actionable suggestions for common failure scenarios. Move from just identifying problems to guiding users toward solutions.

**Acceptance Criteria:**
- [ ] Network errors suggest common allowed domains (e.g., "Try: network: defaults")
- [ ] Tool permission errors show example valid configurations
- [ ] Import resolution failures suggest checking file paths and extensions
- [ ] Type mismatch errors show example of correct type
- [ ] At least 30% of errors include actionable suggestions (up from 13%)

**Code Region:** `pkg/workflow/validation.go`, `pkg/parser/schema.go`, `pkg/workflow/network.go`

```markdown
Enhance error messages with actionable suggestions that guide users to solutions.

Add suggestion patterns for these common scenarios:

1. **Network configuration errors**
   - Current: "network configuration required"
   - Enhanced: "network configuration required. Try: `network: defaults` for basic access, or specify allowed domains. See examples: [link]"

2. **Tool permission mismatches**
   - Current: "tool X requires permission Y"
   - Enhanced: "tool X requires permission Y. Add to frontmatter: `permissions:\n  Y: read` (or 'write' if needed)"

3. **Import resolution failures**
   - Current: "failed to resolve import"
   - Enhanced: "failed to resolve import 'X'. Check: 1) file exists at path, 2) path is relative to .github/workflows/, 3) file has .md extension"

4. **Type mismatches**
   - Current: "got X, want Y"
   - Enhanced: "got X, want Y. Example valid value: [show example based on type]"

Implementation approach:
- Add suggestion generation functions for each error category
- Use switch/case on error type to provide context-specific guidance
- Include code snippet examples where helpful
- Keep suggestions concise (1-2 lines max)

Test with common user errors to ensure suggestions are helpful.

---

Task 4: Implement Error Message Categorization System

Priority: Medium
Estimated Effort: Small
Focus Area: Code Organization

Description:
Introduce an error categorization system that groups related errors and enables better organization in troubleshooting documentation and error reporting.

Acceptance Criteria:

• Define error categories (Schema, Validation, Security, Configuration, Syntax)
• Add category field to CompilerError struct
• Tag existing errors with appropriate categories
• Use categories in compilation summary output
• Enable filtering by category in JSON output mode

Code Region: pkg/console/compiler_error.go, pkg/cli/compile_command.go

Add error categorization to improve organization and troubleshooting.

Step 1: Define error categories in pkg/console/compiler_error.go
```go
type ErrorCategory string

const (
    ErrorCategorySchema        ErrorCategory = "schema"
    ErrorCategoryValidation    ErrorCategory = "validation"
    ErrorCategorySecurity      ErrorCategory = "security"
    ErrorCategoryConfiguration ErrorCategory = "configuration"
    ErrorCategorySyntax        ErrorCategory = "syntax"
    ErrorCategoryNetwork       ErrorCategory = "network"
    ErrorCategoryTools         ErrorCategory = "tools"
)

Step 2: Add Category field to CompilerError struct

type CompilerError struct {
    Position ErrorPosition
    Type     string
    Message  string
    Context  []string
    Category ErrorCategory // New field
}

Step 3: Update error creation sites to include category

• Schema validation errors → ErrorCategorySchema
• Strict mode errors → ErrorCategorySecurity
• Network validation → ErrorCategoryNetwork
• MCP/tool config → ErrorCategoryTools

Step 4: Enhance compilation summary to group by category

✗ Compiled 1 workflow(s): 3 error(s), 0 warning(s)
Schema Errors (2):
  - test-workflow.md: Unknown property 'engien'
  - test-workflow.md: Type mismatch
Security Errors (1):
  - test-workflow.md: Write permission not allowed in strict mode

Step 5: Include categories in JSON output mode for better tooling integration

---

#### Task 5: Create Error Message Testing Framework

**Priority**: Low  
**Estimated Effort**: Medium  
**Focus Area**: Testing

**Description:**
Establish a systematic testing framework for error messages to ensure quality, consistency, and prevent regressions in error message clarity.

**Acceptance Criteria:**
- [ ] Create test suite that validates error message quality
- [ ] Tests check for: presence of suggestions, doc links, actionable guidance
- [ ] Golden file tests for common error scenarios
- [ ] CI integration to catch error message regressions
- [ ] Coverage of top 20 most common error types

**Code Region:** `pkg/parser/schema_error_quality_test.go` (new file), `pkg/workflow/validation_error_quality_test.go` (new file)

```markdown
Create a comprehensive testing framework for error message quality.

Create new test file: pkg/parser/schema_error_quality_test.go

Test categories:

1. **Documentation Link Coverage Tests**
```go
func TestErrorMessagesWithDocLinks(t *testing.T) {
    // High-impact errors should include doc links
    highImpactErrors := []string{
        "strict mode", "network configuration", "MCP server",
        "permission", "safe-outputs",
    }
    // Test that errors matching these patterns include "See: https://"
}

2. Actionable Suggestion Tests

func TestErrorMessagesWithSuggestions(t *testing.T) {
    // Common errors should include suggestions
    // Patterns: "Try:", "Did you mean", "Add:", "Example:"
}

3. Error Message Clarity Tests

func TestErrorMessageClarity(t *testing.T) {
    // Ensure errors are understandable
    // Check: no jargon, includes context, identifies problem, suggests solution
}

4. Golden File Tests for common scenarios

   • Create test workflows with intentional errors
   • Capture expected error output
   • Detect regressions in error message format/quality

5. Error Message Consistency Tests

   • Consistent formatting across all error types
   • Consistent use of terminology
   • Consistent structure (problem → explanation → solution)

Integration with CI:

• Run as part of make test
• Fail build on error message quality regressions
• Generate report of error message coverage statistics

---

## 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2025-11-15 | Workflow Compilation Feedback Quality | Custom | Y | First quality improvement run - established baseline for error message quality, identified documentation gaps |

</details>

---

## 🎯 Recommendations

### Immediate Actions (This Week)
1. **Create troubleshooting guide** (Task 1) - Priority: High - This will immediately reduce support burden and improve developer experience
2. **Add doc links to top 10 error types** (Task 2 - partial) - Priority: High - Quick wins with high impact

### Short-term Actions (This Month)
1. **Complete documentation link enhancement** (Task 2) - Priority: High - Comprehensive coverage of all validation errors
2. **Enhance common errors with suggestions** (Task 3) - Priority: Medium - Improve actionability of error messages
3. **Implement error categorization** (Task 4) - Priority: Medium - Better organization and reporting

### Long-term Actions (This Quarter)
1. **Build error message testing framework** (Task 5) - Priority: Low - Prevent regressions, maintain quality over time
2. **Progressive disclosure UI** - Implement summary/detail/verbose modes for compilation output
3. **Error recovery wizard** - Interactive mode that suggests fixes and validates corrections

---

## 📈 Success Metrics

Track these metrics to measure improvement in **Workflow Compilation Feedback Quality**:

- **Doc Link Coverage**: 5% → 50% (error messages with documentation links)
- **Suggestion Rate**: 13% → 30% (error messages with actionable suggestions)
- **Troubleshooting Coverage**: 0 → 1 comprehensive guide + 20+ common error solutions
- **User Self-Service**: Track reduction in compilation-related support issues
- **Error Resolution Time**: Measure time from error to fix (via user studies)

---

## Next Steps

1. Review and prioritize the tasks above
2. Assign tasks to Copilot agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in Q1 2026 for progress assessment

---

*Generated by Repository Quality Improvement Agent*  
*Next analysis: 2025-11-16 - Focus area will be selected based on diversity algorithm (60% custom, 30% standard, 10% reuse)*


> AI generated by [Repository Quality Improvement Agent](https://github.com/githubnext/gh-aw/actions/runs/19385932313)

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.