[workflow-analysis] Weekly Workflow Analysis - Nov 17-24, 2025

[github-actions[bot]]

Weekly Workflow Analysis Report

Analysis Period: November 17-24, 2025

This report analyzes GitHub Actions workflow runs from the past week, identifying failure patterns, performance issues, and opportunities for improvement across the gh-aw repository.

Key Findings

The Weekly Workflow Analysis workflow itself has experienced consistent failures in recent runs, creating a recursive monitoring problem where our monitoring tool needs monitoring. The last successful run was on November 3rd (§19029318601), with subsequent scheduled runs failing on November 10th, 17th, and the current run in progress.

Critical Issues

1. Self-Referential Monitoring Failure
The Weekly Workflow Analysis workflow has failed in 3 out of its last 5 scheduled runs. This is particularly concerning as this workflow is designed to monitor and analyze other workflows.

2. Tool Integration Challenges
Analysis of failed run §19424194602 reveals systematic issues with the agentic_workflows_logs MCP tool:

• JQ filter parsing errors: Multiple attempts to use jq filters failed with "Invalid numeric literal" errors
• Request timeouts: Repeated timeout errors when fetching workflow logs (MCP error -32001)
• Output size limitations: Tool responses exceeding token limits (73,019 tokens vs 25,000 limit)

Detailed Technical Analysis

Failure Pattern Analysis

Weekly Workflow Analysis Failures

Recent run history shows a troubling pattern:

• Nov 17: §19424194602 - FAILED (10.9m duration)
• Nov 10: §19226397498 - FAILED (11.0m duration)
• Nov 3: §19029318601 - SUCCESS (10.2m duration)
• Oct 27: §18835581756 - SUCCESS (7.9m duration)

Root Cause Analysis - Nov 17 Failure

The audit of §19424194602 identified 19 errors and 9 warnings:

Error Categories:

1. JQ Filter Failures (3+ occurrences)

   • Error: "jq filter failed: exit status 5, stderr: jq: parse error: Invalid numeric literal at line 1, column 4"
   • The agent attempted to use jq filters like .summary, .errors_and_warnings, and .missing_tools
   • Root cause: Likely malformed jq syntax or incompatible filter structure

2. MCP Timeout Errors (3+ occurrences)

   • Error: "MCP error -32001: Request timed out"
   • The logs tool consistently timed out when trying to fetch data
   • Suggests the data volume or processing time exceeds configured limits

3. Output Size Violations (2+ occurrences)

   • Error: "Output size (81,972 tokens) exceeds the limit (12,000 tokens)"
   • Error: "MCP tool 'audit' response (73,019 tokens) exceeds maximum allowed tokens (25,000)"
   • The workflow attempted to fetch too much data without proper pagination

4. Authentication Failure (1 occurrence)

   • Error: "GitHub CLI authentication required. Run 'gh auth login'"
   • Occurred late in the workflow execution, suggesting intermittent auth issues

Tool Usage Patterns

The failed run showed:

• 10 calls to agentic_workflows_logs - highest usage
• 3 calls to agentic_workflows_audit
• 4 calls to TodoWrite - proper task tracking
• 1 call to agentic_workflows_status

Performance Issues

1. Data Fetching Inefficiency

The workflow attempts to fetch large volumes of log data without proper:

• Pagination strategy: No incremental fetching of data
• Filter optimization: JQ filters fail instead of helping reduce payload
• Timeout handling: No retry logic for timeout scenarios

2. Token Budget Management

The workflow repeatedly exceeds token limits:

• Attempted to process 81K+ tokens when limit is 12K
• Audit responses of 73K tokens vs 25K maximum
• No progressive reduction strategy when hitting limits

3. Error Recovery

The workflow shows poor error recovery:

• Multiple retry attempts with the same failing approach
• No fallback to alternative data fetching methods
• Cascading failures due to unhandled errors

Workflow Ecosystem Health

Active Workflows Distribution

The repository contains 124 total workflows with varying AI engines:

• Claude-based workflows: Used for complex analysis tasks
• Copilot-based workflows: Used for code generation and reviews
• Codex-based workflows: Used for specialized tasks

Time-Limited Workflows

Several workflows have stop-after deadlines approaching:

• ci-doctor: 20 days remaining (stops after +1 month)
• daily-team-status: 23 days remaining (stops after +1 month)

Compilation Status

Not all workflows are compiled:

• blog-auditor: Not compiled
• commit-changes-analyzer: Compiled
• daily-repo-chronicle: Not compiled
• dev: Not compiled
• example-permissions-warning: Not compiled
• poem-bot: Not compiled

Recommendations

Immediate Actions (Priority: Critical)

1. Fix the Weekly Workflow Analysis Workflow

• Issue: The monitoring workflow itself is broken
• Action: Revise the workflow to use proper jq syntax and handle large datasets
• Implementation:

  # Use simpler jq filters that work reliably
  # Example: Instead of complex nested queries, use basic filters
  jq: '.runs | length'  # Works
  jq: '.summary'  # Currently fails - needs investigation

2. Implement Proper Pagination

• Issue: Attempting to fetch too much data at once
• Action: Reduce count parameter and use continuation tokens
• Implementation:

  # Fetch in smaller batches
  count: 10  # Instead of 50+
  # Use continuation for additional data

3. Add Timeout Handling

• Issue: No retry logic for MCP timeouts
• Action: Implement exponential backoff and fallback strategies
• Impact: Reduce cascade failures from transient issues

Short-term Improvements (Priority: High)

4. Simplify Data Queries

• Replace complex jq filters with multiple simple queries
• Use GitHub API directly for specific data points when MCP tools fail
• Add validation for filter syntax before execution

5. Implement Token Budget Management

• Pre-calculate expected response sizes
• Request specific fields instead of full objects
• Use minimal_output: true where available

6. Add Monitoring for Monitors

• Create a lightweight health check for the Weekly Workflow Analysis
• Send alerts when the analysis workflow fails
• Consider a backup manual review process

Long-term Optimizations (Priority: Medium)

7. Workflow Pruning

• Review time-limited workflows approaching expiration
• Compile uncompiled workflows to ensure they're ready for use
• Archive or remove workflows that are no longer actively used

8. Improve MCP Tool Reliability

• Add better error messages for jq filter failures
• Implement automatic retry with simplified queries on failure
• Consider caching frequently accessed log data

9. Create Workflow Health Dashboard

• Track success rates by workflow over time
• Identify patterns in failures (time of day, specific triggers, etc.)
• Monitor resource usage trends (tokens, duration, costs)

Success Metrics

To measure improvement, track:

• Weekly Workflow Analysis success rate: Target 100% (currently ~40% recent runs)
• Average execution time: Target <8 minutes (currently 10-11 minutes on failures)
• MCP tool timeout rate: Target <5% (currently ~30% in failed runs)
• Token budget violations: Target 0 per run

Conclusion

The primary issue is a recursive monitoring problem: our workflow analysis tool is experiencing the same types of failures it's designed to detect in other workflows. The root causes are:

1. Inadequate handling of large datasets
2. Fragile jq filter implementation
3. Poor error recovery mechanisms

Fixing these issues in the Weekly Workflow Analysis workflow will not only restore our monitoring capabilities but also provide valuable insights for improving other workflows experiencing similar problems.

---

References:

• §19424194602
• §19226397498
• §19029318601

AI generated by Weekly Workflow Analysis