🔍 Agentic Workflow Audit Report - November 25, 2025

[github-actions[bot]]

This audit analyzes 83 workflow runs from the last 24 hours (November 24-25, 2025) to identify issues, track performance metrics, and monitor workflow health trends.

Executive Summary

Last 24 Hours Performance:

• Total Runs: 83 workflows executed
• Success Rate: 79.5% (66 successful, 9 failed, 8 other)
• Token Usage: 27.8M tokens consumed
• Estimated Cost: $20.70
• Errors Detected: 1,619 errors across all runs
• Missing Tools: 1 unique tool requested (playwright, 3 occurrences)
• MCP Failures: None detected
• Average Performance: 335K tokens/run, $0.25/run

📈 Workflow Health Trends

Success/Failure Patterns

The 5-day trend shows workflow activity has been relatively stable with success rates hovering around 79-80%. The latest data point (Nov 25) shows continued strong performance with the majority of runs succeeding. Notable peaks in successful runs occurred on Nov 23-24, indicating periods of high workflow activity. Failure counts remain low but persistent, warranting investigation of the recurring issues.

Token Usage & Costs

Token consumption shows variability with peaks reaching 9M+ tokens per day, while the 3-day moving average provides a smoother view of resource utilization trends. The cost trend correlates directly with token usage, with daily costs ranging from $2-7. The parallel tracking of run counts shows that higher activity days naturally incur higher costs, with the 5-day total reaching $23.39.

Full Audit Details

Missing Tools Analysis

Playwright Tool Requests

Tool: playwright
Request Count: 3 occurrences
Affected Workflows: Smoke Copilot
Run IDs: §19633673483, §19644306481, §19650577171

Analysis: The Smoke Copilot workflow is requesting the playwright tool which is not currently available. This appears to be a consistent pattern affecting smoke tests.

Recommendation: Evaluate whether playwright browser automation is necessary for the Smoke Copilot workflow. If required, consider:

1. Adding playwright MCP server to workflow configuration
2. Modifying workflow to use alternative browser automation methods
3. Adjusting workflow instructions to avoid browser-dependent operations

Error Analysis

Error Distribution

Total Errors: 1,619 across 83 runs
Average Errors per Run: 19.5 errors

Top Workflows with Errors:

Workflow	Failed Runs	Error Count
Copilot PR Conversation NLP Analysis	1	55
The Daily Repository Chronicle	1	5
AI Triage Campaign	1	2
Glossary Maintainer	3	5 total
Smoke Copilot	2	2 total

Critical Failed Runs

1. §19630410254 - Copilot PR Conversation NLP Analysis

   • Status: Failed
   • Errors: 55
   • Impact: High error count indicates significant issues with this workflow

2. §19640715107 - The Daily Repository Chronicle

   • Status: Failed
   • Errors: 5
   • Impact: Daily scheduled workflow experiencing failures

3. §19650394568 - Glossary Maintainer

   • Status: Failed
   • Errors: 2
   • Impact: Recurring failures (3 failed runs in 24 hours)

4. §19651978426 - Smoke Copilot

   • Status: Failed
   • Errors: 1
   • Impact: Smoke test failure affecting quality checks

5. §19648880219 - Changeset Generator

   • Status: Failed
   • Errors: 1
   • Impact: PR automation workflow experiencing issues

Error Pattern Analysis

Most errors appear to be JSON message parsing or tool result errors based on the error message patterns. The errors contain tool_use_id references, suggesting issues with tool response handling or message formatting in the agentic workflow infrastructure.

MCP Server Status

✅ No MCP Server Failures Detected

All configured MCP servers operated without failures during the audit period. This indicates stable integration with external services.

Tool Usage Statistics

Top 10 Most Used Tools (Last 24 Hours)

Tool	Total Calls	Runs	Purpose
github	620	31	GitHub API interactions
TodoWrite	141	28	Task tracking
Read	107	27	File reading
safeoutputs	106	31	Safe output generation
Edit	65	4	File editing
Grep	31	6	Code search
github_search_pull_requests	21	15	PR searching
Write	19	12	File creation
playwright_browser_navigate	18	17	Browser automation
ast-grep_ast-grep	14	8	AST-based code search

Key Observations:

• GitHub API tools dominate usage (620 calls), indicating heavy integration with GitHub services
• Task management (TodoWrite) is actively used across most workflows
• Browser automation (playwright) is successfully being used despite some missing tool reports
• File operations (Read, Edit, Write) show healthy usage patterns

Performance Metrics

Cost Analysis

• Total 24h Cost: $20.70
• Average Cost per Run: $0.25
• 5-Day Total Cost: $23.39
• Highest Cost Workflow: Copilot PR Conversation NLP Analysis (55 errors likely contributed to high token usage)

Token Efficiency

• Total Tokens (24h): 27,823,546
• Average Tokens per Run: 335,223
• Token Distribution: Varies significantly by workflow type
  • Analytical workflows: Higher token usage (300K-500K)
  • Simple workflows: Lower token usage (50K-150K)

Execution Metrics

• Total Turns: 683 conversational turns across all runs
• Average Turns per Run: 8.2 turns
• Duration: Varies by workflow complexity

Recommendations

Priority 1 - Critical Issues

1. Investigate Copilot PR Conversation NLP Analysis Failures

   • 55 errors in single run indicates major issues
   • Review workflow logs at §19630410254
   • Consider disabling until root cause is identified

2. Address Glossary Maintainer Reliability

   • 3 failures in 24 hours is concerning
   • Review error patterns across runs: §19650394568, §19648902357, §19630529986
   • May need workflow instruction improvements or bug fixes

3. Resolve Playwright Tool Request Issues

   • Add playwright MCP server to Smoke Copilot workflow
   • Or modify workflow to avoid browser automation requirements
   • Affects quality assurance processes

Priority 2 - Performance Optimization

1. Monitor Token Usage Spikes

   • Some workflows consuming excessive tokens
   • Implement token usage alerts for runs exceeding 500K tokens
   • Review and optimize verbose workflows

2. Improve Success Rate Target

   • Current 79.5% is below ideal 90%+ target
   • Focus on eliminating recurring failures
   • Enhance error handling in workflow instructions

3. Error Pattern Investigation

   • 1,619 errors suggest systematic issues
   • Many appear to be tool response formatting issues
   • Review agent SDK error handling

Priority 3 - Monitoring & Maintenance

1. Establish Success Rate Alerts

   • Alert when daily success rate drops below 80%
   • Track trends to catch degradation early

2. Cost Budget Tracking

   • Current $20/day rate is sustainable
   • Establish monthly budget of ~$600
   • Alert if daily costs exceed $30

3. Regular Audit Schedule

   • Continue daily audits to track improvements
   • Build historical trend database
   • Monthly deep-dive analysis

Affected Workflows Summary

Workflows with Issues (Last 24 Hours)

Workflow	Status	Issues
Copilot PR Conversation NLP Analysis	Failed	High error count (55)
The Daily Repository Chronicle	Failed	Execution errors (5)
Glossary Maintainer	Failed (3x)	Recurring reliability issues
Smoke Copilot	Failed (2x)	Missing playwright tool
Changeset Generator	Failed	Tool execution error
AI Triage Campaign	Failed	Minor errors (2)

Workflows Running Successfully

✅ 66 workflows completed successfully including:

• Smoke Claude (multiple runs)
• Go Pattern Detector
• Schema Consistency Checker
• Daily News
• Tidy
• And 61 others

Historical Context

This is the baseline audit report. Future audits will include trend comparisons with previous periods to identify:

• Improving or degrading success rates
• Emerging error patterns
• Cost efficiency trends
• New missing tool requests
• MCP server stability changes

Next Steps

Immediate Actions (Next 24 Hours):

• Investigate Copilot PR Conversation NLP Analysis failure
• Review Glossary Maintainer workflow instructions
• Evaluate playwright tool requirement for Smoke Copilot
• Create GitHub issues for top 3 priority items

Short-term Actions (Next Week):

• Implement token usage monitoring and alerts
• Enhance error handling in high-failure workflows
• Document error patterns and resolutions
• Establish success rate tracking dashboard

Long-term Actions (Next Month):

• Target 90%+ success rate across all workflows
• Optimize high-token workflows for efficiency
• Build comprehensive workflow health metrics
• Automate issue creation for failed workflows

---

Audit Metadata:

• Audit Date: 2025-11-25
• Audit Period: Last 24 hours (2025-11-24 to 2025-11-25)
• Trend Period: 5 days (2025-11-21 to 2025-11-25)
• Total Runs Analyzed: 83 (recent) + 92 (historical trend data)
• Data Source: gh-aw MCP server logs
• Analysis Tool: Python-based log analyzer
• Charts Generated: 2 trend visualizations

---

References:

• §19630410254 - Highest error count run
• §19640715107 - Daily Chronicle failure
• §19650394568 - Recent Glossary Maintainer failure

AI generated by Agentic Workflow Audit Agent