You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Analysis Period: Last 30 days (2026-09-28 to 2026-10-27) Total Tasks Analyzed: 605 Clusters Identified: 7 Overall Success Rate: 84.0% (508 merged / 96 closed / 1 open)
Full Analysis Report
General Insights
Most Common Task Type: C1 — Agent/Workflow Orchestration & GitHub Integration (211 PRs, 35% of total)
Highest Success Rate: C6 — PR Sous Chef Bot-Driven Fixes (89.4% merged)
Most Complex Tasks: C5 — Dependency & Image Version Pinning (avg. 164.5 files changed per PR — large, mechanical bulk-pin diffs) and C6 — PR Sous Chef Bot-Driven Fixes (avg. 13.5 comments, 6.3 reviews — most discussion-heavy, via bot-driven iteration)
Outliers: None flagged this run. Every cluster's merge rate (81.7%–89.4%) sits within ~8 points of the 84.0% overall rate, and none meets the outlier bar (cluster size ≥ 15 PRs and merge rate ≥ 10 points below overall). This is a change from the 2026-10-26 report, which flagged a 63.5%-merge "gh-aw Compiler/CLI Core & Infra Hardening" cluster as an outlier (16.2 points below that run's 79.7% overall rate) — see Key Findings.
Methodology note: clustering used TF-IDF (1-2 grams, 300 features, min document frequency 2) + K-means over cleaned PR title/body text, with k selected by silhouette score across k=3–8 (best: k=7, silhouette 0.054 — a fairly flat, mixed-topic score consistent with this repo's prompts sharing a lot of common vocabulary across task types). Full PR data (comments, reviews, commits, file-change counts) was available in cache for all 605 PRs in the window, so per-cluster engagement stats below reflect complete coverage, not a sample. No workflow-run turn/duration/cost data is included — see Limitations.
Top Keywords: agent, workflow, add, github, repository, coverage, workflows, job
Characteristics: Broadest "glue" cluster — GitHub Actions job/workflow orchestration, Copilot agent job plumbing, firewall/egress scoping, checkout and bootstrap handling. Largest cluster by far yet still converges reliably despite topic diversity.
Top Keywords: model, routing, copilot, claude, models, routed, smoke, agent
Characteristics: Engine/model-routing logic — which AI engine and model a workflow runs under. Holds the dataset's only remaining open PR (Support Claude on Vertex AI with GCP WIF #72273, "Support Claude on Vertex AI with GCP WIF").
Success Rate: 81.7% (58 merged / 13 closed / 0 open) — lowest merge rate this cycle, but only 2.3 points below overall
Avg Comments / Reviews / Files Changed: 4.8 / 2.9 / 13.5 (smallest average diffs of any cluster)
Top Keywords: ledger, memory, test, remove, validation, coverage, findings, fixes
Characteristics: Repo-memory/ledger primitives, targeted test-coverage additions, and regression fixes. Includes a few open-ended "Investigate X" prompts (e.g. Investigate campaign-query memory estimation gap #64583) that closed without merging.
Avg Comments / Reviews / Files Changed: 5.7 / 3.3 / 164.5 (far above any other cluster)
Top Keywords: version, v0, pin, pins, image, aw, gh, gh aw
Characteristics: Automated/semi-automated version-pin bumps across many generated files. High file count but low review friction since changes are mechanical and low-risk.
Success Rate: 89.4% (42 merged / 5 closed / 0 open) — highest of any cluster
Avg Comments / Reviews / Files Changed: 13.5 / 6.3 / 47.1 (highest engagement of any cluster)
Top Keywords: (top TF-IDF terms for this cluster are dominated by PR Sous Chef's run-footer cost/token accounting block — "AIC", "sub" — rather than task content; inspecting raw prompts shows small, well-scoped bug/safety fixes refreshed via the bot's "Branch refresh requested by PR Sous Chef" comments)
Characteristics: Small, targeted fixes iterated through PR Sous Chef bot comments. Highest comment/review volume and the highest merge rate — iteration converges to a mergeable state rather than stalling, consistent with prior reports.
Scope work queue stress runs and shorten PR profiles
C7: Work Queue / Worker System
Merged
...
(22 more C7 PRs omitted for length)
C7: Work Queue / Worker System
—
Key Findings
No outlier cluster this cycle — a reversal from the 2026-10-26 report. That report flagged a 63.5%-merge "gh-aw Compiler/CLI Core & Infra Hardening" cluster (52 PRs, 16.2 points below its 79.7% overall rate), driven mostly by silently-abandoned "Investigate/Assess/Document"-style prompts. This run's merge rates are tightly clustered (81.7%–89.4%, all within ~8 points of the 84.0% overall rate). Related firewall/infra-hardening PRs are now folded into the much larger C1 cluster, which sits at 82.0% — near the floor of this run's range but not an outlier by the sizing/gap thresholds.
High engagement still correlates with higher merge rates, not lower. C6 (PR Sous Chef Bot-Driven Fixes) has the highest comment/review volume (13.5/6.3 avg) and the highest merge rate (89.4%) — the same pattern seen in prior reports: bot-driven iteration converges rather than stalls.
Bulk mechanical changes remain low-risk despite size. C5 (Dependency & Image Version Pinning) touches ~165 files per PR on average — 3-12x any other cluster — yet merges at 86.7%, the second-highest rate, confirming diff size isn't itself a risk signal when the change is mechanical and narrowly scoped.
The largest cluster converges reliably. C1 (Agent/Workflow Orchestration & GitHub Integration) is more than double the size of the next-largest cluster (211 vs. 93 PRs) yet still merges at 82.0%, in line with the rest of the distribution — broad topic diversity isn't suppressing success rate.
Virtually everything in the window has resolved. Only 1 of 605 PRs remains open (Support Claude on Vertex AI with GCP WIF #72273, Model Routing), vs. 37 open in the 2026-10-26 report's 592-PR window — expected given the window has rotated ~2 days forward and previously-open PRs had more time to resolve; not a like-for-like improvement signal (see Limitations).
Lowest-Merge Cluster Root Cause
Not applicable this run. The lowest-performing cluster (C4: Ledger/Memory & Test Coverage Fixes, 81.7% merged) is only 2.3 points below the 84.0% overall rate — well short of the 10-point gap threshold that triggers a root-cause drill-down, regardless of its size (71 PRs, above the 15-PR size floor). No cluster in this run meets both outlier criteria (size ≥ 15 and gap ≥ 10 points), so Phase 4.5's blocker-classification step was not run. This methodology will automatically re-engage the next time a qualifying outlier appears.
Recommendations
No remediation needed this cycle. Every cluster is converging at a healthy rate (82–89%); current prompt-scoping and bot-iteration practices (PR Sous Chef refresh comments) appear to be working across the board.
Keep leaning on PR Sous Chef-style iterative refinement. C6's pattern (highest engagement + highest merge rate) is the strongest repo-wide signal for "what makes a task recoverable" — consider extending similar automated follow-up nudges to clusters with lower review/comment volume (e.g. C4, C1) if their merge rates drift down in future runs.
Watch C1 even though it isn't flagged. It's both the largest cluster (35% of all tasks) and sits at the bottom of this cycle's merge-rate range (82.0%) — the same thematic territory (firewall/infra/compiler-adjacent glue work) that produced last cycle's outlier. Re-running this analysis regularly will catch it early if it drifts toward outlier status again.
Avoid open-ended "Investigate/Assess/Document" prompts. C4 still contains a few closed PRs from this pattern (e.g. Investigate campaign-query memory estimation gap #64583, "Investigate campaign-query memory estimation gap"); converting these into concrete, scoped implementation asks remains good practice even though the cluster isn't an outlier this time.
Limitations
No workflow-run turn/duration/cost metrics are included this run. gh-aw's own workflow-run logs (via the agenticworkflows logs tool) track runs of this repository's own agentic workflows under the copilot engine, which do not correspond 1:1 to the Copilot coding-agent PRs being clustered here — this was established as a dead end in the 2026-10-26 run and was not re-attempted this cycle.
Clustering silhouette score (0.054) is low in absolute terms, reflecting that these prompts share substantial vocabulary across topics (all are gh-aw engineering tasks) rather than being cleanly separable domains — cluster boundaries should be read as thematic groupings, not hard categories.
The 79.7% → 84.0% overall merge-rate change vs. the 2026-10-26 report is not a verified trend: the 30-day window rotated forward by ~2 days (dropping the oldest ~13 PRs and adding newly-created ones), and PRs that were still open in the prior snapshot had two more days to resolve by this run. A like-for-like comparison would require re-scoring the exact same PR set at both points in time.
Full data table capped at 15 PRs per cluster for readability; the complete 605-PR list with cluster assignments is cached at /tmp/gh-aw/cache-memory/prompt-clustering/full-pr-table-2026-10-28.json for future runs.
Generated by Prompt Clustering Analysis (Run: §40666371572)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Summary
Analysis Period: Last 30 days (2026-09-28 to 2026-10-27)
Total Tasks Analyzed: 605
Clusters Identified: 7
Overall Success Rate: 84.0% (508 merged / 96 closed / 1 open)
Full Analysis Report
General Insights
Methodology note: clustering used TF-IDF (1-2 grams, 300 features, min document frequency 2) + K-means over cleaned PR title/body text, with k selected by silhouette score across k=3–8 (best: k=7, silhouette 0.054 — a fairly flat, mixed-topic score consistent with this repo's prompts sharing a lot of common vocabulary across task types). Full PR data (comments, reviews, commits, file-change counts) was available in cache for all 605 PRs in the window, so per-cluster engagement stats below reflect complete coverage, not a sample. No workflow-run turn/duration/cost data is included — see Limitations.
Cluster Analysis
Cluster 1: Agent/Workflow Orchestration & GitHub Integration
Cluster 2: Model Routing (Copilot/Claude/Pi)
Cluster 3: Safe-Output Failure Handling & Diagnostics
Cluster 4: Ledger/Memory & Test Coverage Fixes
Cluster 5: Dependency & Image Version Pinning
Cluster 6: PR Sous Chef Bot-Driven Fixes
Cluster 7: Work Queue / Worker System
Success Rate by Cluster
Full Data Table
All 605 analyzed PRs (click to expand)
Showing up to 15 representative PRs per cluster (605 total analyzed; full list cached at
/tmp/gh-aw/cache-memory/prompt-clustering/).runtimes.nodeaction override in threat-detection and evals...add-labels.maxabove 10 for per-call label limitgh aw envKey Findings
Lowest-Merge Cluster Root Cause
Not applicable this run. The lowest-performing cluster (C4: Ledger/Memory & Test Coverage Fixes, 81.7% merged) is only 2.3 points below the 84.0% overall rate — well short of the 10-point gap threshold that triggers a root-cause drill-down, regardless of its size (71 PRs, above the 15-PR size floor). No cluster in this run meets both outlier criteria (size ≥ 15 and gap ≥ 10 points), so Phase 4.5's blocker-classification step was not run. This methodology will automatically re-engage the next time a qualifying outlier appears.
Recommendations
Limitations
gh-aw's own workflow-run logs (via theagenticworkflows logstool) track runs of this repository's own agentic workflows under thecopilotengine, which do not correspond 1:1 to the Copilot coding-agent PRs being clustered here — this was established as a dead end in the 2026-10-26 run and was not re-attempted this cycle./tmp/gh-aw/cache-memory/prompt-clustering/full-pr-table-2026-10-28.jsonfor future runs.Generated by Prompt Clustering Analysis (Run: §40666371572)
References:
All reactions