[prompt-clustering] 🔬 Copilot Agent Prompt Clustering Analysis - November 18, 2025

[github-actions[bot]]

🔬 Copilot Agent Prompt Clustering Analysis - 2025-11-18

This analysis uses Natural Language Processing (NLP) and K-means clustering to identify patterns in Copilot agent task prompts from the last 30 days. By analyzing 984 PRs, we discovered 7 distinct task clusters with varying success rates and characteristics.

Summary

Analysis Period: Last 30 days
Total Tasks Analyzed: 984
Clusters Identified: 7
Overall Success Rate: 77.2%

Key Insights

• Most Common Task Type: Cluster 5 - feature tasks (282 tasks, 76% success)
• Highest Success Rate: Cluster 2 - bug-fix tasks with 83% success (147 tasks)
• Task Type Distribution: bug-fix (458 tasks), feature (439 tasks), update (87 tasks)

The analysis reveals that bug-fix tasks, particularly those focused on code, workflow, and package modifications (Cluster 2), achieve the highest success rate at 83%. Feature tasks dominate in volume with 439 total tasks across multiple clusters, while update tasks show strong performance at 79% success rate.

Full Analysis Report

Cluster Analysis

Cluster 5: Feature Tasks

• Size: 282 tasks (28.7% of total)
• Success Rate: 75.5% (213 merged, 69 closed, 0 open)
• Top Keywords: add, json, agent, github, copilot
• Category: feature
• Example PRs:
  • #2099 - Add directory creation for copilot engine --add-dir paths
  • #2101 - [WIP] Migrate JavaScript memory server to Wasm component
  • #2102 - Add workflow status badges documentation page

Cluster 7: Feature Tasks

• Size: 157 tasks (16.0% of total)
• Success Rate: 76.4% (120 merged, 37 closed, 0 open)
• Top Keywords: workflow, agentic, workflows, file, use
• Category: feature
• Example PRs:
  • #2100 - Spread scheduled agentic workflows across 24 hours and add 6-hour schedules to smoke tests
  • #2103 - Add smoke-outpost workflow for investigating failed smoke test runs
  • #2109 - Add semantic function refactoring workflow for Go code analysis

Cluster 2: Bug-Fix Tasks

• Size: 147 tasks (14.9% of total)
• Success Rate: 83.0% (122 merged, 25 closed, 0 open)
• Top Keywords: pkg, workflow, code, files, functions
• Category: bug-fix
• Example PRs:
  • #2107 - [WIP] Update variable name to 'log' and modify instructions
  • #2171 - Refactor duplicate MCP code patterns for improved maintainability
  • #2242 - Remove .serena files and refactor isValidReaction to reactions.go

Cluster 6: Bug-Fix Tasks

• Size: 130 tasks (13.2% of total)
• Success Rate: 76.2% (99 merged, 31 closed, 0 open)
• Top Keywords: workflows, workflow, github, githubnext, comments
• Category: bug-fix
• Example PRs:
  • #2193 - Fix heredoc delimiter collision causing workflow compilation failures
  • #2209 - [WIP] Comment on issue [smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1) #2157 regarding recurrence failure
  • #2236 - Confirm gpt-4.1 invalid model issue already resolved - no changes needed

Cluster 4: Bug-Fix Tasks

• Size: 111 tasks (11.3% of total)
• Success Rate: 75.7% (84 merged, 27 closed, 0 open)
• Top Keywords: version, cli, comments, issue, update
• Category: bug-fix
• Example PRs:
  • #2128 - [WIP] Fix missing agent_output.json file in Smoke OpenCode
  • #2156 - Integrate gh-aw-firewall for Copilot engine network access control
  • #2208 - Optimize CLI version checker workflow based on performance analysis

Cluster 3: Update Tasks

• Size: 87 tasks (8.8% of total)
• Success Rate: 79.3% (69 merged, 18 closed, 0 open)
• Top Keywords: update, use, script, workflow, file
• Category: update
• Example PRs:
  • #2097 - Add minimal path format syntax reference to imports documentation
  • #2104 - Add edit tool to commit-changes-analyzer workflow
  • #2113 - Move workflow status documentation page to top level

Cluster 1: Bug-Fix Tasks

• Size: 70 tasks (7.1% of total)
• Success Rate: 75.7% (53 merged, 17 closed, 0 open)
• Top Keywords: error, errors, validation, messages, workflow
• Category: bug-fix
• Example PRs:
  • #2173 - Add internal compiler check for secret redaction step ordering
  • #2212 - Add error/warning summary to logs command with pattern IDs
  • #2244 - Add getValidReactions function to compute valid reaction list dynamically

Success Rate by Cluster

Cluster	Category	Tasks	Success Rate	Top Keywords
5	feature	282	76%	add, json, agent
7	feature	157	76%	workflow, agentic, workflows
2	bug-fix	147	83%	pkg, workflow, code
6	bug-fix	130	76%	workflows, workflow, github
4	bug-fix	111	76%	version, cli, comments
3	update	87	79%	update, use, script
1	bug-fix	70	76%	error, errors, validation

Sample Data (First 50 PRs)

PR #	Title	Cluster	Outcome	Keywords
#2097	Add minimal path format syntax reference to imports document...	3	✅ merged	update, frontmatter
#2099	Add directory creation for copilot engine --add-dir paths	5	✅ merged	folders, those
#2100	Spread scheduled agentic workflows across 24 hours and add 6...	7	✅ merged	workflows, review
#2101	[WIP] Migrate JavaScript memory server to Wasm component	5	❌ closed	javascript, migration
#2102	Add workflow status badges documentation page	5	✅ merged	status, github
#2103	Add smoke-outpost workflow for investigating failed smoke te...	7	✅ merged	workflow, agentic
#2104	Add edit tool to commit-changes-analyzer workflow	3	✅ merged	update, commit
#2107	[WIP] Update variable name to 'log' and modify instructions	2	✅ merged	pkg, workflow
#2108	Remove js-yaml dependency from badge generator, use regex-ba...	5	✅ merged	remove, yml
#2109	Add semantic function refactoring workflow for Go code analy...	7	✅ merged	file, functions
#2110	[WIP] Refactor clusters of functions in Go files	7	❌ closed	file, functions
#2111	Refactor permissions parsing from compiler.go to permissions...	5	❌ closed	permissions, parsing
#2112	Restore AI instructions in README.md wrapped in XML comments	5	✅ merged	restore, instructions
#2113	Move workflow status documentation page to top level	3	✅ merged	move, workflow
#2115	Add scheduling best practices guidance for daily workflows	7	✅ merged	workflow, update
#2118	Merge documentation unbloat style guidance and add import to...	5	✅ merged	documentation, unbloat
#2119	Consolidate permission parsing logic and add support for "al...	3	✅ merged	update, permission
#2120	Update commit changes analyzer: max-turns 100, timeout 30 mi...	3	✅ merged	update, commit
#2122	Update scout agentic workflow to use Claude engine	5	✅ merged	engine, update
#2125	Update dev.md to respond to /dev command in discussions with...	5	✅ merged	dev, discussion
#2126	[WIP] Address feedback on adding debug logging in Go files	5	✅ merged	thanks, feedback
#2127	Fix Smoke OpenCode workflow failure and update to version 0....	5	✅ merged	opencode, job
#2128	[WIP] Fix missing agent_output.json file in Smoke OpenCode	4	❌ closed	issue, details
#2129	Rename smoke-outpost workflow to smoke-detector	7	✅ merged	rename, smoke
#2130	Migrate verbose logging to use logger package in run_command...	5	❌ closed	printf, verbose
#2131	Update AGENTS.md with comprehensive DEBUG environment variab...	3	❌ closed	update, agents
#2132	Add reaction to dev.md and use Claude engine	5	✅ merged	add, reaction
#2133	Fix discussion comment threading by adding replyToId paramet...	5	✅ merged	comment, discussion
#2134	Add reporting instructions for HTML details/summary formatti...	7	✅ merged	instructions, file
#2135	Update smoke genaiscript workflow to use openai:gpt-4.1 mode...	7	✅ merged	update, smoke
#2137	Fix discussion comment replies to use threading instead of c...	5	✅ merged	discussion, comment
#2138	Add allowed domains from network config to sanitization envi...	5	❌ closed	domains, javascript
#2141	Update reporting instructions to require overview before det...	3	✅ merged	update, reporting
#2145	Add trigger-maintenance job to release.yml workflow	3	❌ closed	release, update
#2146	[WIP] Fix missing OPENAI_API_KEY in GenAIScript smoke test	7	✅ merged	genaiscript, workflow
#2147	Add dictation prompt generator agentic workflow	7	✅ merged	prompt, agentic
#2151	Update dictation agent workflow to extract vocabulary from d...	5	✅ merged	remove, update
#2154	Add Dictating Agentic Workflows section to authoring documen...	7	✅ merged	agentic, text
#2155	Update dictation workflow to scan user-facing documentation ...	7	✅ merged	docs, update
#2156	Integrate gh-aw-firewall for Copilot engine network access c...	4	❌ closed	copilot, firewall
#2158	Update workflow status page to use markdown bullet points	3	✅ merged	update, script
#2160	Add blog-auditor workflow for GitHub Next Agentic Workflows ...	7	✅ merged	url, agentic
#2163	Remove tracking issue functionality from blog-auditor workfl...	7	✅ merged	issue, tracking
#2165	Update create-discussion category example to use meaningful ...	5	✅ merged	category, discussion
#2166	Update blog auditor to point to correct URL with plural 'pro...	3	✅ merged	update, blog
#2167	Fix OpenCode MCP server integration - Enable safe-outputs an...	5	✅ merged	opencode, mcp
#2170	Update blog auditor to validate code snippet syntax against ...	7	✅ merged	update, blog
#2171	Refactor duplicate MCP code patterns for improved maintainab...	2	✅ merged	mcp, code
#2173	Add internal compiler check for secret redaction step orderi...	1	✅ merged	compiler, step
#2175	Add shared workflow for checking existing PRs before creatin...	7	❌ closed	workflow, agent

Recommendations

Based on clustering analysis:

1. High Success Pattern: Bug-Fix tasks (Cluster 2) show 83% success rate. Keywords: pkg, workflow, code. Consider this as a model for well-defined tasks.

2. Task Diversity: Identified 7 distinct task patterns. This diversity suggests the agent handles a wide range of work types effectively.

3. Feature Task Patterns: Feature tasks split into two main clusters - general additions (Cluster 5) and workflow-specific features (Cluster 7). Both show similar success rates around 76%.

4. Update Tasks: With 79% success rate, update tasks perform well, suggesting that modification requests with clear scope are effective.

---

Analysis Date: 2025-11-18
Dataset: 984 PRs from last 30 days
Method: TF-IDF vectorization + K-means clustering (k=7)
Silhouette Score: 0.075 (indicates moderate cluster separation)

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.