[prompt-clustering] 🔬 Copilot Agent Prompt Clustering Analysis - 2025-11-18

[github-actions[bot]]

🔬 Copilot Agent Prompt Clustering Analysis

Analysis Date: 2025-11-18
Analysis Period: Last 30 days

NLP-based clustering analysis of 1,020 copilot agent task prompts to identify patterns, trends, and optimization opportunities.

Summary

• Total PRs Found: 1,021
• PRs with Valid Prompts: 1,020
• Clusters Identified: 7
• Overall Success Rate: 76.7% (782 merged, 7 open, 231 closed)

The analysis used TF-IDF vectorization and K-means clustering to group similar tasks together, revealing distinct patterns in how copilot agents are used.

Full Analysis Report

Cluster Analysis

Cluster 4: Documentation Updates

• Size: 288 tasks (28.2% of total)
• Success Rate: 73.6% (212 merged, 3 open, 73 closed)
• Top Keywords: update, agent, mcp, github, output, json, md

Example PRs:

• #2097: Add minimal path format syntax reference to imports documentation
• #2101: [WIP] Migrate JavaScript memory server to Wasm component
• #2102: Add workflow status badges documentation page

Sample Prompt: Update the frontmatter imports documentation under docs with all the supported URL and path formats

Cluster 3: Bug Fixes

• Size: 183 tasks (17.9% of total)
• Success Rate: 74.9% (137 merged, 0 open, 46 closed)
• Top Keywords: workflows, comments, gh, aw, gh aw, github, issue

Example PRs:

• #2189: [WIP] Recompile workflows for updated processes
• #2209: [WIP] Comment on issue [smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1) #2157 regarding recurrence failure
• #2235: Fix: Raise error when agentic workflow hits max-turns limit

Sample Prompt: Recompile workflows.

Cluster 6: Documentation Updates

• Size: 148 tasks (14.5% of total)
• Success Rate: 74.3% (110 merged, 2 open, 36 closed)
• Top Keywords: agentic, workflow, workflows, create, agent, file, update

Example PRs:

• #2100: Spread scheduled agentic workflows across 24 hours and add 6-hour schedules to smoke tests
• #2103: Add smoke-outpost workflow for investigating failed smoke test runs
• #2109: Add semantic function refactoring workflow for Go code analysis

Sample Prompt: Review the scheduled agentic workflows and spread them the entire day. Schedule the smoke workflows every 6h

Cluster 7: CI/CD & Workflows

• Size: 131 tasks (12.8% of total)
• Success Rate: 84.7% (111 merged, 0 open, 20 closed)
• Top Keywords: pkg, pkg workflow, workflow, functions, code, function, comments

Example PRs:

• #2107: [WIP] Update variable name to 'log' and modify instructions
• #2171: Refactor duplicate MCP code patterns for improved maintainability
• #2249: Refactor: Extract duplicate GitHub MCP remote config rendering into shared helper

Sample Prompt: x Rename logger variables to log in all 5 Go files x pkg cli trial_command.go x pkg workflow claude_engine.go

Cluster 2: Documentation Updates

• Size: 97 tasks (9.5% of total)
• Success Rate: 82.5% (80 merged, 1 open, 16 closed)
• Top Keywords: version, changes, cli, update, package, new, github

Example PRs:

• #2104: Add edit tool to commit-changes-analyzer workflow
• #2120: Update commit changes analyzer: max-turns 100, timeout 30 minutes
• #2156: Integrate gh-aw-firewall for Copilot engine network access control

Sample Prompt: Update commit changes analyzer to use the edit tool and allow bash tools.

Cluster 1: Documentation Updates

• Size: 94 tasks (9.2% of total)
• Success Rate: 75.5% (71 merged, 1 open, 22 closed)
• Top Keywords: add, command, logs, compile, update, workflows, workflow

Example PRs:

• #2099: Add directory creation for copilot engine --add-dir paths
• #2125: Update dev.md to respond to /dev command in discussions with 3-word poems
• #2132: Add reaction to dev.md and use Claude engine

Sample Prompt: The copilot engine should review the add dir folders used in the args list and make sure those folders exist before running the cli. Emit mkdir p ... to prepare those directories.

Cluster 5: Documentation Updates

• Size: 79 tasks (7.7% of total)
• Success Rate: 77.2% (61 merged, 0 open, 18 closed)
• Top Keywords: error, fix, errors, tests, permissions, validation, add

Example PRs:

• #2146: [WIP] Fix missing OPENAI_API_KEY in GenAIScript smoke test
• #2153: Fix TestStopTimeResolutionIntegration to check for correct environment variable format
• #2212: Add error/warning summary to logs command with pattern IDs

Sample Prompt: Fix GenAIScript OPENAI_API_KEY Missing Issue Problem The GenAIScript smoke test workflow was failing at run 18722204018 with error OPENAI_API_KEY missing because the workflow uses the openai gpt 4.1 m...

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Merged	Open	Closed	Top Keywords
4	Documentation Updates	288	73.6%	212	3	73	update, agent, mcp
3	Bug Fixes	183	74.9%	137	0	46	workflows, comments, gh
6	Documentation Updates	148	74.3%	110	2	36	agentic, workflow, workflows
7	CI/CD & Workflows	131	84.7%	111	0	20	pkg, pkg workflow, workflow
2	Documentation Updates	97	82.5%	80	1	16	version, changes, cli
1	Documentation Updates	94	75.5%	71	1	22	add, command, logs
5	Documentation Updates	79	77.2%	61	0	18	error, fix, errors

Sample Data Table

PR #	Title	Cluster	Theme	Outcome	Keywords
#2097	Add minimal path format syntax reference to imports document...	4	Documentation Updates	Merged	update, agent
#2099	Add directory creation for copilot engine --add-dir paths	1	Documentation Updates	Merged	add, command
#2100	Spread scheduled agentic workflows across 24 hours and add 6...	6	Documentation Updates	Merged	agentic, workflow
#2102	Add workflow status badges documentation page	4	Documentation Updates	Merged	update, agent
#2103	Add smoke-outpost workflow for investigating failed smoke te...	6	Documentation Updates	Merged	agentic, workflow
#2104	Add edit tool to commit-changes-analyzer workflow	2	Documentation Updates	Merged	version, changes
#2107	[WIP] Update variable name to 'log' and modify instructions	7	CI/CD & Workflows	Merged	pkg, pkg workflow
#2108	Remove js-yaml dependency from badge generator, use regex-ba...	4	Documentation Updates	Merged	update, agent
#2109	Add semantic function refactoring workflow for Go code analy...	6	Documentation Updates	Merged	agentic, workflow
#2112	Restore AI instructions in README.md wrapped in XML comments	4	Documentation Updates	Merged	update, agent

Showing 10 of 1020 total PRs (see full report for complete table)

Key Findings

1. Most Common Task Type: Documentation Updates represents 28.2% of all tasks (288 tasks)
2. Highest Success Rate: CI/CD & Workflows has a 84.7% success rate with 111/131 PRs merged
3. Consistent Performance: Most clusters have success rates between 73-85%, indicating generally reliable agent performance

Recommendations

Based on clustering analysis:

1. Best Practices: CI/CD & Workflows tasks perform well (84.7% success). Keywords: pkg, pkg workflow, workflow, functions, code. Consider using similar prompt patterns for other task types - specifically, prompts that are concrete and reference specific file paths tend to perform better.

2. Prompt Clarity: Analyze successful prompts in high-performing clusters to identify effective prompt patterns and apply them to other task types. Specific, actionable prompts with clear scope tend to have higher success rates.

3. Documentation Tasks: These represent the majority of work but have slightly lower success rates (73-77%). Consider breaking down complex documentation tasks into smaller, more focused subtasks.

---

Methodology: This analysis used TF-IDF vectorization and K-means clustering (k=7) on 1,020 copilot agent task prompts extracted from PR bodies over the last 30 days. Prompts were preprocessed to extract the original task description, cleaned of markup and URLs, and vectorized using unigram and bigram features.

Generated by Copilot Prompt Clustering Analysis

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.