docs(Automation): Improve issue triage agent accuracy and add lib descriptions#9035
Merged
docs(Automation): Improve issue triage agent accuracy and add lib descriptions#9035
Conversation
- Add critical thinking rules to prevent parroting reporter claims - Require independent verification of regression and root cause assertions - Add confidence signaling language rules (reporter claims vs code search confirms) - Add Step 1.5 to read library descriptions before code search - Update regression label rule to require codebase evidence - Add 3 new rules (11-13) for independent analysis - Create .github/lib-descriptions/ with detailed descriptions for all 9 libs - Update triage feedback collector with root cause accuracy tracking - Update architecture table with missing entries (data-mapper v1, MCP, agent, copilot)
takyyon
approved these changes
Apr 8, 2026
🤖 AI PR Validation ReportPR Review ResultsThank you for your submission! Here's detailed feedback on your PR title and body compliance:✅ PR Title
✅ Commit Type
✅ Risk Level
✅ What & Why
✅ Impact of Change
✅ Test Plan
|
| Section | Status | Recommendation |
|---|---|---|
| Title | ✅ | Keep as-is (optionally include file count) |
| Commit Type | ✅ | None |
| Risk Level | ✅ | Matches label (risk:low) |
| What & Why | ✅ | Clear — optionally list key files changed |
| Impact of Change | ✅ | None; consider calling out triage comment behavior change |
| Test Plan | ✅ | Adequate for docs; consider dry-run validation |
| Contributors | Add credits or a short note if none | |
| Screenshots/Videos | ✅ | N/A |
Final Notes
- I reviewed the code diff: the PR adds
.github/lib-descriptions/*files and updates.github/workflows/issue-triage-agent.md. All changes are documentation and workflow guidance only (770 additions, 9 deletions across 12 files) — no production code or tests were changed. Based on that, the advised risk level is low, which matches the PR label and body selection. - Suggested small improvements (optional):
- Add a one-line summary of the most important changed files in the What & Why section to help reviewers quickly find them.
- Fill the Contributors section (or add a short note if no additional contributors).
- If possible, run a short dry-run calibration of the triage agent (or capture an example triage comment before/after) and link to the result in the Test Plan or a follow-up issue.
Please update the Contributors section if appropriate and consider the dry-run suggestion; otherwise this PR is good to go. Thank you for improving the triage agent documentation and adding the library descriptions!
Last updated: Wed, 08 Apr 2026 20:45:48 GMT
📊 Coverage CheckNo source files changed in this PR. |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the GitHub issue triage agent documentation to reduce incorrect “parroting” of reporter diagnoses and to improve component attribution by adding structured library responsibility docs.
Changes:
- Adds “Critical Thinking Rules” to the triage agent playbook to require independent verification of root-cause and regression claims.
- Extends the triage feedback template to track root-cause accuracy/parroting patterns.
- Introduces
.github/lib-descriptions/*to document each library’s scope, boundaries, and common misattribution patterns.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/issue-triage-feedback.md |
Adds a feedback section to track root-cause accuracy and verification failures. |
.github/workflows/issue-triage-agent.md |
Adds critical thinking/verification rules and points to the new library description inventory. |
.github/lib-descriptions/README.md |
Documents the purpose and maintenance expectations for library description files. |
.github/lib-descriptions/designer.md |
Describes libs/designer responsibilities, major subsystems, and common issue patterns. |
.github/lib-descriptions/designer-v2.md |
Describes libs/designer-v2 responsibilities, key differences vs v1, and common issue patterns. |
.github/lib-descriptions/designer-ui.md |
Describes libs/designer-ui scope and what should/shouldn’t be attributed to it. |
.github/lib-descriptions/logic-apps-shared.md |
Describes libs/logic-apps-shared service interfaces/utilities and common issue patterns. |
.github/lib-descriptions/data-mapper.md |
Describes legacy data mapper (v1) scope and when to use it. |
.github/lib-descriptions/data-mapper-v2.md |
Describes current data mapper (v2) scope, architecture, and issue patterns. |
.github/lib-descriptions/chatbot.md |
Describes chatbot UI integration scope vs service/prompt logic boundaries. |
.github/lib-descriptions/vscode-extension.md |
Describes VS Code extension shared library scope and common issue patterns. |
.github/lib-descriptions/a2a-core.md |
Describes A2A client SDK scope and common issue patterns. |
This was referenced Apr 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commit Type
Risk Level
What & Why
Improves the GitHub issue triage agent's accuracy by adding critical thinking rules and detailed library description files.
The triage agent had a pattern of parroting reporter claims as its own analysis — if an issue author said "this is a caching bug" or "this is a regression", the agent would accept that as truth and repeat it back without independent verification. This led to incorrect root cause analysis and mislabeled issues.
Changes:
issue-triage-agent.md— prevents the agent from blindly trusting reporter diagnoses, regression claims, and component attribution.github/lib-descriptions/) — gives the agent detailed context about each library's responsibilities, boundaries, and common misattribution patterns so it can independently map symptoms to the correct componentissue-triage-feedback.mdImpact of Change
Test Plan
Contributors
Screenshots/Videos
N/A — documentation only