Skip to content

docs(Automation): Improve issue triage agent accuracy and add lib descriptions#9035

Merged
rllyy97 merged 1 commit intomainfrom
riley/improve-issue-triage-agent
Apr 8, 2026
Merged

docs(Automation): Improve issue triage agent accuracy and add lib descriptions#9035
rllyy97 merged 1 commit intomainfrom
riley/improve-issue-triage-agent

Conversation

@rllyy97
Copy link
Copy Markdown
Contributor

@rllyy97 rllyy97 commented Apr 8, 2026

Commit Type

  • feature - New functionality
  • fix - Bug fix
  • refactor - Code restructuring without behavior change
  • perf - Performance improvement
  • docs - Documentation update
  • test - Test-related changes
  • chore - Maintenance/tooling

Risk Level

  • Low - Minor changes, limited scope
  • Medium - Moderate changes, some user impact
  • High - Major changes, significant user/system impact

What & Why

Improves the GitHub issue triage agent's accuracy by adding critical thinking rules and detailed library description files.

The triage agent had a pattern of parroting reporter claims as its own analysis — if an issue author said "this is a caching bug" or "this is a regression", the agent would accept that as truth and repeat it back without independent verification. This led to incorrect root cause analysis and mislabeled issues.

Changes:

  • Critical Thinking Rules section added to issue-triage-agent.md — prevents the agent from blindly trusting reporter diagnoses, regression claims, and component attribution
  • Library description files (.github/lib-descriptions/) — gives the agent detailed context about each library's responsibilities, boundaries, and common misattribution patterns so it can independently map symptoms to the correct component
  • Feedback collector update — tracks root cause accuracy and parroting patterns in issue-triage-feedback.md

Impact of Change

  • Users: No user-facing changes — these files only affect the GitHub Actions triage agent
  • Developers: Better triage comments on new issues with independently verified analysis, correct component labels, and explicit uncertainty signaling
  • System: No runtime impact — markdown files only

Test Plan

  • Unit tests added/updated
  • E2E tests added/updated
  • Manual testing completed
  • Tested in: Reviewed all 9 lib descriptions against actual source code to verify accuracy of subsystem inventories, Redux slice lists, service interface counts, and component categories

Contributors

Screenshots/Videos

N/A — documentation only

- Add critical thinking rules to prevent parroting reporter claims
- Require independent verification of regression and root cause assertions
- Add confidence signaling language rules (reporter claims vs code search confirms)
- Add Step 1.5 to read library descriptions before code search
- Update regression label rule to require codebase evidence
- Add 3 new rules (11-13) for independent analysis
- Create .github/lib-descriptions/ with detailed descriptions for all 9 libs
- Update triage feedback collector with root cause accuracy tracking
- Update architecture table with missing entries (data-mapper v1, MCP, agent, copilot)
@rllyy97 rllyy97 added the risk:low Low risk change with minimal impact label Apr 8, 2026
Copilot AI review requested due to automatic review settings April 8, 2026 20:43
@rllyy97 rllyy97 enabled auto-merge (squash) April 8, 2026 20:44
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 8, 2026

🤖 AI PR Validation Report

PR Review Results

Thank you for your submission! Here's detailed feedback on your PR title and body compliance:

PR Title

  • Current: docs(Automation): Improve issue triage agent accuracy and add lib descriptions
  • Issue: None — title is clear, scoped, and describes the change (documentation for automation/triage + added library descriptions).
  • Recommendation: Keep as-is. Optionally you can include a short parenthetical with the number of files added for faster glance (e.g., ... and add lib descriptions (12 files)), but this is optional.

Commit Type

  • Properly selected (docs).
  • Note: Only one commit type selected which is correct for a documentation-only change.

Risk Level

  • The PR body selects Low and the PR is labeled risk:low — these match.

What & Why

  • Current: "Improves the GitHub issue triage agent's accuracy by adding critical thinking rules and detailed library description files..." (present and clear)
  • Issue: None. The description explains the problem (parroting reporter claims) and the concrete changes made.
  • Recommendation: Good. If you want to be extra-helpful to reviewers, add a one-line list of the highest-impact files changed (e.g., .github/workflows/issue-triage-agent.md, .github/lib-descriptions/*) — but this is optional.

Impact of Change

  • The PR clearly states this is documentation-only and indicates no runtime or user-facing changes.
  • Recommendation: None required. Consider calling out that workflow rules changed (so triage agent comments will change) to help reviewers understand triage behavior changes.

Test Plan

  • Assessment: Adequate for documentation-only change. The PR does not add unit/E2E tests (expected for docs), and the author documents that they reviewed all 9 lib descriptions against source code.
  • Recommendation: Consider adding a short note about a dry-run or calibration run (if available) in a follow-up PR or issue — for example: "run triage agent in dry-run on recent issues to verify reduced parroting" — this is optional but useful for operational validation.

⚠️ Contributors

  • Assessment: Blank.
  • Recommendation: Optional but recommended — add contributors (PMs, designers, reviewers) to give credit. If there were genuinely no additional contributors, a short line like "No additional contributors" is fine.

Screenshots/Videos

  • Assessment: N/A — documentation-only. No visual diffs required.

Summary Table

Section Status Recommendation
Title Keep as-is (optionally include file count)
Commit Type None
Risk Level Matches label (risk:low)
What & Why Clear — optionally list key files changed
Impact of Change None; consider calling out triage comment behavior change
Test Plan Adequate for docs; consider dry-run validation
Contributors ⚠️ Add credits or a short note if none
Screenshots/Videos N/A

Final Notes

  • I reviewed the code diff: the PR adds .github/lib-descriptions/* files and updates .github/workflows/issue-triage-agent.md. All changes are documentation and workflow guidance only (770 additions, 9 deletions across 12 files) — no production code or tests were changed. Based on that, the advised risk level is low, which matches the PR label and body selection.
  • Suggested small improvements (optional):
    • Add a one-line summary of the most important changed files in the What & Why section to help reviewers quickly find them.
    • Fill the Contributors section (or add a short note if no additional contributors).
    • If possible, run a short dry-run calibration of the triage agent (or capture an example triage comment before/after) and link to the result in the Test Plan or a follow-up issue.

Please update the Contributors section if appropriate and consider the dry-run suggestion; otherwise this PR is good to go. Thank you for improving the triage agent documentation and adding the library descriptions!


Last updated: Wed, 08 Apr 2026 20:45:48 GMT

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 8, 2026

📊 Coverage Check

No source files changed in this PR.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the GitHub issue triage agent documentation to reduce incorrect “parroting” of reporter diagnoses and to improve component attribution by adding structured library responsibility docs.

Changes:

  • Adds “Critical Thinking Rules” to the triage agent playbook to require independent verification of root-cause and regression claims.
  • Extends the triage feedback template to track root-cause accuracy/parroting patterns.
  • Introduces .github/lib-descriptions/* to document each library’s scope, boundaries, and common misattribution patterns.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
.github/workflows/issue-triage-feedback.md Adds a feedback section to track root-cause accuracy and verification failures.
.github/workflows/issue-triage-agent.md Adds critical thinking/verification rules and points to the new library description inventory.
.github/lib-descriptions/README.md Documents the purpose and maintenance expectations for library description files.
.github/lib-descriptions/designer.md Describes libs/designer responsibilities, major subsystems, and common issue patterns.
.github/lib-descriptions/designer-v2.md Describes libs/designer-v2 responsibilities, key differences vs v1, and common issue patterns.
.github/lib-descriptions/designer-ui.md Describes libs/designer-ui scope and what should/shouldn’t be attributed to it.
.github/lib-descriptions/logic-apps-shared.md Describes libs/logic-apps-shared service interfaces/utilities and common issue patterns.
.github/lib-descriptions/data-mapper.md Describes legacy data mapper (v1) scope and when to use it.
.github/lib-descriptions/data-mapper-v2.md Describes current data mapper (v2) scope, architecture, and issue patterns.
.github/lib-descriptions/chatbot.md Describes chatbot UI integration scope vs service/prompt logic boundaries.
.github/lib-descriptions/vscode-extension.md Describes VS Code extension shared library scope and common issue patterns.
.github/lib-descriptions/a2a-core.md Describes A2A client SDK scope and common issue patterns.

@rllyy97 rllyy97 merged commit f6e674b into main Apr 8, 2026
20 checks passed
@rllyy97 rllyy97 deleted the riley/improve-issue-triage-agent branch April 8, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-validated risk:low Low risk change with minimal impact

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants