refactor(foreman): iterative review loop + CI gate phases by ewega · Pull Request #78 · DevExpGbb/gh-devlake

ewega · 2026-03-02T12:05:40Z

Summary

Restructures the GitHub Foreman workflow from a single-pass review model to an iterative, automated pipeline:

Phase changes

Old	New
Phase 3: Review (single pass)	Phase 3: Code Review Loop (iterative)
Phase 4: Human Gate	Phase 4: CI Gate (replaces QA Enforcer)
Phase 5: Advance	Phase 5: Docs & Consistency
—	Phase 6: Human Gate (renumbered)
—	Phase 7: Advance (renumbered)

Key changes

Phase 3 is now a full loop: mark PRs ready → wait for Copilot Code Review Agent → collect & judge comments → push @copilot\ fixes → loop until no actionable comments remain
Phase 4 replaces the QA Enforcer subagent with CI check polling — CI already runs \go build/vet/test\ on all 3 OSes via .github/workflows/ci.yml; red checks loop back to Phase 3
Phase 5 consolidates Wave Reviewer + Docs Writer into a single post-review pass
Phase 4b Fix Loop removed — absorbed into Phase 3's native loop
\github/request_copilot_review\ kept as fallback if ruleset auto-assign doesn't trigger
All \mcp_github_\ tool references updated to \github/\ style
\qa-enforcer\ removed from agents; CI row added to Your Team table

Validation

No Go files changed — CI passes by default (no build/test/vet to run)
Agent file manually reviewed for orphaned phase references, stale tool names, and numbering consistency

Copilot

Pull request overview

Refactors the GitHub Foreman agent workflow definition to move from a single-pass review process to an iterative review loop, add a CI-based gate, and renumber subsequent phases.

Changes:

Replaces Phase 3 with an iterative “Code Review Loop” that waits for Copilot reviews, judges comments, and requests targeted @copilot fixes until stabilized.
Replaces the previous QA Enforcer/human gate step with a CI check polling gate (Phase 4), looping back to Phase 3 on failures.
Consolidates post-review activities into a single “Docs & Consistency” phase and renumbers Human Gate/Advance phases.

Copilot · 2026-03-02T12:09:56Z

.github/agents/github-foreman.agent.md

-  - qa-enforcer
  - go-developer
  - prettify
  - Explore


The agents: list includes Explore, but there is no corresponding .github/agents/Explore.agent.md (or any other agent definition) in the repo. This will fail when Foreman tries to invoke that subagent; either add the missing agent file or remove/rename this entry to a valid agent name.

Suggested change

- Explore

Copilot · 2026-03-02T12:09:56Z

.github/agents/github-foreman.agent.md

+### Phase 3: Code Review Loop

-When PRs are created by the coding agents:
+This phase is iterative. It runs automatically and loops until Foreman judges there are no more actionable comments across all PRs.

-1. **Automated code review** — Use `mcp_github_request_copilot_review` on each PR. The Code Review Agent automatically applies guidance from `.github/copilot-instructions.md`.
-2. **Cross-PR consistency** (multi-PR waves only) — Run the **Wave Reviewer** subagent to check consistency across all PRs in the wave.
-3. **Quality check** — Run the **QA Enforcer** subagent to verify builds, tests, and coverage on each branch.
-4. **Documentation check** — Run the **Docs Writer** subagent to verify README and AGENTS.md are updated if the wave adds/changes commands.
-5. **Collect Code Review Agent comments** — Use `mcp_github_pull_request_read` with `method: "get_review_comments"` on each PR to pull in all review comments left by the Code Review Agent (and any other reviewers). Summarize findings by severity:
-   - **Blocking** — security issues, logic errors, broken tests
-   - **Suggestions** — style improvements, naming, refactoring opportunities
-   - **Informational** — notes, questions, minor observations
-6. **Synthesize results** — Compile findings from all checks into a summary for the human. Include the Code Review Agent's comments grouped by PR and severity.
+1. **Mark PRs ready for review** — Use `github/pull_request_write` to convert each draft PR to ready-for-review. The repo ruleset automatically triggers the Copilot Code Review Agent. If the ruleset fails to assign the agent within the polling window, use `github/request_copilot_review` as a fallback.


This PR claims all mcp_github_* tool references were updated to github/*, but the file still contains mcp_github_* usages (e.g., Phases 1/1b/2/2b). That’s inconsistent with the tools: list (which only declares github/*) and will cause the Foreman instructions to reference non-existent tools. Update the remaining mcp_github_* references to the corresponding github/* tool names for consistency.

…ub_ refs in phases 1/1b/2/2b

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

.github/agents/github-foreman.agent.md:183

The instructions refer to a todos tool, but the frontmatter declares the tool as todo. This mismatch will cause the agent to look for a non-existent tool name; either update the text to todo or rename the tool entry to todos consistently throughout the file.

This issue also appears on line 196 of the same file.

1. Update your wave tracking (`todos`) — mark all issues as merged
2. Identify the next unblocked wave
3. Return to Phase 1 for the next wave

.github/agents/github-foreman.agent.md:199

The instructions refer to the todos tool here, but the frontmatter lists todo. Align the tool name (either change these references to todo or change the declared tool to todos) so the workflow is executable.

3. **Always complete the full review pipeline** before presenting for human review — don't skip the code review loop (Phase 3), CI gate (Phase 4), or docs check (Phase 5).
4. **Use `custom_instructions`** to give coding agents issue-specific context beyond the issue body — reference relevant skills, architecture decisions, and file locations.
5. **Track everything** with the `todos` tool — every issue should have a trackable status (planned → dispatched → PR created → reviewed → merged).
6. **Keep `.github/copilot-instructions.md` and `AGENTS.md` in sync** — when your wave changes CLI structure, ensure the Docs Writer updates both.

Copilot · 2026-03-02T12:21:02Z

.github/agents/github-foreman.agent.md

 agents:
  - wave-reviewer
  - docs-writer
-  - qa-enforcer
  - go-developer
  - prettify


PR description says "qa-enforcer removed from agents", but the repo still contains .github/agents/qa-enforcer.agent.md and .github/agents/references/foreman-workflows.md still models QA Enforcer as part of the workflow. Either delete/retire the qa-enforcer agent + update the reference doc, or adjust the PR description to clarify it's only removed from the Foreman’s agents: list (not removed from the repo).

Copilot · 2026-03-02T12:21:02Z

.github/agents/github-foreman.agent.md

+2. **Wait for review completion** — Sleep **5 minutes** (`Start-Sleep -Seconds 300`), then poll in **2-minute** cycles (`Start-Sleep -Seconds 120`). Use `github/pull_request_read` with `method: "get_reviews"` on each PR until the Code Review Agent's review shows `status: completed` on all PRs.
+
+3. **Collect and judge comments** — Use `github/pull_request_read` with `method: "get_review_comments"` on each PR. Internally bucket all comments by severity — this summary is for Foreman's judgment only, not presented to the human yet:
+   - **Blocking** — security issues, logic errors, incorrect behaviour


Spelling is inconsistent with the rest of the repo docs (which use US English). Consider changing "behaviour" to "behavior" for consistency.

Suggested change

- **Blocking** — security issues, logic errors, incorrect behaviour

- **Blocking** — security issues, logic errors, incorrect behavior

ewega · 2026-03-02T13:08:10Z

@copilot open a new pull request to apply changes based on the comments in this thread

Copilot · 2026-03-02T13:08:18Z

@ewega I've opened a new pull request, #80, to work on those changes. Once the pull request is ready, I'll request review from you.

… diagrams (#80) * Initial plan * fix(foreman): behavior spelling, remove QA Enforcer from reference diagrams Co-authored-by: ewega <26189114+ewega@users.noreply.github.com> * fix(foreman): update remaining MCP: edge labels to github/* style Co-authored-by: ewega <26189114+ewega@users.noreply.github.com> * fix(foreman): accurate node label + github/pull_request_read CI gate step Co-authored-by: ewega <26189114+ewega@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ewega <26189114+ewega@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

.github/agents/github-foreman.agent.md

Copilot · 2026-03-02T17:25:59Z

.github/agents/references/foreman-workflows.md

    %% Foreman triggers code review
-    F -->|"MCP: request_copilot_review"| CRA
+    F -->|"github/request_copilot_review"| CRA
    CRA -->|"review comments"| PR1
    CRA -->|"review comments"| PR2
-    CRA -.->|"reads review guidance"| CI
+    CRA -.->|"reads review guidance"| INST


In the interaction diagram, Foreman is shown always triggering code review via github/request_copilot_review, but the updated workflow text describes the primary trigger as marking a PR ready-for-review (ruleset auto-assigns the Code Review Agent) with request_copilot_review only as fallback. Please update this arrow/label to reflect the ruleset-driven trigger to keep the diagram consistent with the new Phase 3 loop.

See below for a potential fix:

%% Code review is primarily triggered by PR ready-for-review ruleset PR1 -->|"mark ready for review<br/>(ruleset assigns Code Review Agent)"| CRA PR2 -->|"mark ready for review<br/>(ruleset assigns Code Review Agent)"| CRA F -->|"github/request_copilot_review<br/>(fallback trigger)"| CRA

@copilot open a new pull request to apply changes based on this feedback

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2026-03-02T17:57:36Z

@ewega I've opened a new pull request, #81, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

…#81) * Initial plan * Update interaction diagram to reflect ruleset-driven code review trigger Co-authored-by: ewega <26189114+ewega@users.noreply.github.com> * Update .github/agents/references/foreman-workflows.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ewega <26189114+ewega@users.noreply.github.com> Co-authored-by: Eldrick Wega <ewega@github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

.github/agents/github-foreman.agent.md

.github/agents/references/foreman-workflows.md

refactor(foreman): iterative review loop, CI gate, phase renumber

6554c32

Copilot AI review requested due to automatic review settings March 2, 2026 12:05

Copilot started reviewing on behalf of ewega March 2, 2026 12:06 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

fix(foreman): remove Explore agent (no file backing), update mcp_gith…

d97c3fb

…ub_ refs in phases 1/1b/2/2b

ewega requested a review from Copilot March 2, 2026 12:14

Copilot started reviewing on behalf of ewega March 2, 2026 12:15 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

Copilot AI mentioned this pull request Mar 2, 2026

fix(foreman): US English spelling + retire QA Enforcer from reference diagrams #80

Merged

5 tasks

ewega requested a review from Copilot March 2, 2026 17:21

Copilot started reviewing on behalf of ewega March 2, 2026 17:22 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

Update .github/agents/github-foreman.agent.md

9e3bd7b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 2, 2026 17:56

Copilot started reviewing on behalf of ewega March 2, 2026 17:56 View session

Update .github/agents/github-foreman.agent.md

9c9175b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Mar 2, 2026

Fix interaction diagram to reflect ruleset-driven code review trigger #81

Merged

Copilot AI reviewed Mar 2, 2026

View reviewed changes

ewega requested a review from Copilot March 2, 2026 18:12

Copilot started reviewing on behalf of ewega March 2, 2026 18:13 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

.github/agents/github-foreman.agent.md Show resolved Hide resolved

.github/agents/references/foreman-workflows.md Show resolved Hide resolved

ewega merged commit fbb4d2b into main Mar 2, 2026
7 checks passed

ewega deleted the feat/foreman-review-phases branch March 2, 2026 18:18

	- Blocking — security issues, logic errors, incorrect behaviour
	- Blocking — security issues, logic errors, incorrect behavior

Conversation

ewega commented Mar 2, 2026

Summary

Phase changes

Key changes

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

ewega commented Mar 2, 2026

Uh oh!

Copilot AI commented Mar 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

ewega Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Mar 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants