SWE agent uses `gh` CLI instead of a GitHub tool: PR-centric prompt + missing tool

[k5s-bot]

Summary

Follow-up investigation into an observed behavior: when asked to create a GitHub issue (filing #147), the SWE agent cloned the repo and shelled out to the gh CLI instead of using a structured "GitHub tool." The question raised was whether that was inefficient and whether our prompts need refining.

Finding: the agent behaved correctly given its context — there is no GitHub tool for it to use, and its prompt steers it exclusively toward git + gh. The inefficiency is real, but its root cause is a tooling gap plus a PR-centric prompt, not a bad tool choice by the model.

This ticket records the investigation and proposes prompt + routing improvements. It is deliberately scoped to not duplicate #86 (Create a GitHub tool) and #87 (Allow agents to call tools), which already track the tooling side.

Investigation

Ran against the current main of this repo.

1. The task ran through the claude-code-swe-agent, whose prompt is PR-only

apps/claude-code-swe-agent/src/claude.ts → buildPrompt() is the task prompt handed to Claude Code. It hard-codes a git/PR workflow and only ever references the gh CLI:

• Framing: "Complete the task below end-to-end and open (or update) a GitHub pull request with the result."
• Repository context: "clone it into the current directory … create it with gh repo create."
• Environment block advertises only: "gh (GitHub CLI, authenticated)" as the GitHub-capable tool.
• Rules mandate: "Open a pull request with gh pr create …", feature branch, etc.

Nowhere does the prompt mention a "GitHub tool," the GitHub API, or any non-PR outcome (issue/comment/label). So for a create-an-issue request the model is being told, in effect, that the deliverable is a PR and that gh is how it talks to GitHub. Reaching for gh issue create is the faithful behavior.

There is also a latent goal conflict: the instruction was "create an issue," but the prompt simultaneously insists the result must be "a GitHub pull request." Filing #147 required working against the prompt (no repo change, no branch, no PR). This friction was noted in that session's own summary.

2. There is no GitHub tool available to the agent

• No MCP wiring anywhere. Repo-wide search for mcpServers / --mcp-config / octokit / mcp__ / modelcontextprotocol returns nothing. The SWE agent is launched via claude -p with git/gh on PATH and no MCP servers.
• No github Tool CRD. tools/ and tools-local/ contain kubectl-readonly, recipe-*, web-fetch, web-search, http-get-* — no GitHub tool.
• The only programmatic GitHub surface is internal to the gateway. apps/integration-gateway/src/github-client.ts exposes just postIssueComment() and removeIssueLabel() (the ADR 0025 "starting work" comment / triage-label path). It is a reply client used by the gateway plumbing, not an agent-invokable tool, and it can't create issues or PRs.

So the agent did not choose gh over a GitHub tool — no GitHub tool exists in its context. Given its actual capabilities, gh was the only correct option.

3. Why this is inefficient

Routing a pure issue-creation request to a full SWE-agent Job is heavyweight and mis-framed:

• It provisions a coding-agent Job (workspace, GitHub App/PAT token minting, clone) for what is a single API call (POST /repos/{o}/{r}/issues).
• The PR-centric prompt fights the actual goal, wasting turns reconciling "create an issue" with "open a pull request."
• gh runs with a broad token, whereas a typed tool could scope the exact operation and reuse the gateway's OAuth-delegated identity (per Create a GitHub tool #86).

Recommendations

A. Refine the SWE-agent prompt (claude.ts buildPrompt) — short term, no new infra

• Make the deliverable conditional on the task, not universally "a pull request." Not every GitHub task ends in a PR; allow issue/comment/label/no-op outcomes without forcing clone → branch → PR.
• Make the "Repository context" section acknowledge tasks that need no repo checkout at all.
• Once a GitHub tool exists (Create a GitHub tool #86/Allow agents to call tools #87), name it in the prompt and instruct the model to prefer the structured tool over shelling out to gh for GitHub operations.

B. Close the tooling gap (already tracked — link, don't duplicate)

• Create a GitHub tool #86 Create a GitHub tool — a first-class tool (GH CLI or API) wired for prod, using the integration-gateway OAuth delegation for its token.
• Allow agents to call tools #87 Allow agents to call tools — agent CRD toolRef + SDK support so sub-agents can invoke it.
• Together these let issue/PR/comment operations go through a typed, least-privilege tool instead of ad-hoc gh.

C. Routing / right-sizing

• Consider whether issue-only requests should route to a lighter path than a full SWE-agent Job. The github-issue-labeled-triage IntegrationRoute currently dispatches straight to a SWE Agent CR by default; an issue/comment operation that needs no code change could be served by the orchestrator invoking a GitHub tool directly.

Acceptance criteria

• buildPrompt no longer hard-codes "a pull request" as the only deliverable; non-PR GitHub tasks (issue/comment/label) are handled without a forced clone/branch/PR.
• Prompt references a structured GitHub tool and directs the model to prefer it over gh, gated on Create a GitHub tool #86/Allow agents to call tools #87 landing.
• Decision recorded on right-sizing issue-only requests vs. a full SWE-agent Job (route to orchestrator + GitHub tool, or keep SWE agent).

References

• Prompt: apps/claude-code-swe-agent/src/claude.ts (buildPrompt, environment/rules blocks)
• Config (no MCP): apps/claude-code-swe-agent/src/config.ts
• Gateway GitHub client (reply-only): apps/integration-gateway/src/github-client.ts; ADR docs/adr/0025-triage-agent-starting-work-comment-and-session-page.md
• Tool catalog: tools/, tools-local/
• Related issues: Create a GitHub tool #86, Allow agents to call tools #87 (tooling gap); Update outdated README architecture diagram to explain per-turn tool/skill/agent lookup #147 (the issue whose creation prompted this investigation)