Constrain built-skill allowed-tools to the approved plan - #40
Open
steve wu (wuwangzhang1216) wants to merge 1 commit into
Open
Conversation
Author
|
@microsoft-github-policy-service agree |
`allowed-tools` is the capability grant on an installed SKILL.md. The user approves it on the plan-review screen, but the builder's create turn is an LLM call whose returned list was written into the frontmatter verbatim, so a skill could be installed carrying tools the reviewer never approved. Set-membership is not the right check. The builder's contract deliberately lets the agent tighten the grant to the steps it actually emitted (approved `Bash(gh *)` -> submitted `Bash(gh pr list)`). Comparing strings rejects every such narrowing and falls back to the broader approved pattern, disabling the one behaviour the contract asks for. Decide pattern subsumption instead. common/allowed-tools.ts is pure and errs toward refusal: anything unparseable, ambiguous, or not provably covered is treated as not covered. Bash(gh pr list) <= Bash(gh *) kept (narrowing) Bash(*) !> Bash(gh *) dropped (escalation) Bash(gh *) <= Bash kept (bare name is unrestricted) Bash !> Bash(gh *) dropped (drops an approved restriction) Argument patterns treat `*` as the only wildcard and everything else as literal, so a regex metacharacter in a command cannot silently widen the match. Tool names compare case-insensitively -- a case slip should cost a narrowing rather than widen anything -- while argument patterns stay case-sensitive, because shell commands are. When nothing survives, the approved patterns are re-asserted rather than emitting an empty list: an omitted `allowed-tools` means "use the agent's default set", which may well be wider than what the user approved. Fixes microsoft#8.
steve wu (wuwangzhang1216)
force-pushed
the
fix/constrain-allowed-tools-to-approved-plan
branch
from
August 4, 2026 03:29
25c2a68 to
a8067b7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8.
allowed-toolsis the capability grant on an installedSKILL.md. The user approves it on the plan-review screen, but the builder's create turn is an LLM call whose returned list was written into the frontmatter verbatim:So a skill could be installed carrying tools the reviewer never approved.
Why not set-membership
The issue suggests filtering by
plan.allowedTools.includes(t). That closes the escalation, butallowed-toolsentries are patterns, not fixed strings — and the comment right above this line, plus the builder instructions, deliberately let the agent tighten the grant to the steps it actually emitted. Set-membership rejects every one of those narrowings and falls back to the broader approved pattern:["Bash(gh *)"], submitted["Bash(gh pr list)", "Bash(rm -rf /)"]main["Bash(gh pr list)", "Bash(rm -rf /)"]—rm -rfreaches the frontmatter["Bash(gh *)"]— safe, but the valid narrowing is discarded and the grant ends up wider than the agent asked for["Bash(gh pr list)"], dropped["Bash(rm -rf /)"]So this decides pattern subsumption instead: does the submitted pattern grant anything the approved pattern does not?
What this adds
common/allowed-tools.ts— pure, no Electron imports, so the trust boundary is unit-testable in isolation. It errs toward refusal: anything unparseable, ambiguous, or not provably covered is treated as not covered.A few decisions worth flagging for review:
*is the only wildcard; everything else is literal, so a regex metacharacter in a command can't silently widen the match (echo axb⊄echo a.b).GHis not theghCLI.allowed-toolsmeans "use the agent's default set", which may well be wider than what was approved. The one case that stays empty is an approved list that was itself empty: no explicit pattern can be proven narrower than a default set whose contents we don't know.electron/skillbuilder/builder.tscalls it and logs anything dropped.Tests
common/allowed-tools.test.ts— 10 cases covering narrowing, escalation, the bare-name form, literal metacharacters, the case-sensitivity split, blank/duplicate entries, and both empty-list fallbacks. Added tonpm test.The narrowing test is the specific regression that set-membership would fail, so it pins the behaviour the contract asks for.
Coverage boundary
The merge logic is fully covered, but the wiring in
SkillBuilder.create()is not: it needs a liveCopilotClient, and making it unit-testable would mean injecting the client — out of scope here. Worth a follow-up if you'd like the create turn under test end to end.Validation
tsc --noEmit— clean (commonis in the tsconfig include, so the new files are covered).npm test— the 10 new tests pass, no regressions in the rest of the suite.One note on running the suite locally:
electron/recorder/controller.test.tsfails for me on Node 26, unrelated to this change (it fails the same way on a clean tree). Node 26 removed--experimental-transform-types, and strip-only mode rejects the parameter property atcontroller.ts:111. On Node 24 — what the README and CI use — it's fine. Happy to file that separately if it's useful.