Add the code testing agent and /add-tests workflow#53782
Add the code testing agent and /add-tests workflow#53782JanKrivanek wants to merge 3 commits intodotnet:mainfrom
Conversation
…_0, increase timeout to 60min
There was a problem hiding this comment.
Pull request overview
Adds an agentic /add-tests workflow to generate unit tests for PR changes, including the supporting multi-agent “code testing” skill/agents and the compiled workflow lockfile used by gh-aw.
Changes:
- Introduce
/add-testsworkflow definition (.md) plus repo-specific build setup import. - Check in the compiled gh-aw lock workflow (
add-tests.lock.yml) and mark lockfiles as generated/merge-ours. - Add a multi-agent test-generation skill and supporting agents (research/plan/implement/build/test/fix/lint).
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/shared/repo-build-setup.md | Repo-specific build + PATH setup imported by /add-tests. |
| .github/workflows/add-tests.md | Defines the /add-tests slash-command workflow behavior and instructions. |
| .github/workflows/add-tests.lock.yml | Compiled gh-aw workflow lockfile (generated). |
| .github/aw/actions-lock.json | Locks action SHAs for gh-aw compilation. |
| .github/agents/code-testing-generator.agent.md | Orchestrator agent for Research→Plan→Implement pipeline. |
| .github/agents/code-testing-researcher.agent.md | Research agent producing .testagent/research.md. |
| .github/agents/code-testing-planner.agent.md | Planning agent producing .testagent/plan.md. |
| .github/agents/code-testing-implementer.agent.md | Phase implementer agent that writes tests and validates. |
| .github/agents/code-testing-builder.agent.md | Build agent used by implementer for compilation. |
| .github/agents/code-testing-tester.agent.md | Test execution agent used by implementer. |
| .github/agents/code-testing-fixer.agent.md | Fixer agent for compile/test failures. |
| .github/agents/code-testing-linter.agent.md | Lint/format agent to apply formatting fixes. |
| .gitattributes | Marks workflow lockfiles as linguist-generated and merge=ours. |
| .claude/skills/code-testing-agent/unit-test-generation.prompt.md | Default prompt guidance for test generation quality/coverage. |
| .claude/skills/code-testing-agent/SKILL.md | Skill documentation and usage guidance. |
| .claude/skills/code-testing-agent/extensions/dotnet.md | .NET-specific build/test/lint guidance for the skill. |
Copilot's findings
- Files reviewed: 16/16 changed files
- Comments generated: 13
|
@JanKrivanek This looks like a generic add test flow not specific to the sdk repo. It talks about unit tests which we mostly don't have in this repo. It doesn't appear to have repo specific details on how our tests are typically written (ie I see no details about testassets). We have a bunch of repos in this repo so there's no breakdown of preference for forming a test for the different areas within the repo (ie format, templating, analyzers, runfile, etc). It talks about languages we don't have in this repo. It follows an AW model that doesn't seem to align with the model the runtime team is having everyone align on. I'd talk to @jeffhandley about that. Maybe look into arcade-skills for this but this doesn't seem to fit in this repo unless I'm missing where it provides more repo-specific info that would make it an improvement over (@)copilot |
👍 I'll benchmark the tests generation vs plan CCA generation and go from there Btw. - I'm very aware of the tokens rotation - it's based on our original work couple weeks ago in dotnet/skills :-) - https://github.com/dotnet/skills/pull/96/changes |
Motivation
Add the code testing agent and its hooking into slash comand
This should help driving the test coverage for the changes added by PRs
Functionality tested in fork: JanKrivanek#3
Addapted from: dotnet/machinelearning#7602