Skip to content

Add the code testing agent and /add-tests workflow#53782

Open
JanKrivanek wants to merge 3 commits intodotnet:mainfrom
JanKrivanek:dev/jankrivanek/add-tests-workflow
Open

Add the code testing agent and /add-tests workflow#53782
JanKrivanek wants to merge 3 commits intodotnet:mainfrom
JanKrivanek:dev/jankrivanek/add-tests-workflow

Conversation

@JanKrivanek
Copy link
Copy Markdown
Member

Motivation

Add the code testing agent and its hooking into slash comand
This should help driving the test coverage for the changes added by PRs

Functionality tested in fork: JanKrivanek#3
Addapted from: dotnet/machinelearning#7602

Copilot AI review requested due to automatic review settings April 9, 2026 16:48
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an agentic /add-tests workflow to generate unit tests for PR changes, including the supporting multi-agent “code testing” skill/agents and the compiled workflow lockfile used by gh-aw.

Changes:

  • Introduce /add-tests workflow definition (.md) plus repo-specific build setup import.
  • Check in the compiled gh-aw lock workflow (add-tests.lock.yml) and mark lockfiles as generated/merge-ours.
  • Add a multi-agent test-generation skill and supporting agents (research/plan/implement/build/test/fix/lint).
Show a summary per file
File Description
.github/workflows/shared/repo-build-setup.md Repo-specific build + PATH setup imported by /add-tests.
.github/workflows/add-tests.md Defines the /add-tests slash-command workflow behavior and instructions.
.github/workflows/add-tests.lock.yml Compiled gh-aw workflow lockfile (generated).
.github/aw/actions-lock.json Locks action SHAs for gh-aw compilation.
.github/agents/code-testing-generator.agent.md Orchestrator agent for Research→Plan→Implement pipeline.
.github/agents/code-testing-researcher.agent.md Research agent producing .testagent/research.md.
.github/agents/code-testing-planner.agent.md Planning agent producing .testagent/plan.md.
.github/agents/code-testing-implementer.agent.md Phase implementer agent that writes tests and validates.
.github/agents/code-testing-builder.agent.md Build agent used by implementer for compilation.
.github/agents/code-testing-tester.agent.md Test execution agent used by implementer.
.github/agents/code-testing-fixer.agent.md Fixer agent for compile/test failures.
.github/agents/code-testing-linter.agent.md Lint/format agent to apply formatting fixes.
.gitattributes Marks workflow lockfiles as linguist-generated and merge=ours.
.claude/skills/code-testing-agent/unit-test-generation.prompt.md Default prompt guidance for test generation quality/coverage.
.claude/skills/code-testing-agent/SKILL.md Skill documentation and usage guidance.
.claude/skills/code-testing-agent/extensions/dotnet.md .NET-specific build/test/lint guidance for the skill.

Copilot's findings

  • Files reviewed: 16/16 changed files
  • Comments generated: 13

Comment thread .github/workflows/add-tests.md
Comment thread .github/agents/code-testing-researcher.agent.md Outdated
Comment thread .github/agents/code-testing-implementer.agent.md Outdated
Comment thread .github/agents/code-testing-fixer.agent.md Outdated
Comment thread .github/agents/code-testing-builder.agent.md Outdated
Comment thread .claude/skills/code-testing-agent/extensions/dotnet.md
Comment thread .github/agents/code-testing-generator.agent.md Outdated
Comment thread .github/agents/code-testing-generator.agent.md
Comment thread .claude/skills/code-testing-agent/SKILL.md Outdated
Comment thread .github/workflows/add-tests.md
Comment thread .github/workflows/add-tests.md
@marcpopMSFT
Copy link
Copy Markdown
Member

marcpopMSFT commented Apr 9, 2026

@JanKrivanek This looks like a generic add test flow not specific to the sdk repo. It talks about unit tests which we mostly don't have in this repo. It doesn't appear to have repo specific details on how our tests are typically written (ie I see no details about testassets). We have a bunch of repos in this repo so there's no breakdown of preference for forming a test for the different areas within the repo (ie format, templating, analyzers, runfile, etc). It talks about languages we don't have in this repo. It follows an AW model that doesn't seem to align with the model the runtime team is having everyone align on. I'd talk to @jeffhandley about that.

Maybe look into arcade-skills for this but this doesn't seem to fit in this repo unless I'm missing where it provides more repo-specific info that would make it an improvement over (@)copilot

@JanKrivanek
Copy link
Copy Markdown
Member Author

@JanKrivanek This looks like a generic add test flow not specific to the sdk repo. It talks about unit tests which we mostly don't have in this repo. It doesn't appear to have repo specific details on how our tests are typically written (ie I see no details about testassets). We have a bunch of repos in this repo so there's no breakdown of preference for forming a test for the different areas within the repo (ie format, templating, analyzers, runfile, etc). It talks about languages we don't have in this repo. It follows an AW model that doesn't seem to align with the model the runtime team is having everyone align on. I'd talk to @jeffhandley about that.

Maybe look into arcade-skills for this but this doesn't seem to fit in this repo unless I'm missing where it provides more repo-specific info that would make it an improvement over (@)copilot

👍 I'll benchmark the tests generation vs plan CCA generation and go from there

Btw. - I'm very aware of the tokens rotation - it's based on our original work couple weeks ago in dotnet/skills :-) - https://github.com/dotnet/skills/pull/96/changes
I didn't want to conflict (#53782 (comment))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants