Add skill references for the onboarding process - #114
Conversation
…ictionary guidance
|
Preview deployed to Connect ( Deployed from commit ceefd50. |
|
Preview deployed to Connect ( Deployed from commit ceefd50. |
…semantic/data/context layers
…ata dictionaries categories
simonpcouch
left a comment
simonpcouch
left a comment
There was a problem hiding this comment.
Very much a step forward, thank you! Left some comments, but reviewing as Approve—I trust your judgment on taking some comments and ignoring others.
I'm not sure if this is universally applied in this repo, but I think it's at least the mode; could we soft-wrap text rather than hard-wrap in skills and vignettes? You're welcome to add a line to the tracked AGENTS.md as well to advice future agents to do this.
| ├── agent.R # reusable commons() construction | ||
| ├── DESCRIPTION | ||
| ├── instructions.md # optional concise, always-needed guidance | ||
| ├── onboarding.md # scope, decisions, and unresolved questions |
There was a problem hiding this comment.
Should this just be an AGENTS.md / CLAUDE.md?
| [devrel-agent evaluation](https://github.com/posit-dev/devrel-agent/tree/main/evals) | ||
| as the current worked example of a full commons evaluation. Adapt its | ||
| question, target, scorer, runner, baseline, and transcript-review patterns to | ||
| the agent being evaluated; do not copy its domain-specific rubric. |
There was a problem hiding this comment.
Halfway wondering whether there should be some sort of suggested warning-to-the-user along the lines of 'these will cost a lot of money to run, is that okay?'. I'm well into the tens of dollars for each run in devrel-agent currently.
| questions are realistic, useful, and representative of the agent's intended | ||
| production traffic. | ||
|
|
||
| Expand the set iteratively. Aim for around 30 questions. Include: |
There was a problem hiding this comment.
30 gives me pause. It might be because it appears as early as it does? Maybe we just need to be clear that expanding the set of questions is the outer-most loop; draft a couple/few questions and go through the full implement/pilot/critique/analyze flow, and only then should you add further questions. Otherwise, you're spending more time and money to learn each lesson / debug issues in the eval each time you iterate on it.
| confirm those changes against held-out questions rather than tuning only to | ||
| the visible set. | ||
|
|
||
| Give the user access to the full-run logs. Ask them to review the results and |
There was a problem hiding this comment.
Should this be explicit instructions on how to launch the Inspect log viewer?
| - name: realized | ||
| type: boolean | ||
| description: Orders that were shipped. | ||
| expr: status_cd = 90 |
There was a problem hiding this comment.
Noting that this will change soon (probably this afternoon? #115) and this might go stale before merging.
2 participants
Adds three skill references for the onboarding/agent-creation process:
onboarding.md- Main onboarding reference. Walks the agent through the process of creating a commons agent.data-dictionaries.md- Additional guidance on creating a commons-specific data dictionary.evaluation.md- Guidance on creating evals. This is almost entirely pulled from the eval notes in The onboarding experience #47 and discourage hard-coding numbers in the eval scorer #46 and the work in https://github.com/posit-dev/devrel-agent. I would expect this to change a lot. I'm mostly including it as a placeholder so we can figure out the organization of the skill/references.Also adds some general principles to
SKILL.md.I have not thoroughly tested this by creating a new agent (I did one quick pass, but it was very basic). I would expect things to shift and change once we do so, but I thought it might be helpful to review at this stage anyway.
Closes #47
Closes #46