Repository navigation
Build rewrites from the writer's notes, cold read, fiction rules (1.2.0) - #4
Merged
Merged
Conversation
eval/e2e/ rewrites each draft with a snapshot of the skill through a headless agent, then has fresh judges read every rewrite alone and give P(AI), and compares versions per title with a Wilcoxon signed-rank test. Sets: 20 dev titles, 18 held-out test titles, 30 Spanish, and 30 + 32 WritingPrompts stories. Notes for the new sets were extracted from the human posts. It needs the claude CLI, and codex for GPT judges; the scripts/ apparatus stays standard library only.
…pe rules Measured with the new harness against 1.1.0 (blind Opus judges, P(AI)): blog dev 47.8 to 34.9 (better on 20 of 20 titles), held-out blog titles 36.7 to 31.8, Spanish 38.8 to 29.2, Codex as rewriter 67.0 to 55.3, held-out stories 74.0 to 69.7. Human originals score 4 to 8. - With notes, the post is built from them; draft scaffolding goes. - Sections end on their last fact; no draft residue or date notes. - Sincerity markers named; genuinely joins ai-lean.txt. - A fresh agent reads the text cold once; text-level reasons are fixed. - references/fiction.md: callbacks, clockwork arcs, stated morals. - The Claude text watermark is recorded as a limit. Every rejected candidate is in references/numbers.md with its numbers.
…ewrite # Conflicts: # CLAUDE.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Version 1.2.0 changes how the skill rewrites, and adds a harness that measures a rewrite the way a reader meets it: fresh Claude Opus and GPT judges read each text alone and give the probability it was AI-written.
Mean P(AI) from blind Opus judges, 1.1.0 against this branch. Human originals score 4 to 8.
What changed in the skill:
ai-lean.txt, and held-out human posts are unchanged.references/fiction.mdholds rules for the story's shape: callbacks, a clockwork arc, a stated moral.Six more candidates were measured and left out, including a worked example that helped on dev and hurt on the held-out titles. Each one is in
references/numbers.mdwith its numbers. The harness is ineval/e2e/and needs the claude CLI;scripts/is still standard library only.