fix: improve leaderboard scoring and fix aube package count#95
Merged
darcyclarke merged 1 commit intomainfrom Apr 20, 2026
Merged
fix: improve leaderboard scoring and fix aube package count#95darcyclarke merged 1 commit intomainfrom
darcyclarke merged 1 commit intomainfrom
Conversation
Leaderboard scoring: - DNF penalty: assign slowest successful time (1:1) instead of excluding DNFs from averages, so PMs that fail tests can't cherry-pick artificially low averages - Average view: sort by wins first (most wins = #1), then average time as tiebreaker — ensures PMs that consistently win across many tests rank higher - Specific variant views: keep sorting by average time with wins as tiebreaker - DNF PMs cannot win a fixture (only successful PMs compete for wins) - Skip fixture entirely if ALL PMs DNF Aube package count: - aube uses symlinks inside node_modules/.aube/ (unlike pnpm which uses hard links inside .pnpm/), so find -type f cannot traverse them - Detect node_modules/.aube/ and use find -L scoped to that directory to follow symlinks, then deduplicate unique package names via sed Co-authored-by: Darcy Clarke <darcy@darcyclarke.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Leaderboard scoring fixes
The leaderboard currently ranks by average time, which is unfair — PMs that DNF on most tests get artificially low averages from only their easy wins.
DNF penalty (both views):
Average view (default leaderboard):
Specific variant views:
Aube package count fix
Aube wasn't showing package counts in the benchmarks because:
node_modules/.aube/(unlike pnpm which uses hard links inside.pnpm/)find -type fcannot traverse symlinks, so it found 0 packagesnode_modules/.aube/and usefind -Lscoped to that directory to follow symlinks, then deduplicate unique package namesCo-authored-by: Darcy Clarke darcy@darcyclarke.com