Hi MMLU-Pro team — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results. benchmarks/ already has the original MMLU converted; MMLU-Pro is different enough (10 options instead of 4, CoT-focused, 14 domains) that I think it's worth its own suite rather than folding into the existing MMLU one.
I read the real fields this repo's own eval code (evaluate_from_local.py) uses: question, options (a list, up to 10, with "N/A" entries filtered out), answer (letter), answer_index, category (the 14 domains), cot_content.
That maps onto EvalPort cleanly: question + options -> TestCase.input, answer/answer_index -> TestCase.expected_output, category -> metadata for per-domain breakdowns, graded with exact_match or a small custom grader mirroring your own extract_answer regex chain.
Would your maintainers be open to an EvalPort-format export living in EvalPort's benchmarks/, with attribution, license, and a link to the HF dataset/leaderboard preserved? Happy to do the conversion and open a PR for review first. No worries if this isn't useful.
Hi MMLU-Pro team — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results.
benchmarks/already has the original MMLU converted; MMLU-Pro is different enough (10 options instead of 4, CoT-focused, 14 domains) that I think it's worth its own suite rather than folding into the existing MMLU one.I read the real fields this repo's own eval code (
evaluate_from_local.py) uses:question,options(a list, up to 10, with"N/A"entries filtered out),answer(letter),answer_index,category(the 14 domains),cot_content.That maps onto EvalPort cleanly:
question+options->TestCase.input,answer/answer_index->TestCase.expected_output,category-> metadata for per-domain breakdowns, graded withexact_matchor a small custom grader mirroring your ownextract_answerregex chain.Would your maintainers be open to an EvalPort-format export living in EvalPort's
benchmarks/, with attribution, license, and a link to the HF dataset/leaderboard preserved? Happy to do the conversion and open a PR for review first. No worries if this isn't useful.