Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
166 Open 1,669 Closed
166 Open 1,669 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

[fix] Fix mmlu_redux pulling the wrong _default_template_yaml
#3406 opened Nov 13, 2025 by fxmarty-amd Loading…
1
Resolve deprecation of vllm.utils.get_open_port
#3398 opened Nov 13, 2025 by DarkLight1337 Loading…
Fixes #3391 avoid error on no-output
#3395 opened Nov 12, 2025 by Oseltamivir Loading…
1
fix: resolve duplicate task names and add safeguards.
#3394 opened Nov 11, 2025 by giuliolovisotto Loading…
[bugfix] additional_config parsing
#3393 opened Nov 10, 2025 by brian-dellabetta Loading…
1
Fix MBPP task configuration and code extraction logic
#3387 opened Nov 6, 2025 by tanios13 Loading…
3
feat: refine Chain-of-Thought removal logic
#3386 opened Nov 6, 2025 by Co-Cl2 Loading…
1
[feat] Add Countdown Task
#3384 opened Nov 4, 2025 by StephenXie Loading…
6
Math 500
#3381 opened Nov 1, 2025 by seldereyy Loading…
1
Fix descriptions in the Moral Stories and Histoires Morales tasks.
#3374 opened Oct 28, 2025 by upunaprosk Loading…
Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py
#3372 opened Oct 28, 2025 by vnayakde Loading…
2
Add support for configurable chrF metric parameters in task YAML, fix…
#3363 opened Oct 23, 2025 by augustlakia Loading…
4
Add gsm_symbolic and gsm_symbolic_cot tasks
#3354 opened Oct 19, 2025 by MengAiDev Loading…
1
[AIME24 | AIME25] Enable Multiple Generation Repeats with Pass@k and Majority@k Metrics
#3351 opened Oct 17, 2025 by ihebchaa Loading…
2
fix(tasks):pin correct MMLUSR version
#3350 opened Oct 16, 2025 by christinaexyou Loading…
1
Added ULQA benchmark
#3340 opened Oct 13, 2025 by keramjan Loading…
1
Fix PIL image hashing to use actual bytes instead of object repr
#3331 opened Oct 7, 2025 by tboerstad Loading…
feat: Add support for accelerate-wrapped models in simple_evaluate()
#3313 opened Sep 26, 2025 by DhruvaKashyap Loading…
2
Add MATH500
#3311 opened Sep 26, 2025 by jannalulu Loading…
1
Support empty response for Completions and ChatCompletions API
#3309 opened Sep 22, 2025 by tboerstad Loading…
5
Adding New Task SLR-Bench : Scalable Logical Reasoning Benchmark
#3305 opened Sep 20, 2025 by Ahmad21Omar Loading…
1
Support torchrun vllm DP
#3304 opened Sep 19, 2025 by luccafong Loading…
1
Gemini evaluation support
#3300 opened Sep 15, 2025 by IsraelAbebe Loading…
1
Fix lambada_multilingual_stablelm
#3294 opened Sep 11, 2025 by jmichaelov Loading…
1
Adding SPaRC to lm eval harness
#3262 opened Aug 25, 2025 by lkaesberg Loading…
2
ProTip! Adding no:label will show everything without a label.