Bump lm-eval[api] from 0.4.4 to 0.4.8 #12

dependabot · 2025-04-30T09:36:25Z

Bumps lm-eval[api] from 0.4.4 to 0.4.8.

Release notes

v0.4.8

lm-eval v0.4.8 Release Notes

Key Improvements

New Backend Support:

Added SGLang as new evaluation backend! by @Monstertail

Enabled model steering with vector support via sparsify or sae_lens by @luciaquirke and @AMindToThink

Breaking Change: Python 3.8 support has been dropped as it reached end of life. Please upgrade to Python 3.9 or newer.

Added Support for gen_prefix in config, allowing you to append text after the token (or at the end of non-chat prompts) - particularly effective for evaluating instruct models

New Benchmarks & Tasks

Code Evaluation

HumanEval by @hjlee1371 in #1992

MBPP by @hjlee1371 in #2247

HumanEval+ and MBPP+ by @bzantium in #2734

Multilingual Expansion

Global Coverage:

Global MMLU (Lite version by @shivalika-singh in #2567, Full version by @bzantium in #2636)

MLQA multilingual question answering by @KahnSvaer in #2622

Asian Languages:

HRM8K benchmark for Korean and English by @bzantium in #2627

Updated KorMedMCQA to version 2.0 by @GyoukChu in #2540

Fixed TMLU Taiwan-specific tasks tag by @nike00811 in #2420

European Languages:

Added Evalita-LLM benchmark by @m-resta in #2681

BasqueBench with Basque translations of ARC and PAWS by @naiarapm in #2732

Updated Turkish MMLU configuration by @ArdaYueksel in #2678

Middle Eastern Languages:

Arabic MMLU by @bodasadallah in #2541

AraDICE task by @firojalam in #2507

Ethics & Reasoning

Moral Stories by @upunaprosk in #2653

Histoires Morales by @upunaprosk in #2662

Others

MMLU Pro Plus by @asgsaeid in #2366

GroundCocoa by @HarshKohli in #2724

We extend our thanks to all contributors who made this release possible and to our users for your continued support and feedback.

Thanks, the LM Eval Harness team (@baberabb and @lintangsutawika)

... (truncated)

Commits

6d2abda increment version to 0.4.8 (#2760)
7433295 add debug log (#2757)
8b5c5c1 Add test for a simple Unitxt task (#2742)
d35008f Enable steering HF models (#2749)
14b0bd2 fix doc: generate_until only outputs the generated text! (#2755)
ade0142 Groundcocoa (#2724)
529f480 [Readme change for SGLang] fix error in readme and add OOM solutions for sgla...
a87fe42 fix vllm data parallel (#2746)
af2d2f3 fix log condition (#2737)
29971fa Support SGLang as Potential Backend for Evaluation (#2703)
Additional commits viewable in compare view

You can trigger a rebase of this PR by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Summary by Sourcery

Upgrade lm-eval[api] dependency from version 0.4.4 to 0.4.8

New Features:

Added SGLang as new evaluation backend
Enabled model steering with vector support
Added support for gen_prefix in config

Chores:

Dropped support for Python 3.8
Added new benchmarks and tasks across multiple languages

Note
Automatic rebases have been disabled on this pull request as it has been open for over 30 days.

Bumps [lm-eval[api]](https://github.com/EleutherAI/lm-evaluation-harness) from 0.4.4 to 0.4.8. - [Release notes](https://github.com/EleutherAI/lm-evaluation-harness/releases) - [Commits](EleutherAI/lm-evaluation-harness@v0.4.4...v0.4.8) --- updated-dependencies: - dependency-name: lm-eval[api] dependency-version: 0.4.8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

dependabot · 2025-04-30T09:36:26Z

Reviewers

The following teams could not be added as reviewers: reviewers. Either the team does not exist or it does not have the correct permissions to be added as a reviewer.

Assignees

The following users could not be added as assignees: trustyai-explainability/reviewers. Either the username does not exist or it does not have the correct permissions to be added as an assignee.

Please fix the above issues or remove invalid values from dependabot.yml.

sourcery-ai · 2025-04-30T09:36:29Z

Reviewer's Guide

This pull request updates the lm-eval[api] dependency from version 0.4.4 to 0.4.8.

File-Level Changes

Change	Details	Files
Updated the version of the `lm-eval[api]` dependency.	Changed the version specifier for `lm-eval[api]` from `==0.4.4` to `==0.4.8` in the `eval` extra dependencies section.	`pyproject.toml`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

for more information, see https://pre-commit.ci

dependabot · 2025-06-23T22:33:32Z

A newer version of lm-eval[api] exists, but since this PR has been edited by someone other than Dependabot I haven't updated it. You'll get a PR for the updated version as normal once this PR is merged.

dependabot bot added dependencies Pull requests that update a dependency file python Pull requests that update python code labels Apr 30, 2025

[pre-commit.ci] auto fixes from pre-commit.com hooks

0159c9e

for more information, see https://pre-commit.ci

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bump lm-eval[api] from 0.4.4 to 0.4.8 #12

Bump lm-eval[api] from 0.4.4 to 0.4.8 #12

Uh oh!

dependabot bot commented on behalf of github Apr 30, 2025 •

edited

Loading

Uh oh!

dependabot bot commented on behalf of github Apr 30, 2025

Uh oh!

sourcery-ai bot commented Apr 30, 2025 •

edited

Loading

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

dependabot bot commented on behalf of github Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Bump lm-eval[api] from 0.4.4 to 0.4.8 #12

Are you sure you want to change the base?

Bump lm-eval[api] from 0.4.4 to 0.4.8 #12

Uh oh!

Conversation

dependabot bot commented on behalf of github Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

v0.4.8

lm-eval v0.4.8 Release Notes

Key Improvements

New Benchmarks & Tasks

Code Evaluation

Multilingual Expansion

Ethics & Reasoning

Others

Summary by Sourcery

Uh oh!

dependabot bot commented on behalf of github Apr 30, 2025

Reviewers

Assignees

Uh oh!

sourcery-ai bot commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviewer's Guide

File-Level Changes

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

dependabot bot commented on behalf of github Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dependabot bot commented on behalf of github Apr 30, 2025 •

edited

Loading

sourcery-ai bot commented Apr 30, 2025 •

edited

Loading