✨ One Eval, Evaluation in One ✨

One-Eval is an automated Agent-based evaluation framework for Large Language Models, designed to achieve NL2Eval: automatically orchestrating evaluation workflows and generating reports from natural language requirements.
Built on DataFlow and LangGraph, it emphasizes a traceable, interruptible, and scalable evaluation loop.

English | 简体中文

📰 1. News

• [2026-06] 🧩 One-Eval is now available as a portable Skill!
  Use One-Eval directly inside Claude Code (or Codex) — no frontend/backend to launch. Just point Claude Code at this repo and start evaluating. See Quick Start → Use with Claude Code.
• [2026-03] 🎉 One-Eval (v0.1.0) is officially open-sourced!
  We released the first version, supporting full-link automation from natural language to evaluation reports (NL2Eval). Say goodbye to tedious manual scripts and make LLM evaluation as simple, intuitive, and controllable as chatting. Welcome to Star 🌟 and follow!

💡 Why One-Eval?

Traditional evaluation frameworks often require users to manually search for benchmarks, download data, and fill in extensive configuration parameters.
One-Eval aims to change this: Everything that can be automated is handled by the Agent. From benchmark recommendation to model evaluation, we are committed to providing the most direct and intuitive evaluation experience.

🔍 2. Overview

Traditional evaluation often faces pain points such as complex scripts, fragmented processes, and difficulty in reuse. One-Eval reconstructs evaluation into a graph-based execution process (Graph / Node / State), dedicated to creating the next generation of interactive evaluation experience:

• 🗣️ NL2Eval: Just input a natural language goal (e.g., "Evaluate the model's performance on math reasoning tasks"), and the system automatically parses the intent and plans the execution path.
• 🧩 End-to-End Automation: Automatically completes benchmark recommendation, data preparation, inference execution, metric matching, scoring, and multi-dimensional report generation.
• ⏸️ Human-in-the-Loop: Supports interruption and human intervention at key nodes (such as benchmark selection, result review), facilitating real-time adjustment of evaluation strategies based on feedback.
• 📊 Scalable Architecture: Based on the DataFlow operator system and LangGraph state management, it easily integrates private datasets and custom metrics.

⚡ 3. Quick Start

There are two independent ways to use One-Eval — pick whichever suits you:

• 3.1 Use with Claude Code — zero setup, just paste one line. Best for getting started fast.
• 3.2 Web UI (Frontend + Backend) — a full interactive interface; requires environment setup.

3.1 Use with Claude Code

Zero setup. Just paste this to Claude Code (or Codex, or any coding agent):

Use the one-eval-skill in https://github.com/OpenDCAI/One-Eval to get us started on evaluating my model.

3.2 Web UI (Frontend + Backend)

A full web UI with a separation of frontend and backend architecture.

Step 1: Install the environment

We provide two environment management methods: Conda and uv. Choose one:

# Option A: Conda
conda create -n one-eval python=3.11 -y
conda activate one-eval
pip install -e .

# Option B: uv
uv venv
uv pip install -e .

Step 2: Start Backend (FastAPI)

uvicorn one_eval.server.app:app --host 0.0.0.0 --port 8000

Step 3: Start Frontend (Vite + React)

cd one-eval-web
npm install
npm run dev

Visit http://localhost:5173 to start interactive evaluation.

Note: After starting, please enter the settings interface first to configure parameters such as API, model, and HF Token (to support batch data download), and click save.

🗂️ 4. Bench Gallery

One-Eval has a built-in rich Bench Gallery for unified management of meta-information of various evaluation benchmarks (such as task type, data format, Prompt template).

Currently covering mainstream text-only capability dimensions (no complex sandbox environment required):

• 🧮 Reasoning: MATH, GSM8K, BBH, AIME...
• 🌐 General Knowledge: MMLU, CEval, CMMLU...
• 🔧 Instruction Following: IFEval...

🚀 5. Future Work

We plan to continuously maintain and update One-Eval in the following directions:

• 💻 Support for Complex Evaluation Scenarios: Extend support for LLM evaluation fields that require additional execution environments, such as Code and Text2SQL.
• 🤖 Agentic Evaluation & Sandbox Environments: Support evaluation in Agentic domains (e.g., SWE-bench) that rely on complex sandbox environments.
• 🌐 Online Community & Platform: Deploy an online evaluation platform where users can discuss, build, share, and use their own custom benchmarks.

🙌 Join Us: We warmly welcome co-workers to join our open-source project! Feel free to contact us directly for co-development. We highly support contributors in exploring different directions and achieving their own outputs (e.g., collaborating on paper submissions).

📮 6. Contact & Citation

If you are interested in this project, or have any questions or suggestions, please contact us via Issue or join our WeChat group.

• 📮 GitHub Issues: Submit bugs or feature suggestions.

• 🔧 GitHub Pull Requests: Contribute code improvements.

Citation

@misc{shen2026oneevalagenticautomatedtraceable,
      title={One-Eval: An Agentic System for Automated and Traceable LLM Evaluation}, 
      author={Chengyu Shen and Yanheng Hou and Minghui Pan and Runming He and Zhen Hao Wong and Meiyi Qiang and Zhou Liu and Hao Liang and Peichao Lai and Zeang Sheng and Wentao Zhang},
      year={2026},
      eprint={2603.09821},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.09821}, 
}

@article{liang2025dataflow,
  title={DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI},
  author={Liang, Hao and Ma, Xiaochen and Liu, Zhou and Wong, Zhen Hao and Zhao, Zhengyang and Meng, Zimo and He, Runming and Shen, Chengyu and Cai, Qifeng and Han, Zhaoyang and others},
  journal={arXiv preprint arXiv:2512.16676},
  year={2025}
}