Fix OCR benchmark compatibility and add OmniDocBench v1.6 evaluation

[Yunnglin]

Summary

This PR combines the framework-side fixes for #1527 with a separate OmniDocBench v1.6 integration.

OCR benchmark compatibility (#1527)

• preserve case sensitivity for OCRBench HME samples while keeping non-HME matching case-insensitive
• place images before text prompts for OCRBench and both OmniDocBench adapters
• preserve explicitly configured empty system prompts without inserting duplicates

OmniDocBench integrations

• keep omni_doc_bench on EvalScope's original 981-page evalscope/OmniDocBench_tsv dataset and local scorer
• describe the original TSV integration as legacy without changing its benchmark display name
• recommend omni_doc_bench_v1_6 for new evaluations
• add omni_doc_bench_v1_6 for the latest default revision of OpenDataLab/OmniDocBench on ModelScope
• read the official v1.6 JSON through EvalScope's standard data loader without additional dataset validation; local inputs are expected to follow the v1.6 format
• store the complete page annotation in Sample.target and keep only version and image name in metadata
• use EvalScope's single-file dataset download path for the annotation and selected images

Official v1.6 scoring through ms-enclave

• score one page per sandbox call and reuse a long-lived Docker sandbox pool
• use the official amd64 image as the default and clear its one-shot ENTRYPOINT
• allow users to override the image; incompatible images fail explicitly during scoring
• run the image's official MGAM quick_match, CDM, TEDS/TEDS-S, edit-distance, and reading-order evaluation without vendoring scorer source
• omit page metrics that have no ground-truth component instead of substituting zero
• select the per-page main metric from the explicit PAGE_METRICS order
• average each available page metric globally, then compute Overall from aggregated text Edit, formula CDM, and table TEDS
• fail on disabled sandboxes, scoring failures, missing results, or invalid metric values

Benchmark metadata cleanup

• keep omni_doc_bench displayed as OmniDocBench
• simplify kimi_verifier display name to Kimi-Vendor-Verifier

Compatibility and requirements

• omni_doc_bench and omni_doc_bench_v1_6 use different data and scoring contracts; their scores are not directly comparable.
• omni_doc_bench_v1_6 expects the v1.6 data format even though it follows the latest ModelScope dataset revision.
• v1.6 requires ms-enclave[docker]>=0.0.8. The required ENTRYPOINT support is merged in feat: support overriding Docker image entrypoint ms-enclave#15; the 0.0.8 PyPI release remains a prerequisite for installation.
• Users provide Docker with amd64 support, sufficient disk and memory, ModelScope/GHCR access, and their VLM API endpoint/key.

Validation

• official image runtime verification: Python 3.10.16, TeX Live/CJK, Ghostscript, ImageMagick, official tests, and CDM smoke passed
• official parity: representative formula and table pages matched the official batch results for all seven metrics and Overall
• missing-component behavior: a text-only page returned only text and reading-order metrics
• local Docker VLM E2E: qwen-vl-plus, representative text/formula/table page, inference, sandbox review, and JSON/HTML report passed
• HTTP manager VLM E2E: the same flow passed through a remote ms-enclave manager
• latest ModelScope dataset revision smoke: 1,651 annotations and images were available
• pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings passed
• make docs-pipeline BENCHMARK="omni_doc_bench omni_doc_bench_v1_6 kimi_verifier" FORCE=1 passed
• make lint passed
• git diff --check passed

No API keys, generated reports, Dockerfiles, or third-party scorer sources are committed.

Closes #1527

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[Copilot]

🟡 Not ready to approve

The new v1.6 adapter has a couple of correctness/documentation gaps (unstable main metric selection and missing promised metadata) that should be fixed before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR addresses OCR benchmark parity issues (notably OCRBench/OmniDocBench compatibility) and introduces a new, pinned OmniDocBench v1.6 benchmark integration that scores via an official Docker image executed through ms-enclave.

Changes:

• Fixes adapter prompt/message construction behaviors (empty system prompts, image-before-text ordering) and adds regression tests.
• Updates ModelScope single-file download logic to use the dedicated dataset_file_download API (with tests).
• Adds a new omni_doc_bench_v1_6 benchmark adapter + sandbox scorer, while rebranding the existing omni_doc_bench integration as legacy and updating docs/meta accordingly.

File summaries

File	Description
tests/api/test_default_data_adapter.py	Adds regression tests for empty system prompt insertion behavior.
tests/api/test_dataset_hub.py	Adds coverage for ModelScope single-file download behavior.
requirements/sandbox.txt	Bumps ms-enclave docker extra minimum version.
pyproject.toml	Points omnidoc_bench extra to legacy requirements path.
evalscope/api/dataset/hub.py	Switches ModelScope file resolution from snapshot download to single-file download API.
evalscope/api/benchmark/adapters/default_data_adapter.py	Preserves explicitly configured empty system prompts (treats "" as configured, not falsy).
evalscope/benchmarks/ocr_bench/ocr_bench/ocr_bench_adapter.py	Ensures image content precedes text; preserves case sensitivity for HME matching.
evalscope/benchmarks/omnidoc_bench/legacy/requirements.txt	Adds a dedicated legacy requirements list for the old OmniDocBench integration.
evalscope/benchmarks/omnidoc_bench/legacy/omnidoc_bench_adapter.py	Rebrands as legacy; changes sample target/reference handling; ensures image precedes text.
evalscope/benchmarks/omnidoc_bench/legacy/metrics.py	Adds legacy metric implementation code.
evalscope/benchmarks/omnidoc_bench/legacy/end2end_eval.py	Adds legacy end-to-end evaluator implementation code.
evalscope/benchmarks/omnidoc_bench/legacy/init.py	Initializes legacy benchmark module.
evalscope/benchmarks/omnidoc_bench/v1_6/sandbox_scorer.py	Adds pinned single-page official-scoring program builder + result parser.
evalscope/benchmarks/omnidoc_bench/v1_6/omnidoc_bench_v1_6_adapter.py	Adds v1.6 adapter pinned to dataset revision + official image scoring via sandbox pool.
evalscope/benchmarks/omnidoc_bench/v1_6/init.py	Initializes v1.6 benchmark module.
evalscope/benchmarks/_meta/omni_doc_bench.json	Updates generated legacy benchmark metadata (pretty name/description/statistics/sample example).
evalscope/benchmarks/_meta/omni_doc_bench_v1_6.json	Adds generated metadata for the new v1.6 benchmark.
docs/{en,zh}/get_started/supported_dataset/vlm.md	Lists the legacy benchmark and adds the new v1.6 benchmark entry.
docs/{en,zh}/benchmarks/omni_doc_bench.md	Updates generated legacy benchmark page content.
docs/{en,zh}/benchmarks/omni_doc_bench_v1_6.md	Adds generated v1.6 benchmark page content.
AGENTS.md	Documents required BenchmarkMeta.description section headings and benchmark authoring guidance.

Review details

Suppressed comments (1)

evalscope/benchmarks/omnidoc_bench/legacy/omnidoc_bench_adapter.py:74

• The legacy benchmark description lists Polygon3/zss/rapidfuzz as required, but those packages are not imported anywhere in the legacy implementation and aren’t declared in the legacy extra requirements. This is user-facing documentation and should reflect actual dependencies.

• Files reviewed: 21/25 changed files
• Comments generated: 2
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[qoderai]

👋 Review Summary

This PR solidifies OCRBench compatibility and introduces a well-scoped, versioned OmniDocBench v1.6 integration with pinned data, scorer runtime, and strong validation. The overall design fits cleanly into the existing adapter/sandbox architecture and shows good attention to reproducibility and safety.

🛡️ Key Risks & Issues

• OmniDocBench v1.6 sandbox result parsing is intentionally strict about metric names and value ranges. While this is helpful for catching corrupt runs, it means even benign upstream evaluator changes (e.g., adding a new metric or slightly adjusting ranges) will cause hard failures instead of a more descriptive compatibility error. Given the tight pinning, this is acceptable but makes the integration brittle to upstream evolution.
• The per-sample main_score_name for OmniDocBench v1.6 is derived from next(iter(metrics)), effectively tying the notion of the “main” score to dict insertion order. This works today because metrics are built from a fixed tuple, but future changes to the metric set or parsing could silently change which metric is highlighted downstream.
• The ModelScope single-file download path relies on ValueError as the cache-miss signal when local_files_only=True. If the underlying client changes its exception semantics, EvalScope will stop attempting the remote fallback and would surface the error directly, which could surprise users in new environments.
• The new OmniDocBench v1.6 adapter and sandbox scoring pipeline are not yet covered by targeted automated tests. Given the complexity (pinned revision, SHA-256 enforcement, Docker sandbox, strict metric validation), regressions or environment-specific issues are more likely to be caught only at runtime.

🧪 Verification Advice

• Beyond the existing CLI smoke test and benchmark tests mentioned in the PR description, consider adding focused tests for:
  • _load_annotation digest and structure checks, including wrong-digest and non-list cases.
  • parse_scoring_result success paths and all error branches (missing sentinel, invalid JSON, unexpected metrics, non-numeric/NaN/inf, out-of-range values).
  • match_score for OmniDocBench v1.6 using a monkeypatched sandbox executor to verify how metric dicts map to Score.value and main_score_name.
  • aggregate_scores behavior when some components are missing, and correctness of the Overall formula and component_page_denominators metadata.
• For OCRBench, add small unit tests to lock in the new behavior:
  • Case-sensitive matching for HME100k vs case-insensitive matching for other subsets.
  • Content ordering in record_to_sample (image then text, and text-only when no image is present).
• For the ModelScope hub, extend tests to cover cache-hit behavior with local_files_only=True, force_redownload=True, and forwarding of cache_dir so behavior remains stable if the client library evolves.

💡 Thoughts & Suggestions

• The decision to keep the legacy OmniDocBench TSV adapter under a legacy namespace while introducing a tightly pinned v1.6 adapter is a good balance between reproducibility and forward-looking evaluation; the updated descriptions make the compatibility story clear.
• For OmniDocBench v1.6, you might consider making the handling of unknown metrics in parse_scoring_result a bit more descriptive (e.g., turning “unexpected metrics” into an explicit version/compatibility hint) or limiting strictness to the metrics you actually consume while ignoring truly extra keys.
• It may be worth making the choice of main score explicit (e.g., preferring a specific page metric or omitting main_score_name and relying on aggregated overall) to avoid any surprises if the metric set changes later.
• Overall, the changes are thoughtfully scoped, and the focus on pinning, sandbox isolation, and clear compatibility boundaries is very helpful for users running high-stakes document parsing evaluations.

---

🤖 Generated by Qoder • View workflow run