From 40237d45c118e2bf89147c7f86a95b581e30c015 Mon Sep 17 00:00:00 2001 From: mbagdasarova Date: Fri, 17 Jul 2026 13:59:41 +0200 Subject: [PATCH 1/7] feat(spartqa): add SpartQA spatial-reasoning benchmark MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Native gym port of SpartQA (mteb/SpartQA) — a spatial-reasoning benchmark where the model is shown a scene-description query and must return the matching answer phrase. The scorer is pure rule-based: extract the model's "Final answer: " line, normalize (lowercase, strip punctuation, collapse whitespace), and compare against every accepted phrase in all_targets (exact match sets reward 1.0 and exact=true; substring containment sets reward 1.0 with exact=false; no match → 0.0). - resources_servers/spartqa/app.py — SpartqaResourcesServer.verify() extracts the answer via _extract_answer (strips reasoning blocks, matches "Final answer:" / "Answer:" patterns with fallbacks, cleans markdown artifacts), normalizes, and compares against all_targets (falling back to verifier_metadata when the nemo-evaluator native driver drops the top-level list field). compute_metrics reports mean_reward (= corpus accuracy), exact_match_rate, and parse_rate; get_key_metrics surfaces mean_reward and exact_match_rate. - prepare_spartqa.py joins the MTEB mteb/SpartQA queries/corpus/qrels splits into one row per query, porting build_records from byob_spartqa.py. The all_targets list is mirrored into verifier_metadata so the native driver can forward it intact to verify(). Prompt and scoring logic are ported verbatim from byob_spartqa.py so the metric is identical to the BYOB baseline. - spartqa.yaml wires the server + simple_agent for local eval/training; spartqa_serve.yaml serves the scorer for nemo-evaluator via the gym:// adapter (rule-based verify needs no model on the gym side). - No extra runtime deps — scoring is pure stdlib. Dataset prep requires the HF datasets package (prep-time only, not in requirements.txt). - 25 unit tests covering _extract_answer, _strip_reasoning, _normalize, verify() exact/contains/empty/multi-target/metadata-fallback paths, compute_metrics, get_key_metrics, _response_text, and acceptance parity against every example.jsonl row. Signed-off-by: mbagdasarova --- resources_servers/spartqa/README.md | 78 +++ resources_servers/spartqa/app.py | 242 ++++++++++ .../spartqa/configs/spartqa.yaml | 29 ++ .../spartqa/configs/spartqa_serve.yaml | 19 + resources_servers/spartqa/data/.gitignore | 2 + resources_servers/spartqa/data/example.jsonl | 5 + .../spartqa/data/example_metrics.json | 46 ++ .../spartqa/data/example_rollouts.jsonl | 5 + resources_servers/spartqa/prepare_spartqa.py | 149 ++++++ resources_servers/spartqa/requirements.txt | 4 + .../spartqa/tests/test_acceptance.py | 443 ++++++++++++++++++ resources_servers/spartqa/tests/test_app.py | 304 ++++++++++++ 12 files changed, 1326 insertions(+) create mode 100644 resources_servers/spartqa/README.md create mode 100644 resources_servers/spartqa/app.py create mode 100644 resources_servers/spartqa/configs/spartqa.yaml create mode 100644 resources_servers/spartqa/configs/spartqa_serve.yaml create mode 100644 resources_servers/spartqa/data/.gitignore create mode 100644 resources_servers/spartqa/data/example.jsonl create mode 100644 resources_servers/spartqa/data/example_metrics.json create mode 100644 resources_servers/spartqa/data/example_rollouts.jsonl create mode 100644 resources_servers/spartqa/prepare_spartqa.py create mode 100644 resources_servers/spartqa/requirements.txt create mode 100644 resources_servers/spartqa/tests/test_acceptance.py create mode 100644 resources_servers/spartqa/tests/test_app.py diff --git a/resources_servers/spartqa/README.md b/resources_servers/spartqa/README.md new file mode 100644 index 0000000000..0dd5b5dece --- /dev/null +++ b/resources_servers/spartqa/README.md @@ -0,0 +1,78 @@ +# SpartQA Resources Server + +Spatial-reasoning **answer generation**, ported from the nemo-evaluator BYOB +benchmark `spartqa` (`benchmarks/spartqa/byob_spartqa.py`). The model is shown a +spatial-reasoning query and must return the matching answer phrase, ending with +a `Final answer: ` line. The per-sample reward is `1.0` on an +exact-or-answer-containing match against any accepted answer phrase, else `0.0`. + +Source dataset: [`mteb/SpartQA`](https://huggingface.co/datasets/mteb/SpartQA) +(MTEB retrieval form — `queries` / `corpus` / `qrels` splits joined at prep +time). Upstream eval: `benchmarks/spartqa/byob_spartqa.py`. + +## Scoring + +`verify()` (ported verbatim from the BYOB scorer) extracts the model's final +answer (`_extract_answer` / `_strip_reasoning` / `_clean_candidate`), normalizes +it (`_normalize`: lowercase, strip punctuation, collapse whitespace), and +compares against every accepted phrase in `all_targets` (falling back to +`target`). A strict equality sets `exact`; a substring match sets the reward +without `exact`. Empty output scores `0.0` and never raises. + +## Metrics + +`compute_metrics` reports: + +- `mean_reward` — mean per-sample accuracy (also the reward). +- `exact_match_rate` — fraction with a strict (exact) match. +- `parse_rate` — fraction where a non-empty answer phrase was extracted. + +`get_key_metrics` surfaces `mean_reward` and `exact_match_rate`. + +> **Reasoning models:** `verify()` strips a leading `…` block +> before extracting the answer. + +## Prepare the dataset + +```bash +cd gym +python resources_servers/spartqa/prepare_spartqa.py --split test +``` + +This joins `mteb/SpartQA` (via the HF `datasets` library) and writes the +gitignored `data/spartqa_test.jsonl`. The committed `data/example.jsonl` is a +5-row smoke-test slice. + +## Example rollouts and metrics + +`data/example_rollouts.jsonl` and `data/example_metrics.json` are committed +and can be regenerated at any time with the scripts below (no servers needed): + +```bash +# Regenerate synthetic rollouts (rule-based scorer, no model call) +python resources_servers/spartqa/generate_example_rollouts.py + +# Regenerate dataset stats summary +python resources_servers/spartqa/generate_example_metrics.py + +# Inspect +tail -n 1 resources_servers/spartqa/data/example_rollouts.jsonl | jq .reward +cat resources_servers/spartqa/data/example_metrics.json | jq . +``` + +Note: row 2 (index 2) in the example rollouts is intentionally wrong (reward +0.0) to demonstrate a failed case; the remaining four rows score 1.0. + +## Run + +```bash +gym env start --resources-server spartqa --model-type vllm_model +``` + +No API keys are required — all scoring is rule-based. + +## Test + +```bash +gym env test --resources-server spartqa +``` diff --git a/resources_servers/spartqa/app.py b/resources_servers/spartqa/app.py new file mode 100644 index 0000000000..4abd41affb --- /dev/null +++ b/resources_servers/spartqa/app.py @@ -0,0 +1,242 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""SpartQA resources server — spatial reasoning as direct answer generation. + +Ported from the nemo-evaluator BYOB benchmark ``spartqa`` +(``benchmarks/spartqa/byob_spartqa.py``). The MTEB ``mteb/SpartQA`` retrieval +dataset is joined at prep time (``prepare_spartqa.py``) into one row per query +whose ``target`` is the accepted answer phrase (all accepted phrases in +``all_targets``). The model is shown the query and must return the matching +answer phrase, ending with a ``Final answer: `` line. + +The per-sample reward is ``1.0`` on an exact-or-answer-containing match against +any accepted answer, else ``0.0`` — so ``compute_metrics``'s mean-of-rewards +equals corpus accuracy. ``exact`` (strict match) and ``parsed`` (a non-empty +answer was extracted) ride on each row for downstream inspection. + +The prompt and answer-extraction / scoring logic are ported verbatim from +``byob_spartqa.py`` so the metric is identical. +""" + +from __future__ import annotations + +import re +import string +from typing import Any, Dict, List, Optional + +from pydantic import ConfigDict, Field + +from nemo_gym.base_resources_server import ( + BaseResourcesServerConfig, + BaseRunRequest, + BaseVerifyRequest, + BaseVerifyResponse, + SimpleResourcesServer, +) +from nemo_gym.openai_utils import NeMoGymResponse + + +PROMPT = """\ +Answer the spatial reasoning query below. +Return one concise final answer phrase. If the query gives answer options, +copy the matching option phrase exactly when possible. + +End your response with one line in this exact format: +Final answer: + +Query: +{question} +""" + + +# ── Answer extraction + normalization (verbatim from byob_spartqa.py) ────── + + +def _normalize(text: str) -> str: + table = str.maketrans("", "", string.punctuation) + normalized = text.strip().lower().translate(table) + return " ".join(normalized.split()) + + +def _strip_reasoning(text: str) -> str: + text = re.sub(r".*?", "", text, flags=re.IGNORECASE | re.DOTALL) + text = re.sub( + r"<\|channel\>thought\s*.*?", + "", + text, + flags=re.IGNORECASE | re.DOTALL, + ) + return text.strip() + + +def _clean_candidate(text: str) -> str: + text = text.strip() + text = re.sub(r"^[-*•\s]+", "", text) + text = re.sub(r"^\*+|\*+$", "", text).strip() + text = re.sub(r"", "", text, flags=re.IGNORECASE).strip() + return text.strip().strip("\"'` ") + + +def _extract_answer(text: str) -> str: + text = _strip_reasoning(text) + patterns = [ + r"(?:^|\n)\s*(?:[*_`#>\-\s]*)final\s+answer(?:\s+is)?\s*(?:[*_`\s])*[:\-]\s*(.+)", + r"(?:^|\n)\s*(?:[*_`#>\-\s]*)selected\s+answer(?:\s+is)?\s*(?:[*_`\s])*[:\-]\s*(.+)", + r"(?:^|\n)\s*(?:[*_`#>\-\s]*)answer(?:\s+is)?\s*(?:[*_`\s])*[:\-]\s*(.+)", + r"\b(?:the\s+)?(?:final\s+)?answer\s+(?:is|would\s+be|should\s+be)\s*[:\-]?\s*(.+)", + r"\bselected\s+(?:option|answer)\s+(?:is|would\s+be)\s*[:\-]?\s*(.+)", + ] + extracted = None + for pattern in patterns: + matches = list( + re.finditer(pattern, text, flags=re.IGNORECASE | re.DOTALL | re.MULTILINE) + ) + if matches: + extracted = matches[-1].group(1).strip() + break + if extracted is not None: + text = extracted + + lines = [_clean_candidate(line) for line in text.splitlines() if line.strip()] + lines = [ + line + for line in lines + if line and _normalize(line) not in {"final answer", "answer"} + ] + if not lines: + return "" + + if extracted is None and len(lines) > 1: + first = _normalize(lines[0]) + if first.startswith( + ("thinking process", "analysis", "the user wants", "we need") + ): + return lines[-1] + return lines[0] + + +def _response_text(response: NeMoGymResponse) -> str: + """Best-effort extraction of the assistant text from a NeMoGymResponse.""" + text = getattr(response, "output_text", None) + if isinstance(text, str) and text: + return text + parts: List[str] = [] + for item in getattr(response, "output", None) or []: + if getattr(item, "type", None) != "message": + continue + content = getattr(item, "content", "") + if isinstance(content, str): + parts.append(content) + continue + for c in content or []: + t = c.get("text") if isinstance(c, dict) else getattr(c, "text", None) + if isinstance(t, str): + parts.append(t) + return "".join(parts) + + +# ── Request / response shapes ───────────────────────────────────────────── + + +class SpartqaResourcesServerConfig(BaseResourcesServerConfig): + name: str = "spartqa" + + +class SpartqaRunRequest(BaseRunRequest): + model_config = ConfigDict(extra="allow") + + # The first accepted answer phrase, and all accepted phrases. ``target`` + # (a scalar) survives the nemo-evaluator ``gym://...protocol=native`` driver, + # which forwards a row's top-level scalar fields onto ``/verify`` but DROPS + # list/dict fields (``all_targets`` never arrives that way). The full accepted + # set therefore also rides in ``verifier_metadata``, which the driver forwards + # intact; verify() falls back to it so all phrases are always available. + target: str = "" + all_targets: List[str] = Field(default_factory=list) + verifier_metadata: Optional[Dict[str, Any]] = None + + +class SpartqaVerifyRequest(SpartqaRunRequest, BaseVerifyRequest): + pass + + +class SpartqaVerifyResponse(BaseVerifyResponse): + model_config = ConfigDict(extra="allow") + + exact: bool = False + parsed: bool = False + extracted: str = "" + + +class SpartqaResourcesServer(SimpleResourcesServer): + config: SpartqaResourcesServerConfig + + async def verify(self, body: SpartqaVerifyRequest) -> SpartqaVerifyResponse: + prediction = _extract_answer(_response_text(body.response)) + prediction_norm = _normalize(prediction) + # Prefer the explicit list; fall back to verifier_metadata (the only path + # that survives the native driver) and finally the scalar target. + meta = body.verifier_metadata or {} + targets = body.all_targets or meta.get("all_targets") or [ + body.target or meta.get("target", "") + ] + + exact = False + contains = False + for target in targets: + target_norm = _normalize(str(target)) + if not target_norm: + continue + if prediction_norm == target_norm: + exact = True + contains = True + break + if target_norm in prediction_norm: + contains = True + + return SpartqaVerifyResponse( + **body.model_dump(), + reward=1.0 if (exact or contains) else 0.0, + exact=exact, + parsed=bool(prediction_norm), + extracted=prediction[:200], + ) + + # --- aggregation ----------------------------------------------------- + + def compute_metrics(self, tasks: List[List[Dict[str, Any]]]) -> Dict[str, Any]: + rows = [r for task_rollouts in tasks for r in task_rollouts] + if not rows: + return {} + + metrics: Dict[str, Any] = {} + rewards = [r["reward"] for r in rows if isinstance(r.get("reward"), (int, float))] + if rewards: + metrics["mean_reward"] = sum(rewards) / len(rewards) + metrics["count"] = len(rewards) + metrics["exact_match_rate"] = sum(1 for r in rows if r.get("exact")) / len(rows) + metrics["parse_rate"] = sum(1 for r in rows if r.get("parsed")) / len(rows) + return metrics + + def get_key_metrics(self, agent_metrics: Dict[str, Any]) -> Dict[str, Any]: + return { + k: agent_metrics[k] + for k in ("mean_reward", "exact_match_rate") + if k in agent_metrics + } + + +if __name__ == "__main__": + SpartqaResourcesServer.run_webserver() diff --git a/resources_servers/spartqa/configs/spartqa.yaml b/resources_servers/spartqa/configs/spartqa.yaml new file mode 100644 index 0000000000..6b6949d055 --- /dev/null +++ b/resources_servers/spartqa/configs/spartqa.yaml @@ -0,0 +1,29 @@ +spartqa: + resources_servers: + spartqa: + entrypoint: app.py + domain: reasoning + verified: false + description: SpartQA spatial-reasoning answer generation; reward = 1 on an exact-or-answer-containing match against any accepted answer phrase. + value: Improve spatial and relational reasoning over described scenes. + +spartqa_simple_agent: + responses_api_agents: + simple_agent: + entrypoint: app.py + resources_server: + type: resources_servers + name: spartqa + model_server: + type: responses_api_models + name: policy_model + datasets: + - name: example + type: example + jsonl_fpath: resources_servers/spartqa/data/example.jsonl + license: CC-BY-SA-4.0 + # Build the test split with `python resources_servers/spartqa/prepare_spartqa.py`. + - name: test + type: validation + jsonl_fpath: resources_servers/spartqa/data/spartqa_test.jsonl + license: CC-BY-SA-4.0 diff --git a/resources_servers/spartqa/configs/spartqa_serve.yaml b/resources_servers/spartqa/configs/spartqa_serve.yaml new file mode 100644 index 0000000000..5eb6ab4d89 --- /dev/null +++ b/resources_servers/spartqa/configs/spartqa_serve.yaml @@ -0,0 +1,19 @@ +# Serve ONLY the spartqa resources server (the /verify scorer), for driving +# from nemo-evaluator via the gym:// environment adapter: +# +# ng_run "+config_paths=[resources_servers/spartqa/configs/spartqa_serve.yaml]" +# +# SpartQA needs no model server — verify() extracts the model's final answer +# locally and scores the exact-or-contains match deterministically. +# nemo-evaluator owns the policy model and generation; it POSTs the model's +# answer to this server's /verify. See README.md. +spartqa: + resources_servers: + spartqa: + entrypoint: app.py + host: 0.0.0.0 + port: 8001 + domain: other + verified: false + description: SpartQA spatial-reasoning answer generation; reward = 1 on an exact-or-answer-containing match against any accepted answer phrase. + value: Improve spatial and relational reasoning over described scenes. diff --git a/resources_servers/spartqa/data/.gitignore b/resources_servers/spartqa/data/.gitignore new file mode 100644 index 0000000000..f8d4c51374 --- /dev/null +++ b/resources_servers/spartqa/data/.gitignore @@ -0,0 +1,2 @@ +spartqa_test.jsonl + diff --git a/resources_servers/spartqa/data/example.jsonl b/resources_servers/spartqa/data/example.jsonl new file mode 100644 index 0000000000..3ee960e1d1 --- /dev/null +++ b/resources_servers/spartqa/data/example.jsonl @@ -0,0 +1,5 @@ +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nThere are two blocks, A and B. A small circle is above a large square in block A. A triangle is to the right of the square. Where is the triangle relative to the circle?\nOptions: below the circle; above the circle; left of the circle\n"}]}, "target": "below the circle", "all_targets": ["below the circle", "under the circle"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "below the circle", "all_targets": ["below the circle", "under the circle"]}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nA red ball is to the left of a blue box. The blue box is to the left of a green cup. Which object is farthest to the right?\nOptions: red ball; blue box; green cup\n"}]}, "target": "green cup", "all_targets": ["green cup"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "green cup", "all_targets": ["green cup"]}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nIn the grid, a star is directly north of a moon, and the moon is directly north of a sun. Is the star north of the sun?\nOptions: yes; no\n"}]}, "target": "yes", "all_targets": ["yes"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "yes", "all_targets": ["yes"]}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nA cat sits on a mat. A dog stands to the right of the cat. A bird is above the dog. Relative to the cat, in which direction is the bird?\nOptions: upper right; upper left; lower right\n"}]}, "target": "upper right", "all_targets": ["upper right", "above and to the right"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "upper right", "all_targets": ["upper right", "above and to the right"]}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nBox 1 contains a big black square touching the top edge. A medium white circle is below the square. What is at the top of Box 1?\nOptions: the big black square; the medium white circle\n"}]}, "target": "the big black square", "all_targets": ["the big black square"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "the big black square", "all_targets": ["the big black square"]}} diff --git a/resources_servers/spartqa/data/example_metrics.json b/resources_servers/spartqa/data/example_metrics.json new file mode 100644 index 0000000000..355a62c9dc --- /dev/null +++ b/resources_servers/spartqa/data/example_metrics.json @@ -0,0 +1,46 @@ +{ + "name": "example", + "type": "example", + "jsonl_fpath": "resources_servers/spartqa/data/example.jsonl", + "num_repeats": 1, + "source": null, + "gitlab_identifier": null, + "huggingface_identifier": "mteb/SpartQA", + "license": "CC-BY-SA-4.0", + "Number of examples": 5, + "Number of tools": { + "Total # non-null values": 0, + "Average": 0.0, + "Min": 0.0, + "Max": 0.0, + "Standard deviation": 0.0 + }, + "Json-dumped number of words (proxy for token count)": { + "Total # non-null values": 5, + "Average": 99.0, + "Min": 83.0, + "Max": 113.0, + "Standard deviation": 11.489 + }, + "Number of turns": { + "Total # non-null values": 5, + "Average": 1.0, + "Min": 1.0, + "Max": 1.0, + "Standard deviation": 0.0 + }, + "Temperature": { + "Total # non-null values": 0, + "Average": 0.0, + "Min": 0.0, + "Max": 0.0, + "Standard deviation": 0.0 + }, + "Number of accepted answers (all_targets)": { + "Total # non-null values": 5, + "Average": 1.4, + "Min": 1.0, + "Max": 2.0, + "Standard deviation": 0.548 + } +} diff --git a/resources_servers/spartqa/data/example_rollouts.jsonl b/resources_servers/spartqa/data/example_rollouts.jsonl new file mode 100644 index 0000000000..b4d6f8fcf2 --- /dev/null +++ b/resources_servers/spartqa/data/example_rollouts.jsonl @@ -0,0 +1,5 @@ +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nThere are two blocks, A and B. A small circle is above a large square in block A. A triangle is to the right of the square. Where is the triangle relative to the circle?\nOptions: below the circle; above the circle; left of the circle\n"}]}, "target": "below the circle", "all_targets": ["below the circle", "under the circle"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "below the circle", "all_targets": ["below the circle", "under the circle"]}, "response": {"id": "resp_029628e2a4ef4e149324bbda42b6900b", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_931188755ee742f6b87a93284b8c996b", "content": [{"annotations": [], "text": "The triangle is to the right of the square, which is under the circle.\nFinal answer: below the circle", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "below the circle", "_ng_task_index": 0, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nA red ball is to the left of a blue box. The blue box is to the left of a green cup. Which object is farthest to the right?\nOptions: red ball; blue box; green cup\n"}]}, "target": "green cup", "all_targets": ["green cup"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "green cup", "all_targets": ["green cup"]}, "response": {"id": "resp_b7ca67217aca4a93b31d050380979c9b", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_be94029e039d4284a7bd939ab5caa576", "content": [{"annotations": [], "text": "Following the left-to-right order red, blue, green.\nFinal answer: green cup", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "green cup", "_ng_task_index": 1, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nIn the grid, a star is directly north of a moon, and the moon is directly north of a sun. Is the star north of the sun?\nOptions: yes; no\n"}]}, "target": "yes", "all_targets": ["yes"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "yes", "all_targets": ["yes"]}, "response": {"id": "resp_4104f8c263fc46f3a7d468bf443b3852", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_4b559bd98a0244478e762d5653fce0a8", "content": [{"annotations": [], "text": "The star is north of the moon, and the sun is south of both.\nFinal answer: no", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 0.0, "exact": false, "parsed": true, "extracted": "no", "_ng_task_index": 2, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nA cat sits on a mat. A dog stands to the right of the cat. A bird is above the dog. Relative to the cat, in which direction is the bird?\nOptions: upper right; upper left; lower right\n"}]}, "target": "upper right", "all_targets": ["upper right", "above and to the right"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "upper right", "all_targets": ["upper right", "above and to the right"]}, "response": {"id": "resp_474e797222094d398ee6393293482316", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_333423e11aff4c77b8ff30b1be1c9e9e", "content": [{"annotations": [], "text": "The bird is above the dog, and the dog is right of the cat.\nFinal answer: upper right", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "upper right", "_ng_task_index": 3, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nBox 1 contains a big black square touching the top edge. A medium white circle is below the square. What is at the top of Box 1?\nOptions: the big black square; the medium white circle\n"}]}, "target": "the big black square", "all_targets": ["the big black square"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "the big black square", "all_targets": ["the big black square"]}, "response": {"id": "resp_9c312ab496a9491d9721b74f30aa868f", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_4b10f237db564082bc78a18363e5225a", "content": [{"annotations": [], "text": "The square touches the top edge, the circle is below it.\nFinal answer: the big black square", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "the big black square", "_ng_task_index": 4, "_ng_rollout_index": 0} diff --git a/resources_servers/spartqa/prepare_spartqa.py b/resources_servers/spartqa/prepare_spartqa.py new file mode 100644 index 0000000000..b9c1558ba9 --- /dev/null +++ b/resources_servers/spartqa/prepare_spartqa.py @@ -0,0 +1,149 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Build the SpartQA Gym dataset from the public ``mteb/SpartQA`` HF dataset. + +Ports ``benchmarks/spartqa/byob_spartqa.py::build_records``: joins the MTEB +``queries`` / ``corpus`` / ``qrels`` splits into one row per query whose +``target`` is the accepted answer phrase (all accepted phrases in +``all_targets``), then renders the shared ``PROMPT`` and writes a Gym task per +query. ``target`` / ``all_targets`` ride along as extra fields consumed by +``app.py``'s verify(). + +Requires the ``datasets`` package (HF) at prep time only. + +Usage:: + + python resources_servers/spartqa/prepare_spartqa.py --split test \\ + --output data/spartqa_test.jsonl +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any, List + +from app import PROMPT + + +_HF_DATASET = "mteb/SpartQA" +_DEFAULT_SPLIT = "test" +_AGENT = {"type": "responses_api_agents", "name": "spartqa_simple_agent"} + + +def _load_split(config: str, split: str): + try: + from datasets import load_dataset + except ImportError as exc: # pragma: no cover - environment-dependent + raise SystemExit( + "The 'datasets' package is required to build mteb/SpartQA. " + "Install it (pip install datasets) or stage a prepared JSONL." + ) from exc + return load_dataset(_HF_DATASET, config, split=split) + + +def _unique_preserve_order(values: List[str]) -> List[str]: + seen: set[str] = set() + result: List[str] = [] + for value in values: + key = value.strip().casefold() + if key and key not in seen: + seen.add(key) + result.append(value.strip()) + return result + + +def build_records(split: str = _DEFAULT_SPLIT) -> List[dict[str, Any]]: + """Join the MTEB ``queries`` / ``corpus`` / ``qrels`` splits into rows.""" + corpus = { + row["_id"]: str(row["text"]).strip() + for row in _load_split("corpus", split) + if str(row.get("text", "")).strip() + } + queries = { + row["_id"]: str(row["text"]).strip() + for row in _load_split("queries", split) + if str(row.get("text", "")).strip() + } + + qrels_by_query: dict[str, List[str]] = {} + for row in _load_split("qrels", split): + try: + score = int(row.get("score", 1)) + except (TypeError, ValueError): + score = 1 + if score <= 0: + continue + qrels_by_query.setdefault(row["query-id"], []).append(row["corpus-id"]) + + records: List[dict[str, Any]] = [] + for query_id in sorted(queries): + answer_ids = [ + doc_id for doc_id in qrels_by_query.get(query_id, []) if doc_id in corpus + ] + answers = _unique_preserve_order([corpus[doc_id] for doc_id in answer_ids]) + if not answers: + continue + records.append( + { + "question": queries[query_id], + "target": answers[0], + "all_targets": answers, + } + ) + return records + + +def _to_task(record: dict[str, Any]) -> dict[str, Any]: + return { + "responses_create_params": { + "input": [{"role": "user", "content": PROMPT.format(question=record["question"])}] + }, + "target": record["target"], + "all_targets": record["all_targets"], + # ``all_targets`` is a list and is dropped by the nemo-evaluator + # ``gym://...protocol=native`` driver (it forwards only top-level scalar + # fields to /verify). Mirror the full accepted set into verifier_metadata, + # which the driver forwards intact, so verify() sees every phrase. + "verifier_metadata": { + "target": record["target"], + "all_targets": record["all_targets"], + }, + "agent_ref": _AGENT, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description="Build the SpartQA Gym dataset.") + parser.add_argument("--split", default=_DEFAULT_SPLIT, help="MTEB split (default: test)") + parser.add_argument( + "--output", + default=str(Path(__file__).parent / "data" / "spartqa_test.jsonl"), + help="Output JSONL path", + ) + args = parser.parse_args() + + records = build_records(args.split) + out_path = Path(args.output) + out_path.parent.mkdir(parents=True, exist_ok=True) + with out_path.open("w", encoding="utf-8") as fh: + for record in records: + fh.write(json.dumps(_to_task(record), ensure_ascii=False) + "\n") + print(f"SpartQA: wrote {len(records)} rows -> {out_path}") + + +if __name__ == "__main__": + main() diff --git a/resources_servers/spartqa/requirements.txt b/resources_servers/spartqa/requirements.txt new file mode 100644 index 0000000000..9fd546ac45 --- /dev/null +++ b/resources_servers/spartqa/requirements.txt @@ -0,0 +1,4 @@ +-e nemo-gym[dev] @ ../../ +# SpartQA scoring is pure-stdlib (re/string) — no extra runtime dependencies. +# Building the dataset (prepare_spartqa.py) additionally needs `datasets` (HF), +# used only at prep time. diff --git a/resources_servers/spartqa/tests/test_acceptance.py b/resources_servers/spartqa/tests/test_acceptance.py new file mode 100644 index 0000000000..b02d6fa7e4 --- /dev/null +++ b/resources_servers/spartqa/tests/test_acceptance.py @@ -0,0 +1,443 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Acceptance tests for the SpartQA gym benchmark. + +These verify the approved user story's acceptance criteria against the built +implementation. They are independent of the builder's unit tests +(``tests/test_app.py``): fixtures are re-declared here and data files are loaded +by path relative to this test file. One ``test_ac_*`` per acceptance criterion, +one ``test_edge_*`` per story edge case. +""" + +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path +from typing import Any, List +from unittest.mock import MagicMock + +import pytest +import yaml +from pytest import approx + +from nemo_gym.openai_utils import ( + NeMoGymResponse, + NeMoGymResponseCreateParamsNonStreaming, + NeMoGymResponseOutputMessage, + NeMoGymResponseOutputText, +) +from nemo_gym.server_utils import ServerClient +from resources_servers.spartqa import app as spartqa_app +from resources_servers.spartqa.app import ( + PROMPT, + SpartqaResourcesServer, + SpartqaResourcesServerConfig, + SpartqaVerifyRequest, + SpartqaVerifyResponse, + SimpleResourcesServer, + _clean_candidate, + _extract_answer, + _normalize, + _strip_reasoning, +) + +# ── Paths (relative to this test file) ───────────────────────────────────── + +_SERVER_DIR = Path(__file__).resolve().parent.parent +_EXAMPLE_JSONL = _SERVER_DIR / "data" / "example.jsonl" +_ROLLOUTS_JSONL = _SERVER_DIR / "data" / "example_rollouts.jsonl" +_CONFIG_YAML = _SERVER_DIR / "configs" / "spartqa.yaml" +_PREPARE_PY = _SERVER_DIR / "prepare_spartqa.py" + + +# ── Fixture builders (mirror tests/test_app.py idioms) ───────────────────── + + +def _make_response(text: str) -> NeMoGymResponse: + return NeMoGymResponse( + id="resp", + created_at=0.0, + model="policy_model", + object="response", + output=[ + NeMoGymResponseOutputMessage( + id="msg", + content=[NeMoGymResponseOutputText(annotations=[], text=text, type="output_text")], + role="assistant", + status="completed", + type="message", + ) + ], + parallel_tool_calls=False, + tool_choice="none", + tools=[], + ) + + +def _config() -> SpartqaResourcesServerConfig: + return SpartqaResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint="", name="spartqa") + + +def _make_request( + text: str, *, target: str = "", all_targets: list[str] | None = None +) -> SpartqaVerifyRequest: + return SpartqaVerifyRequest( + responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), + response=_make_response(text), + target=target, + all_targets=all_targets or [], + ) + + +def _server() -> SpartqaResourcesServer: + return SpartqaResourcesServer(config=_config(), server_client=MagicMock(spec=ServerClient)) + + +def _read_jsonl(path: Path) -> List[dict[str, Any]]: + return [json.loads(line) for line in path.read_text().splitlines() if line.strip()] + + +def _load_prepare_module() -> Any: + """Load ``prepare_spartqa.py`` (which does ``from app import PROMPT``). + + The prepare script imports the sibling ``app`` module by bare name, so the + already-imported ``resources_servers.spartqa.app`` is registered under + ``app`` for the duration of the load, then removed to avoid polluting + ``sys.modules`` for other servers' prepare scripts. + """ + saved = sys.modules.get("app") + sys.modules["app"] = spartqa_app + try: + spec = importlib.util.spec_from_file_location("spartqa_prepare", _PREPARE_PY) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + finally: + if saved is not None: + sys.modules["app"] = saved + else: + sys.modules.pop("app", None) + + +# ── AC1: server subclasses SimpleResourcesServer, async verify -> reward ──── + + +class TestAcServerContract: + def test_ac_subclasses_simple_resources_server(self) -> None: + assert issubclass(SpartqaResourcesServer, SimpleResourcesServer) + + async def test_ac_verify_is_async_and_returns_reward(self) -> None: + result = await _server().verify(_make_request("Final answer: yes", target="yes")) + assert isinstance(result, SpartqaVerifyResponse) + assert hasattr(result, "reward") + assert result.reward == approx(1.0) + + +# ── AC2: metric parity with byob ──────────────────────────────────────────── + +# Shared fixtures: (response_text, target, all_targets, expected_correct). +_PARITY_FIXTURES: list[tuple[str, str, list[str], bool]] = [ + ("Final answer: green cup", "green cup", ["green cup"], True), + ("Final answer: the green cup on the table", "green cup", ["green cup"], True), + ("Final answer: red ball", "green cup", ["green cup"], False), + ( + "Final answer: under the circle", + "below the circle", + ["below the circle", "under the circle"], + True, + ), + (" ", "green cup", ["green cup"], False), + ("the star is northFinal answer: yes", "yes", ["yes"], True), +] + + +class TestAcMetricParity: + def test_ac_prompt_text_matches_byob(self) -> None: + # PROMPT is ported verbatim; assert the exact contract text the prep + # step renders (the query slot plus the required final-answer line). + assert "Final answer: " in PROMPT + assert PROMPT.startswith("Answer the spatial reasoning query below.") + assert PROMPT.rstrip().endswith("{question}") + + def test_ac_normalize_matches_byob_logic(self) -> None: + assert _normalize(" The Big, BLACK Square!! ") == "the big black square" + assert _normalize("") == "" + + def test_ac_strip_reasoning_matches_byob_logic(self) -> None: + assert _strip_reasoning("hiddenvisible") == "visible" + assert _strip_reasoning("plain") == "plain" + + def test_ac_clean_candidate_matches_byob_logic(self) -> None: + assert _clean_candidate("- *green cup*") == "green cup" + assert _clean_candidate(' "yes" ') == "yes" + + def test_ac_extract_answer_matches_byob_logic(self) -> None: + assert _extract_answer("Reasoning.\nFinal answer: below the circle") == "below the circle" + assert _extract_answer("xFinal answer: yes") == "yes" + assert _extract_answer(" ") == "" + + @pytest.mark.parametrize("text,target,all_targets,expected", _PARITY_FIXTURES) + async def test_ac_scoring_matches_byob_expected( + self, text: str, target: str, all_targets: list[str], expected: bool + ) -> None: + result = await _server().verify( + _make_request(text, target=target, all_targets=all_targets) + ) + assert bool(result.reward) is expected + + async def test_ac_agrees_with_byob_module_when_importable(self) -> None: + # If nemo_evaluator (byob's hard dependency) is installed, compare the + # ported helpers and scorer against the byob source of truth directly. + pytest.importorskip("nemo_evaluator") + import benchmarks.spartqa.byob_spartqa as byob + from nemo_evaluator import ScorerInput + + server = _server() + for text, target, all_targets, _expected in _PARITY_FIXTURES: + assert byob._normalize(text) == _normalize(text) + assert byob._strip_reasoning(text) == _strip_reasoning(text) + assert byob._extract_answer(text) == _extract_answer(text) + byob_correct = byob.spartqa( + ScorerInput( + response=text, target=target, metadata={"all_targets": all_targets} + ) + )["correct"] + result = await server.verify( + _make_request(text, target=target, all_targets=all_targets) + ) + assert bool(result.reward) == bool(byob_correct) + + +# ── AC3: reward is strictly 1.0 or 0.0 ───────────────────────────────────── + + +class TestAcBinaryReward: + @pytest.mark.parametrize( + "text,target,all_targets", + [ + ("Final answer: green cup", "green cup", ["green cup"]), + ("Final answer: the green cup on the table", "green cup", ["green cup"]), + ("Final answer: red ball", "green cup", ["green cup"]), + ("", "green cup", ["green cup"]), + ("no marker at all", "green cup", ["green cup"]), + ("Final answer: under the circle", "below the circle", ["below the circle", "under the circle"]), + ], + ) + async def test_ac_reward_is_binary( + self, text: str, target: str, all_targets: list[str] + ) -> None: + result = await _server().verify( + _make_request(text, target=target, all_targets=all_targets) + ) + assert result.reward in {0.0, 1.0} + + +# ── AC4: all_targets read from field; falls back to [target] ──────────────── + + +class TestAcAllTargets: + async def test_ac_uses_all_targets_field(self) -> None: + # target does not match; a non-first accepted phrase does. + result = await _server().verify( + _make_request( + "Final answer: under the circle", + target="something else", + all_targets=["something else", "under the circle"], + ) + ) + assert result.reward == approx(1.0) + + async def test_ac_falls_back_to_target_when_all_targets_empty(self) -> None: + result = await _server().verify( + _make_request("Final answer: yes", target="yes", all_targets=[]) + ) + assert result.reward == approx(1.0) + + async def test_ac_falls_back_to_target_when_all_targets_absent(self) -> None: + # all_targets defaults to [] when not supplied -> uses [target]. + request = SpartqaVerifyRequest( + responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), + response=_make_response("Final answer: green cup"), + target="green cup", + ) + result = await _server().verify(request) + assert result.all_targets == [] + assert result.reward == approx(1.0) + + +# ── AC5: response carries exact, parsed, extracted extras ─────────────────── + + +class TestAcExtraFields: + async def test_ac_extra_fields_present(self) -> None: + result = await _server().verify( + _make_request("Final answer: green cup", target="green cup") + ) + assert result.exact is True + assert result.parsed is True + assert result.extracted == "green cup" + + async def test_ac_exact_false_on_contains_only(self) -> None: + result = await _server().verify( + _make_request("Final answer: the green cup on the table", target="green cup") + ) + assert result.reward == approx(1.0) + assert result.exact is False + assert result.parsed is True + + +# ── AC6: empty / whitespace output -> reward 0.0, no exception ─────────────── + + +class TestAcEmptyOutput: + @pytest.mark.parametrize("text", ["", " ", "\n\t "]) + async def test_ac_empty_output_scores_zero(self, text: str) -> None: + result = await _server().verify(_make_request(text, target="green cup")) + assert result.reward == approx(0.0) + assert result.parsed is False + assert result.exact is False + + +# ── AC7: example.jsonl row shape ──────────────────────────────────────────── + + +class TestAcExampleDataset: + def test_ac_example_rows_conform(self) -> None: + rows = _read_jsonl(_EXAMPLE_JSONL) + assert len(rows) >= 5 + for row in rows: + params = row["responses_create_params"] + messages = params["input"] + assert isinstance(messages, list) and messages + assert any(m.get("role") == "user" for m in messages) + assert isinstance(row["target"], str) and row["target"] + assert isinstance(row["all_targets"], list) and row["all_targets"] + assert row["agent_ref"]["name"] == "spartqa_simple_agent" + + +# ── AC8: config wires server + agent ──────────────────────────────────────── + + +class TestAcConfig: + def test_ac_config_parses_and_wires_server_and_agent(self) -> None: + config = yaml.safe_load(_CONFIG_YAML.read_text()) + assert "spartqa" in config + assert "spartqa" in config["spartqa"]["resources_servers"] + assert "spartqa_simple_agent" in config + agent = config["spartqa_simple_agent"]["responses_api_agents"]["simple_agent"] + assert agent["resources_server"]["name"] == "spartqa" + + +# ── AC9: prepare_spartqa.py record-building logic ─────────────────────────── + + +class TestAcPrepareScript: + def test_ac_prepare_defines_build_helpers(self) -> None: + prepare = _load_prepare_module() + assert callable(prepare.build_records) + assert callable(prepare._unique_preserve_order) + + def test_ac_unique_preserve_order_dedupes_casefold_preserving_order(self) -> None: + prepare = _load_prepare_module() + result = prepare._unique_preserve_order( + ["Below the circle", " below the circle ", "Under the Circle", "", " "] + ) + assert result == ["Below the circle", "Under the Circle"] + + +# ── AC10: round-trip over example.jsonl -> reward 1.0 ─────────────────────── + + +class TestAcRoundTrip: + async def test_ac_example_targets_score_one(self) -> None: + server = _server() + rows = _read_jsonl(_EXAMPLE_JSONL) + assert rows + for row in rows: + result = await server.verify( + _make_request( + f"Final answer: {row['target']}", + target=row["target"], + all_targets=row["all_targets"], + ) + ) + assert result.reward == approx(1.0), row["target"] + + +# ── Committed rollouts are self-consistent with the scorer ────────────────── + + +class TestAcRolloutsSelfConsistent: + async def test_ac_rollouts_reproduce_committed_fields(self) -> None: + server = _server() + rows = _read_jsonl(_ROLLOUTS_JSONL) + assert rows + for row in rows: + text = row["response"]["output"][0]["content"][0]["text"] + result = await server.verify( + _make_request(text, target=row["target"], all_targets=row["all_targets"]) + ) + assert result.reward == approx(row["reward"]), text + assert result.exact is row["exact"] + assert result.parsed is row["parsed"] + assert result.extracted == row["extracted"] + + +# ── Story edge cases ──────────────────────────────────────────────────────── + + +class TestEdgeCases: + async def test_edge_reasoning_wrapped_output_is_stripped_and_scores(self) -> None: + result = await _server().verify( + _make_request( + "the star is north of the moonFinal answer: yes", + target="yes", + ) + ) + assert result.reward == approx(1.0) + assert result.extracted == "yes" + + async def test_edge_multiple_answers_any_match_scores_one(self) -> None: + result = await _server().verify( + _make_request( + "Final answer: above and to the right", + target="upper right", + all_targets=["upper right", "above and to the right"], + ) + ) + assert result.reward == approx(1.0) + + async def test_edge_exact_true_only_on_strict_equality(self) -> None: + strict = await _server().verify( + _make_request( + "Final answer: above and to the right", + target="upper right", + all_targets=["upper right", "above and to the right"], + ) + ) + assert strict.exact is True + + loose = await _server().verify( + _make_request( + "Final answer: it is above and to the right of the cat", + target="upper right", + all_targets=["upper right", "above and to the right"], + ) + ) + assert loose.reward == approx(1.0) + assert loose.exact is False diff --git a/resources_servers/spartqa/tests/test_app.py b/resources_servers/spartqa/tests/test_app.py new file mode 100644 index 0000000000..16b911fad4 --- /dev/null +++ b/resources_servers/spartqa/tests/test_app.py @@ -0,0 +1,304 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import json +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest +from pytest import approx + +from nemo_gym.openai_utils import ( + NeMoGymResponse, + NeMoGymResponseCreateParamsNonStreaming, + NeMoGymResponseOutputMessage, + NeMoGymResponseOutputText, +) +from nemo_gym.server_utils import ServerClient +from resources_servers.spartqa.app import ( + SpartqaResourcesServer, + SpartqaResourcesServerConfig, + SpartqaVerifyRequest, + _extract_answer, + _normalize, + _response_text, + _strip_reasoning, +) + + +_EXAMPLE_JSONL = Path(__file__).resolve().parent.parent / "data" / "example.jsonl" + + +def _make_response(text: str) -> NeMoGymResponse: + return NeMoGymResponse( + id="resp", + created_at=0.0, + model="policy_model", + object="response", + output=[ + NeMoGymResponseOutputMessage( + id="msg", + content=[NeMoGymResponseOutputText(annotations=[], text=text, type="output_text")], + role="assistant", + status="completed", + type="message", + ) + ], + parallel_tool_calls=False, + tool_choice="none", + tools=[], + ) + + +def _config() -> SpartqaResourcesServerConfig: + return SpartqaResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint="", name="spartqa") + + +def _make_request( + text: str, + *, + target: str = "", + all_targets: list[str] | None = None, + verifier_metadata: dict | None = None, +) -> SpartqaVerifyRequest: + return SpartqaVerifyRequest( + responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), + response=_make_response(text), + target=target, + all_targets=all_targets or [], + verifier_metadata=verifier_metadata, + ) + + +def _server() -> SpartqaResourcesServer: + return SpartqaResourcesServer(config=_config(), server_client=MagicMock(spec=ServerClient)) + + +# ── Pure helpers ───────────────────────────────────────────────────────── + + +class TestExtractAnswer: + def test_pulls_phrase_after_final_answer(self) -> None: + assert _extract_answer("Reasoning here.\nFinal answer: below the circle") == ( + "below the circle" + ) + + def test_strips_think_reasoning(self) -> None: + text = "the star is northFinal answer: yes" + assert _extract_answer(text) == "yes" + + def test_thinking_process_prefix_returns_last_line(self) -> None: + # No explicit "Final answer:" marker; a multi-line "thinking process" + # prefix means the answer is the last non-empty line. + text = "Thinking process: I consider the layout\ngreen cup" + assert _extract_answer(text) == "green cup" + + def test_empty_returns_empty(self) -> None: + assert _extract_answer(" ") == "" + + +class TestStripReasoning: + def test_removes_think_block(self) -> None: + assert _strip_reasoning("hiddenvisible") == "visible" + + def test_passthrough(self) -> None: + assert _strip_reasoning("plain") == "plain" + + +class TestNormalize: + def test_lowercases_strips_punctuation_collapses_ws(self) -> None: + assert _normalize(" The Big, BLACK Square!! ") == "the big black square" + + def test_empty(self) -> None: + assert _normalize("") == "" + + +# ── verify() ───────────────────────────────────────────────────────────── + + +class TestVerify: + async def test_exact_match(self) -> None: + result = await _server().verify( + _make_request("Final answer: green cup", target="green cup") + ) + assert result.reward == approx(1.0) + assert result.exact is True + assert result.parsed is True + assert result.extracted == "green cup" + + async def test_contains_match_not_exact(self) -> None: + result = await _server().verify( + _make_request("Final answer: the green cup on the table", target="green cup") + ) + assert result.reward == approx(1.0) + assert result.exact is False + + async def test_no_match(self) -> None: + result = await _server().verify( + _make_request("Final answer: red ball", target="green cup") + ) + assert result.reward == approx(0.0) + assert result.exact is False + assert result.parsed is True + + async def test_empty_output_reward_zero_no_raise(self) -> None: + result = await _server().verify(_make_request(" ", target="green cup")) + assert result.reward == approx(0.0) + assert result.parsed is False + + async def test_all_targets_empty_falls_back_to_target(self) -> None: + result = await _server().verify( + _make_request("Final answer: yes", target="yes", all_targets=[]) + ) + assert result.reward == approx(1.0) + + async def test_multiple_targets_any_match(self) -> None: + result = await _server().verify( + _make_request( + "Final answer: under the circle", + target="below the circle", + all_targets=["below the circle", "under the circle"], + ) + ) + assert result.reward == approx(1.0) + assert result.exact is True + + async def test_empty_targets_are_skipped(self) -> None: + result = await _server().verify( + _make_request("Final answer: yes", target="", all_targets=["", " "]) + ) + assert result.reward == approx(0.0) + + async def test_all_targets_from_verifier_metadata(self) -> None: + # The native driver drops the top-level ``all_targets`` list; the full + # accepted set must be recoverable from verifier_metadata. + result = await _server().verify( + _make_request( + "Final answer: under the circle", + target="below the circle", + all_targets=[], + verifier_metadata={ + "target": "below the circle", + "all_targets": ["below the circle", "under the circle"], + }, + ) + ) + assert result.reward == approx(1.0) + assert result.exact is True + + async def test_target_from_verifier_metadata_when_top_level_missing(self) -> None: + result = await _server().verify( + _make_request( + "Final answer: yes", + target="", + all_targets=[], + verifier_metadata={"target": "yes"}, + ) + ) + assert result.reward == approx(1.0) + + +# ── compute_metrics() / get_key_metrics() ────────────────────────────────── + + +class TestComputeMetrics: + def test_mean_and_rates(self) -> None: + tasks = [ + [{"reward": 1.0, "exact": True, "parsed": True}], + [{"reward": 0.0, "exact": False, "parsed": True}], + [{"reward": 1.0, "exact": False, "parsed": True}], + [{"reward": 0.0, "exact": False, "parsed": False}], + ] + metrics = _server().compute_metrics(tasks) + assert metrics["mean_reward"] == approx(0.5) + assert metrics["count"] == 4 + assert metrics["exact_match_rate"] == approx(0.25) + assert metrics["parse_rate"] == approx(0.75) + + def test_empty(self) -> None: + assert _server().compute_metrics([]) == {} + + +class TestGetKeyMetrics: + def test_selects_headline(self) -> None: + out = _server().get_key_metrics( + {"mean_reward": 0.5, "exact_match_rate": 0.25, "parse_rate": 0.75, "count": 4} + ) + assert out == {"mean_reward": approx(0.5), "exact_match_rate": approx(0.25)} + + +# ── _response_text() ─────────────────────────────────────────────────────── + + +class TestResponseText: + def test_output_text_fast_path(self) -> None: + assert _response_text(_make_response("hello")) == "hello" + + def test_fallback_joins_message_content(self) -> None: + message = SimpleNamespace( + type="message", + content=[SimpleNamespace(text="a"), {"text": "b"}], + ) + reasoning = SimpleNamespace(type="reasoning", content="ignored") + response = SimpleNamespace(output_text=None, output=[reasoning, message]) + assert _response_text(response) == "ab" + + def test_fallback_string_content(self) -> None: + message = SimpleNamespace(type="message", content="plain") + response = SimpleNamespace(output_text="", output=[message]) + assert _response_text(response) == "plain" + + +# ── Acceptance / parity ──────────────────────────────────────────────────── + + +class TestAcceptance: + async def test_each_example_target_scores_one(self) -> None: + server = _server() + rows = [ + json.loads(line) for line in _EXAMPLE_JSONL.read_text().splitlines() if line.strip() + ] + assert len(rows) >= 5 + for row in rows: + resp_text = f"Final answer: {row['target']}" + result = await server.verify( + _make_request( + resp_text, target=row["target"], all_targets=row.get("all_targets", []) + ) + ) + assert result.reward == approx(1.0) + + async def test_agrees_with_byob_scorer(self) -> None: + pytest.importorskip("nemo_evaluator") + import benchmarks.spartqa.byob_spartqa as byob + from nemo_evaluator import ScorerInput + + fixtures = [ + ("Final answer: green cup", "green cup", ["green cup"]), + ("Final answer: the green cup on the table", "green cup", ["green cup"]), + ("Final answer: red ball", "green cup", ["green cup"]), + ("Final answer: under the circle", "below the circle", + ["below the circle", "under the circle"]), + (" ", "green cup", ["green cup"]), + ] + server = _server() + for text, target, all_targets in fixtures: + expected = byob.spartqa( + ScorerInput(response=text, target=target, metadata={"all_targets": all_targets}) + )["correct"] + result = await server.verify( + _make_request(text, target=target, all_targets=all_targets) + ) + assert bool(result.reward) == bool(expected) From 5a7436836da999dbf9df1d112e6e77d4cb93333b Mon Sep 17 00:00:00 2001 From: mbagdasarova Date: Fri, 17 Jul 2026 15:27:27 +0200 Subject: [PATCH 2/7] refactor: improve test naming and remove redundant checks in acceptance tests Signed-off-by: mbagdasarova --- resources_servers/spartqa/README.md | 7 ++-- resources_servers/spartqa/app.py | 16 +++----- resources_servers/spartqa/prepare_spartqa.py | 5 +-- .../spartqa/tests/test_acceptance.py | 38 ++++--------------- resources_servers/spartqa/tests/test_app.py | 22 ----------- 5 files changed, 18 insertions(+), 70 deletions(-) diff --git a/resources_servers/spartqa/README.md b/resources_servers/spartqa/README.md index 0dd5b5dece..430ec0515b 100644 --- a/resources_servers/spartqa/README.md +++ b/resources_servers/spartqa/README.md @@ -1,18 +1,17 @@ # SpartQA Resources Server -Spatial-reasoning **answer generation**, ported from the nemo-evaluator BYOB -benchmark `spartqa` (`benchmarks/spartqa/byob_spartqa.py`). The model is shown a +Spatial-reasoning **answer generation** benchmark. The model is shown a spatial-reasoning query and must return the matching answer phrase, ending with a `Final answer: ` line. The per-sample reward is `1.0` on an exact-or-answer-containing match against any accepted answer phrase, else `0.0`. Source dataset: [`mteb/SpartQA`](https://huggingface.co/datasets/mteb/SpartQA) (MTEB retrieval form — `queries` / `corpus` / `qrels` splits joined at prep -time). Upstream eval: `benchmarks/spartqa/byob_spartqa.py`. +time by `prepare_spartqa.py`). ## Scoring -`verify()` (ported verbatim from the BYOB scorer) extracts the model's final +`verify()` extracts the model's final answer (`_extract_answer` / `_strip_reasoning` / `_clean_candidate`), normalizes it (`_normalize`: lowercase, strip punctuation, collapse whitespace), and compares against every accepted phrase in `all_targets` (falling back to diff --git a/resources_servers/spartqa/app.py b/resources_servers/spartqa/app.py index 4abd41affb..5db7c956b4 100644 --- a/resources_servers/spartqa/app.py +++ b/resources_servers/spartqa/app.py @@ -14,20 +14,16 @@ # limitations under the License. """SpartQA resources server — spatial reasoning as direct answer generation. -Ported from the nemo-evaluator BYOB benchmark ``spartqa`` -(``benchmarks/spartqa/byob_spartqa.py``). The MTEB ``mteb/SpartQA`` retrieval -dataset is joined at prep time (``prepare_spartqa.py``) into one row per query -whose ``target`` is the accepted answer phrase (all accepted phrases in -``all_targets``). The model is shown the query and must return the matching -answer phrase, ending with a ``Final answer: `` line. +The MTEB ``mteb/SpartQA`` retrieval dataset is joined at prep time +(``prepare_spartqa.py``) into one row per query whose ``target`` is the +accepted answer phrase (all accepted phrases in ``all_targets``). The model is +shown the query and must return the matching answer phrase, ending with a +``Final answer: `` line. The per-sample reward is ``1.0`` on an exact-or-answer-containing match against any accepted answer, else ``0.0`` — so ``compute_metrics``'s mean-of-rewards equals corpus accuracy. ``exact`` (strict match) and ``parsed`` (a non-empty answer was extracted) ride on each row for downstream inspection. - -The prompt and answer-extraction / scoring logic are ported verbatim from -``byob_spartqa.py`` so the metric is identical. """ from __future__ import annotations @@ -61,7 +57,7 @@ """ -# ── Answer extraction + normalization (verbatim from byob_spartqa.py) ────── +# ── Answer extraction + normalization ─────────────────────────────────────── def _normalize(text: str) -> str: diff --git a/resources_servers/spartqa/prepare_spartqa.py b/resources_servers/spartqa/prepare_spartqa.py index b9c1558ba9..966401f7d2 100644 --- a/resources_servers/spartqa/prepare_spartqa.py +++ b/resources_servers/spartqa/prepare_spartqa.py @@ -14,9 +14,8 @@ # limitations under the License. """Build the SpartQA Gym dataset from the public ``mteb/SpartQA`` HF dataset. -Ports ``benchmarks/spartqa/byob_spartqa.py::build_records``: joins the MTEB -``queries`` / ``corpus`` / ``qrels`` splits into one row per query whose -``target`` is the accepted answer phrase (all accepted phrases in +Joins the MTEB ``queries`` / ``corpus`` / ``qrels`` splits into one row per +query whose ``target`` is the accepted answer phrase (all accepted phrases in ``all_targets``), then renders the shared ``PROMPT`` and writes a Gym task per query. ``target`` / ``all_targets`` ride along as extra fields consumed by ``app.py``'s verify(). diff --git a/resources_servers/spartqa/tests/test_acceptance.py b/resources_servers/spartqa/tests/test_acceptance.py index b02d6fa7e4..bf15f4720e 100644 --- a/resources_servers/spartqa/tests/test_acceptance.py +++ b/resources_servers/spartqa/tests/test_acceptance.py @@ -148,7 +148,7 @@ async def test_ac_verify_is_async_and_returns_reward(self) -> None: assert result.reward == approx(1.0) -# ── AC2: metric parity with byob ──────────────────────────────────────────── +# ── AC2: scoring correctness ───────────────────────────────────────────────── # Shared fixtures: (response_text, target, all_targets, expected_correct). _PARITY_FIXTURES: list[tuple[str, str, list[str], bool]] = [ @@ -167,32 +167,30 @@ async def test_ac_verify_is_async_and_returns_reward(self) -> None: class TestAcMetricParity: - def test_ac_prompt_text_matches_byob(self) -> None: - # PROMPT is ported verbatim; assert the exact contract text the prep - # step renders (the query slot plus the required final-answer line). + def test_ac_prompt_text_contract(self) -> None: assert "Final answer: " in PROMPT assert PROMPT.startswith("Answer the spatial reasoning query below.") assert PROMPT.rstrip().endswith("{question}") - def test_ac_normalize_matches_byob_logic(self) -> None: + def test_ac_normalize_logic(self) -> None: assert _normalize(" The Big, BLACK Square!! ") == "the big black square" assert _normalize("") == "" - def test_ac_strip_reasoning_matches_byob_logic(self) -> None: + def test_ac_strip_reasoning_logic(self) -> None: assert _strip_reasoning("hiddenvisible") == "visible" assert _strip_reasoning("plain") == "plain" - def test_ac_clean_candidate_matches_byob_logic(self) -> None: + def test_ac_clean_candidate_logic(self) -> None: assert _clean_candidate("- *green cup*") == "green cup" assert _clean_candidate(' "yes" ') == "yes" - def test_ac_extract_answer_matches_byob_logic(self) -> None: + def test_ac_extract_answer_logic(self) -> None: assert _extract_answer("Reasoning.\nFinal answer: below the circle") == "below the circle" assert _extract_answer("xFinal answer: yes") == "yes" assert _extract_answer(" ") == "" @pytest.mark.parametrize("text,target,all_targets,expected", _PARITY_FIXTURES) - async def test_ac_scoring_matches_byob_expected( + async def test_ac_scoring_correctness( self, text: str, target: str, all_targets: list[str], expected: bool ) -> None: result = await _server().verify( @@ -200,28 +198,6 @@ async def test_ac_scoring_matches_byob_expected( ) assert bool(result.reward) is expected - async def test_ac_agrees_with_byob_module_when_importable(self) -> None: - # If nemo_evaluator (byob's hard dependency) is installed, compare the - # ported helpers and scorer against the byob source of truth directly. - pytest.importorskip("nemo_evaluator") - import benchmarks.spartqa.byob_spartqa as byob - from nemo_evaluator import ScorerInput - - server = _server() - for text, target, all_targets, _expected in _PARITY_FIXTURES: - assert byob._normalize(text) == _normalize(text) - assert byob._strip_reasoning(text) == _strip_reasoning(text) - assert byob._extract_answer(text) == _extract_answer(text) - byob_correct = byob.spartqa( - ScorerInput( - response=text, target=target, metadata={"all_targets": all_targets} - ) - )["correct"] - result = await server.verify( - _make_request(text, target=target, all_targets=all_targets) - ) - assert bool(result.reward) == bool(byob_correct) - # ── AC3: reward is strictly 1.0 or 0.0 ───────────────────────────────────── diff --git a/resources_servers/spartqa/tests/test_app.py b/resources_servers/spartqa/tests/test_app.py index 16b911fad4..dbb3529bb1 100644 --- a/resources_servers/spartqa/tests/test_app.py +++ b/resources_servers/spartqa/tests/test_app.py @@ -280,25 +280,3 @@ async def test_each_example_target_scores_one(self) -> None: ) assert result.reward == approx(1.0) - async def test_agrees_with_byob_scorer(self) -> None: - pytest.importorskip("nemo_evaluator") - import benchmarks.spartqa.byob_spartqa as byob - from nemo_evaluator import ScorerInput - - fixtures = [ - ("Final answer: green cup", "green cup", ["green cup"]), - ("Final answer: the green cup on the table", "green cup", ["green cup"]), - ("Final answer: red ball", "green cup", ["green cup"]), - ("Final answer: under the circle", "below the circle", - ["below the circle", "under the circle"]), - (" ", "green cup", ["green cup"]), - ] - server = _server() - for text, target, all_targets in fixtures: - expected = byob.spartqa( - ScorerInput(response=text, target=target, metadata={"all_targets": all_targets}) - )["correct"] - result = await server.verify( - _make_request(text, target=target, all_targets=all_targets) - ) - assert bool(result.reward) == bool(expected) From 28988ba5dd6afad0ca2513e9e84eb73b759d0904 Mon Sep 17 00:00:00 2001 From: mbagdasarova Date: Mon, 27 Jul 2026 17:44:52 +0200 Subject: [PATCH 3/7] feat(spartqa): enhance CO question handling and update documentation Signed-off-by: mbagdasarova --- resources_servers/spartqa/README.md | 76 ++++-- resources_servers/spartqa/app.py | 202 +++++++++++----- .../spartqa/configs/spartqa.yaml | 2 +- resources_servers/spartqa/data/example.jsonl | 10 +- .../spartqa/data/example_metrics.json | 22 +- .../spartqa/data/example_rollouts.jsonl | 10 +- resources_servers/spartqa/prepare_spartqa.py | 88 ++++--- .../spartqa/tests/test_acceptance.py | 220 +++++++++++------- resources_servers/spartqa/tests/test_app.py | 208 ++++++++++++----- 9 files changed, 566 insertions(+), 272 deletions(-) diff --git a/resources_servers/spartqa/README.md b/resources_servers/spartqa/README.md index 430ec0515b..084daad431 100644 --- a/resources_servers/spartqa/README.md +++ b/resources_servers/spartqa/README.md @@ -1,30 +1,72 @@ # SpartQA Resources Server -Spatial-reasoning **answer generation** benchmark. The model is shown a -spatial-reasoning query and must return the matching answer phrase, ending with -a `Final answer: ` line. The per-sample reward is `1.0` on an -exact-or-answer-containing match against any accepted answer phrase, else `0.0`. +Spatial-reasoning benchmark covering SpartQA's **CO (choose-object)** question +type. The model is shown a story, a question of the form "… X or Y?", and four +candidate answers; it must copy one of them, ending with a +`Final answer: ` line. The per-sample reward is `1.0` iff the +answer resolves to the gold label, else `0.0`. Source dataset: [`mteb/SpartQA`](https://huggingface.co/datasets/mteb/SpartQA) (MTEB retrieval form — `queries` / `corpus` / `qrels` splits joined at prep time by `prepare_spartqa.py`). +## Task definition + +Per the SpartQA paper (Mirzaee et al., NAACL 2021, Table 8 fn.), CO is a +**four-label single-choice** task: + +| Label | Meaning | +|-------|---------| +| `X` | only the first object named in the question | +| `Y` | only the second object | +| `both of them` | both objects satisfy the asked relation | +| `none of them` | neither does (the paper's DK / None / `[]`) | + +The retrieval encoding has no way to express "both", so a `both of them` gold +is flattened into **three** relevant qrels documents (the phrase plus each +object). `prepare_spartqa.py` (`resolve_gold`) undoes that flattening back into +one gold label, and renders all four candidates into the prompt so +`both of them` / `none of them` are reachable answers. + +> Scoring the three flattened phrases as interchangeable accepted answers — +> i.e. crediting a single object for a `both of them` gold — is not the paper's +> metric and is trivially gameable: echoing the question's two options back +> ("X or Y") scores **94.9%** that way, versus **0.0%** here. + +Corpus composition (`test`, 3594 rows): 1579 `both of them` (43.9%), +183 `none of them` (5.1%), 1832 a single object. The majority-class baseline is +therefore **43.9%**; always answering the first option scores **26.1%**. + ## Scoring -`verify()` extracts the model's final -answer (`_extract_answer` / `_strip_reasoning` / `_clean_candidate`), normalizes -it (`_normalize`: lowercase, strip punctuation, collapse whitespace), and -compares against every accepted phrase in `all_targets` (falling back to -`target`). A strict equality sets `exact`; a substring match sets the reward -without `exact`. Empty output scores `0.0` and never raises. +`verify()` extracts the model's final answer (`_extract_answer` / +`_strip_reasoning` / `_clean_candidate`), then resolves it to one of the four +candidate labels (`match_label`): + +1. **Verbatim match** — article- and punctuation-insensitive equality with a + candidate (`_label_key`). Sets `exact`. Aliases (`both`, `neither`, `DK`, …) + each map to exactly one fixed label. +2. **Unambiguous containment** — the answer contains exactly one candidate + (when one candidate nests inside another, the most specific wins). Scores + without `exact`. +3. Otherwise the answer resolves to **no label** and scores `0.0`. An answer + naming two candidates is ambiguous by construction, so it never scores. + +Reward is `1.0` iff the resolved label is the gold label. Empty output scores +`0.0` and never raises. ## Metrics `compute_metrics` reports: -- `mean_reward` — mean per-sample accuracy (also the reward). -- `exact_match_rate` — fraction with a strict (exact) match. +- `mean_reward` — mean per-sample CO accuracy (also the reward). +- `exact_match_rate` — fraction that answered the gold label verbatim. - `parse_rate` — fraction where a non-empty answer phrase was extracted. +- `label_resolve_rate` — fraction that resolved to any candidate label; a low + value means the model is not following the copy-a-candidate format. +- `accuracy_both_of_them` / `accuracy_none_of_them` — accuracy on those gold + slices. A model that never answers `both of them` — 44% of the corpus — + shows up as a near-zero slice even when the headline number looks healthy. `get_key_metrics` surfaces `mean_reward` and `exact_match_rate`. @@ -40,7 +82,8 @@ python resources_servers/spartqa/prepare_spartqa.py --split test This joins `mteb/SpartQA` (via the HF `datasets` library) and writes the gitignored `data/spartqa_test.jsonl`. The committed `data/example.jsonl` is a -5-row smoke-test slice. +5-row slice sampled from that file, covering a `both of them` gold, a +`none of them` gold, and both single-object cases. ## Example rollouts and metrics @@ -59,8 +102,11 @@ tail -n 1 resources_servers/spartqa/data/example_rollouts.jsonl | jq .reward cat resources_servers/spartqa/data/example_metrics.json | jq . ``` -Note: row 2 (index 2) in the example rollouts is intentionally wrong (reward -0.0) to demonstrate a failed case; the remaining four rows score 1.0. +Note: row 3 (index 3) in the example rollouts is intentionally wrong (reward +0.0) — it echoes both of the question's options back, the degenerate answer the +scorer must not credit. The remaining four rows score 1.0 and cover a verbatim +match, an alias (`neither` → `none of them`), and a label recovered from a +longer sentence. ## Run diff --git a/resources_servers/spartqa/app.py b/resources_servers/spartqa/app.py index 5db7c956b4..c7e9bbba39 100644 --- a/resources_servers/spartqa/app.py +++ b/resources_servers/spartqa/app.py @@ -12,18 +12,25 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -"""SpartQA resources server — spatial reasoning as direct answer generation. - -The MTEB ``mteb/SpartQA`` retrieval dataset is joined at prep time -(``prepare_spartqa.py``) into one row per query whose ``target`` is the -accepted answer phrase (all accepted phrases in ``all_targets``). The model is -shown the query and must return the matching answer phrase, ending with a -``Final answer: `` line. - -The per-sample reward is ``1.0`` on an exact-or-answer-containing match against -any accepted answer, else ``0.0`` — so ``compute_metrics``'s mean-of-rewards -equals corpus accuracy. ``exact`` (strict match) and ``parsed`` (a non-empty -answer was extracted) ride on each row for downstream inspection. +"""SpartQA resources server — the CO (choose-object) question type. + +``mteb/SpartQA`` is the MTEB retrieval form of SpartQA (Mirzaee et al., NAACL +2021) and contains CO questions only: a story plus "… or ?". Per the paper (Table 8 fn.), CO is a **four-label single-choice** task — +``X``, ``Y``, ``both of them``, ``none of them`` — and the metric is accuracy +over that label set. + +``prepare_spartqa.py`` recovers that label set from the retrieval qrels (which +flatten "both of them" into three relevant documents), renders the four +candidates into the prompt, and stores the single gold label in ``target`` and +the two object labels in ``options``. + +The per-sample reward is ``1.0`` iff the model's answer resolves to the gold +label, else ``0.0`` — so ``compute_metrics``'s mean-of-rewards equals CO +accuracy. Resolution is a verbatim match against one candidate, or a +containment match when it is *unambiguous*: a response naming two candidates +(e.g. echoing "X or Y" back) resolves to nothing and scores 0.0. Scoring any of +the three flattened qrels phrases as correct would let that echo score ~95%. """ from __future__ import annotations @@ -45,17 +52,44 @@ PROMPT = """\ -Answer the spatial reasoning query below. -Return one concise final answer phrase. If the query gives answer options, -copy the matching option phrase exactly when possible. +Answer the spatial reasoning question below. +Choose exactly one of the candidate answers and copy it verbatim. Answer +"both of them" if both listed objects satisfy the question, and "none of them" +if neither of them does. End your response with one line in this exact format: -Final answer: +Final answer: -Query: +Story and question: {question} + +Candidate answers: +{candidates} """ +# The two story-independent CO labels. Aliases let a model phrase them its own +# way; each alias maps to exactly one label, so this adds no leniency across +# labels. +BOTH_LABEL = "both of them" +NONE_LABEL = "none of them" + +_BOTH_ALIASES = frozenset({"both", "both of them", "both objects", "both of the objects", "both of these"}) +# The paper treats DK / None / [] alike: none of the candidate objects hold. +_NONE_ALIASES = frozenset( + { + "none", + "none of them", + "none of the objects", + "neither", + "neither of them", + "neither object", + "no object", + "dk", + "do not know", + "dont know", + } +) + # ── Answer extraction + normalization ─────────────────────────────────────── @@ -96,9 +130,7 @@ def _extract_answer(text: str) -> str: ] extracted = None for pattern in patterns: - matches = list( - re.finditer(pattern, text, flags=re.IGNORECASE | re.DOTALL | re.MULTILINE) - ) + matches = list(re.finditer(pattern, text, flags=re.IGNORECASE | re.DOTALL | re.MULTILINE)) if matches: extracted = matches[-1].group(1).strip() break @@ -106,23 +138,67 @@ def _extract_answer(text: str) -> str: text = extracted lines = [_clean_candidate(line) for line in text.splitlines() if line.strip()] - lines = [ - line - for line in lines - if line and _normalize(line) not in {"final answer", "answer"} - ] + lines = [line for line in lines if line and _normalize(line) not in {"final answer", "answer"}] if not lines: return "" if extracted is None and len(lines) > 1: first = _normalize(lines[0]) - if first.startswith( - ("thinking process", "analysis", "the user wants", "we need") - ): + if first.startswith(("thinking process", "analysis", "the user wants", "we need")): return lines[-1] return lines[0] +def _label_key(text: str) -> str: + """Normalize a candidate label or a prediction for comparison. + + Article-insensitive, because the qrels phrases and the story's own wording + disagree on leading articles ("medium yellow square …" vs "the medium + yellow square …"). + """ + return re.sub(r"^(?:a|an|the)\s+", "", _normalize(text)).strip() + + +def candidate_labels(options: List[str]) -> List[str]: + """The four CO labels: the two story objects, then ``both`` / ``none``.""" + labels: List[str] = [] + seen: set[str] = set() + for label in [*options, BOTH_LABEL, NONE_LABEL]: + key = _label_key(str(label)) + if key and key not in seen: + seen.add(key) + labels.append(str(label).strip()) + return labels + + +def match_label(prediction: str, labels: List[str]) -> Optional[str]: + """Resolve a free-text answer to exactly one candidate label, or ``None``. + + Verbatim (article-insensitive) equality wins outright. Otherwise the answer + must *contain* exactly one label — a response naming two of them is + ambiguous and resolves to ``None`` rather than being credited for whichever + one happens to be gold. + """ + key = _label_key(prediction) + if not key: + return None + + by_key = {_label_key(label): label for label in labels} + if key in by_key: + return by_key[key] + if key in _BOTH_ALIASES and BOTH_LABEL in by_key.values(): + return by_key[_label_key(BOTH_LABEL)] + if key in _NONE_ALIASES and NONE_LABEL in by_key.values(): + return by_key[_label_key(NONE_LABEL)] + + hits = [label_key for label_key in by_key if label_key and label_key in key] + # One option can nest inside the other ("a triangle in block B" inside "a + # big blue triangle in block B"); the most specific match is the intended + # one, so drop labels wholly contained in another hit. + hits = [h for h in hits if not any(other != h and h in other for other in hits)] + return by_key[hits[0]] if len(hits) == 1 else None + + def _response_text(response: NeMoGymResponse) -> str: """Best-effort extraction of the assistant text from a NeMoGymResponse.""" text = getattr(response, "output_text", None) @@ -153,14 +229,15 @@ class SpartqaResourcesServerConfig(BaseResourcesServerConfig): class SpartqaRunRequest(BaseRunRequest): model_config = ConfigDict(extra="allow") - # The first accepted answer phrase, and all accepted phrases. ``target`` - # (a scalar) survives the nemo-evaluator ``gym://...protocol=native`` driver, - # which forwards a row's top-level scalar fields onto ``/verify`` but DROPS - # list/dict fields (``all_targets`` never arrives that way). The full accepted - # set therefore also rides in ``verifier_metadata``, which the driver forwards - # intact; verify() falls back to it so all phrases are always available. + # ``target`` is the single gold CO label; ``options`` are the two story + # objects offered by the question. ``target`` (a scalar) survives the + # nemo-evaluator ``gym://...protocol=native`` driver, which forwards a row's + # top-level scalar fields onto ``/verify`` but DROPS list/dict fields + # (``options`` never arrives that way). The options therefore also ride in + # ``verifier_metadata``, which the driver forwards intact; verify() falls + # back to it so the full candidate set is always available. target: str = "" - all_targets: List[str] = Field(default_factory=list) + options: List[str] = Field(default_factory=list) verifier_metadata: Optional[Dict[str, Any]] = None @@ -174,6 +251,7 @@ class SpartqaVerifyResponse(BaseVerifyResponse): exact: bool = False parsed: bool = False extracted: str = "" + predicted_label: str = "" class SpartqaResourcesServer(SimpleResourcesServer): @@ -181,33 +259,26 @@ class SpartqaResourcesServer(SimpleResourcesServer): async def verify(self, body: SpartqaVerifyRequest) -> SpartqaVerifyResponse: prediction = _extract_answer(_response_text(body.response)) - prediction_norm = _normalize(prediction) - # Prefer the explicit list; fall back to verifier_metadata (the only path - # that survives the native driver) and finally the scalar target. + # Prefer the explicit fields; fall back to verifier_metadata, the only + # path that survives the native driver. meta = body.verifier_metadata or {} - targets = body.all_targets or meta.get("all_targets") or [ - body.target or meta.get("target", "") - ] - - exact = False - contains = False - for target in targets: - target_norm = _normalize(str(target)) - if not target_norm: - continue - if prediction_norm == target_norm: - exact = True - contains = True - break - if target_norm in prediction_norm: - contains = True + gold = body.target or str(meta.get("target", "")) + options = body.options or list(meta.get("options") or []) + + labels = candidate_labels(options) if options else candidate_labels([gold]) + predicted = match_label(prediction, labels) + gold_key = _label_key(gold) + correct = bool(gold_key) and predicted is not None and _label_key(predicted) == gold_key return SpartqaVerifyResponse( **body.model_dump(), - reward=1.0 if (exact or contains) else 0.0, - exact=exact, - parsed=bool(prediction_norm), + reward=1.0 if correct else 0.0, + # ``exact`` means the gold label was copied verbatim, as instructed, + # rather than recovered from a longer sentence. + exact=correct and _label_key(prediction) == gold_key, + parsed=bool(_normalize(prediction)), extracted=prediction[:200], + predicted_label=predicted or "", ) # --- aggregation ----------------------------------------------------- @@ -224,14 +295,21 @@ def compute_metrics(self, tasks: List[List[Dict[str, Any]]]) -> Dict[str, Any]: metrics["count"] = len(rewards) metrics["exact_match_rate"] = sum(1 for r in rows if r.get("exact")) / len(rows) metrics["parse_rate"] = sum(1 for r in rows if r.get("parsed")) / len(rows) + # Fraction of answers that resolved to some candidate label at all; a low + # value means the model is not following the copy-a-candidate format. + metrics["label_resolve_rate"] = sum(1 for r in rows if r.get("predicted_label")) / len(rows) + # Accuracy split by gold label. A model that ignores "both of them" — + # the gold for 44% of the corpus — shows up here as a near-zero slice + # even when the headline number looks healthy. + for label in (BOTH_LABEL, NONE_LABEL): + slice_rows = [r for r in rows if _label_key(str(r.get("target", ""))) == _label_key(label)] + if slice_rows: + key = label.replace(" ", "_") + metrics[f"accuracy_{key}"] = sum(1 for r in slice_rows if r.get("reward") == 1.0) / len(slice_rows) return metrics def get_key_metrics(self, agent_metrics: Dict[str, Any]) -> Dict[str, Any]: - return { - k: agent_metrics[k] - for k in ("mean_reward", "exact_match_rate") - if k in agent_metrics - } + return {k: agent_metrics[k] for k in ("mean_reward", "exact_match_rate") if k in agent_metrics} if __name__ == "__main__": diff --git a/resources_servers/spartqa/configs/spartqa.yaml b/resources_servers/spartqa/configs/spartqa.yaml index 6b6949d055..2d718ef529 100644 --- a/resources_servers/spartqa/configs/spartqa.yaml +++ b/resources_servers/spartqa/configs/spartqa.yaml @@ -4,7 +4,7 @@ spartqa: entrypoint: app.py domain: reasoning verified: false - description: SpartQA spatial-reasoning answer generation; reward = 1 on an exact-or-answer-containing match against any accepted answer phrase. + description: SpartQA spatial reasoning, CO (choose-object) question type; four candidate answers per question (two story objects, "both of them", "none of them"); reward = 1 iff the answer resolves to the single gold label. value: Improve spatial and relational reasoning over described scenes. spartqa_simple_agent: diff --git a/resources_servers/spartqa/data/example.jsonl b/resources_servers/spartqa/data/example.jsonl index 3ee960e1d1..b4d0c85215 100644 --- a/resources_servers/spartqa/data/example.jsonl +++ b/resources_servers/spartqa/data/example.jsonl @@ -1,5 +1,5 @@ -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nThere are two blocks, A and B. A small circle is above a large square in block A. A triangle is to the right of the square. Where is the triangle relative to the circle?\nOptions: below the circle; above the circle; left of the circle\n"}]}, "target": "below the circle", "all_targets": ["below the circle", "under the circle"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "below the circle", "all_targets": ["below the circle", "under the circle"]}} -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nA red ball is to the left of a blue box. The blue box is to the left of a green cup. Which object is farthest to the right?\nOptions: red ball; blue box; green cup\n"}]}, "target": "green cup", "all_targets": ["green cup"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "green cup", "all_targets": ["green cup"]}} -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nIn the grid, a star is directly north of a moon, and the moon is directly north of a sun. Is the star north of the sun?\nOptions: yes; no\n"}]}, "target": "yes", "all_targets": ["yes"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "yes", "all_targets": ["yes"]}} -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nA cat sits on a mat. A dog stands to the right of the cat. A bird is above the dog. Relative to the cat, in which direction is the bird?\nOptions: upper right; upper left; lower right\n"}]}, "target": "upper right", "all_targets": ["upper right", "above and to the right"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "upper right", "all_targets": ["upper right", "above and to the right"]}} -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nBox 1 contains a big black square touching the top edge. A medium white circle is below the square. What is at the top of Box 1?\nOptions: the big black square; the medium white circle\n"}]}, "target": "the big black square", "all_targets": ["the big black square"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "the big black square", "all_targets": ["the big black square"]}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nThere are two blocks. We call them A and B. Block A has two big black triangles. Below a medium black triangle and a small blue triangle there is big black triangle number one. The small blue triangle is above and to the left of the medium black triangle. Block B is below block A. It has a big yellow square. What is above the yellow shape? a small shape or a big black triangle?\n\nCandidate answers:\n- a small shape\n- a big black triangle\n- both of them\n- none of them\n"}]}, "target": "both of them", "options": ["a small shape", "a big black triangle"], "verifier_metadata": {"target": "both of them", "options": ["a small shape", "a big black triangle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nThere are two small black squares in a block. Near to and to the left of a medium blue triangle there is a big blue circle. Below, far from and to the left of small black square number two is a medium blue circle. Near to a big yellow triangle is the medium blue triangle. The medium blue circle is touching the left edge of this block. Which object is far from a small black square? the big blue circle or the yellow triangle?\n\nCandidate answers:\n- the big blue circle\n- the yellow triangle\n- both of them\n- none of them\n"}]}, "target": "none of them", "options": ["the big blue circle", "the yellow triangle"], "verifier_metadata": {"target": "none of them", "options": ["the big blue circle", "the yellow triangle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have three blocks. We call them A, B and C. A small blue triangle is in block A. Below block A is block B which has one small blue square. To the left of block A there is block C which has two medium yellow triangles. Medium yellow triangle number one is above medium yellow triangle number two and a big blue square. What is to the left of the small blue triangle? a medium yellow triangle or a small blue square?\n\nCandidate answers:\n- a medium yellow triangle\n- a small blue square\n- both of them\n- none of them\n"}]}, "target": "a medium yellow triangle", "options": ["a medium yellow triangle", "a small blue square"], "verifier_metadata": {"target": "a medium yellow triangle", "options": ["a medium yellow triangle", "a small blue square"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have three blocks. We call them A, B and C. A small blue triangle is in block A. Below block A is block B which has one small blue square. To the left of block A there is block C which has two medium yellow triangles. Medium yellow triangle number one is above medium yellow triangle number two and a big blue square. What is to the left of the small blue triangle? a small blue square or a big blue square?\n\nCandidate answers:\n- a small blue square\n- a big blue square\n- both of them\n- none of them\n"}]}, "target": "a big blue square", "options": ["a small blue square", "a big blue square"], "verifier_metadata": {"target": "a big blue square", "options": ["a small blue square", "a big blue square"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have two blocks, A and B. Block A is to the left of B. Block A contains two small black squares. To the right of a big black circle there is small black square number one. Below the big black circle and a medium black circle is small black square number two. Block B has one big black square. What is to the left of the big black square? a medium black circle or a big black circle?\n\nCandidate answers:\n- a medium black circle\n- a big black circle\n- both of them\n- none of them\n"}]}, "target": "both of them", "options": ["a medium black circle", "a big black circle"], "verifier_metadata": {"target": "both of them", "options": ["a medium black circle", "a big black circle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} diff --git a/resources_servers/spartqa/data/example_metrics.json b/resources_servers/spartqa/data/example_metrics.json index 355a62c9dc..a80d13b772 100644 --- a/resources_servers/spartqa/data/example_metrics.json +++ b/resources_servers/spartqa/data/example_metrics.json @@ -17,10 +17,10 @@ }, "Json-dumped number of words (proxy for token count)": { "Total # non-null values": 5, - "Average": 99.0, - "Min": 83.0, - "Max": 113.0, - "Standard deviation": 11.489 + "Average": 179.4, + "Min": 171.0, + "Max": 184.0, + "Standard deviation": 5.367 }, "Number of turns": { "Total # non-null values": 5, @@ -36,11 +36,13 @@ "Max": 0.0, "Standard deviation": 0.0 }, - "Number of accepted answers (all_targets)": { + "Number of candidate answers": { "Total # non-null values": 5, - "Average": 1.4, - "Min": 1.0, - "Max": 2.0, - "Standard deviation": 0.548 - } + "Average": 4.0, + "Min": 4.0, + "Max": 4.0, + "Standard deviation": 0.0 + }, + "Number of 'both of them' golds": 2, + "Number of 'none of them' golds": 1 } diff --git a/resources_servers/spartqa/data/example_rollouts.jsonl b/resources_servers/spartqa/data/example_rollouts.jsonl index b4d6f8fcf2..5f1b17e31a 100644 --- a/resources_servers/spartqa/data/example_rollouts.jsonl +++ b/resources_servers/spartqa/data/example_rollouts.jsonl @@ -1,5 +1,5 @@ -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nThere are two blocks, A and B. A small circle is above a large square in block A. A triangle is to the right of the square. Where is the triangle relative to the circle?\nOptions: below the circle; above the circle; left of the circle\n"}]}, "target": "below the circle", "all_targets": ["below the circle", "under the circle"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "below the circle", "all_targets": ["below the circle", "under the circle"]}, "response": {"id": "resp_029628e2a4ef4e149324bbda42b6900b", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_931188755ee742f6b87a93284b8c996b", "content": [{"annotations": [], "text": "The triangle is to the right of the square, which is under the circle.\nFinal answer: below the circle", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "below the circle", "_ng_task_index": 0, "_ng_rollout_index": 0} -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nA red ball is to the left of a blue box. The blue box is to the left of a green cup. Which object is farthest to the right?\nOptions: red ball; blue box; green cup\n"}]}, "target": "green cup", "all_targets": ["green cup"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "green cup", "all_targets": ["green cup"]}, "response": {"id": "resp_b7ca67217aca4a93b31d050380979c9b", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_be94029e039d4284a7bd939ab5caa576", "content": [{"annotations": [], "text": "Following the left-to-right order red, blue, green.\nFinal answer: green cup", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "green cup", "_ng_task_index": 1, "_ng_rollout_index": 0} -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nIn the grid, a star is directly north of a moon, and the moon is directly north of a sun. Is the star north of the sun?\nOptions: yes; no\n"}]}, "target": "yes", "all_targets": ["yes"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "yes", "all_targets": ["yes"]}, "response": {"id": "resp_4104f8c263fc46f3a7d468bf443b3852", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_4b559bd98a0244478e762d5653fce0a8", "content": [{"annotations": [], "text": "The star is north of the moon, and the sun is south of both.\nFinal answer: no", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 0.0, "exact": false, "parsed": true, "extracted": "no", "_ng_task_index": 2, "_ng_rollout_index": 0} -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nA cat sits on a mat. A dog stands to the right of the cat. A bird is above the dog. Relative to the cat, in which direction is the bird?\nOptions: upper right; upper left; lower right\n"}]}, "target": "upper right", "all_targets": ["upper right", "above and to the right"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "upper right", "all_targets": ["upper right", "above and to the right"]}, "response": {"id": "resp_474e797222094d398ee6393293482316", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_333423e11aff4c77b8ff30b1be1c9e9e", "content": [{"annotations": [], "text": "The bird is above the dog, and the dog is right of the cat.\nFinal answer: upper right", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "upper right", "_ng_task_index": 3, "_ng_rollout_index": 0} -{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning query below.\nReturn one concise final answer phrase. If the query gives answer options,\ncopy the matching option phrase exactly when possible.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nQuery:\nBox 1 contains a big black square touching the top edge. A medium white circle is below the square. What is at the top of Box 1?\nOptions: the big black square; the medium white circle\n"}]}, "target": "the big black square", "all_targets": ["the big black square"], "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "verifier_metadata": {"target": "the big black square", "all_targets": ["the big black square"]}, "response": {"id": "resp_9c312ab496a9491d9721b74f30aa868f", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_4b10f237db564082bc78a18363e5225a", "content": [{"annotations": [], "text": "The square touches the top edge, the circle is below it.\nFinal answer: the big black square", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "the big black square", "_ng_task_index": 4, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nThere are two blocks. We call them A and B. Block A has two big black triangles. Below a medium black triangle and a small blue triangle there is big black triangle number one. The small blue triangle is above and to the left of the medium black triangle. Block B is below block A. It has a big yellow square. What is above the yellow shape? a small shape or a big black triangle?\n\nCandidate answers:\n- a small shape\n- a big black triangle\n- both of them\n- none of them\n"}]}, "target": "both of them", "options": ["a small shape", "a big black triangle"], "verifier_metadata": {"target": "both of them", "options": ["a small shape", "a big black triangle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_20078df25d2140509cf9fcd601151a42", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_fd65a5b507254909b1ce3088f6be3924", "content": [{"annotations": [], "text": "Both listed shapes sit above the block's bottom edge.\nFinal answer: both of them", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "both of them", "predicted_label": "both of them", "_ng_task_index": 0, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nThere are two small black squares in a block. Near to and to the left of a medium blue triangle there is a big blue circle. Below, far from and to the left of small black square number two is a medium blue circle. Near to a big yellow triangle is the medium blue triangle. The medium blue circle is touching the left edge of this block. Which object is far from a small black square? the big blue circle or the yellow triangle?\n\nCandidate answers:\n- the big blue circle\n- the yellow triangle\n- both of them\n- none of them\n"}]}, "target": "none of them", "options": ["the big blue circle", "the yellow triangle"], "verifier_metadata": {"target": "none of them", "options": ["the big blue circle", "the yellow triangle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_10e3ebd7c7e1468aa58f76d2b988b380", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_e71953cbc16042699ba7e591a4cc48fe", "content": [{"annotations": [], "text": "Neither of the two objects satisfies the asked relation.\nFinal answer: neither", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": false, "parsed": true, "extracted": "neither", "predicted_label": "none of them", "_ng_task_index": 1, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have three blocks. We call them A, B and C. A small blue triangle is in block A. Below block A is block B which has one small blue square. To the left of block A there is block C which has two medium yellow triangles. Medium yellow triangle number one is above medium yellow triangle number two and a big blue square. What is to the left of the small blue triangle? a medium yellow triangle or a small blue square?\n\nCandidate answers:\n- a medium yellow triangle\n- a small blue square\n- both of them\n- none of them\n"}]}, "target": "a medium yellow triangle", "options": ["a medium yellow triangle", "a small blue square"], "verifier_metadata": {"target": "a medium yellow triangle", "options": ["a medium yellow triangle", "a small blue square"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_5db9b1c4943b4d13b7990449043e7a65", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_dac0a2cd643e4b90b909c7b229e507e8", "content": [{"annotations": [], "text": "Only the first candidate matches the relation.\nFinal answer: a medium yellow triangle", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "a medium yellow triangle", "predicted_label": "a medium yellow triangle", "_ng_task_index": 2, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have three blocks. We call them A, B and C. A small blue triangle is in block A. Below block A is block B which has one small blue square. To the left of block A there is block C which has two medium yellow triangles. Medium yellow triangle number one is above medium yellow triangle number two and a big blue square. What is to the left of the small blue triangle? a small blue square or a big blue square?\n\nCandidate answers:\n- a small blue square\n- a big blue square\n- both of them\n- none of them\n"}]}, "target": "a big blue square", "options": ["a small blue square", "a big blue square"], "verifier_metadata": {"target": "a big blue square", "options": ["a small blue square", "a big blue square"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_1b59b532aa3e42eb911e29388528be9b", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_1234a148f406404e886436efbc250277", "content": [{"annotations": [], "text": "It could be either one.\nFinal answer: a small blue square or a big blue square", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 0.0, "exact": false, "parsed": true, "extracted": "a small blue square or a big blue square", "predicted_label": "", "_ng_task_index": 3, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have two blocks, A and B. Block A is to the left of B. Block A contains two small black squares. To the right of a big black circle there is small black square number one. Below the big black circle and a medium black circle is small black square number two. Block B has one big black square. What is to the left of the big black square? a medium black circle or a big black circle?\n\nCandidate answers:\n- a medium black circle\n- a big black circle\n- both of them\n- none of them\n"}]}, "target": "both of them", "options": ["a medium black circle", "a big black circle"], "verifier_metadata": {"target": "both of them", "options": ["a medium black circle", "a big black circle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_05e53effc96e471abc7e98d1e7eef406", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_a7a34dc091604be582d53db4467ece1d", "content": [{"annotations": [], "text": "Checking each candidate in turn, the relation holds for both.\nFinal answer: I would say both of them.", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": false, "parsed": true, "extracted": "I would say both of them.", "predicted_label": "both of them", "_ng_task_index": 4, "_ng_rollout_index": 0} diff --git a/resources_servers/spartqa/prepare_spartqa.py b/resources_servers/spartqa/prepare_spartqa.py index 966401f7d2..61f08b9405 100644 --- a/resources_servers/spartqa/prepare_spartqa.py +++ b/resources_servers/spartqa/prepare_spartqa.py @@ -14,11 +14,21 @@ # limitations under the License. """Build the SpartQA Gym dataset from the public ``mteb/SpartQA`` HF dataset. -Joins the MTEB ``queries`` / ``corpus`` / ``qrels`` splits into one row per -query whose ``target`` is the accepted answer phrase (all accepted phrases in -``all_targets``), then renders the shared ``PROMPT`` and writes a Gym task per -query. ``target`` / ``all_targets`` ride along as extra fields consumed by -``app.py``'s verify(). +``mteb/SpartQA`` holds SpartQA's CO (choose-object) questions in retrieval +form. Per the paper, CO is a four-label single-choice task — the two objects +named in the question, ``both of them``, and ``none of them`` — but the +retrieval encoding has no way to express "both", so a "both of them" gold is +flattened into *three* relevant documents (the ``both of them`` phrase plus +each object). This script undoes that flattening: + +* ``options`` — the two objects, parsed from the question's "… X or Y?" tail. +* ``target`` — the single gold label: ``both of them`` when qrels marks the + pair, ``none of them`` when qrels says so, else the one gold object (snapped + to the option's own wording so the label set is self-consistent). + +The four candidates are rendered into the prompt so ``both of them`` and +``none of them`` are actually reachable answers; ``target`` / ``options`` ride +along as extra fields consumed by ``app.py``'s verify(). Requires the ``datasets`` package (HF) at prep time only. @@ -32,10 +42,11 @@ import argparse import json +import re from pathlib import Path -from typing import Any, List +from typing import Any, List, Optional -from app import PROMPT +from app import BOTH_LABEL, NONE_LABEL, PROMPT, _label_key _HF_DATASET = "mteb/SpartQA" @@ -65,6 +76,34 @@ def _unique_preserve_order(values: List[str]) -> List[str]: return result +def parse_options(question: str) -> Optional[List[str]]: + """Parse the two candidate objects out of a CO question's "… X or Y?" tail. + + Returns ``None`` when the final sentence does not offer exactly two + alternatives, so the caller can drop the row rather than guess. + """ + last_sentence = re.split(r"(?<=[.?])\s+", question.strip())[-1].strip().rstrip("?") + parts = [part.strip() for part in last_sentence.split(" or ")] + if len(parts) != 2 or not all(parts): + return None + return parts + + +def resolve_gold(answers: List[str], options: List[str]) -> Optional[str]: + """Collapse the flattened qrels phrases back into one CO label.""" + keys = {_label_key(a) for a in answers} + if _label_key(BOTH_LABEL) in keys or len(keys & {_label_key(o) for o in options}) >= 2: + return BOTH_LABEL + if _label_key(NONE_LABEL) in keys: + return NONE_LABEL + for option in options: + if _label_key(option) in keys: + # Snap to the option's own wording: qrels and the story disagree on + # leading articles, and the prompt shows the option text. + return option + return None + + def build_records(split: str = _DEFAULT_SPLIT) -> List[dict[str, Any]]: """Join the MTEB ``queries`` / ``corpus`` / ``qrels`` splits into rows.""" corpus = { @@ -90,36 +129,33 @@ def build_records(split: str = _DEFAULT_SPLIT) -> List[dict[str, Any]]: records: List[dict[str, Any]] = [] for query_id in sorted(queries): - answer_ids = [ - doc_id for doc_id in qrels_by_query.get(query_id, []) if doc_id in corpus - ] + question = queries[query_id] + answer_ids = [doc_id for doc_id in qrels_by_query.get(query_id, []) if doc_id in corpus] answers = _unique_preserve_order([corpus[doc_id] for doc_id in answer_ids]) - if not answers: + options = parse_options(question) + if not answers or not options: + continue + target = resolve_gold(answers, options) + if not target: continue - records.append( - { - "question": queries[query_id], - "target": answers[0], - "all_targets": answers, - } - ) + records.append({"question": question, "target": target, "options": options}) return records def _to_task(record: dict[str, Any]) -> dict[str, Any]: + candidates = "\n".join(f"- {label}" for label in [*record["options"], BOTH_LABEL, NONE_LABEL]) + content = PROMPT.format(question=record["question"], candidates=candidates) return { - "responses_create_params": { - "input": [{"role": "user", "content": PROMPT.format(question=record["question"])}] - }, + "responses_create_params": {"input": [{"role": "user", "content": content}]}, "target": record["target"], - "all_targets": record["all_targets"], - # ``all_targets`` is a list and is dropped by the nemo-evaluator + "options": record["options"], + # ``options`` is a list and is dropped by the nemo-evaluator # ``gym://...protocol=native`` driver (it forwards only top-level scalar - # fields to /verify). Mirror the full accepted set into verifier_metadata, - # which the driver forwards intact, so verify() sees every phrase. + # fields to /verify). Mirror the candidate set into verifier_metadata, + # which the driver forwards intact, so verify() sees every label. "verifier_metadata": { "target": record["target"], - "all_targets": record["all_targets"], + "options": record["options"], }, "agent_ref": _AGENT, } diff --git a/resources_servers/spartqa/tests/test_acceptance.py b/resources_servers/spartqa/tests/test_acceptance.py index bf15f4720e..cadd5ddd96 100644 --- a/resources_servers/spartqa/tests/test_acceptance.py +++ b/resources_servers/spartqa/tests/test_acceptance.py @@ -43,18 +43,24 @@ from nemo_gym.server_utils import ServerClient from resources_servers.spartqa import app as spartqa_app from resources_servers.spartqa.app import ( + BOTH_LABEL, + NONE_LABEL, PROMPT, + SimpleResourcesServer, SpartqaResourcesServer, SpartqaResourcesServerConfig, SpartqaVerifyRequest, SpartqaVerifyResponse, - SimpleResourcesServer, _clean_candidate, _extract_answer, + _label_key, _normalize, _strip_reasoning, + candidate_labels, + match_label, ) + # ── Paths (relative to this test file) ───────────────────────────────────── _SERVER_DIR = Path(__file__).resolve().parent.parent @@ -92,17 +98,19 @@ def _config() -> SpartqaResourcesServerConfig: return SpartqaResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint="", name="spartqa") -def _make_request( - text: str, *, target: str = "", all_targets: list[str] | None = None -) -> SpartqaVerifyRequest: +def _make_request(text: str, *, target: str = "", options: list[str] | None = None) -> SpartqaVerifyRequest: return SpartqaVerifyRequest( responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), response=_make_response(text), target=target, - all_targets=all_targets or [], + options=options or [], ) +# A representative CO row: two story objects plus the two fixed labels. +_OPTIONS = ["a big blue square", "a small blue square"] + + def _server() -> SpartqaResourcesServer: return SpartqaResourcesServer(config=_config(), server_client=MagicMock(spec=ServerClient)) @@ -142,7 +150,9 @@ def test_ac_subclasses_simple_resources_server(self) -> None: assert issubclass(SpartqaResourcesServer, SimpleResourcesServer) async def test_ac_verify_is_async_and_returns_reward(self) -> None: - result = await _server().verify(_make_request("Final answer: yes", target="yes")) + result = await _server().verify( + _make_request(f"Final answer: {BOTH_LABEL}", target=BOTH_LABEL, options=_OPTIONS) + ) assert isinstance(result, SpartqaVerifyResponse) assert hasattr(result, "reward") assert result.reward == approx(1.0) @@ -150,27 +160,34 @@ async def test_ac_verify_is_async_and_returns_reward(self) -> None: # ── AC2: scoring correctness ───────────────────────────────────────────────── -# Shared fixtures: (response_text, target, all_targets, expected_correct). +# Shared fixtures: (response_text, gold_label, options, expected_correct). _PARITY_FIXTURES: list[tuple[str, str, list[str], bool]] = [ - ("Final answer: green cup", "green cup", ["green cup"], True), - ("Final answer: the green cup on the table", "green cup", ["green cup"], True), - ("Final answer: red ball", "green cup", ["green cup"], False), - ( - "Final answer: under the circle", - "below the circle", - ["below the circle", "under the circle"], - True, - ), - (" ", "green cup", ["green cup"], False), - ("the star is northFinal answer: yes", "yes", ["yes"], True), + # Verbatim candidate. + ("Final answer: a big blue square", _OPTIONS[0], _OPTIONS, True), + # Candidate recovered from a longer sentence. + ("Final answer: it is a big blue square.", _OPTIONS[0], _OPTIONS, True), + # The other candidate. + ("Final answer: a small blue square", _OPTIONS[0], _OPTIONS, False), + # "both of them" gold is NOT earned by naming one object. + (f"Final answer: {_OPTIONS[0]}", BOTH_LABEL, _OPTIONS, False), + # Echoing the question's options is ambiguous. + (f"Final answer: {_OPTIONS[0]} or {_OPTIONS[1]}", _OPTIONS[0], _OPTIONS, False), + # Aliases of the two fixed labels. + ("Final answer: Both", BOTH_LABEL, _OPTIONS, True), + ("Final answer: neither", NONE_LABEL, _OPTIONS, True), + (" ", _OPTIONS[0], _OPTIONS, False), + (f"reasoningFinal answer: {BOTH_LABEL}", BOTH_LABEL, _OPTIONS, True), ] class TestAcMetricParity: def test_ac_prompt_text_contract(self) -> None: - assert "Final answer: " in PROMPT - assert PROMPT.startswith("Answer the spatial reasoning query below.") - assert PROMPT.rstrip().endswith("{question}") + assert "Final answer: " in PROMPT + assert PROMPT.startswith("Answer the spatial reasoning question below.") + assert "{question}" in PROMPT + # The four candidates must be rendered, else "both of them" / "none of + # them" are unreachable answers. + assert PROMPT.rstrip().endswith("{candidates}") def test_ac_normalize_logic(self) -> None: assert _normalize(" The Big, BLACK Square!! ") == "the big black square" @@ -189,13 +206,9 @@ def test_ac_extract_answer_logic(self) -> None: assert _extract_answer("xFinal answer: yes") == "yes" assert _extract_answer(" ") == "" - @pytest.mark.parametrize("text,target,all_targets,expected", _PARITY_FIXTURES) - async def test_ac_scoring_correctness( - self, text: str, target: str, all_targets: list[str], expected: bool - ) -> None: - result = await _server().verify( - _make_request(text, target=target, all_targets=all_targets) - ) + @pytest.mark.parametrize("text,target,options,expected", _PARITY_FIXTURES) + async def test_ac_scoring_correctness(self, text: str, target: str, options: list[str], expected: bool) -> None: + result = await _server().verify(_make_request(text, target=target, options=options)) assert bool(result.reward) is expected @@ -204,55 +217,46 @@ async def test_ac_scoring_correctness( class TestAcBinaryReward: @pytest.mark.parametrize( - "text,target,all_targets", - [ - ("Final answer: green cup", "green cup", ["green cup"]), - ("Final answer: the green cup on the table", "green cup", ["green cup"]), - ("Final answer: red ball", "green cup", ["green cup"]), - ("", "green cup", ["green cup"]), - ("no marker at all", "green cup", ["green cup"]), - ("Final answer: under the circle", "below the circle", ["below the circle", "under the circle"]), - ], + "text,target,options", + [(f[0], f[1], f[2]) for f in _PARITY_FIXTURES] + + [("no marker at all", _OPTIONS[0], _OPTIONS), ("", _OPTIONS[0], _OPTIONS)], ) - async def test_ac_reward_is_binary( - self, text: str, target: str, all_targets: list[str] - ) -> None: - result = await _server().verify( - _make_request(text, target=target, all_targets=all_targets) - ) + async def test_ac_reward_is_binary(self, text: str, target: str, options: list[str]) -> None: + result = await _server().verify(_make_request(text, target=target, options=options)) assert result.reward in {0.0, 1.0} -# ── AC4: all_targets read from field; falls back to [target] ──────────────── +# ── AC4: options read from field / verifier_metadata; falls back to [target] ─ -class TestAcAllTargets: - async def test_ac_uses_all_targets_field(self) -> None: - # target does not match; a non-first accepted phrase does. +class TestAcCandidateSet: + async def test_ac_uses_options_field(self) -> None: + # The non-gold option must resolve to itself, not to the gold. result = await _server().verify( - _make_request( - "Final answer: under the circle", - target="something else", - all_targets=["something else", "under the circle"], - ) + _make_request(f"Final answer: {_OPTIONS[1]}", target=_OPTIONS[0], options=_OPTIONS) ) - assert result.reward == approx(1.0) + assert result.reward == approx(0.0) + assert result.predicted_label == _OPTIONS[1] - async def test_ac_falls_back_to_target_when_all_targets_empty(self) -> None: - result = await _server().verify( - _make_request("Final answer: yes", target="yes", all_targets=[]) + async def test_ac_reads_options_from_verifier_metadata(self) -> None: + # The native driver drops list fields; verifier_metadata is the fallback. + request = SpartqaVerifyRequest( + responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), + response=_make_response(f"Final answer: {BOTH_LABEL}"), + target="", + verifier_metadata={"target": BOTH_LABEL, "options": _OPTIONS}, ) + result = await _server().verify(request) assert result.reward == approx(1.0) - async def test_ac_falls_back_to_target_when_all_targets_absent(self) -> None: - # all_targets defaults to [] when not supplied -> uses [target]. + async def test_ac_falls_back_to_target_when_options_absent(self) -> None: request = SpartqaVerifyRequest( responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), - response=_make_response("Final answer: green cup"), - target="green cup", + response=_make_response(f"Final answer: {_OPTIONS[0]}"), + target=_OPTIONS[0], ) result = await _server().verify(request) - assert result.all_targets == [] + assert result.options == [] assert result.reward == approx(1.0) @@ -262,15 +266,20 @@ async def test_ac_falls_back_to_target_when_all_targets_absent(self) -> None: class TestAcExtraFields: async def test_ac_extra_fields_present(self) -> None: result = await _server().verify( - _make_request("Final answer: green cup", target="green cup") + _make_request(f"Final answer: {_OPTIONS[0]}", target=_OPTIONS[0], options=_OPTIONS) ) assert result.exact is True assert result.parsed is True - assert result.extracted == "green cup" + assert result.extracted == _OPTIONS[0] + assert result.predicted_label == _OPTIONS[0] - async def test_ac_exact_false_on_contains_only(self) -> None: + async def test_ac_exact_false_when_recovered_from_a_sentence(self) -> None: result = await _server().verify( - _make_request("Final answer: the green cup on the table", target="green cup") + _make_request( + f"Final answer: it is {_OPTIONS[0]} on the left.", + target=_OPTIONS[0], + options=_OPTIONS, + ) ) assert result.reward == approx(1.0) assert result.exact is False @@ -283,7 +292,7 @@ async def test_ac_exact_false_on_contains_only(self) -> None: class TestAcEmptyOutput: @pytest.mark.parametrize("text", ["", " ", "\n\t "]) async def test_ac_empty_output_scores_zero(self, text: str) -> None: - result = await _server().verify(_make_request(text, target="green cup")) + result = await _server().verify(_make_request(text, target=_OPTIONS[0], options=_OPTIONS)) assert result.reward == approx(0.0) assert result.parsed is False assert result.exact is False @@ -302,8 +311,15 @@ def test_ac_example_rows_conform(self) -> None: assert isinstance(messages, list) and messages assert any(m.get("role") == "user" for m in messages) assert isinstance(row["target"], str) and row["target"] - assert isinstance(row["all_targets"], list) and row["all_targets"] + assert isinstance(row["options"], list) and len(row["options"]) == 2 + assert row["verifier_metadata"]["options"] == row["options"] assert row["agent_ref"]["name"] == "spartqa_simple_agent" + # The four candidates must be visible to the model. + content = messages[0]["content"] + for label in [*row["options"], BOTH_LABEL, NONE_LABEL]: + assert label in content + # The gold must be one of the four candidates. + assert _label_key(row["target"]) in {_label_key(c) for c in candidate_labels(row["options"])} # ── AC8: config wires server + agent ──────────────────────────────────────── @@ -327,6 +343,26 @@ def test_ac_prepare_defines_build_helpers(self) -> None: prepare = _load_prepare_module() assert callable(prepare.build_records) assert callable(prepare._unique_preserve_order) + assert callable(prepare.parse_options) + assert callable(prepare.resolve_gold) + + def test_ac_parse_options_splits_the_question_tail(self) -> None: + prepare = _load_prepare_module() + question = "Block A is above block B. What is below the circle? a big square or a small square?" + assert prepare.parse_options(question) == ["a big square", "a small square"] + assert prepare.parse_options("A statement with no alternatives.") is None + + def test_ac_resolve_gold_unflattens_the_retrieval_qrels(self) -> None: + prepare = _load_prepare_module() + options = ["a big square", "a small square"] + # qrels marks all three phrases -> the single label is "both of them". + assert prepare.resolve_gold([BOTH_LABEL, *options], options) == BOTH_LABEL + # Two objects and no sentinel is still "both of them". + assert prepare.resolve_gold(options, options) == BOTH_LABEL + assert prepare.resolve_gold([NONE_LABEL], options) == NONE_LABEL + # A single object gold snaps to the option's own wording. + assert prepare.resolve_gold(["big square"], options) == "a big square" + assert prepare.resolve_gold(["a green cup"], options) is None def test_ac_unique_preserve_order_dedupes_casefold_preserving_order(self) -> None: prepare = _load_prepare_module() @@ -349,7 +385,7 @@ async def test_ac_example_targets_score_one(self) -> None: _make_request( f"Final answer: {row['target']}", target=row["target"], - all_targets=row["all_targets"], + options=row["options"], ) ) assert result.reward == approx(1.0), row["target"] @@ -365,13 +401,12 @@ async def test_ac_rollouts_reproduce_committed_fields(self) -> None: assert rows for row in rows: text = row["response"]["output"][0]["content"][0]["text"] - result = await server.verify( - _make_request(text, target=row["target"], all_targets=row["all_targets"]) - ) + result = await server.verify(_make_request(text, target=row["target"], options=row["options"])) assert result.reward == approx(row["reward"]), text assert result.exact is row["exact"] assert result.parsed is row["parsed"] assert result.extracted == row["extracted"] + assert result.predicted_label == row["predicted_label"] # ── Story edge cases ──────────────────────────────────────────────────────── @@ -381,38 +416,47 @@ class TestEdgeCases: async def test_edge_reasoning_wrapped_output_is_stripped_and_scores(self) -> None: result = await _server().verify( _make_request( - "the star is north of the moonFinal answer: yes", - target="yes", + f"the star is north of the moonFinal answer: {BOTH_LABEL}", + target=BOTH_LABEL, + options=_OPTIONS, ) ) assert result.reward == approx(1.0) - assert result.extracted == "yes" + assert result.extracted == BOTH_LABEL + + async def test_edge_single_object_never_satisfies_a_both_gold(self) -> None: + for option in _OPTIONS: + result = await _server().verify( + _make_request(f"Final answer: {option}", target=BOTH_LABEL, options=_OPTIONS) + ) + assert result.reward == approx(0.0), option - async def test_edge_multiple_answers_any_match_scores_one(self) -> None: + async def test_edge_ambiguous_answer_resolves_to_no_label(self) -> None: result = await _server().verify( _make_request( - "Final answer: above and to the right", - target="upper right", - all_targets=["upper right", "above and to the right"], + f"Final answer: {_OPTIONS[0]} or {_OPTIONS[1]}", + target=BOTH_LABEL, + options=_OPTIONS, ) ) - assert result.reward == approx(1.0) + assert result.reward == approx(0.0) + assert result.predicted_label == "" + + async def test_edge_nested_options_pick_the_most_specific(self) -> None: + options = ["a triangle in block B", "a big blue triangle in block B"] + assert match_label(options[1], candidate_labels(options)) == options[1] - async def test_edge_exact_true_only_on_strict_equality(self) -> None: + async def test_edge_exact_true_only_on_verbatim_answer(self) -> None: strict = await _server().verify( - _make_request( - "Final answer: above and to the right", - target="upper right", - all_targets=["upper right", "above and to the right"], - ) + _make_request(f"Final answer: {_OPTIONS[0]}", target=_OPTIONS[0], options=_OPTIONS) ) assert strict.exact is True loose = await _server().verify( _make_request( - "Final answer: it is above and to the right of the cat", - target="upper right", - all_targets=["upper right", "above and to the right"], + f"Final answer: I believe it is {_OPTIONS[0]}.", + target=_OPTIONS[0], + options=_OPTIONS, ) ) assert loose.reward == approx(1.0) diff --git a/resources_servers/spartqa/tests/test_app.py b/resources_servers/spartqa/tests/test_app.py index dbb3529bb1..2253876b43 100644 --- a/resources_servers/spartqa/tests/test_app.py +++ b/resources_servers/spartqa/tests/test_app.py @@ -17,7 +17,6 @@ from types import SimpleNamespace from unittest.mock import MagicMock -import pytest from pytest import approx from nemo_gym.openai_utils import ( @@ -28,13 +27,18 @@ ) from nemo_gym.server_utils import ServerClient from resources_servers.spartqa.app import ( + BOTH_LABEL, + NONE_LABEL, SpartqaResourcesServer, SpartqaResourcesServerConfig, SpartqaVerifyRequest, _extract_answer, + _label_key, _normalize, _response_text, _strip_reasoning, + candidate_labels, + match_label, ) @@ -70,18 +74,22 @@ def _make_request( text: str, *, target: str = "", - all_targets: list[str] | None = None, + options: list[str] | None = None, verifier_metadata: dict | None = None, ) -> SpartqaVerifyRequest: return SpartqaVerifyRequest( responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), response=_make_response(text), target=target, - all_targets=all_targets or [], + options=options or [], verifier_metadata=verifier_metadata, ) +# A representative CO row: two story objects plus the two fixed labels. +_OPTIONS = ["a big blue square", "a small blue square"] + + def _server() -> SpartqaResourcesServer: return SpartqaResourcesServer(config=_config(), server_client=MagicMock(spec=ServerClient)) @@ -91,9 +99,7 @@ def _server() -> SpartqaResourcesServer: class TestExtractAnswer: def test_pulls_phrase_after_final_answer(self) -> None: - assert _extract_answer("Reasoning here.\nFinal answer: below the circle") == ( - "below the circle" - ) + assert _extract_answer("Reasoning here.\nFinal answer: below the circle") == ("below the circle") def test_strips_think_reasoning(self) -> None: text = "the star is northFinal answer: yes" @@ -129,86 +135,137 @@ def test_empty(self) -> None: class TestVerify: - async def test_exact_match(self) -> None: + async def test_verbatim_gold_scores_one_and_exact(self) -> None: result = await _server().verify( - _make_request("Final answer: green cup", target="green cup") + _make_request("Final answer: a big blue square", target=_OPTIONS[0], options=_OPTIONS) ) assert result.reward == approx(1.0) assert result.exact is True assert result.parsed is True - assert result.extracted == "green cup" + assert result.extracted == "a big blue square" + assert result.predicted_label == _OPTIONS[0] - async def test_contains_match_not_exact(self) -> None: + async def test_gold_inside_a_sentence_scores_one_but_not_exact(self) -> None: result = await _server().verify( - _make_request("Final answer: the green cup on the table", target="green cup") + _make_request( + "Final answer: it must be a big blue square.", + target=_OPTIONS[0], + options=_OPTIONS, + ) ) assert result.reward == approx(1.0) assert result.exact is False + assert result.predicted_label == _OPTIONS[0] - async def test_no_match(self) -> None: + async def test_wrong_option_scores_zero(self) -> None: result = await _server().verify( - _make_request("Final answer: red ball", target="green cup") + _make_request("Final answer: a small blue square", target=_OPTIONS[0], options=_OPTIONS) ) assert result.reward == approx(0.0) - assert result.exact is False - assert result.parsed is True + assert result.predicted_label == _OPTIONS[1] - async def test_empty_output_reward_zero_no_raise(self) -> None: - result = await _server().verify(_make_request(" ", target="green cup")) - assert result.reward == approx(0.0) - assert result.parsed is False + async def test_both_of_them_gold(self) -> None: + result = await _server().verify( + _make_request(f"Final answer: {BOTH_LABEL}", target=BOTH_LABEL, options=_OPTIONS) + ) + assert result.reward == approx(1.0) + assert result.exact is True - async def test_all_targets_empty_falls_back_to_target(self) -> None: + async def test_none_of_them_gold(self) -> None: result = await _server().verify( - _make_request("Final answer: yes", target="yes", all_targets=[]) + _make_request(f"Final answer: {NONE_LABEL}", target=NONE_LABEL, options=_OPTIONS) ) assert result.reward == approx(1.0) - async def test_multiple_targets_any_match(self) -> None: + async def test_article_difference_still_matches(self) -> None: + # qrels phrases and the story wording disagree on leading articles. result = await _server().verify( _make_request( - "Final answer: under the circle", - target="below the circle", - all_targets=["below the circle", "under the circle"], + "Final answer: the big blue square", + target="a big blue square", + options=_OPTIONS, ) ) assert result.reward == approx(1.0) assert result.exact is True - async def test_empty_targets_are_skipped(self) -> None: - result = await _server().verify( - _make_request("Final answer: yes", target="", all_targets=["", " "]) - ) - assert result.reward == approx(0.0) - - async def test_all_targets_from_verifier_metadata(self) -> None: - # The native driver drops the top-level ``all_targets`` list; the full - # accepted set must be recoverable from verifier_metadata. + async def test_alias_resolves_to_fixed_label(self) -> None: + for alias, gold in (("Both", BOTH_LABEL), ("Neither", NONE_LABEL), ("DK", NONE_LABEL)): + result = await _server().verify(_make_request(f"Final answer: {alias}", target=gold, options=_OPTIONS)) + assert result.reward == approx(1.0), alias + assert result.exact is False, alias + + async def test_naming_a_single_option_does_not_earn_a_both_gold(self) -> None: + # Regression: the retrieval qrels list "both of them" alongside each + # object, so any-of matching used to credit this. + for option in _OPTIONS: + result = await _server().verify( + _make_request(f"Final answer: {option}", target=BOTH_LABEL, options=_OPTIONS) + ) + assert result.reward == approx(0.0), option + + async def test_echoing_both_options_is_ambiguous_and_scores_zero(self) -> None: + # Regression: "X or Y" copied straight out of the question used to score + # 1.0 on ~95% of the corpus via substring matching. + text = f"Final answer: {_OPTIONS[0]} or {_OPTIONS[1]}" + for gold in (_OPTIONS[0], _OPTIONS[1], BOTH_LABEL): + result = await _server().verify(_make_request(text, target=gold, options=_OPTIONS)) + assert result.reward == approx(0.0), gold + assert result.predicted_label == "" + + async def test_nested_options_resolve_to_the_most_specific(self) -> None: + options = ["a triangle that is in block B", "a big blue triangle that is in block B"] result = await _server().verify( _make_request( - "Final answer: under the circle", - target="below the circle", - all_targets=[], - verifier_metadata={ - "target": "below the circle", - "all_targets": ["below the circle", "under the circle"], - }, + "Final answer: a big blue triangle that is in block B", + target=options[1], + options=options, ) ) assert result.reward == approx(1.0) - assert result.exact is True + assert result.predicted_label == options[1] + + async def test_empty_output_reward_zero_no_raise(self) -> None: + result = await _server().verify(_make_request(" ", target=_OPTIONS[0], options=_OPTIONS)) + assert result.reward == approx(0.0) + assert result.parsed is False + assert result.predicted_label == "" - async def test_target_from_verifier_metadata_when_top_level_missing(self) -> None: + async def test_empty_target_scores_zero(self) -> None: + result = await _server().verify(_make_request("Final answer: yes", target="", options=[])) + assert result.reward == approx(0.0) + + async def test_fields_from_verifier_metadata(self) -> None: + # The native driver drops the top-level ``options`` list; the candidate + # set must be recoverable from verifier_metadata. result = await _server().verify( _make_request( - "Final answer: yes", + f"Final answer: {BOTH_LABEL}", target="", - all_targets=[], - verifier_metadata={"target": "yes"}, + options=[], + verifier_metadata={"target": BOTH_LABEL, "options": _OPTIONS}, ) ) assert result.reward == approx(1.0) + async def test_options_missing_falls_back_to_target_only(self) -> None: + result = await _server().verify(_make_request("Final answer: a big blue square", target=_OPTIONS[0])) + assert result.reward == approx(1.0) + + +class TestLabelMatching: + def test_candidate_labels_appends_fixed_labels_and_dedupes(self) -> None: + assert candidate_labels(_OPTIONS) == [*_OPTIONS, BOTH_LABEL, NONE_LABEL] + assert candidate_labels([BOTH_LABEL]) == [BOTH_LABEL, NONE_LABEL] + + def test_label_key_strips_articles_and_punctuation(self) -> None: + assert _label_key(" The Big, BLUE Square! ") == "big blue square" + assert _label_key("") == "" + + def test_match_label_returns_none_when_nothing_matches(self) -> None: + assert match_label("a green cup", candidate_labels(_OPTIONS)) is None + assert match_label("", candidate_labels(_OPTIONS)) is None + # ── compute_metrics() / get_key_metrics() ────────────────────────────────── @@ -216,16 +273,35 @@ async def test_target_from_verifier_metadata_when_top_level_missing(self) -> Non class TestComputeMetrics: def test_mean_and_rates(self) -> None: tasks = [ - [{"reward": 1.0, "exact": True, "parsed": True}], - [{"reward": 0.0, "exact": False, "parsed": True}], - [{"reward": 1.0, "exact": False, "parsed": True}], - [{"reward": 0.0, "exact": False, "parsed": False}], + [{"reward": 1.0, "exact": True, "parsed": True, "target": BOTH_LABEL, "predicted_label": BOTH_LABEL}], + [ + { + "reward": 0.0, + "exact": False, + "parsed": True, + "target": BOTH_LABEL, + "predicted_label": "a big blue square", + } + ], + [ + { + "reward": 1.0, + "exact": False, + "parsed": True, + "target": "a big blue square", + "predicted_label": "a big blue square", + } + ], + [{"reward": 0.0, "exact": False, "parsed": False, "target": NONE_LABEL, "predicted_label": ""}], ] metrics = _server().compute_metrics(tasks) assert metrics["mean_reward"] == approx(0.5) assert metrics["count"] == 4 assert metrics["exact_match_rate"] == approx(0.25) assert metrics["parse_rate"] == approx(0.75) + assert metrics["label_resolve_rate"] == approx(0.75) + assert metrics["accuracy_both_of_them"] == approx(0.5) + assert metrics["accuracy_none_of_them"] == approx(0.0) def test_empty(self) -> None: assert _server().compute_metrics([]) == {} @@ -233,9 +309,7 @@ def test_empty(self) -> None: class TestGetKeyMetrics: def test_selects_headline(self) -> None: - out = _server().get_key_metrics( - {"mean_reward": 0.5, "exact_match_rate": 0.25, "parse_rate": 0.75, "count": 4} - ) + out = _server().get_key_metrics({"mean_reward": 0.5, "exact_match_rate": 0.25, "parse_rate": 0.75, "count": 4}) assert out == {"mean_reward": approx(0.5), "exact_match_rate": approx(0.25)} @@ -265,18 +339,32 @@ def test_fallback_string_content(self) -> None: class TestAcceptance: - async def test_each_example_target_scores_one(self) -> None: + async def test_each_example_gold_scores_one(self) -> None: server = _server() - rows = [ - json.loads(line) for line in _EXAMPLE_JSONL.read_text().splitlines() if line.strip() - ] + rows = [json.loads(line) for line in _EXAMPLE_JSONL.read_text().splitlines() if line.strip()] assert len(rows) >= 5 for row in rows: - resp_text = f"Final answer: {row['target']}" result = await server.verify( _make_request( - resp_text, target=row["target"], all_targets=row.get("all_targets", []) + f"Final answer: {row['target']}", + target=row["target"], + options=row["options"], ) ) - assert result.reward == approx(1.0) + assert result.reward == approx(1.0), row["target"] + + async def test_example_rows_cover_both_and_none_golds(self) -> None: + rows = [json.loads(line) for line in _EXAMPLE_JSONL.read_text().splitlines() if line.strip()] + golds = {row["target"] for row in rows} + assert BOTH_LABEL in golds + assert NONE_LABEL in golds + # Every row must offer exactly two story objects to choose between. + assert all(len(row["options"]) == 2 for row in rows) + async def test_echoing_the_question_options_never_scores(self) -> None: + server = _server() + rows = [json.loads(line) for line in _EXAMPLE_JSONL.read_text().splitlines() if line.strip()] + for row in rows: + text = f"Final answer: {row['options'][0]} or {row['options'][1]}" + result = await server.verify(_make_request(text, target=row["target"], options=row["options"])) + assert result.reward == approx(0.0), row["target"] From 9cb8cd25a8b8f25167292ac15dbbdc835bf4e243 Mon Sep 17 00:00:00 2001 From: mbagdasarova Date: Wed, 29 Jul 2026 14:09:51 +0200 Subject: [PATCH 4/7] feat(app): improve answer extraction logic to handle multiple final answers and ignore template echoes Signed-off-by: mbagdasarova --- resources_servers/spartqa/app.py | 13 ++++++++++--- resources_servers/spartqa/tests/test_app.py | 17 +++++++++++++++++ 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/resources_servers/spartqa/app.py b/resources_servers/spartqa/app.py index c7e9bbba39..7b1ae7c5c0 100644 --- a/resources_servers/spartqa/app.py +++ b/resources_servers/spartqa/app.py @@ -130,9 +130,16 @@ def _extract_answer(text: str) -> str: ] extracted = None for pattern in patterns: - matches = list(re.finditer(pattern, text, flags=re.IGNORECASE | re.DOTALL | re.MULTILINE)) - if matches: - extracted = matches[-1].group(1).strip() + matches = list(re.finditer(pattern, text, flags=re.IGNORECASE | re.MULTILINE)) + # Drop the instruction-template echo (``Final answer: ``) and empty captures; keep the last remaining real answer. + candidates = [ + captured + for m in matches + if (captured := m.group(1).strip()) and _normalize(captured) != "candidate answer" + ] + if candidates: + extracted = candidates[-1] break if extracted is not None: text = extracted diff --git a/resources_servers/spartqa/tests/test_app.py b/resources_servers/spartqa/tests/test_app.py index 2253876b43..3b9aeb1b63 100644 --- a/resources_servers/spartqa/tests/test_app.py +++ b/resources_servers/spartqa/tests/test_app.py @@ -111,6 +111,23 @@ def test_thinking_process_prefix_returns_last_line(self) -> None: text = "Thinking process: I consider the layout\ngreen cup" assert _extract_answer(text) == "green cup" + def test_ignores_template_echo_and_takes_last_answer(self) -> None: + # Reasoning models often restate the prompt's format instruction + # ("Final answer: ") mid-thought. The extractor must + # skip that placeholder echo and return the real concluding answer, + # not the first "Final answer:" line it sees. + text = ( + "To answer I must end with 'Final answer: '.\n" + "Working through the layout, the yellow circle is below.\n" + "Final answer: a big yellow circle that is touching the bottom edge of a block" + ) + assert _extract_answer(text) == "a big yellow circle that is touching the bottom edge of a block" + + def test_multiple_final_answer_lines_takes_last(self) -> None: + # A model that drafts an answer then revises it: the last line wins. + text = "Final answer: a small blue shape\nOn reflection:\nFinal answer: both of them" + assert _extract_answer(text) == "both of them" + def test_empty_returns_empty(self) -> None: assert _extract_answer(" ") == "" From 0501c4d371198d7673a9f2dbc009a7846665344b Mon Sep 17 00:00:00 2001 From: mbagdasarova Date: Fri, 31 Jul 2026 14:05:30 +0200 Subject: [PATCH 5/7] feat(readme): add SpartQA spatial-reasoning answer generation details Signed-off-by: mbagdasarova --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 69185488bb..bcc4d3e25e 100644 --- a/README.md +++ b/README.md @@ -307,6 +307,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace). | Single Step Tool Use With Argument Comparison | agent | General function-calling RL dataset using expert trajectories; behavior cloning to match expert tool calls per step. | - | ✓ | ✓ | Creative Commons Attribution 4.0 International | toolcall_schema_single_step_tool_use_with_argument_comparison.yaml | Nemotron-RL-Agentic-Function-Calling-Pivot-v1 | | Single Step Tool Use With Argument Comparison | agent | GitHub-issue dataset for software-engineering agents; refactored from SWE-Gym and SWE-Bench-Verified for NeMo Gym. | - | ✓ | ✓ | Creative Commons Attribution 4.0 International | swe_pivot_single_step_tool_use_with_argument_comparison.yaml | Nemotron-RL-Agentic-SWE-Pivot-v1 | | Single Step Tool Use With Argument Comparison | agent | The model must output the next correct call in a given trajectory involving search tools. | Improve agentic search capability. | ✓ | ✓ | Apache 2.0 | search_pivot_single_step_tool_use_with_argument_comparison.yaml | - | +| Spartqa | reasoning | SpartQA spatial-reasoning answer generation; reward = 1 on an exact-or-answer-containing match against any accepted answer phrase. | Improve spatial and relational reasoning over described scenes. | - | ✓ | - | spartqa.yaml | - | | Speed Bench | other | Speculative-decoding throughput benchmark. Reads vLLM `/metrics` Prometheus counters before/after generation to compute acceptance length and acceptance rate. | Measure inference-time speculative-decoding effectiveness for serving research and regression tests. | - | - | - | speed_bench.yaml | - | | Spider2 Lite | coding | Text-to-SQL with execution-based evaluation on Spider 2.0-Lite (135 SQLite tasks). Binary reward based on result-set equivalence. | Improve text-to-SQL capabilities for real-world enterprise queries using execution-based binary reward without an LLM judge. | - | ✓ | - | spider2_lite.yaml | - | | Structeval | instruction_following | StructEval non-renderable format verification (JSON, YAML, CSV, TOML, XML) | Improve structured output generation quality | ✓ | - | Apache 2.0 | structeval_nonrenderable.yaml | - | From da3ff35b1ab5a4da8b9b713f9627e266e9e66fb0 Mon Sep 17 00:00:00 2001 From: mbagdasarova Date: Fri, 31 Jul 2026 14:20:33 +0200 Subject: [PATCH 6/7] feat(readme): update SpartQA spatial reasoning description for CO question type Signed-off-by: mbagdasarova --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index bcc4d3e25e..f9141b532b 100644 --- a/README.md +++ b/README.md @@ -307,7 +307,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace). | Single Step Tool Use With Argument Comparison | agent | General function-calling RL dataset using expert trajectories; behavior cloning to match expert tool calls per step. | - | ✓ | ✓ | Creative Commons Attribution 4.0 International | toolcall_schema_single_step_tool_use_with_argument_comparison.yaml | Nemotron-RL-Agentic-Function-Calling-Pivot-v1 | | Single Step Tool Use With Argument Comparison | agent | GitHub-issue dataset for software-engineering agents; refactored from SWE-Gym and SWE-Bench-Verified for NeMo Gym. | - | ✓ | ✓ | Creative Commons Attribution 4.0 International | swe_pivot_single_step_tool_use_with_argument_comparison.yaml | Nemotron-RL-Agentic-SWE-Pivot-v1 | | Single Step Tool Use With Argument Comparison | agent | The model must output the next correct call in a given trajectory involving search tools. | Improve agentic search capability. | ✓ | ✓ | Apache 2.0 | search_pivot_single_step_tool_use_with_argument_comparison.yaml | - | -| Spartqa | reasoning | SpartQA spatial-reasoning answer generation; reward = 1 on an exact-or-answer-containing match against any accepted answer phrase. | Improve spatial and relational reasoning over described scenes. | - | ✓ | - | spartqa.yaml | - | +| Spartqa | reasoning | SpartQA spatial reasoning, CO (choose-object) question type; four candidate answers per question (two story objects, "both of them", "none of them"); reward = 1 iff the answer resolves to the single gold label. | Improve spatial and relational reasoning over described scenes. | - | ✓ | - | spartqa.yaml | - | | Speed Bench | other | Speculative-decoding throughput benchmark. Reads vLLM `/metrics` Prometheus counters before/after generation to compute acceptance length and acceptance rate. | Measure inference-time speculative-decoding effectiveness for serving research and regression tests. | - | - | - | speed_bench.yaml | - | | Spider2 Lite | coding | Text-to-SQL with execution-based evaluation on Spider 2.0-Lite (135 SQLite tasks). Binary reward based on result-set equivalence. | Improve text-to-SQL capabilities for real-world enterprise queries using execution-based binary reward without an LLM judge. | - | ✓ | - | spider2_lite.yaml | - | | Structeval | instruction_following | StructEval non-renderable format verification (JSON, YAML, CSV, TOML, XML) | Improve structured output generation quality | ✓ | - | Apache 2.0 | structeval_nonrenderable.yaml | - | From 5aee79e928203ad988a2833a64e89b324f85330d Mon Sep 17 00:00:00 2001 From: mbagdasarova Date: Wed, 12 Aug 2026 11:51:01 +0200 Subject: [PATCH 7/7] fix(readme): update training servers table descriptions for clarity Signed-off-by: mbagdasarova --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2098fddaea..ea332e2b20 100644 --- a/README.md +++ b/README.md @@ -210,7 +210,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace). | Environment | Domain | Description | Value | Train | Validation | License | Config | Dataset | -| --------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----- | ---------- | --------------------------------------------------------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| --------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----- | ---------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Aalcr | other | - | - | - | - | - | aalcr.yaml | - | | Abstention | rlhf | Train models to abstain when unsure using three-tier reward on HotPotQA with LLM judge | Improve calibration by rewarding abstention over incorrect answers | ✓ | ✓ | Creative Commons Attribution-ShareAlike 4.0 International | abstention.yaml | - | | Anyswe Agent | coding | SWE-bench run by Claude Code natively inside the task container. | Eval software engineering capabilities on SWE-bench with any Gym agent. | - | - | - | anyswe_claude_code.yaml | - |