diff --git a/README.md b/README.md index 6e7c48d238..ea332e2b20 100644 --- a/README.md +++ b/README.md @@ -337,6 +337,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace). | Single Step Tool Use With Argument Comparison | agent | GitHub-issue dataset for software-engineering agents; refactored from SWE-Gym and SWE-Bench-Verified for NeMo Gym. | - | ✓ | ✓ | Creative Commons Attribution 4.0 International | swe_pivot_single_step_tool_use_with_argument_comparison.yaml | Nemotron-RL-Agentic-SWE-Pivot-v1 | | Single Step Tool Use With Argument Comparison | agent | Parallel tool-call verification; the model must emit the expected set of tool calls in a single turn, in any order. | - | - | - | - | parallel_tool_calls_single_step_tool_use_with_argument_comparison.yaml | - | | Single Step Tool Use With Argument Comparison | agent | The model must output the next correct call in a given trajectory involving search tools. | Improve agentic search capability. | ✓ | ✓ | Apache 2.0 | search_pivot_single_step_tool_use_with_argument_comparison.yaml | - | +| Spartqa | reasoning | SpartQA spatial reasoning, CO (choose-object) question type; four candidate answers per question (two story objects, "both of them", "none of them"); reward = 1 iff the answer resolves to the single gold label. | Improve spatial and relational reasoning over described scenes. | - | ✓ | - | spartqa.yaml | - | | Speed Bench | other | Speculative-decoding throughput benchmark. Reads vLLM `/metrics` Prometheus counters before/after generation to compute acceptance length and acceptance rate. | Measure inference-time speculative-decoding effectiveness for serving research and regression tests. | - | - | - | speed_bench.yaml | - | | Spider2 Lite | coding | Text-to-SQL with execution-based evaluation on Spider 2.0-Lite (135 SQLite tasks). Binary reward based on result-set equivalence. | Improve text-to-SQL capabilities for real-world enterprise queries using execution-based binary reward without an LLM judge. | - | ✓ | - | spider2_lite.yaml | - | | String Match | knowledge | General-purpose string matching verifier for free-form text answers | Verify answers via case-insensitive string matching with multiple extraction strategies | - | - | - | string_match.yaml | - | diff --git a/resources_servers/spartqa/README.md b/resources_servers/spartqa/README.md new file mode 100644 index 0000000000..084daad431 --- /dev/null +++ b/resources_servers/spartqa/README.md @@ -0,0 +1,123 @@ +# SpartQA Resources Server + +Spatial-reasoning benchmark covering SpartQA's **CO (choose-object)** question +type. The model is shown a story, a question of the form "… X or Y?", and four +candidate answers; it must copy one of them, ending with a +`Final answer: ` line. The per-sample reward is `1.0` iff the +answer resolves to the gold label, else `0.0`. + +Source dataset: [`mteb/SpartQA`](https://huggingface.co/datasets/mteb/SpartQA) +(MTEB retrieval form — `queries` / `corpus` / `qrels` splits joined at prep +time by `prepare_spartqa.py`). + +## Task definition + +Per the SpartQA paper (Mirzaee et al., NAACL 2021, Table 8 fn.), CO is a +**four-label single-choice** task: + +| Label | Meaning | +|-------|---------| +| `X` | only the first object named in the question | +| `Y` | only the second object | +| `both of them` | both objects satisfy the asked relation | +| `none of them` | neither does (the paper's DK / None / `[]`) | + +The retrieval encoding has no way to express "both", so a `both of them` gold +is flattened into **three** relevant qrels documents (the phrase plus each +object). `prepare_spartqa.py` (`resolve_gold`) undoes that flattening back into +one gold label, and renders all four candidates into the prompt so +`both of them` / `none of them` are reachable answers. + +> Scoring the three flattened phrases as interchangeable accepted answers — +> i.e. crediting a single object for a `both of them` gold — is not the paper's +> metric and is trivially gameable: echoing the question's two options back +> ("X or Y") scores **94.9%** that way, versus **0.0%** here. + +Corpus composition (`test`, 3594 rows): 1579 `both of them` (43.9%), +183 `none of them` (5.1%), 1832 a single object. The majority-class baseline is +therefore **43.9%**; always answering the first option scores **26.1%**. + +## Scoring + +`verify()` extracts the model's final answer (`_extract_answer` / +`_strip_reasoning` / `_clean_candidate`), then resolves it to one of the four +candidate labels (`match_label`): + +1. **Verbatim match** — article- and punctuation-insensitive equality with a + candidate (`_label_key`). Sets `exact`. Aliases (`both`, `neither`, `DK`, …) + each map to exactly one fixed label. +2. **Unambiguous containment** — the answer contains exactly one candidate + (when one candidate nests inside another, the most specific wins). Scores + without `exact`. +3. Otherwise the answer resolves to **no label** and scores `0.0`. An answer + naming two candidates is ambiguous by construction, so it never scores. + +Reward is `1.0` iff the resolved label is the gold label. Empty output scores +`0.0` and never raises. + +## Metrics + +`compute_metrics` reports: + +- `mean_reward` — mean per-sample CO accuracy (also the reward). +- `exact_match_rate` — fraction that answered the gold label verbatim. +- `parse_rate` — fraction where a non-empty answer phrase was extracted. +- `label_resolve_rate` — fraction that resolved to any candidate label; a low + value means the model is not following the copy-a-candidate format. +- `accuracy_both_of_them` / `accuracy_none_of_them` — accuracy on those gold + slices. A model that never answers `both of them` — 44% of the corpus — + shows up as a near-zero slice even when the headline number looks healthy. + +`get_key_metrics` surfaces `mean_reward` and `exact_match_rate`. + +> **Reasoning models:** `verify()` strips a leading `…` block +> before extracting the answer. + +## Prepare the dataset + +```bash +cd gym +python resources_servers/spartqa/prepare_spartqa.py --split test +``` + +This joins `mteb/SpartQA` (via the HF `datasets` library) and writes the +gitignored `data/spartqa_test.jsonl`. The committed `data/example.jsonl` is a +5-row slice sampled from that file, covering a `both of them` gold, a +`none of them` gold, and both single-object cases. + +## Example rollouts and metrics + +`data/example_rollouts.jsonl` and `data/example_metrics.json` are committed +and can be regenerated at any time with the scripts below (no servers needed): + +```bash +# Regenerate synthetic rollouts (rule-based scorer, no model call) +python resources_servers/spartqa/generate_example_rollouts.py + +# Regenerate dataset stats summary +python resources_servers/spartqa/generate_example_metrics.py + +# Inspect +tail -n 1 resources_servers/spartqa/data/example_rollouts.jsonl | jq .reward +cat resources_servers/spartqa/data/example_metrics.json | jq . +``` + +Note: row 3 (index 3) in the example rollouts is intentionally wrong (reward +0.0) — it echoes both of the question's options back, the degenerate answer the +scorer must not credit. The remaining four rows score 1.0 and cover a verbatim +match, an alias (`neither` → `none of them`), and a label recovered from a +longer sentence. + +## Run + +```bash +gym env start --resources-server spartqa --model-type vllm_model +``` + +No API keys are required — all scoring is rule-based. + +## Test + +```bash +gym env test --resources-server spartqa +``` diff --git a/resources_servers/spartqa/app.py b/resources_servers/spartqa/app.py new file mode 100644 index 0000000000..7b1ae7c5c0 --- /dev/null +++ b/resources_servers/spartqa/app.py @@ -0,0 +1,323 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""SpartQA resources server — the CO (choose-object) question type. + +``mteb/SpartQA`` is the MTEB retrieval form of SpartQA (Mirzaee et al., NAACL +2021) and contains CO questions only: a story plus "… or ?". Per the paper (Table 8 fn.), CO is a **four-label single-choice** task — +``X``, ``Y``, ``both of them``, ``none of them`` — and the metric is accuracy +over that label set. + +``prepare_spartqa.py`` recovers that label set from the retrieval qrels (which +flatten "both of them" into three relevant documents), renders the four +candidates into the prompt, and stores the single gold label in ``target`` and +the two object labels in ``options``. + +The per-sample reward is ``1.0`` iff the model's answer resolves to the gold +label, else ``0.0`` — so ``compute_metrics``'s mean-of-rewards equals CO +accuracy. Resolution is a verbatim match against one candidate, or a +containment match when it is *unambiguous*: a response naming two candidates +(e.g. echoing "X or Y" back) resolves to nothing and scores 0.0. Scoring any of +the three flattened qrels phrases as correct would let that echo score ~95%. +""" + +from __future__ import annotations + +import re +import string +from typing import Any, Dict, List, Optional + +from pydantic import ConfigDict, Field + +from nemo_gym.base_resources_server import ( + BaseResourcesServerConfig, + BaseRunRequest, + BaseVerifyRequest, + BaseVerifyResponse, + SimpleResourcesServer, +) +from nemo_gym.openai_utils import NeMoGymResponse + + +PROMPT = """\ +Answer the spatial reasoning question below. +Choose exactly one of the candidate answers and copy it verbatim. Answer +"both of them" if both listed objects satisfy the question, and "none of them" +if neither of them does. + +End your response with one line in this exact format: +Final answer: + +Story and question: +{question} + +Candidate answers: +{candidates} +""" + +# The two story-independent CO labels. Aliases let a model phrase them its own +# way; each alias maps to exactly one label, so this adds no leniency across +# labels. +BOTH_LABEL = "both of them" +NONE_LABEL = "none of them" + +_BOTH_ALIASES = frozenset({"both", "both of them", "both objects", "both of the objects", "both of these"}) +# The paper treats DK / None / [] alike: none of the candidate objects hold. +_NONE_ALIASES = frozenset( + { + "none", + "none of them", + "none of the objects", + "neither", + "neither of them", + "neither object", + "no object", + "dk", + "do not know", + "dont know", + } +) + + +# ── Answer extraction + normalization ─────────────────────────────────────── + + +def _normalize(text: str) -> str: + table = str.maketrans("", "", string.punctuation) + normalized = text.strip().lower().translate(table) + return " ".join(normalized.split()) + + +def _strip_reasoning(text: str) -> str: + text = re.sub(r".*?", "", text, flags=re.IGNORECASE | re.DOTALL) + text = re.sub( + r"<\|channel\>thought\s*.*?", + "", + text, + flags=re.IGNORECASE | re.DOTALL, + ) + return text.strip() + + +def _clean_candidate(text: str) -> str: + text = text.strip() + text = re.sub(r"^[-*•\s]+", "", text) + text = re.sub(r"^\*+|\*+$", "", text).strip() + text = re.sub(r"", "", text, flags=re.IGNORECASE).strip() + return text.strip().strip("\"'` ") + + +def _extract_answer(text: str) -> str: + text = _strip_reasoning(text) + patterns = [ + r"(?:^|\n)\s*(?:[*_`#>\-\s]*)final\s+answer(?:\s+is)?\s*(?:[*_`\s])*[:\-]\s*(.+)", + r"(?:^|\n)\s*(?:[*_`#>\-\s]*)selected\s+answer(?:\s+is)?\s*(?:[*_`\s])*[:\-]\s*(.+)", + r"(?:^|\n)\s*(?:[*_`#>\-\s]*)answer(?:\s+is)?\s*(?:[*_`\s])*[:\-]\s*(.+)", + r"\b(?:the\s+)?(?:final\s+)?answer\s+(?:is|would\s+be|should\s+be)\s*[:\-]?\s*(.+)", + r"\bselected\s+(?:option|answer)\s+(?:is|would\s+be)\s*[:\-]?\s*(.+)", + ] + extracted = None + for pattern in patterns: + matches = list(re.finditer(pattern, text, flags=re.IGNORECASE | re.MULTILINE)) + # Drop the instruction-template echo (``Final answer: ``) and empty captures; keep the last remaining real answer. + candidates = [ + captured + for m in matches + if (captured := m.group(1).strip()) and _normalize(captured) != "candidate answer" + ] + if candidates: + extracted = candidates[-1] + break + if extracted is not None: + text = extracted + + lines = [_clean_candidate(line) for line in text.splitlines() if line.strip()] + lines = [line for line in lines if line and _normalize(line) not in {"final answer", "answer"}] + if not lines: + return "" + + if extracted is None and len(lines) > 1: + first = _normalize(lines[0]) + if first.startswith(("thinking process", "analysis", "the user wants", "we need")): + return lines[-1] + return lines[0] + + +def _label_key(text: str) -> str: + """Normalize a candidate label or a prediction for comparison. + + Article-insensitive, because the qrels phrases and the story's own wording + disagree on leading articles ("medium yellow square …" vs "the medium + yellow square …"). + """ + return re.sub(r"^(?:a|an|the)\s+", "", _normalize(text)).strip() + + +def candidate_labels(options: List[str]) -> List[str]: + """The four CO labels: the two story objects, then ``both`` / ``none``.""" + labels: List[str] = [] + seen: set[str] = set() + for label in [*options, BOTH_LABEL, NONE_LABEL]: + key = _label_key(str(label)) + if key and key not in seen: + seen.add(key) + labels.append(str(label).strip()) + return labels + + +def match_label(prediction: str, labels: List[str]) -> Optional[str]: + """Resolve a free-text answer to exactly one candidate label, or ``None``. + + Verbatim (article-insensitive) equality wins outright. Otherwise the answer + must *contain* exactly one label — a response naming two of them is + ambiguous and resolves to ``None`` rather than being credited for whichever + one happens to be gold. + """ + key = _label_key(prediction) + if not key: + return None + + by_key = {_label_key(label): label for label in labels} + if key in by_key: + return by_key[key] + if key in _BOTH_ALIASES and BOTH_LABEL in by_key.values(): + return by_key[_label_key(BOTH_LABEL)] + if key in _NONE_ALIASES and NONE_LABEL in by_key.values(): + return by_key[_label_key(NONE_LABEL)] + + hits = [label_key for label_key in by_key if label_key and label_key in key] + # One option can nest inside the other ("a triangle in block B" inside "a + # big blue triangle in block B"); the most specific match is the intended + # one, so drop labels wholly contained in another hit. + hits = [h for h in hits if not any(other != h and h in other for other in hits)] + return by_key[hits[0]] if len(hits) == 1 else None + + +def _response_text(response: NeMoGymResponse) -> str: + """Best-effort extraction of the assistant text from a NeMoGymResponse.""" + text = getattr(response, "output_text", None) + if isinstance(text, str) and text: + return text + parts: List[str] = [] + for item in getattr(response, "output", None) or []: + if getattr(item, "type", None) != "message": + continue + content = getattr(item, "content", "") + if isinstance(content, str): + parts.append(content) + continue + for c in content or []: + t = c.get("text") if isinstance(c, dict) else getattr(c, "text", None) + if isinstance(t, str): + parts.append(t) + return "".join(parts) + + +# ── Request / response shapes ───────────────────────────────────────────── + + +class SpartqaResourcesServerConfig(BaseResourcesServerConfig): + name: str = "spartqa" + + +class SpartqaRunRequest(BaseRunRequest): + model_config = ConfigDict(extra="allow") + + # ``target`` is the single gold CO label; ``options`` are the two story + # objects offered by the question. ``target`` (a scalar) survives the + # nemo-evaluator ``gym://...protocol=native`` driver, which forwards a row's + # top-level scalar fields onto ``/verify`` but DROPS list/dict fields + # (``options`` never arrives that way). The options therefore also ride in + # ``verifier_metadata``, which the driver forwards intact; verify() falls + # back to it so the full candidate set is always available. + target: str = "" + options: List[str] = Field(default_factory=list) + verifier_metadata: Optional[Dict[str, Any]] = None + + +class SpartqaVerifyRequest(SpartqaRunRequest, BaseVerifyRequest): + pass + + +class SpartqaVerifyResponse(BaseVerifyResponse): + model_config = ConfigDict(extra="allow") + + exact: bool = False + parsed: bool = False + extracted: str = "" + predicted_label: str = "" + + +class SpartqaResourcesServer(SimpleResourcesServer): + config: SpartqaResourcesServerConfig + + async def verify(self, body: SpartqaVerifyRequest) -> SpartqaVerifyResponse: + prediction = _extract_answer(_response_text(body.response)) + # Prefer the explicit fields; fall back to verifier_metadata, the only + # path that survives the native driver. + meta = body.verifier_metadata or {} + gold = body.target or str(meta.get("target", "")) + options = body.options or list(meta.get("options") or []) + + labels = candidate_labels(options) if options else candidate_labels([gold]) + predicted = match_label(prediction, labels) + gold_key = _label_key(gold) + correct = bool(gold_key) and predicted is not None and _label_key(predicted) == gold_key + + return SpartqaVerifyResponse( + **body.model_dump(), + reward=1.0 if correct else 0.0, + # ``exact`` means the gold label was copied verbatim, as instructed, + # rather than recovered from a longer sentence. + exact=correct and _label_key(prediction) == gold_key, + parsed=bool(_normalize(prediction)), + extracted=prediction[:200], + predicted_label=predicted or "", + ) + + # --- aggregation ----------------------------------------------------- + + def compute_metrics(self, tasks: List[List[Dict[str, Any]]]) -> Dict[str, Any]: + rows = [r for task_rollouts in tasks for r in task_rollouts] + if not rows: + return {} + + metrics: Dict[str, Any] = {} + rewards = [r["reward"] for r in rows if isinstance(r.get("reward"), (int, float))] + if rewards: + metrics["mean_reward"] = sum(rewards) / len(rewards) + metrics["count"] = len(rewards) + metrics["exact_match_rate"] = sum(1 for r in rows if r.get("exact")) / len(rows) + metrics["parse_rate"] = sum(1 for r in rows if r.get("parsed")) / len(rows) + # Fraction of answers that resolved to some candidate label at all; a low + # value means the model is not following the copy-a-candidate format. + metrics["label_resolve_rate"] = sum(1 for r in rows if r.get("predicted_label")) / len(rows) + # Accuracy split by gold label. A model that ignores "both of them" — + # the gold for 44% of the corpus — shows up here as a near-zero slice + # even when the headline number looks healthy. + for label in (BOTH_LABEL, NONE_LABEL): + slice_rows = [r for r in rows if _label_key(str(r.get("target", ""))) == _label_key(label)] + if slice_rows: + key = label.replace(" ", "_") + metrics[f"accuracy_{key}"] = sum(1 for r in slice_rows if r.get("reward") == 1.0) / len(slice_rows) + return metrics + + def get_key_metrics(self, agent_metrics: Dict[str, Any]) -> Dict[str, Any]: + return {k: agent_metrics[k] for k in ("mean_reward", "exact_match_rate") if k in agent_metrics} + + +if __name__ == "__main__": + SpartqaResourcesServer.run_webserver() diff --git a/resources_servers/spartqa/configs/spartqa.yaml b/resources_servers/spartqa/configs/spartqa.yaml new file mode 100644 index 0000000000..2d718ef529 --- /dev/null +++ b/resources_servers/spartqa/configs/spartqa.yaml @@ -0,0 +1,29 @@ +spartqa: + resources_servers: + spartqa: + entrypoint: app.py + domain: reasoning + verified: false + description: SpartQA spatial reasoning, CO (choose-object) question type; four candidate answers per question (two story objects, "both of them", "none of them"); reward = 1 iff the answer resolves to the single gold label. + value: Improve spatial and relational reasoning over described scenes. + +spartqa_simple_agent: + responses_api_agents: + simple_agent: + entrypoint: app.py + resources_server: + type: resources_servers + name: spartqa + model_server: + type: responses_api_models + name: policy_model + datasets: + - name: example + type: example + jsonl_fpath: resources_servers/spartqa/data/example.jsonl + license: CC-BY-SA-4.0 + # Build the test split with `python resources_servers/spartqa/prepare_spartqa.py`. + - name: test + type: validation + jsonl_fpath: resources_servers/spartqa/data/spartqa_test.jsonl + license: CC-BY-SA-4.0 diff --git a/resources_servers/spartqa/configs/spartqa_serve.yaml b/resources_servers/spartqa/configs/spartqa_serve.yaml new file mode 100644 index 0000000000..5eb6ab4d89 --- /dev/null +++ b/resources_servers/spartqa/configs/spartqa_serve.yaml @@ -0,0 +1,19 @@ +# Serve ONLY the spartqa resources server (the /verify scorer), for driving +# from nemo-evaluator via the gym:// environment adapter: +# +# ng_run "+config_paths=[resources_servers/spartqa/configs/spartqa_serve.yaml]" +# +# SpartQA needs no model server — verify() extracts the model's final answer +# locally and scores the exact-or-contains match deterministically. +# nemo-evaluator owns the policy model and generation; it POSTs the model's +# answer to this server's /verify. See README.md. +spartqa: + resources_servers: + spartqa: + entrypoint: app.py + host: 0.0.0.0 + port: 8001 + domain: other + verified: false + description: SpartQA spatial-reasoning answer generation; reward = 1 on an exact-or-answer-containing match against any accepted answer phrase. + value: Improve spatial and relational reasoning over described scenes. diff --git a/resources_servers/spartqa/data/.gitignore b/resources_servers/spartqa/data/.gitignore new file mode 100644 index 0000000000..f8d4c51374 --- /dev/null +++ b/resources_servers/spartqa/data/.gitignore @@ -0,0 +1,2 @@ +spartqa_test.jsonl + diff --git a/resources_servers/spartqa/data/example.jsonl b/resources_servers/spartqa/data/example.jsonl new file mode 100644 index 0000000000..b4d0c85215 --- /dev/null +++ b/resources_servers/spartqa/data/example.jsonl @@ -0,0 +1,5 @@ +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nThere are two blocks. We call them A and B. Block A has two big black triangles. Below a medium black triangle and a small blue triangle there is big black triangle number one. The small blue triangle is above and to the left of the medium black triangle. Block B is below block A. It has a big yellow square. What is above the yellow shape? a small shape or a big black triangle?\n\nCandidate answers:\n- a small shape\n- a big black triangle\n- both of them\n- none of them\n"}]}, "target": "both of them", "options": ["a small shape", "a big black triangle"], "verifier_metadata": {"target": "both of them", "options": ["a small shape", "a big black triangle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nThere are two small black squares in a block. Near to and to the left of a medium blue triangle there is a big blue circle. Below, far from and to the left of small black square number two is a medium blue circle. Near to a big yellow triangle is the medium blue triangle. The medium blue circle is touching the left edge of this block. Which object is far from a small black square? the big blue circle or the yellow triangle?\n\nCandidate answers:\n- the big blue circle\n- the yellow triangle\n- both of them\n- none of them\n"}]}, "target": "none of them", "options": ["the big blue circle", "the yellow triangle"], "verifier_metadata": {"target": "none of them", "options": ["the big blue circle", "the yellow triangle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have three blocks. We call them A, B and C. A small blue triangle is in block A. Below block A is block B which has one small blue square. To the left of block A there is block C which has two medium yellow triangles. Medium yellow triangle number one is above medium yellow triangle number two and a big blue square. What is to the left of the small blue triangle? a medium yellow triangle or a small blue square?\n\nCandidate answers:\n- a medium yellow triangle\n- a small blue square\n- both of them\n- none of them\n"}]}, "target": "a medium yellow triangle", "options": ["a medium yellow triangle", "a small blue square"], "verifier_metadata": {"target": "a medium yellow triangle", "options": ["a medium yellow triangle", "a small blue square"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have three blocks. We call them A, B and C. A small blue triangle is in block A. Below block A is block B which has one small blue square. To the left of block A there is block C which has two medium yellow triangles. Medium yellow triangle number one is above medium yellow triangle number two and a big blue square. What is to the left of the small blue triangle? a small blue square or a big blue square?\n\nCandidate answers:\n- a small blue square\n- a big blue square\n- both of them\n- none of them\n"}]}, "target": "a big blue square", "options": ["a small blue square", "a big blue square"], "verifier_metadata": {"target": "a big blue square", "options": ["a small blue square", "a big blue square"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have two blocks, A and B. Block A is to the left of B. Block A contains two small black squares. To the right of a big black circle there is small black square number one. Below the big black circle and a medium black circle is small black square number two. Block B has one big black square. What is to the left of the big black square? a medium black circle or a big black circle?\n\nCandidate answers:\n- a medium black circle\n- a big black circle\n- both of them\n- none of them\n"}]}, "target": "both of them", "options": ["a medium black circle", "a big black circle"], "verifier_metadata": {"target": "both of them", "options": ["a medium black circle", "a big black circle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}} diff --git a/resources_servers/spartqa/data/example_metrics.json b/resources_servers/spartqa/data/example_metrics.json new file mode 100644 index 0000000000..a80d13b772 --- /dev/null +++ b/resources_servers/spartqa/data/example_metrics.json @@ -0,0 +1,48 @@ +{ + "name": "example", + "type": "example", + "jsonl_fpath": "resources_servers/spartqa/data/example.jsonl", + "num_repeats": 1, + "source": null, + "gitlab_identifier": null, + "huggingface_identifier": "mteb/SpartQA", + "license": "CC-BY-SA-4.0", + "Number of examples": 5, + "Number of tools": { + "Total # non-null values": 0, + "Average": 0.0, + "Min": 0.0, + "Max": 0.0, + "Standard deviation": 0.0 + }, + "Json-dumped number of words (proxy for token count)": { + "Total # non-null values": 5, + "Average": 179.4, + "Min": 171.0, + "Max": 184.0, + "Standard deviation": 5.367 + }, + "Number of turns": { + "Total # non-null values": 5, + "Average": 1.0, + "Min": 1.0, + "Max": 1.0, + "Standard deviation": 0.0 + }, + "Temperature": { + "Total # non-null values": 0, + "Average": 0.0, + "Min": 0.0, + "Max": 0.0, + "Standard deviation": 0.0 + }, + "Number of candidate answers": { + "Total # non-null values": 5, + "Average": 4.0, + "Min": 4.0, + "Max": 4.0, + "Standard deviation": 0.0 + }, + "Number of 'both of them' golds": 2, + "Number of 'none of them' golds": 1 +} diff --git a/resources_servers/spartqa/data/example_rollouts.jsonl b/resources_servers/spartqa/data/example_rollouts.jsonl new file mode 100644 index 0000000000..5f1b17e31a --- /dev/null +++ b/resources_servers/spartqa/data/example_rollouts.jsonl @@ -0,0 +1,5 @@ +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nThere are two blocks. We call them A and B. Block A has two big black triangles. Below a medium black triangle and a small blue triangle there is big black triangle number one. The small blue triangle is above and to the left of the medium black triangle. Block B is below block A. It has a big yellow square. What is above the yellow shape? a small shape or a big black triangle?\n\nCandidate answers:\n- a small shape\n- a big black triangle\n- both of them\n- none of them\n"}]}, "target": "both of them", "options": ["a small shape", "a big black triangle"], "verifier_metadata": {"target": "both of them", "options": ["a small shape", "a big black triangle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_20078df25d2140509cf9fcd601151a42", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_fd65a5b507254909b1ce3088f6be3924", "content": [{"annotations": [], "text": "Both listed shapes sit above the block's bottom edge.\nFinal answer: both of them", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "both of them", "predicted_label": "both of them", "_ng_task_index": 0, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nThere are two small black squares in a block. Near to and to the left of a medium blue triangle there is a big blue circle. Below, far from and to the left of small black square number two is a medium blue circle. Near to a big yellow triangle is the medium blue triangle. The medium blue circle is touching the left edge of this block. Which object is far from a small black square? the big blue circle or the yellow triangle?\n\nCandidate answers:\n- the big blue circle\n- the yellow triangle\n- both of them\n- none of them\n"}]}, "target": "none of them", "options": ["the big blue circle", "the yellow triangle"], "verifier_metadata": {"target": "none of them", "options": ["the big blue circle", "the yellow triangle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_10e3ebd7c7e1468aa58f76d2b988b380", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_e71953cbc16042699ba7e591a4cc48fe", "content": [{"annotations": [], "text": "Neither of the two objects satisfies the asked relation.\nFinal answer: neither", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": false, "parsed": true, "extracted": "neither", "predicted_label": "none of them", "_ng_task_index": 1, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have three blocks. We call them A, B and C. A small blue triangle is in block A. Below block A is block B which has one small blue square. To the left of block A there is block C which has two medium yellow triangles. Medium yellow triangle number one is above medium yellow triangle number two and a big blue square. What is to the left of the small blue triangle? a medium yellow triangle or a small blue square?\n\nCandidate answers:\n- a medium yellow triangle\n- a small blue square\n- both of them\n- none of them\n"}]}, "target": "a medium yellow triangle", "options": ["a medium yellow triangle", "a small blue square"], "verifier_metadata": {"target": "a medium yellow triangle", "options": ["a medium yellow triangle", "a small blue square"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_5db9b1c4943b4d13b7990449043e7a65", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_dac0a2cd643e4b90b909c7b229e507e8", "content": [{"annotations": [], "text": "Only the first candidate matches the relation.\nFinal answer: a medium yellow triangle", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": true, "parsed": true, "extracted": "a medium yellow triangle", "predicted_label": "a medium yellow triangle", "_ng_task_index": 2, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have three blocks. We call them A, B and C. A small blue triangle is in block A. Below block A is block B which has one small blue square. To the left of block A there is block C which has two medium yellow triangles. Medium yellow triangle number one is above medium yellow triangle number two and a big blue square. What is to the left of the small blue triangle? a small blue square or a big blue square?\n\nCandidate answers:\n- a small blue square\n- a big blue square\n- both of them\n- none of them\n"}]}, "target": "a big blue square", "options": ["a small blue square", "a big blue square"], "verifier_metadata": {"target": "a big blue square", "options": ["a small blue square", "a big blue square"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_1b59b532aa3e42eb911e29388528be9b", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_1234a148f406404e886436efbc250277", "content": [{"annotations": [], "text": "It could be either one.\nFinal answer: a small blue square or a big blue square", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 0.0, "exact": false, "parsed": true, "extracted": "a small blue square or a big blue square", "predicted_label": "", "_ng_task_index": 3, "_ng_rollout_index": 0} +{"responses_create_params": {"input": [{"role": "user", "content": "Answer the spatial reasoning question below.\nChoose exactly one of the candidate answers and copy it verbatim. Answer\n\"both of them\" if both listed objects satisfy the question, and \"none of them\"\nif neither of them does.\n\nEnd your response with one line in this exact format:\nFinal answer: \n\nStory and question:\nWe have two blocks, A and B. Block A is to the left of B. Block A contains two small black squares. To the right of a big black circle there is small black square number one. Below the big black circle and a medium black circle is small black square number two. Block B has one big black square. What is to the left of the big black square? a medium black circle or a big black circle?\n\nCandidate answers:\n- a medium black circle\n- a big black circle\n- both of them\n- none of them\n"}]}, "target": "both of them", "options": ["a medium black circle", "a big black circle"], "verifier_metadata": {"target": "both of them", "options": ["a medium black circle", "a big black circle"]}, "agent_ref": {"type": "responses_api_agents", "name": "spartqa_simple_agent"}, "response": {"id": "resp_05e53effc96e471abc7e98d1e7eef406", "created_at": 1784055202.0, "error": null, "incomplete_details": null, "instructions": null, "metadata": null, "model": "synthetic", "object": "response", "output": [{"id": "msg_a7a34dc091604be582d53db4467ece1d", "content": [{"annotations": [], "text": "Checking each candidate in turn, the relation holds for both.\nFinal answer: I would say both of them.", "type": "output_text", "logprobs": null}], "role": "assistant", "status": "completed", "type": "message"}], "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "tools": [], "top_p": null}, "reward": 1.0, "exact": false, "parsed": true, "extracted": "I would say both of them.", "predicted_label": "both of them", "_ng_task_index": 4, "_ng_rollout_index": 0} diff --git a/resources_servers/spartqa/prepare_spartqa.py b/resources_servers/spartqa/prepare_spartqa.py new file mode 100644 index 0000000000..61f08b9405 --- /dev/null +++ b/resources_servers/spartqa/prepare_spartqa.py @@ -0,0 +1,184 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Build the SpartQA Gym dataset from the public ``mteb/SpartQA`` HF dataset. + +``mteb/SpartQA`` holds SpartQA's CO (choose-object) questions in retrieval +form. Per the paper, CO is a four-label single-choice task — the two objects +named in the question, ``both of them``, and ``none of them`` — but the +retrieval encoding has no way to express "both", so a "both of them" gold is +flattened into *three* relevant documents (the ``both of them`` phrase plus +each object). This script undoes that flattening: + +* ``options`` — the two objects, parsed from the question's "… X or Y?" tail. +* ``target`` — the single gold label: ``both of them`` when qrels marks the + pair, ``none of them`` when qrels says so, else the one gold object (snapped + to the option's own wording so the label set is self-consistent). + +The four candidates are rendered into the prompt so ``both of them`` and +``none of them`` are actually reachable answers; ``target`` / ``options`` ride +along as extra fields consumed by ``app.py``'s verify(). + +Requires the ``datasets`` package (HF) at prep time only. + +Usage:: + + python resources_servers/spartqa/prepare_spartqa.py --split test \\ + --output data/spartqa_test.jsonl +""" + +from __future__ import annotations + +import argparse +import json +import re +from pathlib import Path +from typing import Any, List, Optional + +from app import BOTH_LABEL, NONE_LABEL, PROMPT, _label_key + + +_HF_DATASET = "mteb/SpartQA" +_DEFAULT_SPLIT = "test" +_AGENT = {"type": "responses_api_agents", "name": "spartqa_simple_agent"} + + +def _load_split(config: str, split: str): + try: + from datasets import load_dataset + except ImportError as exc: # pragma: no cover - environment-dependent + raise SystemExit( + "The 'datasets' package is required to build mteb/SpartQA. " + "Install it (pip install datasets) or stage a prepared JSONL." + ) from exc + return load_dataset(_HF_DATASET, config, split=split) + + +def _unique_preserve_order(values: List[str]) -> List[str]: + seen: set[str] = set() + result: List[str] = [] + for value in values: + key = value.strip().casefold() + if key and key not in seen: + seen.add(key) + result.append(value.strip()) + return result + + +def parse_options(question: str) -> Optional[List[str]]: + """Parse the two candidate objects out of a CO question's "… X or Y?" tail. + + Returns ``None`` when the final sentence does not offer exactly two + alternatives, so the caller can drop the row rather than guess. + """ + last_sentence = re.split(r"(?<=[.?])\s+", question.strip())[-1].strip().rstrip("?") + parts = [part.strip() for part in last_sentence.split(" or ")] + if len(parts) != 2 or not all(parts): + return None + return parts + + +def resolve_gold(answers: List[str], options: List[str]) -> Optional[str]: + """Collapse the flattened qrels phrases back into one CO label.""" + keys = {_label_key(a) for a in answers} + if _label_key(BOTH_LABEL) in keys or len(keys & {_label_key(o) for o in options}) >= 2: + return BOTH_LABEL + if _label_key(NONE_LABEL) in keys: + return NONE_LABEL + for option in options: + if _label_key(option) in keys: + # Snap to the option's own wording: qrels and the story disagree on + # leading articles, and the prompt shows the option text. + return option + return None + + +def build_records(split: str = _DEFAULT_SPLIT) -> List[dict[str, Any]]: + """Join the MTEB ``queries`` / ``corpus`` / ``qrels`` splits into rows.""" + corpus = { + row["_id"]: str(row["text"]).strip() + for row in _load_split("corpus", split) + if str(row.get("text", "")).strip() + } + queries = { + row["_id"]: str(row["text"]).strip() + for row in _load_split("queries", split) + if str(row.get("text", "")).strip() + } + + qrels_by_query: dict[str, List[str]] = {} + for row in _load_split("qrels", split): + try: + score = int(row.get("score", 1)) + except (TypeError, ValueError): + score = 1 + if score <= 0: + continue + qrels_by_query.setdefault(row["query-id"], []).append(row["corpus-id"]) + + records: List[dict[str, Any]] = [] + for query_id in sorted(queries): + question = queries[query_id] + answer_ids = [doc_id for doc_id in qrels_by_query.get(query_id, []) if doc_id in corpus] + answers = _unique_preserve_order([corpus[doc_id] for doc_id in answer_ids]) + options = parse_options(question) + if not answers or not options: + continue + target = resolve_gold(answers, options) + if not target: + continue + records.append({"question": question, "target": target, "options": options}) + return records + + +def _to_task(record: dict[str, Any]) -> dict[str, Any]: + candidates = "\n".join(f"- {label}" for label in [*record["options"], BOTH_LABEL, NONE_LABEL]) + content = PROMPT.format(question=record["question"], candidates=candidates) + return { + "responses_create_params": {"input": [{"role": "user", "content": content}]}, + "target": record["target"], + "options": record["options"], + # ``options`` is a list and is dropped by the nemo-evaluator + # ``gym://...protocol=native`` driver (it forwards only top-level scalar + # fields to /verify). Mirror the candidate set into verifier_metadata, + # which the driver forwards intact, so verify() sees every label. + "verifier_metadata": { + "target": record["target"], + "options": record["options"], + }, + "agent_ref": _AGENT, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description="Build the SpartQA Gym dataset.") + parser.add_argument("--split", default=_DEFAULT_SPLIT, help="MTEB split (default: test)") + parser.add_argument( + "--output", + default=str(Path(__file__).parent / "data" / "spartqa_test.jsonl"), + help="Output JSONL path", + ) + args = parser.parse_args() + + records = build_records(args.split) + out_path = Path(args.output) + out_path.parent.mkdir(parents=True, exist_ok=True) + with out_path.open("w", encoding="utf-8") as fh: + for record in records: + fh.write(json.dumps(_to_task(record), ensure_ascii=False) + "\n") + print(f"SpartQA: wrote {len(records)} rows -> {out_path}") + + +if __name__ == "__main__": + main() diff --git a/resources_servers/spartqa/requirements.txt b/resources_servers/spartqa/requirements.txt new file mode 100644 index 0000000000..9fd546ac45 --- /dev/null +++ b/resources_servers/spartqa/requirements.txt @@ -0,0 +1,4 @@ +-e nemo-gym[dev] @ ../../ +# SpartQA scoring is pure-stdlib (re/string) — no extra runtime dependencies. +# Building the dataset (prepare_spartqa.py) additionally needs `datasets` (HF), +# used only at prep time. diff --git a/resources_servers/spartqa/tests/test_acceptance.py b/resources_servers/spartqa/tests/test_acceptance.py new file mode 100644 index 0000000000..cadd5ddd96 --- /dev/null +++ b/resources_servers/spartqa/tests/test_acceptance.py @@ -0,0 +1,463 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Acceptance tests for the SpartQA gym benchmark. + +These verify the approved user story's acceptance criteria against the built +implementation. They are independent of the builder's unit tests +(``tests/test_app.py``): fixtures are re-declared here and data files are loaded +by path relative to this test file. One ``test_ac_*`` per acceptance criterion, +one ``test_edge_*`` per story edge case. +""" + +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path +from typing import Any, List +from unittest.mock import MagicMock + +import pytest +import yaml +from pytest import approx + +from nemo_gym.openai_utils import ( + NeMoGymResponse, + NeMoGymResponseCreateParamsNonStreaming, + NeMoGymResponseOutputMessage, + NeMoGymResponseOutputText, +) +from nemo_gym.server_utils import ServerClient +from resources_servers.spartqa import app as spartqa_app +from resources_servers.spartqa.app import ( + BOTH_LABEL, + NONE_LABEL, + PROMPT, + SimpleResourcesServer, + SpartqaResourcesServer, + SpartqaResourcesServerConfig, + SpartqaVerifyRequest, + SpartqaVerifyResponse, + _clean_candidate, + _extract_answer, + _label_key, + _normalize, + _strip_reasoning, + candidate_labels, + match_label, +) + + +# ── Paths (relative to this test file) ───────────────────────────────────── + +_SERVER_DIR = Path(__file__).resolve().parent.parent +_EXAMPLE_JSONL = _SERVER_DIR / "data" / "example.jsonl" +_ROLLOUTS_JSONL = _SERVER_DIR / "data" / "example_rollouts.jsonl" +_CONFIG_YAML = _SERVER_DIR / "configs" / "spartqa.yaml" +_PREPARE_PY = _SERVER_DIR / "prepare_spartqa.py" + + +# ── Fixture builders (mirror tests/test_app.py idioms) ───────────────────── + + +def _make_response(text: str) -> NeMoGymResponse: + return NeMoGymResponse( + id="resp", + created_at=0.0, + model="policy_model", + object="response", + output=[ + NeMoGymResponseOutputMessage( + id="msg", + content=[NeMoGymResponseOutputText(annotations=[], text=text, type="output_text")], + role="assistant", + status="completed", + type="message", + ) + ], + parallel_tool_calls=False, + tool_choice="none", + tools=[], + ) + + +def _config() -> SpartqaResourcesServerConfig: + return SpartqaResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint="", name="spartqa") + + +def _make_request(text: str, *, target: str = "", options: list[str] | None = None) -> SpartqaVerifyRequest: + return SpartqaVerifyRequest( + responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), + response=_make_response(text), + target=target, + options=options or [], + ) + + +# A representative CO row: two story objects plus the two fixed labels. +_OPTIONS = ["a big blue square", "a small blue square"] + + +def _server() -> SpartqaResourcesServer: + return SpartqaResourcesServer(config=_config(), server_client=MagicMock(spec=ServerClient)) + + +def _read_jsonl(path: Path) -> List[dict[str, Any]]: + return [json.loads(line) for line in path.read_text().splitlines() if line.strip()] + + +def _load_prepare_module() -> Any: + """Load ``prepare_spartqa.py`` (which does ``from app import PROMPT``). + + The prepare script imports the sibling ``app`` module by bare name, so the + already-imported ``resources_servers.spartqa.app`` is registered under + ``app`` for the duration of the load, then removed to avoid polluting + ``sys.modules`` for other servers' prepare scripts. + """ + saved = sys.modules.get("app") + sys.modules["app"] = spartqa_app + try: + spec = importlib.util.spec_from_file_location("spartqa_prepare", _PREPARE_PY) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + finally: + if saved is not None: + sys.modules["app"] = saved + else: + sys.modules.pop("app", None) + + +# ── AC1: server subclasses SimpleResourcesServer, async verify -> reward ──── + + +class TestAcServerContract: + def test_ac_subclasses_simple_resources_server(self) -> None: + assert issubclass(SpartqaResourcesServer, SimpleResourcesServer) + + async def test_ac_verify_is_async_and_returns_reward(self) -> None: + result = await _server().verify( + _make_request(f"Final answer: {BOTH_LABEL}", target=BOTH_LABEL, options=_OPTIONS) + ) + assert isinstance(result, SpartqaVerifyResponse) + assert hasattr(result, "reward") + assert result.reward == approx(1.0) + + +# ── AC2: scoring correctness ───────────────────────────────────────────────── + +# Shared fixtures: (response_text, gold_label, options, expected_correct). +_PARITY_FIXTURES: list[tuple[str, str, list[str], bool]] = [ + # Verbatim candidate. + ("Final answer: a big blue square", _OPTIONS[0], _OPTIONS, True), + # Candidate recovered from a longer sentence. + ("Final answer: it is a big blue square.", _OPTIONS[0], _OPTIONS, True), + # The other candidate. + ("Final answer: a small blue square", _OPTIONS[0], _OPTIONS, False), + # "both of them" gold is NOT earned by naming one object. + (f"Final answer: {_OPTIONS[0]}", BOTH_LABEL, _OPTIONS, False), + # Echoing the question's options is ambiguous. + (f"Final answer: {_OPTIONS[0]} or {_OPTIONS[1]}", _OPTIONS[0], _OPTIONS, False), + # Aliases of the two fixed labels. + ("Final answer: Both", BOTH_LABEL, _OPTIONS, True), + ("Final answer: neither", NONE_LABEL, _OPTIONS, True), + (" ", _OPTIONS[0], _OPTIONS, False), + (f"reasoningFinal answer: {BOTH_LABEL}", BOTH_LABEL, _OPTIONS, True), +] + + +class TestAcMetricParity: + def test_ac_prompt_text_contract(self) -> None: + assert "Final answer: " in PROMPT + assert PROMPT.startswith("Answer the spatial reasoning question below.") + assert "{question}" in PROMPT + # The four candidates must be rendered, else "both of them" / "none of + # them" are unreachable answers. + assert PROMPT.rstrip().endswith("{candidates}") + + def test_ac_normalize_logic(self) -> None: + assert _normalize(" The Big, BLACK Square!! ") == "the big black square" + assert _normalize("") == "" + + def test_ac_strip_reasoning_logic(self) -> None: + assert _strip_reasoning("hiddenvisible") == "visible" + assert _strip_reasoning("plain") == "plain" + + def test_ac_clean_candidate_logic(self) -> None: + assert _clean_candidate("- *green cup*") == "green cup" + assert _clean_candidate(' "yes" ') == "yes" + + def test_ac_extract_answer_logic(self) -> None: + assert _extract_answer("Reasoning.\nFinal answer: below the circle") == "below the circle" + assert _extract_answer("xFinal answer: yes") == "yes" + assert _extract_answer(" ") == "" + + @pytest.mark.parametrize("text,target,options,expected", _PARITY_FIXTURES) + async def test_ac_scoring_correctness(self, text: str, target: str, options: list[str], expected: bool) -> None: + result = await _server().verify(_make_request(text, target=target, options=options)) + assert bool(result.reward) is expected + + +# ── AC3: reward is strictly 1.0 or 0.0 ───────────────────────────────────── + + +class TestAcBinaryReward: + @pytest.mark.parametrize( + "text,target,options", + [(f[0], f[1], f[2]) for f in _PARITY_FIXTURES] + + [("no marker at all", _OPTIONS[0], _OPTIONS), ("", _OPTIONS[0], _OPTIONS)], + ) + async def test_ac_reward_is_binary(self, text: str, target: str, options: list[str]) -> None: + result = await _server().verify(_make_request(text, target=target, options=options)) + assert result.reward in {0.0, 1.0} + + +# ── AC4: options read from field / verifier_metadata; falls back to [target] ─ + + +class TestAcCandidateSet: + async def test_ac_uses_options_field(self) -> None: + # The non-gold option must resolve to itself, not to the gold. + result = await _server().verify( + _make_request(f"Final answer: {_OPTIONS[1]}", target=_OPTIONS[0], options=_OPTIONS) + ) + assert result.reward == approx(0.0) + assert result.predicted_label == _OPTIONS[1] + + async def test_ac_reads_options_from_verifier_metadata(self) -> None: + # The native driver drops list fields; verifier_metadata is the fallback. + request = SpartqaVerifyRequest( + responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), + response=_make_response(f"Final answer: {BOTH_LABEL}"), + target="", + verifier_metadata={"target": BOTH_LABEL, "options": _OPTIONS}, + ) + result = await _server().verify(request) + assert result.reward == approx(1.0) + + async def test_ac_falls_back_to_target_when_options_absent(self) -> None: + request = SpartqaVerifyRequest( + responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), + response=_make_response(f"Final answer: {_OPTIONS[0]}"), + target=_OPTIONS[0], + ) + result = await _server().verify(request) + assert result.options == [] + assert result.reward == approx(1.0) + + +# ── AC5: response carries exact, parsed, extracted extras ─────────────────── + + +class TestAcExtraFields: + async def test_ac_extra_fields_present(self) -> None: + result = await _server().verify( + _make_request(f"Final answer: {_OPTIONS[0]}", target=_OPTIONS[0], options=_OPTIONS) + ) + assert result.exact is True + assert result.parsed is True + assert result.extracted == _OPTIONS[0] + assert result.predicted_label == _OPTIONS[0] + + async def test_ac_exact_false_when_recovered_from_a_sentence(self) -> None: + result = await _server().verify( + _make_request( + f"Final answer: it is {_OPTIONS[0]} on the left.", + target=_OPTIONS[0], + options=_OPTIONS, + ) + ) + assert result.reward == approx(1.0) + assert result.exact is False + assert result.parsed is True + + +# ── AC6: empty / whitespace output -> reward 0.0, no exception ─────────────── + + +class TestAcEmptyOutput: + @pytest.mark.parametrize("text", ["", " ", "\n\t "]) + async def test_ac_empty_output_scores_zero(self, text: str) -> None: + result = await _server().verify(_make_request(text, target=_OPTIONS[0], options=_OPTIONS)) + assert result.reward == approx(0.0) + assert result.parsed is False + assert result.exact is False + + +# ── AC7: example.jsonl row shape ──────────────────────────────────────────── + + +class TestAcExampleDataset: + def test_ac_example_rows_conform(self) -> None: + rows = _read_jsonl(_EXAMPLE_JSONL) + assert len(rows) >= 5 + for row in rows: + params = row["responses_create_params"] + messages = params["input"] + assert isinstance(messages, list) and messages + assert any(m.get("role") == "user" for m in messages) + assert isinstance(row["target"], str) and row["target"] + assert isinstance(row["options"], list) and len(row["options"]) == 2 + assert row["verifier_metadata"]["options"] == row["options"] + assert row["agent_ref"]["name"] == "spartqa_simple_agent" + # The four candidates must be visible to the model. + content = messages[0]["content"] + for label in [*row["options"], BOTH_LABEL, NONE_LABEL]: + assert label in content + # The gold must be one of the four candidates. + assert _label_key(row["target"]) in {_label_key(c) for c in candidate_labels(row["options"])} + + +# ── AC8: config wires server + agent ──────────────────────────────────────── + + +class TestAcConfig: + def test_ac_config_parses_and_wires_server_and_agent(self) -> None: + config = yaml.safe_load(_CONFIG_YAML.read_text()) + assert "spartqa" in config + assert "spartqa" in config["spartqa"]["resources_servers"] + assert "spartqa_simple_agent" in config + agent = config["spartqa_simple_agent"]["responses_api_agents"]["simple_agent"] + assert agent["resources_server"]["name"] == "spartqa" + + +# ── AC9: prepare_spartqa.py record-building logic ─────────────────────────── + + +class TestAcPrepareScript: + def test_ac_prepare_defines_build_helpers(self) -> None: + prepare = _load_prepare_module() + assert callable(prepare.build_records) + assert callable(prepare._unique_preserve_order) + assert callable(prepare.parse_options) + assert callable(prepare.resolve_gold) + + def test_ac_parse_options_splits_the_question_tail(self) -> None: + prepare = _load_prepare_module() + question = "Block A is above block B. What is below the circle? a big square or a small square?" + assert prepare.parse_options(question) == ["a big square", "a small square"] + assert prepare.parse_options("A statement with no alternatives.") is None + + def test_ac_resolve_gold_unflattens_the_retrieval_qrels(self) -> None: + prepare = _load_prepare_module() + options = ["a big square", "a small square"] + # qrels marks all three phrases -> the single label is "both of them". + assert prepare.resolve_gold([BOTH_LABEL, *options], options) == BOTH_LABEL + # Two objects and no sentinel is still "both of them". + assert prepare.resolve_gold(options, options) == BOTH_LABEL + assert prepare.resolve_gold([NONE_LABEL], options) == NONE_LABEL + # A single object gold snaps to the option's own wording. + assert prepare.resolve_gold(["big square"], options) == "a big square" + assert prepare.resolve_gold(["a green cup"], options) is None + + def test_ac_unique_preserve_order_dedupes_casefold_preserving_order(self) -> None: + prepare = _load_prepare_module() + result = prepare._unique_preserve_order( + ["Below the circle", " below the circle ", "Under the Circle", "", " "] + ) + assert result == ["Below the circle", "Under the Circle"] + + +# ── AC10: round-trip over example.jsonl -> reward 1.0 ─────────────────────── + + +class TestAcRoundTrip: + async def test_ac_example_targets_score_one(self) -> None: + server = _server() + rows = _read_jsonl(_EXAMPLE_JSONL) + assert rows + for row in rows: + result = await server.verify( + _make_request( + f"Final answer: {row['target']}", + target=row["target"], + options=row["options"], + ) + ) + assert result.reward == approx(1.0), row["target"] + + +# ── Committed rollouts are self-consistent with the scorer ────────────────── + + +class TestAcRolloutsSelfConsistent: + async def test_ac_rollouts_reproduce_committed_fields(self) -> None: + server = _server() + rows = _read_jsonl(_ROLLOUTS_JSONL) + assert rows + for row in rows: + text = row["response"]["output"][0]["content"][0]["text"] + result = await server.verify(_make_request(text, target=row["target"], options=row["options"])) + assert result.reward == approx(row["reward"]), text + assert result.exact is row["exact"] + assert result.parsed is row["parsed"] + assert result.extracted == row["extracted"] + assert result.predicted_label == row["predicted_label"] + + +# ── Story edge cases ──────────────────────────────────────────────────────── + + +class TestEdgeCases: + async def test_edge_reasoning_wrapped_output_is_stripped_and_scores(self) -> None: + result = await _server().verify( + _make_request( + f"the star is north of the moonFinal answer: {BOTH_LABEL}", + target=BOTH_LABEL, + options=_OPTIONS, + ) + ) + assert result.reward == approx(1.0) + assert result.extracted == BOTH_LABEL + + async def test_edge_single_object_never_satisfies_a_both_gold(self) -> None: + for option in _OPTIONS: + result = await _server().verify( + _make_request(f"Final answer: {option}", target=BOTH_LABEL, options=_OPTIONS) + ) + assert result.reward == approx(0.0), option + + async def test_edge_ambiguous_answer_resolves_to_no_label(self) -> None: + result = await _server().verify( + _make_request( + f"Final answer: {_OPTIONS[0]} or {_OPTIONS[1]}", + target=BOTH_LABEL, + options=_OPTIONS, + ) + ) + assert result.reward == approx(0.0) + assert result.predicted_label == "" + + async def test_edge_nested_options_pick_the_most_specific(self) -> None: + options = ["a triangle in block B", "a big blue triangle in block B"] + assert match_label(options[1], candidate_labels(options)) == options[1] + + async def test_edge_exact_true_only_on_verbatim_answer(self) -> None: + strict = await _server().verify( + _make_request(f"Final answer: {_OPTIONS[0]}", target=_OPTIONS[0], options=_OPTIONS) + ) + assert strict.exact is True + + loose = await _server().verify( + _make_request( + f"Final answer: I believe it is {_OPTIONS[0]}.", + target=_OPTIONS[0], + options=_OPTIONS, + ) + ) + assert loose.reward == approx(1.0) + assert loose.exact is False diff --git a/resources_servers/spartqa/tests/test_app.py b/resources_servers/spartqa/tests/test_app.py new file mode 100644 index 0000000000..3b9aeb1b63 --- /dev/null +++ b/resources_servers/spartqa/tests/test_app.py @@ -0,0 +1,387 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import json +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock + +from pytest import approx + +from nemo_gym.openai_utils import ( + NeMoGymResponse, + NeMoGymResponseCreateParamsNonStreaming, + NeMoGymResponseOutputMessage, + NeMoGymResponseOutputText, +) +from nemo_gym.server_utils import ServerClient +from resources_servers.spartqa.app import ( + BOTH_LABEL, + NONE_LABEL, + SpartqaResourcesServer, + SpartqaResourcesServerConfig, + SpartqaVerifyRequest, + _extract_answer, + _label_key, + _normalize, + _response_text, + _strip_reasoning, + candidate_labels, + match_label, +) + + +_EXAMPLE_JSONL = Path(__file__).resolve().parent.parent / "data" / "example.jsonl" + + +def _make_response(text: str) -> NeMoGymResponse: + return NeMoGymResponse( + id="resp", + created_at=0.0, + model="policy_model", + object="response", + output=[ + NeMoGymResponseOutputMessage( + id="msg", + content=[NeMoGymResponseOutputText(annotations=[], text=text, type="output_text")], + role="assistant", + status="completed", + type="message", + ) + ], + parallel_tool_calls=False, + tool_choice="none", + tools=[], + ) + + +def _config() -> SpartqaResourcesServerConfig: + return SpartqaResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint="", name="spartqa") + + +def _make_request( + text: str, + *, + target: str = "", + options: list[str] | None = None, + verifier_metadata: dict | None = None, +) -> SpartqaVerifyRequest: + return SpartqaVerifyRequest( + responses_create_params=NeMoGymResponseCreateParamsNonStreaming(input=[]), + response=_make_response(text), + target=target, + options=options or [], + verifier_metadata=verifier_metadata, + ) + + +# A representative CO row: two story objects plus the two fixed labels. +_OPTIONS = ["a big blue square", "a small blue square"] + + +def _server() -> SpartqaResourcesServer: + return SpartqaResourcesServer(config=_config(), server_client=MagicMock(spec=ServerClient)) + + +# ── Pure helpers ───────────────────────────────────────────────────────── + + +class TestExtractAnswer: + def test_pulls_phrase_after_final_answer(self) -> None: + assert _extract_answer("Reasoning here.\nFinal answer: below the circle") == ("below the circle") + + def test_strips_think_reasoning(self) -> None: + text = "the star is northFinal answer: yes" + assert _extract_answer(text) == "yes" + + def test_thinking_process_prefix_returns_last_line(self) -> None: + # No explicit "Final answer:" marker; a multi-line "thinking process" + # prefix means the answer is the last non-empty line. + text = "Thinking process: I consider the layout\ngreen cup" + assert _extract_answer(text) == "green cup" + + def test_ignores_template_echo_and_takes_last_answer(self) -> None: + # Reasoning models often restate the prompt's format instruction + # ("Final answer: ") mid-thought. The extractor must + # skip that placeholder echo and return the real concluding answer, + # not the first "Final answer:" line it sees. + text = ( + "To answer I must end with 'Final answer: '.\n" + "Working through the layout, the yellow circle is below.\n" + "Final answer: a big yellow circle that is touching the bottom edge of a block" + ) + assert _extract_answer(text) == "a big yellow circle that is touching the bottom edge of a block" + + def test_multiple_final_answer_lines_takes_last(self) -> None: + # A model that drafts an answer then revises it: the last line wins. + text = "Final answer: a small blue shape\nOn reflection:\nFinal answer: both of them" + assert _extract_answer(text) == "both of them" + + def test_empty_returns_empty(self) -> None: + assert _extract_answer(" ") == "" + + +class TestStripReasoning: + def test_removes_think_block(self) -> None: + assert _strip_reasoning("hiddenvisible") == "visible" + + def test_passthrough(self) -> None: + assert _strip_reasoning("plain") == "plain" + + +class TestNormalize: + def test_lowercases_strips_punctuation_collapses_ws(self) -> None: + assert _normalize(" The Big, BLACK Square!! ") == "the big black square" + + def test_empty(self) -> None: + assert _normalize("") == "" + + +# ── verify() ───────────────────────────────────────────────────────────── + + +class TestVerify: + async def test_verbatim_gold_scores_one_and_exact(self) -> None: + result = await _server().verify( + _make_request("Final answer: a big blue square", target=_OPTIONS[0], options=_OPTIONS) + ) + assert result.reward == approx(1.0) + assert result.exact is True + assert result.parsed is True + assert result.extracted == "a big blue square" + assert result.predicted_label == _OPTIONS[0] + + async def test_gold_inside_a_sentence_scores_one_but_not_exact(self) -> None: + result = await _server().verify( + _make_request( + "Final answer: it must be a big blue square.", + target=_OPTIONS[0], + options=_OPTIONS, + ) + ) + assert result.reward == approx(1.0) + assert result.exact is False + assert result.predicted_label == _OPTIONS[0] + + async def test_wrong_option_scores_zero(self) -> None: + result = await _server().verify( + _make_request("Final answer: a small blue square", target=_OPTIONS[0], options=_OPTIONS) + ) + assert result.reward == approx(0.0) + assert result.predicted_label == _OPTIONS[1] + + async def test_both_of_them_gold(self) -> None: + result = await _server().verify( + _make_request(f"Final answer: {BOTH_LABEL}", target=BOTH_LABEL, options=_OPTIONS) + ) + assert result.reward == approx(1.0) + assert result.exact is True + + async def test_none_of_them_gold(self) -> None: + result = await _server().verify( + _make_request(f"Final answer: {NONE_LABEL}", target=NONE_LABEL, options=_OPTIONS) + ) + assert result.reward == approx(1.0) + + async def test_article_difference_still_matches(self) -> None: + # qrels phrases and the story wording disagree on leading articles. + result = await _server().verify( + _make_request( + "Final answer: the big blue square", + target="a big blue square", + options=_OPTIONS, + ) + ) + assert result.reward == approx(1.0) + assert result.exact is True + + async def test_alias_resolves_to_fixed_label(self) -> None: + for alias, gold in (("Both", BOTH_LABEL), ("Neither", NONE_LABEL), ("DK", NONE_LABEL)): + result = await _server().verify(_make_request(f"Final answer: {alias}", target=gold, options=_OPTIONS)) + assert result.reward == approx(1.0), alias + assert result.exact is False, alias + + async def test_naming_a_single_option_does_not_earn_a_both_gold(self) -> None: + # Regression: the retrieval qrels list "both of them" alongside each + # object, so any-of matching used to credit this. + for option in _OPTIONS: + result = await _server().verify( + _make_request(f"Final answer: {option}", target=BOTH_LABEL, options=_OPTIONS) + ) + assert result.reward == approx(0.0), option + + async def test_echoing_both_options_is_ambiguous_and_scores_zero(self) -> None: + # Regression: "X or Y" copied straight out of the question used to score + # 1.0 on ~95% of the corpus via substring matching. + text = f"Final answer: {_OPTIONS[0]} or {_OPTIONS[1]}" + for gold in (_OPTIONS[0], _OPTIONS[1], BOTH_LABEL): + result = await _server().verify(_make_request(text, target=gold, options=_OPTIONS)) + assert result.reward == approx(0.0), gold + assert result.predicted_label == "" + + async def test_nested_options_resolve_to_the_most_specific(self) -> None: + options = ["a triangle that is in block B", "a big blue triangle that is in block B"] + result = await _server().verify( + _make_request( + "Final answer: a big blue triangle that is in block B", + target=options[1], + options=options, + ) + ) + assert result.reward == approx(1.0) + assert result.predicted_label == options[1] + + async def test_empty_output_reward_zero_no_raise(self) -> None: + result = await _server().verify(_make_request(" ", target=_OPTIONS[0], options=_OPTIONS)) + assert result.reward == approx(0.0) + assert result.parsed is False + assert result.predicted_label == "" + + async def test_empty_target_scores_zero(self) -> None: + result = await _server().verify(_make_request("Final answer: yes", target="", options=[])) + assert result.reward == approx(0.0) + + async def test_fields_from_verifier_metadata(self) -> None: + # The native driver drops the top-level ``options`` list; the candidate + # set must be recoverable from verifier_metadata. + result = await _server().verify( + _make_request( + f"Final answer: {BOTH_LABEL}", + target="", + options=[], + verifier_metadata={"target": BOTH_LABEL, "options": _OPTIONS}, + ) + ) + assert result.reward == approx(1.0) + + async def test_options_missing_falls_back_to_target_only(self) -> None: + result = await _server().verify(_make_request("Final answer: a big blue square", target=_OPTIONS[0])) + assert result.reward == approx(1.0) + + +class TestLabelMatching: + def test_candidate_labels_appends_fixed_labels_and_dedupes(self) -> None: + assert candidate_labels(_OPTIONS) == [*_OPTIONS, BOTH_LABEL, NONE_LABEL] + assert candidate_labels([BOTH_LABEL]) == [BOTH_LABEL, NONE_LABEL] + + def test_label_key_strips_articles_and_punctuation(self) -> None: + assert _label_key(" The Big, BLUE Square! ") == "big blue square" + assert _label_key("") == "" + + def test_match_label_returns_none_when_nothing_matches(self) -> None: + assert match_label("a green cup", candidate_labels(_OPTIONS)) is None + assert match_label("", candidate_labels(_OPTIONS)) is None + + +# ── compute_metrics() / get_key_metrics() ────────────────────────────────── + + +class TestComputeMetrics: + def test_mean_and_rates(self) -> None: + tasks = [ + [{"reward": 1.0, "exact": True, "parsed": True, "target": BOTH_LABEL, "predicted_label": BOTH_LABEL}], + [ + { + "reward": 0.0, + "exact": False, + "parsed": True, + "target": BOTH_LABEL, + "predicted_label": "a big blue square", + } + ], + [ + { + "reward": 1.0, + "exact": False, + "parsed": True, + "target": "a big blue square", + "predicted_label": "a big blue square", + } + ], + [{"reward": 0.0, "exact": False, "parsed": False, "target": NONE_LABEL, "predicted_label": ""}], + ] + metrics = _server().compute_metrics(tasks) + assert metrics["mean_reward"] == approx(0.5) + assert metrics["count"] == 4 + assert metrics["exact_match_rate"] == approx(0.25) + assert metrics["parse_rate"] == approx(0.75) + assert metrics["label_resolve_rate"] == approx(0.75) + assert metrics["accuracy_both_of_them"] == approx(0.5) + assert metrics["accuracy_none_of_them"] == approx(0.0) + + def test_empty(self) -> None: + assert _server().compute_metrics([]) == {} + + +class TestGetKeyMetrics: + def test_selects_headline(self) -> None: + out = _server().get_key_metrics({"mean_reward": 0.5, "exact_match_rate": 0.25, "parse_rate": 0.75, "count": 4}) + assert out == {"mean_reward": approx(0.5), "exact_match_rate": approx(0.25)} + + +# ── _response_text() ─────────────────────────────────────────────────────── + + +class TestResponseText: + def test_output_text_fast_path(self) -> None: + assert _response_text(_make_response("hello")) == "hello" + + def test_fallback_joins_message_content(self) -> None: + message = SimpleNamespace( + type="message", + content=[SimpleNamespace(text="a"), {"text": "b"}], + ) + reasoning = SimpleNamespace(type="reasoning", content="ignored") + response = SimpleNamespace(output_text=None, output=[reasoning, message]) + assert _response_text(response) == "ab" + + def test_fallback_string_content(self) -> None: + message = SimpleNamespace(type="message", content="plain") + response = SimpleNamespace(output_text="", output=[message]) + assert _response_text(response) == "plain" + + +# ── Acceptance / parity ──────────────────────────────────────────────────── + + +class TestAcceptance: + async def test_each_example_gold_scores_one(self) -> None: + server = _server() + rows = [json.loads(line) for line in _EXAMPLE_JSONL.read_text().splitlines() if line.strip()] + assert len(rows) >= 5 + for row in rows: + result = await server.verify( + _make_request( + f"Final answer: {row['target']}", + target=row["target"], + options=row["options"], + ) + ) + assert result.reward == approx(1.0), row["target"] + + async def test_example_rows_cover_both_and_none_golds(self) -> None: + rows = [json.loads(line) for line in _EXAMPLE_JSONL.read_text().splitlines() if line.strip()] + golds = {row["target"] for row in rows} + assert BOTH_LABEL in golds + assert NONE_LABEL in golds + # Every row must offer exactly two story objects to choose between. + assert all(len(row["options"]) == 2 for row in rows) + + async def test_echoing_the_question_options_never_scores(self) -> None: + server = _server() + rows = [json.loads(line) for line in _EXAMPLE_JSONL.read_text().splitlines() if line.strip()] + for row in rows: + text = f"Final answer: {row['options'][0]} or {row['options'][1]}" + result = await server.verify(_make_request(text, target=row["target"], options=row["options"])) + assert result.reward == approx(0.0), row["target"]