add new crmarena dataset

[sampatsk]

Add CRMArena dataset

Description

Adds datasets/crmarena/, a new dataset of CRM agent trajectories derived from
Salesforce AI Research's CRMArena benchmark. The public Salesforce/CRMArena
dataset ships only tasks + ground-truth answers (no trajectories), so this
integration converts agent rollouts: the CRMArena tool-calling agent
(--agent_strategy tool_call --org_type original) was run over the benchmark
tasks against a live Salesforce sandbox, and each rollout's OpenAI-style message
history (system prompt → user query → assistant tool calls → tool observations →
final respond) is converted to ATIF and standardized ATIF, then to OpenHands v0
and OpenHands SDK SFT.

Type of Change

• New dataset
• New agent
• Bug fix
• Documentation update
• Other

Dataset/Agent Information

• Name: crmarena
• Source: dataset Salesforce/CRMArena
  (tasks/answers) + agent harness
  SalesforceAIResearch/CRMArena
  (tools + rollout runner)
• License: CC BY-NC 4.0 (research use only) — recorded in datasets/crmarena/LICENSE
• Domain: Customer Relationship Management (Salesforce) — multi-turn
  function-calling over 27 CRM tools incl. SOQL/SOSL
• Split/size: rollouts over the test split across all 9 original task
  categories; committed sample = one representative trajectory per category
  (9 trajectories, 6–11 ATIF steps and 4–9 tool calls each). Rollouts generated
  with a GPT-5.1 tool-calling model.

Files Added

datasets/crmarena/
├── README.md                     # source, license, schema mapping, regen + validate commands
├── LICENSE                       # CC BY-NC 4.0 notice
├── requirements.txt              # deps to REGENERATE rollouts (not needed by the ADP scripts)
├── extract_raw.py                # reads CRMArena result logs; renames `traj` → `messages`
├── schema_raw.py                 # Pydantic model of the raw rollout record
├── raw_to_atif.py                # shim → scripts/raw_to_atif_common.py
├── atif_to_std.py                # shim → scripts/atif_to_std_common.py
├── metadata.json                 # 27 CRMArena tools as custom_tools + sample_expectations
├── sample_raw.json               # 9 records (one per task category)
├── sample_atif.json              # 9 records
├── sample_std.json               # 9 records
└── sample_sft/
    ├── openhands_v0.json         # 9 records
    └── openhands_sdk.json        # 9 records

Schema Mapping Summary

• Each rollout's traj (renamed to messages by extract_raw.py) is an OpenAI
  chat list, converted by the shared scripts/raw_to_atif_common.py.
• system / user messages → ATIF system / user steps.
• assistant messages with tool_calls → ATIF agent steps; ToolCalls
  preserve the original CRMArena function names and JSON arguments.
• tool messages → ATIF observations linked to the preceding tool call via
  source_call_id (the original tool_call_id). Example:
  call_3dLY… → "2024-04-22T23:59:59Z".
• Task context (task_id, task_type, gt_answer, reward, agent_info)
  preserved in ATIF extra.raw.
• atif_to_std.py applies shared normalization while retaining the CRMArena
  custom tools declared in metadata.json. respond (the agent's final-answer
  tool) is kept as a custom tool rather than remapped to the built-in finish.

Testing

Run with the venv activated and export PYTHONPATH=$(pwd) (repo convention):

• pytest tests/ -q → 1158 passed, 1 skipped, 0 failed
• Focused: test_dataset_structure, test_raw_schemas,
  test_standardized_schemas, test_std_to_sft_conversion,
  test_sample_expectations, test_schema_version (all -k crmarena) pass
• scripts/check_sample_fixtures.py --dataset crmarena → ok crmarena
• ruff check / ruff format --check clean; all files end with a trailing newline
• Sample files regenerated end-to-end from the committed scripts (not hand-edited)

[sampatsk]

The workflow pins OpenHands/extensions/plugins/pr-review@main, which installs openhands-sdk + openhands-tools + lmnr via uv run --with ... but not openhands-agent-server (the package that provides fastapi). openhands-sdk 1.36.0/1.36.1 (released 2026-07-13/15) started importing fastapi at runtime without declaring it as a dependency, so every PR review now crashes.