SplatReasoner

Project page | Paper

This is the official implementation of SplatReasoner: Enhancing Embodied Reasoning and Grounding by Novel View Synthesis, accepted to ECCV 2026.

Because the full benchmarks contain many scenes and questions, this release provides complete end-to-end pipeline results for a manageable subset of 10 HM3D and 10 ScanNet episodic-memory scenes.

Processed queries / full evaluation queries:

• EM-EQA: 219 / 1,623
• ScanRefer: 312 / 9,508
• Multi3DRefer: 338 / 10,214 non-zero-target queries
• 3D Referring Segmentation: 49 / 49 (introduced in this work)

To evaluate the full benchmarks, run the provided pipeline on all scenes and questions in the corresponding datasets.

📋 Prerequisites

Download Habitat-Matterport 3D (HM3D) and ScanNet, then extract them to the following paths relative to the repository root.

• HM3D → data/raw/hm3d/
• ScanNet → data/raw/scannet/

Download the required SAM, YOLO-World, and PQGS assets:

mkdir -p data/assets

wget -c -O data/assets/index_1105_128_1024.faiss https://raw.githubusercontent.com/kaist-ami/Dr-Splat/main/my_utils/index_1105_128_1024.faiss
wget -c -O data/assets/text_negative.npy https://raw.githubusercontent.com/kaist-ami/Dr-Splat/main/assets/text_negative.npy
wget -c -O data/assets/sam_vit_h_4b8939.pth https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
wget -c -O data/assets/sam_l.pt https://github.com/ultralytics/assets/releases/download/v8.2.0/sam_l.pt
wget -c -O data/assets/yolov8x-world.pt https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-world.pt

HM3D frame rendering also requires the OpenEQA HM3D agent states.

mkdir -p data/raw/open-eqa-hm3d-states
tar -xzf open-eqa-hm3d-states-v0.tgz -C data/raw/open-eqa-hm3d-states

The extracted directories should follow this layout:

data/raw/
├── hm3d/
│   └── val/{scene}/{scene}.basis.glb
├── open-eqa-hm3d-states/
│   └── hm3d-v0/{episode}/*.pkl
└── scannet/
    └── scans/{scene}/{scene}.sens

Raw HM3D and ScanNet data are not distributed with this repository.

📁 Data

The scene and task split is fixed in data/manifests/scenes.json.

data/
├── annotations/        # Task annotations, selected classes, and class lists
├── assets/             # SAM, YOLO-World, FAISS PQ, negative text features
├── frames/             # Generated RGB, depth, and camera trajectories
│   ├── hm3d-v0/        # 10 selected episodic histories
│   └── scannet-v0/     # 10 selected episodic histories
├── ground_truth/       # Generated Gaussian GT masks
├── manifests/          # Scene subset and source-path metadata
├── raw/                # Locally downloaded raw datasets
└── snapshots/          # Episodic snapshots used by EQA prefiltering

All scripts use data/ and outputs/ by default. To use custom locations:

export SPLATREASONER_DATA_ROOT=/path/to/data
export SPLATREASONER_OUTPUT_ROOT=/path/to/outputs  # optional

After creating the environment below, generate the RGB, depth, camera, initial point-cloud, and aligned ScanNet depth inputs. Depth is optional and is used only for 3DGS depth regularization.

python 3dgs/preprocess/hm3d/extract-frames_transforms.py
python 3dgs/preprocess/hm3d/extract_points.py

python 3dgs/preprocess/scannet/extract-frames_transforms.py
python 3dgs/preprocess/scannet/extract_points.py
python 3dgs/utils/make_depth_scale_scannet_all.py

The frame renderers process the 10 HM3D and 10 ScanNet manifest scenes by default. Use --scene to render a smaller subset:

python 3dgs/preprocess/hm3d/extract-frames_transforms.py --scene AYpsNQsWncn
python 3dgs/preprocess/scannet/extract-frames_transforms.py --scene scene0203_01

The frame renderers write the selected trajectories to data/frames/. Place the ScanRefer-format inputs used by ScanRefer and Multi3DRefer under data/raw/scanrefer/data/. Generated data/frames/ and data/ground_truth/ are not included in the Git repository or source archive.

ScanRefer and Multi3DRefer annotations remain unchanged and are filtered in memory using the manifest.

The downloaded assets follow this layout:

data/assets/
├── index_1105_128_1024.faiss
├── sam_l.pt
├── sam_vit_h_4b8939.pth
├── text_negative.npy
└── yolov8x-world.pt

🛠️ Environment

3DGS and PQGS, following the Dr. Splat (CVPR 2025) implementation, share one Python 3.9, PyTorch 2.2.1, CUDA 11.8 environment. Their CUDA rasterizers use different package names: diff_gaussian_rasterization and pqgs_diff_gaussian_rasterization. The environment pins NumPy 1.25 for compatibility with the Habitat-Sim and PQGS stages.

conda env create -f environment.yml
conda activate splatreasoner
./install_extensions.sh

For GPT-based view selection and evaluation, set OPENAI_API_KEY directly or copy the example key file:

cp key/openai_api_key.example.txt key/openai_api_key.txt
# Replace the placeholder in key/openai_api_key.txt before loading it.
source key/load_key.sh

🧠 3DGS and PQGS

All commands read data/manifests/scenes.json. Use --partition and --num-partitions to distribute scenes across GPUs.

Reconstruct depth-regularized 3DGS:

python 3dgs/runs/reconstruct.py hm3d --gpu 0
python 3dgs/runs/reconstruct.py scannet --gpu 0
python 3dgs/runs/reconstruct.py scanrefer --gpu 0

Extract SAM/OpenCLIP image features for PQGS:

python pqgs/runs/extract_features.py hm3d --gpu 0
python pqgs/runs/extract_features.py scannet --gpu 0
python pqgs/runs/extract_features.py scanrefer --gpu 0

Feature extraction is resumable per frame. Long scenes can be split without changing their features:

python pqgs/runs/extract_features.py hm3d \
  --frame-partition 0 --num-frame-partitions 2 --gpu 0
python pqgs/runs/extract_features.py hm3d \
  --frame-partition 1 --num-frame-partitions 2 --gpu 1

Optimize discrete category PQGS for EQA. This stage does not require the SAM/OpenCLIP feature-extraction step:

python pqgs/runs/train_pqgs.py label hm3d --gpu 0
python pqgs/runs/train_pqgs.py label scannet --gpu 0

Optimize continuous CLIP PQGS for grounding:

python pqgs/runs/train_pqgs.py image scannet --gpu 0
python pqgs/runs/train_pqgs.py image scanrefer --gpu 0

Base and semantic checkpoints are written below:

outputs/3dgs/{hm3d,scannet,scanrefer}/{scene}/
├── chkpnt30000.pth
├── pqgs_label_200_feature1_topk1/chkpnt0.pth
└── pqgs_image_1_pq_openclip_fix_topk45_weight_128/chkpnt0.pth

❓ EQA Evaluation

The release subset contains 219 OpenEQA questions.

Evidence-category extraction can be reproduced with:

bash vlm-as-judge/shells/prefilter_eqa.sh

Render initial and query-conditioned novel views:

python pqgs/runs/render_candidates.py eqa no-novel --gpu 0
python pqgs/runs/render_candidates.py eqa novel --gpu 0

For EM-EQA, Gaussian clustering uses distance thresholds of 3.0 for HM3D and 0.4 for ScanNet. Cross-cluster merging uses a threshold of 1.1.

Generate answers and run verification:

bash vlm-as-judge/shells/eqa_no_novel.sh
bash vlm-as-judge/shells/eqa_novel.sh
bash vlm-as-judge/shells/eqa_verify.sh

Compute LLM-Match:

# Initial view selection
bash eqa_eval/run_llm_match.sh \
  outputs/eqa/answers/no_novel/gpt_answer.json initial_view_selection

# Novel view synthesis + verification
bash eqa_eval/run_llm_match.sh \
  outputs/eqa/answers/verified/gpt_answer.json verified

Outputs:

outputs/views/eqa/{no_novel,novel}/{question_id}/images/
outputs/eqa/answers/{no_novel,novel,verified}/gpt_answer.json
outputs/eqa/metrics/

Setting	Questions (evaluated / full)	LLM-Match
Initial view selection	219 / 1,623	65.41
Novel view synthesis + verification	219 / 1,623	67.35

GPT-4o is used for answer generation, GPT-5 mini for verification, and GPT-4 Turbo as the LLM-Match judge. The initial view selection result does not use novel view synthesis or the verification model.

The full EM-EQA evaluation set contains 1,623 questions. It is derived from the 1,636 OpenEQA questions by excluding 13 questions from scannet-v0/109-scannet-scene0648_01, for which the author-provided episode metadata required by the pipeline is unavailable.

The release scores are in results/eqa/initial_view_selection.json and results/eqa/novel_view_synthesis_and_verification.json. These values are computed on the selected 20-scene subset and are not directly comparable to the full-set results reported in the paper.

🎯 3D Grounding Evaluation

Novel view synthesis can also be applied to 3D grounding. This release focuses on initial view selection, which already provides strong grounding performance without additional per-query novel-view rendering.

ScanRefer

ScanRefer uses 312 queries from scene0015_00, scene0406_00, and scene0645_00. The full benchmark contains 9,508 queries, so this release evaluates the selected three-scene subset. Novel views are omitted because synthesizing them for every query in the full benchmark is computationally expensive.

# 1. Render initial-view candidates
python pqgs/runs/render_candidates.py scanrefer no-novel --gpu 0

# 2. Select a view and convert it to Gaussian indices
bash vlm-as-judge/shells/scanrefer_no_novel.sh
python pqgs/runs/extract_gaussian_indices.py scanrefer no-novel --gpu 0

# 3. Distill ground truth and evaluate
bash grounding_eval/run_eval.sh distill --task scanrefer
bash grounding_eval/run_eval.sh scanrefer --method ours

Multi3DRefer

Multi3DRefer uses 338 queries from the same three-scene intersection. The full benchmark contains 10,214 non-zero-target queries, so this release evaluates the selected three-scene subset. Novel views are omitted for the same reason: the full benchmark contains too many queries for practical per-query novel-view synthesis.

# 1. Render initial views and their Gaussian index arrays
python pqgs/runs/render_candidates.py multi3drefer no-novel --gpu 0

# 2. Select initial views
bash vlm-as-judge/shells/multi3drefer_no_novel.sh

# 3. Distill ground truth and evaluate
bash grounding_eval/run_eval.sh distill --task multi3drefer
bash grounding_eval/run_eval.sh multi3drefer --method ours --eval-type all

3D Referring Segmentation

3D Referring Segmentation uses all 49 questions from five ScanNet scenes.

# 1. Render initial-view candidates
python pqgs/runs/render_candidates.py referring-segmentation no-novel --gpu 0

# 2. Select a view
bash vlm-as-judge/shells/referring_segmentation_no_novel.sh

# 3. Convert the selected view to Gaussian indices
python pqgs/runs/extract_gaussian_indices.py \
  referring-segmentation no-novel --gpu 0

# 4. Distill ground-truth labels onto the Gaussians
bash grounding_eval/run_eval.sh distill --task referring_segmentation

# 5. Evaluate initial view selection
bash grounding_eval/run_eval.sh referring-segmentation

Results

Task	Queries	mIoU	Acc@8	Acc@10
ScanRefer	312 / 312	0.0669	0.2363	0.2328
Multi3DRefer	338 / 338	0.0613	0.2630	0.2503
3D Referring Segmentation	49 / 49	0.1526	0.4857	0.4571

The reported values match the included evaluation outputs. Results from API-based view selection may vary across runs. The release summaries are in results/grounding/.

Gaussian index outputs

Each task keeps the final Gaussian index arrays and writes per-query IoU plus scene-macro 3D mIoU and accuracy aggregates under outputs/grounding/.

# ScanRefer: initial view selection
outputs/views/scanrefer/no_novel/{target_id}/gaussians/instance_all_indexes.npy

# Multi3DRefer: initial view selection; one file per predicted instance
outputs/views/multi3drefer/no_novel/{target_id}/gaussians_frustum/instance_*_indexes.npy

# 3D Referring Segmentation: initial view selection
outputs/views/referring_segmentation/no_novel/{target_id}/gaussians/instance_all_indexes.npy

⚖️ License

SplatReasoner code is released under the Apache License 2.0. Bundled third-party components retain their original licenses as provided in their respective directories.

📖 Citation

@inproceedings{yu2026splatreasoner,
    title={SplatReasoner: Enhancing Embodied Reasoning and Grounding by Novel View Synthesis},
    author={Yu-Ji, Kim and Lee, Dahye and Jun-Seong, Kim and Hyeon-Woo, Nam and Kim, GeonU and Kwon, Yongjin and Wang, Yu-Chiang Frank and Choe, Jaesung and Oh, Tae-Hyun},
    booktitle={ECCV},
    year={2026}
}