4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere

Yihang Luo¹  Shangchen Zhou¹  Yushi Lan²  Xingang Pan^1,3  Chen Change Loy^1,3 

¹S-Lab, Nanyang Technological University  ²University of Oxford  ³ACE Robotics 

4RC (pronounced "ARC") enables unified and complete 4D reconstruction via conditional querying from monocular videos in a single feed-forward pass.

🎇 For more visual results, go checkout our project page

---

Introducing 4RC

We present 4RC, a unified feed-forward framework for 4D reconstruction from monocular videos. Unlike existing methods that typically decouple motion from geometry or produce limited 4D attributes, such as sparse trajectories or two-view scene flow, 4RC learns a holistic 4D representation that jointly captures dense scene geometry and motion dynamics. At its core, 4RC introduces a novel encode-once, query-anywhere and anytime paradigm: a transformer backbone encodes the entire video into a compact spatio-temporal latent space, from which a conditional decoder can efficiently query 3D geometry and motion for any query frame at any target timestamp. To facilitate learning, we represent per-view 4D attributes in a minimally factorized form, decomposing them into base geometry and time-dependent relative motion. Extensive experiments demonstrate that 4RC outperforms prior and concurrent methods across a wide range of 4D reconstruction tasks.

🔥 News

• [2026/04/13] Our inference code and weights are released!

🔧 Installation

1. Clone Repo

   git clone https://github.com/Luo-Yihang/4RC
   cd 4RC

2. Create Conda Environment

   conda create -n 4rc python=3.11 cmake=3.14.0 -y
   conda activate 4rc

3. Install Python Dependencies

   Important: Install Torch based on your CUDA version. For example, for Torch 2.8.0 + CUDA 12.6:

   # Install Torch
   pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
   
   # Install other dependencies
   pip install -r requirements.txt
   
   # Install 4RC as a package
   pip install -e .

💻 Inference

You can now try 4RC with the following code. The checkpoint will be downloaded automatically from Hugging Face.

import torch

from arc.models.arc.arc import Arc
from arc.dust3r.inference_multiview import inference
from arc.dust3r.utils.image import load_images

device = "cuda" if torch.cuda.is_available() else "cpu"

model = Arc.from_pretrained("Luo-Yihang/4RC").to(device)
model.eval()

example_dir = "examples/robot_arm"
images = load_images(example_dir, size=512, patch_size=14, verbose=True)

with torch.no_grad():
    predictions, profiling = inference(
        images,
        model,
        device,
        dtype="bf16-mixed",
        profiling=True,
        verbose=True,
        use_center_as_anchor=False,
    )

⚡ Demo

Launch the interactive Gradio demo:

python app.py

🔍 CLI

For the command-line workflow without the Gradio UI, use the two-step pipeline:

Step 1: Run inference and save to .npz:

python inference.py --input ./examples/robot_arm --save result.npz

[Optional]

• Use --refine_track_visualization to enable VLA + SAM2 to auto-segment dynamic objects and filter their trajectories for better visulization.
• Use --checkpoint_dir Luo-Yihang/4RC_geofinetune to use the checkpoint finetuned on more geometry datasets for even better geometry prediction.

Step 2: Visualize with viser directly from .npz:

python arc/viz/viser_visualizer_track.py --npz_path result.npz --port 8020

Open http://localhost:8020 in your browser to interact with the 3D visualization.

📁 Code Structure

4RC/
├── arc/
│   ├── models/
│   │   └── arc/
│   ├── dust3r/
│   ├── croco/
│   └── viz/
├── assets/
├── eval/
├── examples/
├── app.py
├── inference.py
├── requirements.txt
├── setup.py
└── README.md

📈 Evaluation

1. Prepare Evaluation Dataset

   For dense tracking, please download the evaluation data from Hugging Face. For point tracking, please download the dataset used by WorldTrack from Google Drive. For camera pose estimation, multi-view 3D reconstruction, and video depth evaluation, we follow the dataset preparation guides provided by MonST3R and Spann3R. For convenience, we also provide the preprocessed datasets on Hugging Face, which can be downloaded directly.

   All datasets should be organized as follows under the project's data directory:

   data/
   ├── 7scenes
   ├── bonn
   ├── kitti
   ├── neural_rgbd
   ├── nyu-v2
   ├── scannetv2
   ├── sintel
   ├── tum
   ├── kubric
   ├── waymo
   ├── po_mini
   ├── ds_mini
   ├── adt_mini
   └── pstudio_mini
   

2. Run Evaluation

   Use the provided scripts to evaluate different tasks.

   3D Tracking

   # Dense tracking
   bash eval/track/eval_dense_track.sh
   
   # Sparse point tracking
   bash eval/track/eval_point_track.sh

   Results will be saved in eval_results/{dense_track,point_track}/${data}/track_eval_${data}.

   Video Depth

   bash eval/video_depth/run.sh

   Results will be saved in eval_results/video_depth/4rc/${data}/result_scale.json.

   Camera Pose Estimation

   bash eval/relpose/run.sh

   Results will be saved in eval_results/relpose/4rc/${data}/_error_log.txt.

   Multi-view Reconstruction

   bash eval/mv_recon/run.sh

   Results will be saved in eval_results/mv_recon/4rc/${data}/logs_all.txt.

📆 TODO

🐎 Pushing the bandwidth limit!

• Release evaluation code.
• Release training code.

📝 Citation

If you find our repo useful for your research, please consider citing our paper:

@inproceedings{luo20264rc,
   title     = {4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere},
   author    = {Yihang Luo and Shangchen Zhou and Yushi Lan and Xingang Pan and Chen Change Loy},
   booktitle = {ICML},
   year      = {2026}
}

📝 Acknowledgments

We recognize several concurrent works on the 4D reconstruction. We encourage you to check them out:

St4RTrack  |  TraceAnything  |  V-DPM  |  Any4D  |  D4RT

4RC is built on the shoulders of several outstanding open-source projects. Many thanks to the following exceptional projects:

DA3  |  VGGT  |  Fast3R  |  DUSt3R  |  Viser

📫 Contact

If you have any questions, please feel free to reach us at luo_yihang@outlook.com.