Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions evaluation/paibench_g/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: OpenMDW-1.1 -->

# Cosmos3 PAI-Bench (Generation) Reproduction

End-to-end recipe for generating the PAI-Bench (Physical AI Bench) generation
set with Cosmos3-Super using the native Cosmos Framework PyTorch entrypoint
(`python -m cosmos_framework.scripts.inference`).

PAI-Bench covers Physical AI domains (AV driving, robotics, industry, physics,
human, common sense) across 1044 samples. The notebook runs both generation
tasks:

- **Text-to-Video (T2V)**: generate from the prompt only (no condition image);
generate 189 frames.
- **Image-to-Video (I2V)**: condition on the per-sample image
(`condition_image/<image_name>`); generate 189 frames.

Both tasks generate at 24 FPS, 720p, 16:9, and keep the raw output (no staging).

## Files

- `run_with_cosmos_framework.ipynb` — main notebook (demos for both tasks + two full-sweep cells).
- `assets/i2v_prompts.json` — 1044 I2V entries with `json_upsampled_prompt` and `negative_prompt`.
- `assets/t2v_prompts.json` — 1044 T2V entries with `json_upsampled_prompt` and `negative_prompt`.

## Dataset

The condition images come from the Hugging Face dataset
[`shi-labs/physical-ai-bench-generation`](https://huggingface.co/datasets/shi-labs/physical-ai-bench-generation),
cloned via `git clone` (Git LFS). Only the condition images are read from the
dataset; the prompts come from the local `assets/` files.

## Sampling settings

| Setting | Value |
| ----------- | ------------: |
| num_frames | 189 |
| fps | 24 |
| resolution | 720 |
| num_steps | 50 |
| guidance | 6.0 |
| shift | 10.0 |
| seed | 0 |

## Requirements

- 4-GPU Linux node (configurable via `COSMOS3_NUM_GPUS`, default 4)
- `uv >= 0.11.3`
- `git`, `git-lfs`
- Hugging Face access to the Cosmos3 model family
18,794 changes: 18,794 additions & 0 deletions evaluation/paibench_g/assets/i2v_prompts.json

Large diffs are not rendered by default.

18,794 changes: 18,794 additions & 0 deletions evaluation/paibench_g/assets/t2v_prompts.json

Large diffs are not rendered by default.

Loading