Question about Cosmos3-Nano-Policy-DROID

[ssj9596]

Hi, thanks for releasing Cosmos3 and the DROID policy.

I noticed that mid-training uses a unified EEF pose action format, but the DROID post-training policy seems to switch to joint-space actions. Is that because joint actions work better on RoboLab, or is there another reason behind it?

Would appreciate any insight. Thanks!

[fwd4]

Good observation. We tried both action spaces during DROID post-training. In our RoboLab runs, the unified EEF pose action format and joint-space action format had very similar success rates, with only about a 2% difference. So the switch to joint-space actions was not because joint actions were substantially better on RoboLab.
The useful takeaway is that Cosmos3 can tolerate an action-space change at post-training time in this setting, while still reaching comparable policy performance. This is likely helped by the pretrained model’s general reasoning and video-generation capabilities, and we encourage users to explore more options on our Cosmos3 platform.

[ssj9596]

Thanks, this is helpful.

I’m trying to reproduce the DROID policy result. I’m using the public LeRobot dataset lerobot/droid_1.0.1, but I noticed a few potential data issues: the wrist camera resolution looks lower than expected, and some episodes seem to have inconsistent left/right camera streams, similar to this discussion: https://huggingface.co/datasets/lerobot/droid_1.0.1/discussions/6

I trained with the official examples/launch_sft_action_policy_droid.sh on 16 A800s, with batch size 32 per GPU, for around 5k steps, but still got 0 success on the 64 RoboLab simple tasks.

Could you clarify which DROID dataset was used for the released policy? Also, roughly how many training steps should be enough to start seeing non-zero success?

Any guidance would be very helpful!

[fwd4]

Thank you for trying it out. For your questions:

• The droid dataset we use has higher resolutions, generated from raw data. It is undergoing internal process and will be released soon. Progress is tracked here
• For reproduction, you can pay attention to pr action dataloader: episode-shuffle stream (fix DROID grad-norm instability) #37 that fix a data shuffle bug. With that fix, you are supposed to get reasonable results on droid. This doc gives more detailed steps in reproducing the results.