DancingMate: COORDINATED AND SYNCHRONIZED DANCE ACCOMPANIMENT GENERATION

Code release for the ICASSP 2026 work DancingMate: Coordinated and Synchronized Dance Accompaniment Generation.

Overview

DancingMate studies dance accompaniment: given music and a leader dancer, synthesize the motion of a follower dancer that is rhythmically synchronized with the music and spatially coordinated with the partner. This repository packages the training, evaluation, preprocessing, metric, and visualization code used for the DancingMate release.

The repository is organized as a standalone codebase so that the main experiments can be reproduced after preparing the required data, SMPL-X assets, and checkpoints. Large datasets, pretrained weights, generated videos, and restricted third-party assets are intentionally not stored in git.

Installation

The original code was developed with an older CUDA/PyTorch stack. Python 3.8 and the provided conda environment are recommended.

conda env create -f environment.yml
conda activate dancingmate

Some dependencies are platform-specific:

• Install pytorch3d for the CUDA/PyTorch version on your machine.
• Download SMPL-X models separately from the official SMPL-X website.
• Install Blender and the SMPL-X Blender add-on only if you need mesh/video visualization.
• The contact metric uses an optional external mesh-intersection dependency.

Data

DD100 data and generated caches are not stored in git. The expected local layout is documented in DATA.md. A typical prepared workspace looks like:

data/
  motion/
    smplx/{train,test}/*.npy
    pos3d/{train,test}/*.npy
    rotmat/{train,test}/*.npy
  music/
    mp3/{train,test}/*.mp3
    feature/{train,test}/*.npy

Prepare motion and music features with:

export SMPLX_MODEL_PATH=/path/to/smplx/models
python _train_test_split.py
python _prepare_motion_data.py --motion-root ./data/motion
python _prepare_music_data.py --mp3-root ./data/music/mp3 --feature-root ./data/music/feature

Training

The release follows a four-stage training pipeline. Commands below assume they are run from the repository root.

# 1. Pose motion VQ-VAE
python main_mix.py --config configs/sep_vqvaexm_full_final.yaml --train

# 2. Translation VQ-VAE
python main_transl.py --config configs/transl_vqvaex_final.yaml --train

# 3. Follower GPT
python main_gpt2t.py --config configs/follower_gpt_beta0.9_final.yaml --train

# 4. Off-policy RL fine-tuning
python main_ac_new.py --config configs/rl_final_debug_reward3_random_5mem_lr3e-5.yaml --train

The srun_*.sh wrappers run locally by default. To use Slurm, set DANCINGMATE_LAUNCHER=slurm:

DANCINGMATE_LAUNCHER=slurm ./srun_mix.sh configs/sep_vqvaexm_full_final.yaml train PARTITION 1

Evaluation

Solo and duet metrics are provided under utils/:

python utils/metrics.py
python utils/metrics_duet.py
python utils/metric_footskating_dur.py

The contact-frequency metric depends on an external mesh-intersection package that is intentionally not vendored in this release. See CONTACT_METRIC.md.

Visualization

SMPL-X rendering requires Blender and the SMPL-X Blender add-on.

export BLENDER_BIN=/path/to/blender
export SMPLX_BLENDER_ADDON=/path/to/SMPL_blender_addon.zip
./visualize_01.sh data/motion/smplx/test_vis outputs/vis

You can also set DANCINGMATE_SMPLX_VIS_DIR and DANCINGMATE_VIDEO_DIR to choose default input and output folders for visualize_01.sh.

Repository Layout

configs/              Training and evaluation configs
datasets/             DD100-style motion/music dataloaders
models/               VQ-VAE, GPT, actor-critic, reward, and helper modules
utils/                Metrics, logging, saving, visualization, and feature helpers
tools/vis/            Optional Blender SMPL-X renderer
docs/                 Data, model zoo, contact metric, and release notes
data/                 Local data root, ignored by git
checkpoints/          Local checkpoint root, ignored by git
outputs/              Local rendering/evaluation outputs, ignored by git

Baseline

Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment (ICLR 2024) is used as the baseline method. The original Duolando project page is available here.

Acknowledgements

This release uses Duolando as a baseline and also relies on the broader music-to-dance and motion-generation ecosystem, including PyTorch, PyTorch3D, SMPL-X, Blender, and related evaluation utilities. Third-party assets and restricted dependencies are not vendored in this release. See NOTICE.md for details.

License

See LICENSE.md and NOTICE.md for license and third-party notices. SMPL-X, Blender, datasets, pretrained weights, and optional metric dependencies have their own licenses and terms of use.