ViReSkill

This repository is the official implementation of "ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learning".

Installation

To run ViReSkill on LIBERO, install LIBERO dependancies by running the following commands:

git clone https://github.com/PanasonicConnect/vireskill.git
conda create -n "vireskill" python=3.8.13
conda activate vireskill
# install LIBERO
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -r requirements.txt
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
pip install -e .
# download datasets
# Use the following LIBERO datasets: libero_spatial, libero_object, libero_goal, and libero-10. Do not use libero-90.
python benchmark_scripts/download_libero_datasets.py
# install ViReSkill dependencies
cd ../vireskill
pip install -r requirements.txt

Setup

• Save your OpenAI API key to "OpenAI_api_key.txt".
• To reproduce our evaluation results, comment out the lines in the LIBERO code.

Run Training and Evaluation

We provided scripts for the following baselines used in the paper:

• ViReSkill: Our method, which uses skill memory and vision-grounded replanning.
• VoxPoser: The base code-generation pipeline without replanning. A single control plan is generated per task and executed to compute success rate.

./run_voxposer.sh    # Baseline(VoxPoser)
./run_vireskill.sh   # ViReSkill
python3 ./extract_success_rates.py  # check the results

Acknowledgements

This project was made possible thanks to the open-source contributions of the following repositories:

• LIBERO: Used as the base environment and dataset for training and evaluation.
• VoxPoser: Provided inspiration for vision-language action planning baselines.
• OpenVLA: We referred to its evaluation scripts for designing our own evaluation pipeline.