grutopia

GRUtopia

🔥 News

• [2025-02] GRUtopia 2.0 released!
• [2024-07] We release the paper and demos of GRUtopia.

🚀 New Features in 2.0 release

• Gym compatible env implementation.
• Easy-to-use pythonic config system to use out-of-the-box sensors, controllers, robots and tasks.
• Examples of driving diverse robots and the corresponding policies.
• Benchmark and baseline for social navigation and mobile manipulation task.
• Teleportation tools with Mocap and Apple VisionPro.
• Physically accurate interactive object assets that are ready for simulation.
• Procedural Indoor Scene Generation with GRScenes-100.

📋 Contents

• 🏠 About
• 📚 Getting Started
• 🏙️ Assets
• 📦 Benchmark & Method
• 👥 Support
• 📝 TODO List
• 🔗 Citation
• 📄 License
• 👏 Acknowledgements

🏠 About

Recent works have been exploring the scaling laws in the field of Embodied AI. Given the prohibitive costs of collecting real-world data, we believe the Simulation-to-Real (Sim2Real) paradigm is a more feasible path for scaling the learning of embodied models.

We introduce project GRUtopia (aka. 桃源 in Chinese), a general-purpose research platform for embodied AGI. It features several advancements:

• 🏙️ GRScenes, the scene dataset, includes 100k interactive finely annotated scenes. GRScenes covers 89 diverse scene categories, facilitating deployment of general robots across different scenarios.
• 🧑‍🤝‍🧑 GRResidents, a Large Language Model (LLM) driven Non-Player Character (NPC) system that enables social interaction, task generation, and task assignment, thus simulating social scenarios for embodied AI applications.
• 🤖 GRBench, a collection of embodied AI benchmarks for assessing various capabilities of solving embodied tasks.

We hope that this work can alleviate the scarcity of high-quality data in this field and provide a more comprehensive assessment of embodied AI research.

📚 Getting Started

Prerequisites

• Ubuntu 20.04, 22.04
• NVIDIA Omniverse Isaac Sim 4.2.0
  • Ubuntu 20.04/22.04 Operating System
  • NVIDIA GPU (RTX 2070 or higher)
  • NVIDIA GPU Driver (recommended version 535.216.01+)
  • Docker (Optional)
  • NVIDIA Container Toolkit (Optional)
• Conda
  • Python 3.10.16 (3.10.* should be ok)

Installation

We provide the installation guide here. You can install locally or use docker and verify the installation easily.

Documentation & Tutorial

We provide detailed docs for the basic usage of different modules supported in GRUtopia. Welcome to try and post your suggestions!

🏙️ Assets

[!NOTE] 📝First of all you MUST complete the User Agreement for GRScenes-100 Dataset Access.

Then you can choose to download all assets (~80GB) or a minimum set (~500MB) to examine installation by running the following script with GRUtopia installed:

$ python -m grutopia.download_assets

The default path to store the downloaded assets is ${PATH/TO/GRUTOPIA/ROOT}/grutopia/assets. Users have two ways to configure the asset path:

• Spcecify a custom path during download using python -m grutopia.download_assets.
• Set it later by running python -m grutopia.set_assets_path and entering the preferred directory.

GRScenes-100

If you want to separately download the GRScenes-100 scene assets, you can manually download them from OpenDataLab, ModelScope and HuggingFace. Please refer to the instructions for scene usage.

Robots & Weights

If you want to separately download robots and policy weights, you can manually download the robot directory from from OpenDataLab, ModelScope and HuggingFace and move it to the root of the asset path.

📦 Benchmark & Method

We preliminarily establish three benchmarks for evaluating the capabilities of embodied agents from different aspects: Object Loco-Navigation, Social Loco-Navigation, and Loco-Manipulation. Please refer to the documentation for running the benchmarks.

👥 Support

Join our WeChat support group for any help.

📝 TODO List

• [x] Release the paper with demos.
• [x] Release the platform with basic functions and demo scenes.
• [x] Release 100 curated scenes.
• [x] Polish APIs and related codes.
• [x] Full release and further updates.
• [x] Release the baseline methods and benchmark data.
• [ ] Support multiple episodes.
• [ ] Vectorized env and batch execution.
• [ ] Training framework.

🔗 Citation

If you find our work helpful, please cite:

@inproceedings{grutopia,
    title={GRUtopia: Dream General Robots in a City at Scale},
    author={Wang, Hanqing and Chen, Jiahe and Huang, Wensi and Ben, Qingwei and Wang, Tai and Mi, Boyu and Huang, Tao and Zhao, Siheng and Chen, Yilun and Yang, Sizhe and Cao, Peizhou and Yu, Wenye and Ye, Zichao and Li, Jialun and Long, Junfeng and Wang, ZiRui and Wang, Huiling and Zhao, Ying and Tu, Zhongying and Qiao, Yu and Lin, Dahua and Pang Jiangmiao},
    year={2024},
    booktitle={arXiv},
}

📄 License

GRUtopia's simulation platform is MIT licensed. The open-sourced GRScenes are under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License .

👏 Acknowledgements

• OmniGibson: We refer to OmniGibson for designs of oracle actions.
• RSL_RL: We use rsl_rl library to train the control policies for legged robots.
• ReferIt3D: We refer to the Sr3D's approach to extract spatial relationship.
• Isaac Lab: We use some utilities from Orbit (Isaac Lab) for driving articulated joints in Isaac Sim.
• Open-TeleVision: We use Open-TeleVision to teleoperate with Apple VisionPro.
• HaMeR: We use HaMeR to recognize hand gesture in teleoperate with camera.
• Infinigen: We use Infinigen to procedurally generate indoor scenes upon GRScenes-100 dataset.
• VLFM: We refer to VLFM to implement our benchmark baselines.
• Grounding DINO: We use Grounding DINO in our benchmark baselines.
• YOLOv7: We use YOLOv7 in our benchmark baselines.
• MobileSAM: We use MobileSAM in our benchmark baselines.