We present DISCOVERSE, the first unified, modular, open-source 3DGS-based simulation framework for Real2Sim2Real robot learning. It features a holistic Real2Sim pipeline that synthesizes hyper-realistic geometry and appearance of complex real-world scenarios, paving the way for analyzing and bridging the Sim2Real gap. Powered by Gaussian Splatting and MuJoCo, DISCOVERSE enables massively parallel simulation of multiple sensor modalities and accurate physics, with inclusive supports for existing 3D assets, robot models, and ROS plugins, empowering large-scale robot learning and complex robotic benchmarks. Through extensive experiments on imitation learning, DISCOVERSE demonstrates state-of-the-art zero-shot Sim2Real transfer performance compared to existing simulators.
DISCOVERSE unifies real-world captures, 3D AIGC, and any existing 3D assets in formats of 3DGS (.ply), mesh (.obj/.stl), and MJCF physical models (.xml), enabling their use as interactive scene nodes (objects and robots) or the background node. We leverage Gaussian splatting as our rendering engine to generate hyper-realistic radiance field rendering of multiple sensor modalities and use MuJoCo as the physical engine to ensure accurate physics. Benefiting from the efficiency and fidelity, DISCOVERSE enables user-definable data generation strategy, evaluation metrics, and algorithms for robotics and embodied AI, empowering a variety of applications, e.g., parallel training, complex robotic benchmarks, etc.
DISCOVERSE achieves a total of 650 FPS for rendering hyper-realistic RGB-D frames at 640x480 resolution with 5 cameras on a desktop with Ubuntu 20.04, on 3.1 GHz Intel Xeon w5-3435x CPU and an Nvidia 6000 Ada GPU, and achieves 240 FPS with the same setup on a laptop with Ubuntu 20.04, on 3.2 GHz AMD R7-5800H CPU and an Nvidia GeForce RTX 3060 GPU.
DISCOVERSE uses 3DGS as a universal visual representation and integrate laser scanning, state-of-the-art generative models, and physically-based relighting to boost the geometry and appearance fidelity of the reconstructed radiance fields. Please refer to this documentation for detailed implementations of the Real2Sim solution.
We leverage XGRIDS for scene-level Real2Sim generation. In collaboration with XGRIDS, we offer users a specialized service for hardware and software supports. If you are interested, please visit this link to submit your information and view pricing details.
DISCOVERSE enables zero-shot transfer of contact-rich real-world manipulation tasks. The video plays at normal speed (1x).
@misc{discoverse2024,
title={DISCOVERSE: Efficient Robot Simulation in Complex High-Fidelity Environments},
author={Yufei Jia and Guangyu Wang and Yuhang Dong and Junzhe Wu and Yupei Zeng and Haizhou Ge and Kairui Ding and Zike Yan and Weibin Gu and Chuxuan Li and Ziming Wang and Yunjie Cheng and Wei Sui and Ruqi Huang and Guyue Zhou},
url={https://air-discoverse.github.io/},
year={2024}
}This work is supported by funding from XIAOMI FOUNDATION. The authors would like to acknowledge DISCOVER Lab and DISCOVER Robotics for technical and hardware supports. The authors would also like to thank Deemos for their invaluable supports on 3D generation. We also thank Nerfies for templates of this website.