World Model Planners

Documentation for data preparation, model architectures, training, and the ROS 2 planner bridge.

← Project home

Guides

• Getting started
  • Python workspace setup
  • Build the ROS 2 wrapper
• Data pipeline
  • Run the pipeline
  • Stage 0: extract bags
  • Stage 1: build episodes
  • Stage 2: select episodes
  • Stage 3: save or upload
  • Main CLI options
  • Output
• World-model pipeline
  • Model families and training objectives
  • Loading episodes
  • Standard WorldModel: CNN, DINOv2, V-JEPA2, and RSSM
  • DinoWM
  • Planning and the ROS 2 bridge
• ROS 2 planner wrapper
  • Setup and launch
  • What the launch starts
  • Goals, outputs, and node parameters
• Training and evaluation
  • Standard WorldModel: CNN or DINOv2, MLP or RSSM
  • JEPA-lite and state decoder
  • DinoWM
  • Checkpoints and evaluation

References

[ABF+25]

Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-JEPA 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. URL: https://arxiv.org/abs/2506.09985.

[ADM+23]

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. arXiv preprint arXiv:2301.08243, 2023. URL: https://arxiv.org/abs/2301.08243.

[Gon85]

Teofilo F. Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38:293–306, 1985. doi:10.1016/0304-3975(85)90224-5.

[HLF+19]

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning. 2019. URL: https://arxiv.org/abs/1811.04551.

[O+23]

Maxime Oquab and others. DINOv2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. URL: https://arxiv.org/abs/2304.07193.

[Rub99]

Reuven Y. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1:127–190, 1999. doi:10.1023/A:1010091220143.

[ZPLP24]

Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: world models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983, 2024. URL: https://arxiv.org/abs/2411.04983.