comma.ai

A driving policy training in the world model. Strong steering noise causes drift, and the policy learns to recover with further training. Shown at 2× speed.

Things I am working on at comma.ai.

End-to-End Machine Learning and World Models for Autonomous Driving

Adapted from Learning to Drive from a World Model.

By now, most autonomous driving labs agree that building a fully autonomous driving policy based on hard-coded rules and engineered features is doomed to fail. The only realistic way to build an autonomous driving policy that scales to arbitrarily complex and diverse environments is to use methods that scale arbitrarily with computation and data: search and learning.¹ We want a driving policy that is trained end-to-end and learns to drive from experience like we do.

A key challenge in end-to-end learning is how to train a policy that can perform well outside the i.i.d. setting assumed by most supervised learning algorithms, such as behavior cloning. In the real world, the policy's predictions influence its future observations. Small errors accumulate over time, leading to a compounding effect that drives the system into states it never encountered during pure imitation-learning training.

To overcome this, the driving policy needs to be trained on-policy, allowing it to learn from its own interactions with the environment and enabling it to recover from its own mistakes.

World models^{2, 3,
4} are data-driven simulators. They are generative models that predict the next world state given a history of past states and actions.

World models can take many forms. The key idea is to represent the state as a lower-dimensional latent representation using a “compressor model,” and to model the dynamics of the latent space using a “dynamics model.”

We can control the world model by providing a desired six-degree-of-freedom (6-DoF) relative transformation between the current and next state.

The driving policy is trained in the (dreams of the) world model and then deployed in the real world.

The holy trinity of scaling: inference, training, data

The driving world models we train are “World-Action Models”: they predict both future frames and future actions. Their performance scales predictably over more than three orders of magnitude in training FLOPs. Over the project's lifetime, we have scaled the models from 500 million to 4 billion parameters, expanded the training set from 500,000 to 10 million minutes of driving, and grown the training mesh from 8-way Distributed Data Parallel (DDP) to a 32 × 8 Hybrid Sharded Data Parallel (HSDP) mesh.

Log-scale chart showing frame-prediction loss decreasing as training FLOPs increase — World model loss follows a consistent scaling law as training compute grows.

Frame-prediction performance correlates almost perfectly with action-prediction performance across model sizes, dataset sizes, and training epochs.

Scatter plot showing a close correlation between frame-prediction loss and action-prediction loss — Validation world model loss closely tracks validation action-prediction loss.

Inference-time efficiency is also important: during driving-policy training, thousands of episode workers continually request frames from the world model. To serve them efficiently, we quantize the diffusion transformer (DiT) to FP4/FP8, use an FP8 key-value (KV) cache, and run classifier-free guidance (CFG) packed across the context dimension.

The driving policy also follows consistent scaling laws as we increase compute at inference time and during training. We use the ConvNeXt family of models with depth/width maximal update parametrization (μP).

Log-scale chart showing path-planning error decreasing as inference FLOPs increase — Path-planning error follows a consistent scaling law as inference-time compute grows.

Fun fact: inspecting the Git history of a file called racks.py in our codebase gives a nice visualization of the growth of our data center.

Step chart showing peak data center FP16 throughput increasing from 2022 to 2026 — Peak data center FP16 throughput over time. GPUs go brrr.

Another fun fact: Codex complained about querying a database to get the number of logged minutes of openpilot data, saying that “the records get denser in recent months.” I guess Codex is right! FYI, the data is not cumulative.

Line chart showing logged monthly openpilot segments trending upward from 2022 to 2026 — Logged monthly openpilot data segments. Values are not cumulative.

Watch it drive

There are lots of videos of openpilot driving out there. In fact, dear reader, you can buy one today and try it yourself! But my all time favorite video is the Taco Bell drive!

Driving to Taco Bell with openpilot end-to-end.

References

Richard Sutton. “The Bitter Lesson.” 2019. ↩︎
David Ha and Jürgen Schmidhuber. “Recurrent World Models Facilitate Policy Evolution.” Advances in Neural Information Processing Systems 31, 2018. ↩︎
Anthony Hu et al. “GAIA-1: A Generative World Model for Autonomous Driving.” 2023. ↩︎
Eder Santana and George Hotz. “Learning a Driving Simulator.” 2016. ↩︎

Other Fun Things I Am Doing at comma.ai

Some cool stuff. Weekend projects. Side quests.

CTFs

comma.ai/tinycorp NeurIPS CTF
Organized a CTF for a NeurIPS 2025 social. Had a good turn out of >100 people
Random 1 flagger
Short but hard

Coding Challenges

Mostly for hiring.

Datasets

Models

Papers

Goff, M., et al. “Learning to Drive from a World Model.” Proceedings of the Computer Vision and Pattern Recognition Conference, CVF, 2025, pp. 1964–1973.

Notable Blog Posts

Learning to Drive from a World Model
Companion blog post of a 2025 CVPR paper
openpilot 0.11
the first point release using only the world model as the simulation stack
openpilot 0.10
the first point release using the world model as part of the simulation stack
Driving to Taco Bell with openpilot end-to-end
lateral control, longitudinal control, and point-to-point navigation
openpilot 0.9.0
experimental end-to-end longitudinal control
openpilot 0.8.10
my first openpilot release hehe