Beyond Imitation Learning: How E2E Self-Play Reinforcement Learning Scales for Autonomous Driving
How self-play DAgger distills a privileged Reinforcement Learning driving policy into a camera-based policy

TL;DR
- On the highway, the situations that matter most are the ones least likely to be represented in human driving logs. Models that learn by imitating those logs eventually stop improving.
- Self-play reinforcement learning lets a driving policy generate its own experience. Our paper, Scaling Self-Play for End-to-End Driving, written with Mila, NYU and collaborators and to be presented at CoRL 2026, shows how to do that for end-to-end models that drive from camera images.
- Robust driving doesn’t need photorealism: our simplified simulator renders about 1,000× faster than a photorealistic one. The resulting camera-based policy, trained without human driving trajectories, keeps improving where imitation plateaus and scores best among the methods evaluated on the closed-loop benchmark.
The data runs out where it matters most
At 100 km/h, a truck covers close to 30 meters every second. Most of those seconds are uneventful. The ones that decide whether a driver is good (a stalled car in the lane ahead, a cut-in at speed, a merge that closes faster than expected) are rare on the road, so they are rare in recorded data too.
Most end-to-end driving models today are trained with imitation learning, also called behavior cloning: the model is shown recorded human driving and learns to copy what the human did in each situation. It never drives on its own during training and never gets feedback on the outcome of its own choices. That is what separates it from reinforcement learning.
A model trained to imitate human driving learns the common case very well and the rare case barely at all. It also never practices recovering from its own mistakes, because a log only contains what the human did. Once the model drives on its own, a small error puts it somewhere the data never went, and the next error is bigger.
Recording more miles helps less than you might expect, since most new miles are more examples of driving that was already going fine. In our experiments, an end-to-end model trained by imitation stopped improving at around 100 million agent steps, where one agent step is one vehicle taking one action in the simulator.

Experience you generate, not data you collect
Reinforcement learning (RL) takes a different route. Imitation learning copies what a human did; RL lets the model drive in a simulator, try actions, and learn from a reward that scores what happened: progress made, collisions avoided, a smooth ride. Self-play reinforcement learning is a specific kind of RL in which the model doesn’t drive against recorded traffic. Instead, one policy drives every vehicle in a simulated scene and learns from all of them at once. Because everyone on the road is learning, the traffic keeps getting harder in the ways that matter: gaps close faster, merges get tighter, the car ahead brakes. The rare situations stop being rare, because the agents keep creating them for each other.
With self-play, the ceiling moves from how much data you can collect to how much simulation you can run.
So why isn’t self-play reinforcement learning already the standard way to train driving models? So far, it has worked mainly for policies that see the world as a structured list of objects and lanes. End-to-end models that drive from camera images face two obstacles. First, every self-play step needs fresh camera images, and photorealistic simulators produce them roughly a thousand times slower than the renderer we describe below. Second, RL is sample-inefficient: its policy updates need billions of agent steps of experience, and with a large vision model, every step costs a forward and a backward pass. This work addresses both.
Three ideas that make it work
You don’t need photorealism to learn to drive
What a driver needs from a scene is where things are and how they move, not what they look like. So we built Gigapixel, a simulator that draws the world as plain boxes for vehicles, thin strips for lanes and small spheres for traffic lights, and renders thousands of these worlds at once on a GPU, fast enough for self-play reinforcement learning from pixels. Its renderer sustains about 50,000 agent steps per second on a single NVIDIA A100 GPU, roughly 1,000 times faster than a photorealistic reconstruction simulator. To put that in perspective: the camera-based student in this work trained on 150 million agent steps. At Gigapixel’s rate, one GPU renders all of them in under an hour; at the speed of the photorealistic simulator we compared against, the same rendering would take more than a month. Throughput also scales nearly linearly as GPUs are added.
Learn with RL where it’s cheap; distill where it’s expensive
Reinforcement learning needs billions of agent steps of simulated experience; the teacher in this work trained for 25 billion. That is affordable for a small policy that reads a list of objects, and far too costly for a large vision model that has to process four camera images at every step. So we split the job into two steps. Step one is RL: it trains a compact teacher that sees the scene as structured data, a list of objects and lanes, which is cheap to simulate. Step two is the distillation: instead of training the camera-based student with RL, we train it with supervised learning to match the teacher’s planned trajectories. The student drives every vehicle in the simulation, and wherever it ends up, the teacher shows it what it should do from there. This technique is known as DAgger; because we run it in self-play, we call it self-play DAgger (see panel (b) in the figure below). Because the student learns in the situations it actually creates, including its own mistakes, it learns to recover from them. It reaches the same level of driving in roughly 3,000 times fewer agent steps than training the camera model with RL directly.
Only the eyes need to adapt
A student trained on boxes still has to see real roads. The skill of driving lives in the part of the network that plans, so we leave that part alone and retrain only the part that turns images into an internal picture of the scene. Using pairs of real camera frames and renderings of the same moment, we teach it to see a real road the way it saw the simulated one. No human driving trajectories are used at any stage.
How the pieces fit together
The figure below, from the paper, shows the full training pipeline, from left to right.

- The teacher: Vectorized Self-Play RL. A compact network of about 2.7 million parameters, trained with reinforcement learning (PPO algorithm) in self-play for 25 billion agent steps across 335,000 driving scenarios from nuPlan. Its reward weights for things like collisions, comfort and progress are randomized for each vehicle and given to the policy as an input. One model therefore learns a whole range of driving styles, from cautious to assertive, which gives the student varied traffic to drive against.
- The student: Pixel-based Self-Play DAgger in Gigapixel. A transformer-based E2E planner (adapted from the DrivoR paper) that sees four rendered camera views and outputs a 4-second trajectory. It controls every vehicle in the scene. At each situation it reaches, the simulator is copied and the teacher drives on from that exact point for 4 seconds; that rollout becomes the student’s target. Early in training the teacher still takes some of the actions, and control passes fully to the student over time, which keeps self-play stable while the student is still learning.
- The adaptation: Sim-to-Real Perception Adaptation. The trained student is duplicated. One copy looks at the rendering of a scene, the other at the matching real camera frame, and only the real-image copy’s perception backbone is updated, so that it produces the same internal features and the same planned trajectory. The planning head is frozen throughout. Both choices matter: without the feature-matching term, the HUGSIM score drops from 33.2 to 18.5.
The full method, ablations and benchmark tables are on the project page and in the paper.
What we learned
Self-play changes what data you get, not just how much
This is where the approach answers the problem we started with. Human driving logs are thin exactly where it matters: the stalled car, the cut-in at speed, the merge that closes too fast. The obvious benefit of self-play reinforcement learning is volume: every vehicle in a rollout becomes a training example, so each simulated second yields many. The bigger benefit is content. Because every driver is learning, the scenarios shift as training goes on, and the student keeps running into interactions that a fixed dataset would show it once, if ever. Self-play turns the sparse long tail of the logs into a dense, steady supply of exactly the situations that improve driving behavior. That is why it beat single-agent training at every scale we tested and kept improving past the point where imitation stalled.
You can see the difference in behavior. When a lead vehicle slows to a stop, the self-play policy eases off and plans a smooth path around it; the imitation-trained policy keeps going straight and rear-ends it.
“Imitation learning is limited by the sparsity of useful long-tail driving scenarios, so it tends to plateau in performance. Self-play reinforcement learning greatly increases the density of these high-value scenarios to avoid this plateau, showing that it can enable much more robust, safe and performant planning that scales.”
Felix Heide
Head of AI, Torc Robotics
Professor at Princeton
A higher score isn’t the same as safer driving
We evaluated our approach on the HUGSIM benchmark, which reconstructs real driving scenes in a photorealistic simulator and lets a policy drive through them in closed loop, with surrounding traffic that becomes more adversarial across four tiers, from Easy to Extreme.
Overall, our self-play policy scored best among the methods evaluated in the paper, without using any human trajectories.* The exception was the Extreme tier, where surrounding vehicles actively try to cause collisions. There, the imitation-trained version of the same model scored higher.
Inspecting the runs showed why. The imitation-trained model tends to drive fast, which lets it outpace the aggressive traffic before a collision happens. Ours drove more cautiously, yielded to those vehicles and often got stuck, which lowers route completion and, with it, the score. Across the full evaluation set, the imitation-trained model’s average speed at collision was 2.7 times higher than ours.
Average speed at collision on HUGSIM (lower is better)
For a heavy truck, that trade-off isn’t close. At the same speed, a loaded truck carries far more energy into a collision than a passenger car, so a policy that sometimes waits behind aggressive traffic shows much safer behavior.
What benchmarks can and can’t tell you
Benchmarks like HUGSIM are valuable in a research setting. They let us compare methods on equal terms and check, at a high level, whether an idea holds up outside our own simulator. But as the example above shows, a single aggregate number can’t fully capture good driving behavior, and it can even reward the wrong behavior. That is why at Torc we evaluate how a policy drives from many perspectives and at different levels of depth, using a wide range of metrics and observers, both in simulation and in real-world truck tests.
The renderer stopped being the bottleneck
Once a full-size end-to-end model is in the training loop, most of the cost is the model itself, not drawing the images. That leads to the most practical lesson of this work: the best simulator for training isn’t necessarily the most realistic one; it is the one that keeps the model learning from as many useful situations as possible per GPU-hour. That is what self-play reinforcement learning in a simple box world gives us. It generates a much higher density of rare, useful scenes than any log can, and that keeps the model learning well past the point where imitation plateaus. What is learned in the box world doesn’t stay there, either: the self-play RL policy trained on structured inputs distills efficiently into an end-to-end policy that drives from camera pixels, and a light perception adaptation carries that behavior over to real camera images.
What a box world can’t teach
Boxes and lane strips can’t represent debris, unusual obstacles, weather or lighting, so the policy never practices those in simulation. The student also has to infer from pixels what the teacher was simply given, which isn’t always possible when something is hidden. And the adaptation step needs paired real and rendered views. These limitations also show where the research goes from here: adding human driving data back into training so policies stay human-like, generating rarer and more informative starting scenarios, and eventually training camera-based policies with RL directly, without a teacher.
Why this matters for Torc Robotics
Highway trucking raises the stakes on everything above. Popular urban datasets concentrate on objects within 100 m and speeds under 50 km/h; long-range highway data such as the TruckDrive dataset extends labels to 400 m, speeds above 100 km/h, and trajectories to 800 m. At that range and speed, the interactions that matter are the rarest in any log, which is exactly where generated experience should pay off most. Trucks also move differently from cars: this work was evaluated on passenger-car benchmarks, and adapting it to heavy, articulated vehicles is part of the work ahead.
This research also connects directly to what we run today. The teacher in this paper is the same kind of structured self-play reinforcement learning policy we use in our production work, and end-to-end RL is where our research is heading. The paper shows one path between the two: keep the robustness that self-play builds, and carry it into models that drive directly from cameras. We’ll share more about RL on our trucks in an upcoming post.
Ongoing collaboration with Mila, NYU and partners
None of this happens alone. The work is a collaboration between Torc, Mila, NYU’s EMERGE Lab and partner universities, and it builds on open-source tools: Gigapixel extends PufferDrive, and its code is coming soon. If these are the problems you want to work on, we’re hiring.

Meet the Torc Robotics RL team at CoRL 2026
Poster Session 3 | Poster 207
Wednesday, November 11, 2026 • 10:30 AM–12:00 PM • JW Marriott
We’ll be presenting Scaling Self-Play for End-to-End Driving as a poster at the Conference on Robot Learning, and members of our RL team will be there to meet. Come find us to talk about:
- Self-play reinforcement learning and closed-loop training for driving, from structured policies to end-to-end models
- How we take RL from research into engineering for autonomous trucks
- Open roles on our RL and ML behaviors, End-2-End, Simulation and ML infrastructure teams
Go deeper
Paper: Scaling Self-Play for End-to-End Driving (CoRL 2026, Austin, TX)
Project page, with videos, full results and method details
Code (coming soon)
Related Research



















To enable SAE Level 4 autonomous driving, the company has purposefully designed and built redundancy into the Freightliner Cascadia platform for safety-critical systems for safe, driverless operations. With over 1,500 engineering requirements, all translated into features, and a second set of electronically controlled systems like an integrated power network, the autonomous-ready Cascadia sets an industry standard for autonomous systems integration. – Daimler Truck press release April 2025
After 20 years in the robotics business, and six years of software research and development, Torc officially opened the company’s first commercial hub in May 2025, getting our trucks officially on the I-35 corridor. Our fleet has moved from our testing grounds in Albuquerque to focus on the Laredo to Fort Worth route, a crucial freight lane for many large freight customers. We’re already working in concert with them as we continue to look for more opportunities for the latter half of 2026.
There are still technology refinements and milestones to hit for our virtual driver and AI development as well, and we’ll be doing that with our new technology center in Ann Arbor, Michigan. Situated close to closed course test tracks and surrounded by top-notch robotics university programs, we continue to onboard the Torc’rs needed to get to our market entry timeframe.