Is Energy Guidance All You Need?
Training-Free Norm Injection for Driving World Models

Overview

Driving world models built on large video-diffusion backbones generate realistic scenes but are hard to control: enforcing a traffic norm usually means retraining the backbone or conditioning it on hand-built layouts. We ask whether controllability requires training at all.

Add docs/static/images/overview.png (paper Figure 1) to show the method overview here.

Abstract

A rectified-flow driving world model that jointly generates future video and a planned ego trajectory can have its planned trajectory steered entirely at sampling time by differentiable energy functions that encode driving norms, without knowledge-specific retraining of the diffusion backbone. We demonstrate that a world model built on an Open-Sora 2.0 MM-DiT backbone can be steered to brake at a counterfactual target by injecting energy guidance at sampling time. However, we find that the generated video does not yet follow the steered trajectory through the backbone's joint self-attention, and identify the cross-stream coupling as a crucial requirement for end-to-end-controllable rollouts.

The trajectory steers, the video does not

Add docs/static/images/fig_aggregate.png (paper Figure 3) here.

What we show

• Driving-norm compliance can be framed as training-free energy guidance on a frozen rectified-flow world model that jointly generates video and an ego plan.
• A single sampling-time energy steers the ego trajectory to a counterfactual braking target, simulating a rear-end-collision driving norm.
• Under the current architecture the steered trajectory does not propagate to the generated video; we trace this to the cross-stream coupling and propose deep joint denoising of the two streams as the next step.

@inproceedings{energyguidance2026,
  title     = {Is Energy Guidance All You Need? Training-Free Norm
               Injection for Driving World Models},
  author    = {Su, Xiyan and Diermeyer, Frank and Lienkamp, Markus},
  booktitle = {RSS 2026 Workshop on Robot World Models},
  year      = {2026}
}