AR-WAM: Efficient Streaming World-Action Modeling for Autonomous Driving

Baiting Luo, Shatadal Mishra, Ahmadreza Moradipari, Nejib Ammar

Toyota Motor North America R&D - InfoTech Labs

Toyota Motor North America Toyota InfoTech

Abstract

Three panels. (a) Future frames predicted by AR-WAM on NAVSIM and nuScenes with one and with ten video passes, next to a plot of NAVHARD EPDMS against latency for AR-WAM, SimWAM and UNIVERSE. (b) A single-frame policy, a sliding-window policy that re-encodes its window, and the streaming policy that encodes only the new frame and reuses cached history. (c) Plans sampled for one scene, colored by reward, and NAVTEST EPDMS rising during refinement with Dr. GRPO.
AR-WAM plans from cached visual history and a single predictive pass. (a) Predicted futures and NAVHARD EPDMS against latency. One video pass scores above the variant without future conditioning (38.6 versus 37.3) and matches ten video passes (38.8) at under a quarter of their latency, while SimWAM and UNIVERSE score lower at higher latency. (b) AR-WAM encodes only the new frame and reuses its cached history, whereas a sliding-window policy re-encodes its window at every decision. (c) Reward refinement with Dr. GRPO raises NAVTEST EPDMS to 92.2; the scored samples show 32 plans for one scene, colored by reward.

Autonomous driving requires anticipating how traffic will evolve while responding quickly to new observations. Video foundation models offer the first ability, but repeated history encoding and iterative video generation make them costly to use for driving. We present AR-WAM, an efficient streaming world-action model that combines autoregressive temporal modeling with single-pass future conditioning. Because its video backbone is causal, AR-WAM updates its visual context incrementally, reusing past computation and conditioning future prediction on observed motion. We further hypothesize that future features computed from recent observations can improve generalization in dynamic traffic without iterative video generation. AR-WAM therefore obtains these features in one forward pass and uses them to condition trajectory generation, and reinforcement learning then improves the policy through direct optimization of driving rewards. With a single front camera, AR-WAM achieves 91.5 PDMS after supervised training and 94.0 after reinforcement learning on NAVSIM-v1, outperforming the compared strong end-to-end world-action models. In a dynamic setting of continuous driving, supervised AR-WAM keeps its visual history across decisions and outperforms strong world-action baselines.

Method Overview

AR-WAM architecture. (a) A causal video DiT runs one pass over the current frame and noised future slots while attending to the cached keys and values of earlier frames. (b) The trajectory expert reads the shared visual keys and values at matching backbone depths, together with image features, the ego state and the driving command. (c) The predicted trajectory in the camera view and in the bird's-eye view.

Cached visual history. A policy that re-encodes a window of recent frames repeats most of its visual computation at every decision. AR-WAM keeps its observation history in the causal form of its backbone, a pretrained causal video diffusion transformer. Because a frame never depends on later ones under causal attention, the attention keys and values (K/V) of earlier frames are computed once, cached and reused. A sliding window bounds the cache, and only the newest, incomplete chunk is recomputed when a frame arrives, so memory and per-decision work stay bounded however long the stream runs.

One predictive pass per decision. Generating future video by iterative denoising multiplies the backbone cost of each decision. AR-WAM instead runs one backbone pass over the current frame and future slots left at pure noise, attending to the cached history (a). Because the backbone is trained to predict future video, this pass yields prediction-trained features without generating video. The ego state and driving command change at every decision, so they enter only this pass and the trajectory expert, which keeps the cache valid as driving intent changes.

Layerwise trajectory expert. The trajectory expert has one block per backbone block. At each depth, its action tokens attend to the keys and values of the cached history and of the predictive pass, together with higher-resolution features of the current image and the ego state and driving command (b). This attention is decision-relative: only the expert's own positions are shifted, so it sees the same relative positions however long the stream has run, no cached key is modified, and the same weights apply at the start of a stream and deep into a long one. The expert generates the trajectory by flow matching and reuses the decision's visual K/V at every integration step.

Joint training and reward refinement. The backbone and the trajectory expert are trained jointly with two flow-matching losses, one on the trajectory and one on future video. Reinforcement learning then improves the policy through direct optimization of driving rewards. Because the visual pass does not depend on the trajectory, each scene needs one frozen pass, shared by all sampled plans and policy updates. Group-relative advantages are commonly divided by the standard deviation of the rewards in their group. With rule-based driving scores this is a poor fit, because nearly tied groups are the norm, so AR-WAM removes the division, as Dr. GRPO does, and trains on a mix of mined hard scenes and ordinary scenes with one shared reward scale.

Does Future Modeling Improve Decisions?

We compare the single predictive pass with two alternatives. In the no-future variant, the trajectory expert reads only the observed history and the current latent, although the backbone is still trained to predict future video, as in current-frame WAMs that drop future tokens at inference. In the clear-future variant, AR-WAM's own weights denoise the future video jointly with the action, so the last action steps read an almost clean predicted future at the cost of one backbone pass per action step.

FuturePDMS ↑EPDMS ↑L2 ↓PassesLatency
No future91.537.30.8710.8×
One pass (AR-WAM)91.538.60.8711.0×
Clear future91.538.80.88104.2×

Future modeling at inference. PDMS: NAVTEST; EPDMS: NAVHARD two-stage; L2: nuScenes zero-shot average in metres; Passes: backbone world passes per decision; Latency: median per decision relative to one pass, which takes 146 ms on an H200. Bold: best result in each column.

Test-time future prediction improves decisions at a small cost in latency. One predictive pass raises EPDMS on the perturbed two-stage scenes of NAVHARD by 1.3 over the no-future variant, with unchanged NAVTEST and nuScenes accuracy, while latency grows only from 0.8× to 1.0×. Denoising a clear future adds a further 0.2 EPDMS but takes more than four times as long, so one pass recovers most of its benefit at under a quarter of the latency.

Evaluation in a Dynamic Setting

Each standard NAVTEST query rebuilds the policy's history from four frames, so it cannot measure the effect of history carried across decisions. We therefore also evaluate a dynamic setting in which each policy runs through 138 full NAVTEST source logs in chronological order (69,711 frames at 2 Hz) and keeps its history across frames. Observations come from the logs and do not depend on the predicted actions. Plans are scored with one-stage EPDMS, including extended comfort (EC) between consecutive plans. Full logs keep the frames in which the vehicle is waiting, which NAVSIM largely leaves out of NAVTRAIN and NAVTEST, so to compare policies on the benchmark's scene distribution we score the 47,220 non-waiting frames.

MethodHist.Latency (ms)NCDACDDCTLCEPTTCLKHCECEPDMS
SimWAM1499.198.099.199.499.588.098.298.198.787.290.4
UNIVERSE5256.998.599.099.599.887.898.797.998.664.588.3
DriveVA5258.898.599.099.599.887.798.698.298.557.487.4
DriveWAM1895.798.099.199.499.488.398.198.398.791.890.9
AR-WAM, current frame1106.998.299.199.599.788.098.398.598.784.890.7
AR-WAM5146.398.799.199.599.688.098.898.898.788.091.5
AR-WAM + RL, EPDMS5132.898.799.699.599.588.698.998.098.789.992.2

Dynamic setting. The 47,220 non-waiting frames of 138 NAVTEST logs, run in order at 2 Hz with a persistent history cache. Columns are one-stage NAVSIM-v2 sub-scores. Hist.: camera frames used per decision. Latency: median per decision at batch size one on an H200. Current frame: AR-WAM retrained without visual history. Bold: best result in each column.

AR-WAM outscores all four baselines and runs faster. Its lead over SimWAM, which replans from the current frame, comes mainly from fewer at-fault collisions and time-to-collision violations, and visual memory accounts for most of it: the current-frame variant falls between the two, closer to SimWAM. This memory costs about 40 ms per decision. Refinement with the EPDMS reward raises both EPDMS and extended comfort further.

UNIVERSE re-encodes its five-frame window at every decision, whereas AR-WAM encodes only the newest frame and reuses its cache. With the same five frames, AR-WAM is 1.8 times faster than UNIVERSE and DriveVA in median latency, although it also computes predictive features. It is 3.4 times faster than SimWAM, which plans from the current frame with ten flow steps, and 6.1 times faster than DriveWAM.

Comparison on NAVTEST and NAVHARD

AR-WAM is trained on all 103,288 NAVTRAIN scenes with one front camera. We report NAVSIM-v1 PDMS and its components, NAVSIM-v2 EPDMS and the two-stage NAVHARD score on the 12,146 NAVTEST and 5,912 NAVHARD scenes. Direct methods output one trajectory per decision; proposal-scoring methods select one of up to 64 candidates with a scorer trained on PDM sub-scores.

NAVTEST, NAVSIM-v1NAVTESTNAVHARD
MethodRef.Cam.NCDACTTCComf.EPPDMSEPDMSEPDMS
Reference planners without a world model
Human driver––100.0100.0100.099.987.594.8––
DiffusionDriveCVPR'253+L98.296.294.7100.082.288.1––
ReCogDrive†ICLR'26197.997.394.9100.087.390.8––
DrivoR‡CVPR'26498.998.396.2100.089.193.1–48.3
World-action models, direct prediction, supervised
EponaICCV'25197.995.193.899.980.486.2––
DriveVLA-W0ICLR'261––––––86.124.4
PWMNeurIPS'25198.695.995.4100.081.888.1––
DriveLaWCVPR'26199.097.196.7100.081.389.188.630.6
DriveWAMNeurIPS'26198.398.195.2100.084.390.1––
SimWAM, Wan2.2arXiv198.698.095.8–84.090.390.237.6
DriveVAECCV'26199.097.398.4100.083.290.5––
GeoWAMarXivM––––––90.739.6
SUVarXiv199.197.896.7100.084.690.891.036.9
UNIVERSENeurIPS'26199.197.698.5100.083.691.0––
WA-JEPAarXiv499.598.397.7100.085.091.891.7–
World-action models, post-trained or proposal scoring
SimWAM, Wan2.2†arXiv198.599.196.199.886.591.9––
SimWAM, Cosmos2.5†arXiv198.899.296.4–87.092.4––
DA-WAM‡arXiv199.198.996.899.890.093.789.2–
AR-WAM199.498.298.0100.084.591.590.938.6
AR-WAM + RL, PDMS†199.099.396.1100.090.794.0––
AR-WAM + RL, EPDMS†199.399.397.1100.086.692.792.242.3

Comparison on NAVTEST and NAVHARD. Baseline values are publication-reported. Cam.: cameras (L: LiDAR; M: multiview). †: reward post-training. ‡: proposal scoring. Bold: best result in each column, not counting the human driver and ‡ rows.

Among the compared supervised direct methods, AR-WAM has the highest NAVSIM-v1 PDMS with a single camera, and only the four-camera WA-JEPA scores higher. Its NAVSIM-v2 EPDMS is within 0.1 of the best supervised single-camera result. With the PDMS reward, AR-WAM reaches 94.0 PDMS, above every direct method and above the proposal-scoring DA-WAM and DrivoR. With the EPDMS reward, EPDMS rises from 90.9 to 92.2 on NAVTEST and from 38.6 to 42.3 on NAVHARD, the best NAVSIM-v2 scores among direct methods on both.

Zero-shot Transfer to nuScenes

To test transfer beyond NAVSIM, we apply AR-WAM to nuScenes without fine-tuning, following DriveVA's evaluation protocol on 4,519 samples from 150 validation scenes. SimWAM, UNIVERSE, DriveVA and DriveWAM are evaluated on the same samples, together with a constant-velocity reference. The table also separates the waiting samples, in which the logged vehicle moves less than 1 m within 3 s. NAVSIM's training protocol removes most scenes in which the vehicle stands still, so the non-waiting samples correspond to the kind of scenes AR-WAM is trained on.

All samplesNon-waitingWaiting
L2Col.L2Col.L2Col.
Method1 s2 s3 sAvg.Avg.Avg.Avg.Avg.Avg.
Constant velocity0.331.132.341.270.471.460.540.150.06
SimWAM0.290.831.770.970.050.940.051.120.05
UNIVERSE0.300.751.440.830.070.910.080.370.01
DriveVA0.340.791.480.870.100.970.080.300.20
DriveWAM0.270.771.680.910.030.880.031.080.01
Adaptive-WAM0.350.711.580.880.08––––
AR-WAM0.240.731.630.870.050.880.040.820.14

NAVSIM-to-nuScenes transfer. Lower is better. L2 is in metres; Col. is the horizon-averaged collision percentage. The Adaptive-WAM row is publication-reported; the others are our evaluations on 4,519 samples. Of the 4,437 samples with all horizons, 635 are waiting. Bold: best result in each column.

On the non-waiting samples, AR-WAM and DriveWAM transfer best, with the lowest L2 error and about half the collision rate of UNIVERSE and DriveVA. Over all samples, including the 14% of waiting samples that its training largely excludes, AR-WAM stays close to UNIVERSE (0.87 versus 0.83 m average L2), matches DriveVA and has the lowest one-second error.

Citation

@article{luo2026arwam,
  title={AR-WAM: Efficient Streaming World-Action Modeling for Autonomous Driving},
  author={Luo, Baiting and Mishra, Shatadal and Moradipari, Ahmadreza and Ammar, Nejib},
  journal={arXiv preprint},
  year={2026}
}