Prediction quality changes when feedback enters the test
A predictive model can trace a plausible robot trajectory and still be the wrong choice for controlling that robot. The reason is feedback. A working robot repeatedly estimates its position, acts, receives new measurements and corrects its estimate. An offline test that asks the model to predict for a long interval without those corrections measures a different problem.
A preprint submitted on September 2 isolates that difference in a simulated mobile-robot task. Dharini Raghavan and Amritpal Singh compared six state estimators across 24 sensing conditions. A conventional replay metric agreed more closely with closed-loop tracking performance than a 20-step rollout that withheld landmark measurements. The measurement-free score selected an estimator other than the best closed-loop choice in 18 conditions. Replay position error did so in five.
This is not a benchmark of large visual world models, video generators or physical robots. The system represents a differential-drive vehicle with a three-value state: horizontal position, vertical position and heading. Its controlled simplicity is the point. By fixing the simulated dynamics and controller, the researchers could vary the estimator and measurement schedule without mixing in image quality, learned action generation or hardware variability.
Six estimators face four kinds of degraded sensing
The simulated robot followed a figure-eight path using a pure-pursuit controller. The controller received an estimated pose rather than the simulator’s ground truth. Wheel odometry supplied incremental movement, while range and bearing observations of known landmarks provided periodic absolute corrections. The experiments degraded sensing through four axes: landmark range noise, the number of visible landmarks, gyroscope bias and complete landmark-sensing outages.
The six primary estimators covered analytic and learned approaches. Dead reckoning propagated odometry without external correction. An extended Kalman filter, or EKF, combined a mathematical motion model with landmark observations. Four recurrent estimators used either a gated recurrent unit or a selective state-space model to predict corrections on top of dead reckoning or the EKF. Their names were GRU-DR, SSM-DR, GRU-EKF and SSM-EKF.
That separation matters when reading the results. The main ranking analysis contains six estimators multiplied by 24 sensing conditions, producing 144 estimator-condition pairs. Its replay, measurement-free rollout and closed-loop comparisons were aggregated over ten evaluation seeds per condition. Additional long-outage training variants were evaluated separately and should not be counted as part of the original six.
Three protocols measure different errors
The replay protocol gave every estimator the same previously recorded simulated trajectory, including its odometry and available landmark observations. An estimator could update itself whenever measurements appeared, but its errors could not change the future route because the trajectory was fixed. The authors measured position and heading root-mean-square error, which summarize the typical magnitude of estimation error.
The measurement-free protocol began from saved checkpoints along that trajectory. From each checkpoint, an estimator predicted 20 steps using odometry while receiving no new landmark correction. With a simulation interval of 0.1 seconds, this represented two seconds of uncorrected propagation. Position error was measured at the final horizon and aggregated across checkpoints and ten evaluation seeds.
Closed-loop testing changed the causal structure. Each estimator supplied the pose used by the controller, and the resulting command moved the simulated robot. An estimation error therefore changed the next action, trajectory and sensing geometry. Performance was measured primarily with cross-track RMSE, the distance between the robot’s resulting route and its reference path.
Those metrics have different denominators and meanings. Replay position RMSE and rollout position RMSE assess estimated pose, while closed-loop cross-track RMSE assesses path following. The study does not claim that their numerical values should match. It asks whether the ordering they assign to six estimators resembles the ordering produced by closed-loop control.
Replay preserved the ranking more often
Across the 144 estimator-condition pairs, replay position RMSE had a Spearman rank correlation of 0.923 with closed-loop cross-track RMSE. The 20-step measurement-free rollout reached 0.774. Spearman correlation measures how similarly two criteria order cases, not whether their errors have equal values.
Table 2 provides the more operational result. Replay position error failed to select the closed-loop-best estimator in five of 24 sensing conditions. Measurement-free rollout error selected a different estimator in 18. Its maximum selection regret, the additional closed-loop tracking error incurred by following the offline choice, was 0.1208 meters, compared with 0.0347 meters for replay position error. Mean regret was 0.0265 meters versus 0.0051.
Replay was not infallible, and not every replay metric worked equally well. Replay heading RMSE had a pooled rank correlation of 0.881 yet missed the top closed-loop estimator in 19 conditions. That contrast shows why a strong aggregate correlation does not guarantee reliable model selection for each operating condition. The paper’s condition-level analysis likewise found replay position error strongest among the three tested scores, without turning it into a universal metric.
The correction schedule explains the mismatch
A measurement-free rollout tests how rapidly an estimator drifts when external observations disappear. That is useful if the intended task genuinely requires long periods without correction. The simulated feedback system normally received intermittent landmark measurements, however, and some estimators used those updates more effectively than their uncorrected rollout would suggest.
The researchers held the rollout horizon at 20 steps and varied the interval between measurement updates. With measurements incorporated at every step, rank correlation with closed-loop performance was 0.916. With measurements withheld for the entire rollout, it fell to 0.774. Top-choice failures rose from five of 24 with updates every step to 18 of 24 without updates, reaching 19 at several intermediate intervals.
The result is not an argument against long-horizon prediction. It shows that horizon length cannot describe an evaluation by itself. A model tested for 20 or 80 steps while receiving regular corrections is being asked a different question from one propagated for the same duration without sensing. Planning systems that deliberately imagine long futures may still need long uncorrected tests. State estimators used inside measurement-rich feedback loops need tests that include those updates.
More outage training did not produce a universal gain
The study also trained additional versions of three recurrent estimators on longer sensing outages. The standard recurrent models used 200 simulated training trajectories. The extended-outage variants used 300, with half containing a 32-to-58-step outage instead of the standard six-to-27-step range.
Under one combined degradation condition, which joined high wheel slip, gyroscope bias, range noise, sparse landmarks and a long outage, the authors report that GRU-EKF cross-track RMSE fell from 1.717 to 1.061 meters after extended-outage training. SSM-EKF fell from 1.936 to 1.419 meters. The corresponding SSM model anchored to dead reckoning worsened slightly, from 5.290 to 5.411 meters.
The isolated outage sweep produced a different pattern. The extended-outage GRU-EKF improved mainly during the longest outages, while the extended-outage SSM-EKF performed worse than its standard counterpart at every tested outage duration. These are separate training comparisons, not additional estimators in the main six-model ranking. They reinforce a practical lesson: exposing a model to more difficult simulated conditions does not guarantee a better feedback system across architectures and sensing regimes.
A useful evaluation warning, still confined to simulation
The public repository contains simulator code, checkpoints, result files, unit tests, figure scripts and a script that checks bundled results against the paper’s headline numbers. That makes the numerical trail inspectable and lowers the cost of reproducing or challenging the study. It is not independent replication, because the artifacts and paper come from the same authors.
The study also leaves a substantial transfer gap. It uses one simulated differential-drive platform, one reference-path family and a pure-pursuit controller. It does not evaluate camera-based representations, learned policies, manipulation tasks, drones or physical closed-loop operation. Existing physical sensor logs mentioned by the authors replay a fixed historical trajectory and therefore cannot test the central feedback effect.
Even within that boundary, the work advances robot evaluation by identifying a controllable source of misleading rankings. Developers can vary measurement timing in simulation before committing to expensive hardware trials, then examine whether the preferred estimator remains preferred when sensing, correction and control are interleaved. The defensible conclusion is methodological: imagined rollouts become more informative when their information schedule matches the system they are meant to represent.