RoboRMBench Exposes Wording-Sensitive Reward Scores in Robot Learning

A useful robot must follow meaning, not preferred wording

Robots that learn from language need more than a broad ability to recognize objects and actions. They also need to treat two instructions with the same meaning as the same objective. A new preprint introduces RoboRMBench, a test designed around that deceptively simple requirement. Its authors report that several vision-language models changed their assessment of an unchanged robot trajectory when the instruction was reworded. In some cases, a model moved across the boundary between failure and success even though the visual evidence and intended task remained fixed. That result does not show that deployed robots are routinely making these mistakes. It identifies a measurable weakness in one proposed component of robot learning: using a model that sees video and reads language to supply a reward signal.

In reinforcement learning, a reward is the score that tells a learning system which behavior moved it closer to a goal. Engineers can write those scoring rules by hand, or people can label large numbers of attempts. A capable vision-language model offers another route. It can inspect frames from a robot trajectory, read an instruction and estimate progress on a scale. This could make it easier to specify many household, warehouse or laboratory tasks in ordinary language. The attraction is practical, but so is the risk. If wording changes the score while behavior does not, the learner may reinforce a linguistic accident instead of the physical result a person requested.

How RoboRMBench isolates the language variable

The team built RoboRMBench from the test split of RoboRewardBench, whose real-robot trajectories originated in Open X-Embodiment and RoboArena. According to the paper, the source material spans 14 robot embodiments, first-person and external camera views, varied manipulation tasks and human-verified progress scores. The researchers downsampled the overrepresented lowest score to produce a more balanced evaluation set. The finished benchmark contains 2,390 trajectories and 21,673 paraphrased instructions. These are records of physical robot activity, but the new experiment is an offline comparison of model judgments over recorded trajectories. It is not a trial in which robots learned new behavior autonomously in homes or factories.

For each original instruction, the researchers generated alternatives in three categories. Lexical substitution changes individual words. Syntactic restructuring changes sentence construction. An action-goal perspective shift recasts a sequence of actions as an intended end state, or does the reverse. That last category matters because people naturally describe the same task from different viewpoints. A person may ask a robot to pick up a radish and put it in a bowl, while another may describe the desired final arrangement. The benchmark asks whether the model preserves its judgment across those forms.

Meaning preservation is the hard part of such a test. A rewrite that subtly changes an object, direction, sequence or constraint should produce a different score. The authors screened candidates with three model validators and rejected a paraphrase if any validator detected a semantic difference. They then compared the filter with human annotations on 210 candidates. The ensemble caught 24 of the 25 candidates that at least one of two human annotators had flagged, while also rejecting 17 that the annotators did not flag. This supports a conservative filter, but it is not proof that every retained paraphrase is perfectly equivalent.

RoboRMBench separates consistency from accuracy. Its score-crossing rate asks how often one trajectory receives at least one failure-level score and one success-level score across equivalent instructions. Scores one and two count as failure, four and five as success, and the middle score is excluded. The flip rate instead compares each paraphrase with the original wording. Mean error measures how far the average prediction sits from the human progress label. These definitions matter because a model can be consistently wrong, or accurate on average while changing categories when phrasing changes. The authors also calculated an extreme version that counts only direct crossings between scores one and five.

What the model comparisons found

The reported spread across models is large. Under the action-goal perspective shifts, Gemini 2.5 Flash-Lite had a score-crossing rate of 0.607 and Llama 4 Scout had 0.557. In the authors' evaluation, each therefore crossed the failure-success boundary on more than half of the relevant trajectories. GPT-5.1 rose from 0.153 under lexical substitutions to 0.300 under perspective shifts. Results for dedicated RoboReward models were lower: RoboReward-4B ranged from 0.034 to 0.086 across the three rewrite categories, while RoboReward-8B ranged from 0.034 to 0.114. Those comparisons suggest that task-specific, trajectory-grounded supervision may matter more for this property than model size alone. They do not establish a universal ranking of the named models.

Average accuracy did not resolve the problem. The paper notes that Claude Sonnet 4.6 produced the lowest score-crossing rate among the general-purpose models for lexical substitutions, 0.051, while its mean error remained close to one point. Gemini 3 Flash had a lower mean error of 0.681 in that setting but a higher crossing rate of 0.271. A reliability assessment that reports only average error could therefore miss contradictory decisions. For robot developers, the useful lesson is methodological: measure whether a reward model responds to the outcome, and separately test whether equivalent language moves its decision.

The main evaluation used one model call for each trajectory-instruction pair and retained provider or model-family sampling configurations rather than imposing one decoding setup. Hosted services may introduce ordinary run-to-run variation. To estimate that effect, the researchers selected 300 pairs, queried each ten times with the same frames and instruction, and compared the resulting variation with variation across paraphrases. They report an average repeated-call standard deviation of 0.09 and an average across-paraphrase standard deviation of 0.52. This makes random sampling an incomplete explanation for the observed gap, but it does not turn every individual model response into a deterministic result.

A benchmark connected to an offline decision

The paper connects consistency to a downstream decision through an offline best-of-20 test. A reward model scored 20 candidate trajectories for a task, and the analysis compared the chosen trajectory with human progress labels. Across 11 models, score-crossing rate correlated with regret, the gap between the selected trajectory and the best available one, with Pearson r of 0.882 and Spearman rho of 0.856. Lower crossing rates were generally associated with better selections. This is useful evidence that the metric tracks something operationally relevant within the dataset. It remains a retrospective selection experiment, not evidence that a robot trained with the benchmark will perform safely or reliably in an open environment.

The authors tested two possible responses. At inference time, averaging predictions across additional equivalent phrasings reduced both crossings and prediction error in the evaluated cases, but required more model calls and did not eliminate crossings. In a separate training experiment, they added a penalty for variance across paraphrases to Qwen3-VL-4B. Its perspective-shift crossing rate fell from 0.200 to 0.057, while mean error in that category fell from 1.020 to 0.372. That single reported experiment is a promising mechanism, not independent confirmation or a general solution. Future work would need replication across architectures, tasks and robot-learning pipelines.

What the study does not establish

Several boundaries should travel with the headline. The work is a first-version preprint and has not been peer reviewed. All benchmark instructions are in English, so multilingual and code-switched commands remain untested. The study evaluates end-of-episode scores, not step-by-step reward shaping or direct trajectory comparisons. Its conclusions also depend on the selected prompt, the one-to-five scale, failure-success thresholds, paraphrase-generation process, semantic filter, recorded task distribution and model configurations available at the time. The source label distribution was deliberately rebalanced, which improves comparison across progress levels but changes it from the original distribution.

Within those limits, RoboRMBench demonstrates why a falsifiable reliability test can accelerate rather than obstruct useful robotics. It converts a vague expectation, that a robot should understand intent instead of preferred wording, into measurements that model builders can reproduce, challenge and improve. A sensible development gate would pair ordinary accuracy with paraphrase consistency, repeat tests under controlled decoding, extend them to more languages and then verify that gains survive closed-loop physical training. Passing this benchmark alone would never certify a robot. Failing it, however, would reveal a correctable weakness before an unstable reward signal is allowed to steer costly physical learning.