Opus 5.5 leads RoboColiseum four-model test

RoboColiseum is a simulated benchmark for embodied models that scores vision-language-action models, world/action models and agent systems across four tracks: instruction following, spatial reasoning, general manipulation and robustness. On October 20, 2026, Da Huang, Jichao Wang and nine co-authors posted a cross-model evaluation of it to arXiv under cs.RO. The report builds Agent2Act, a control harness that gives a multimodal LLM observation, action and feedback interfaces, and runs GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5 and Gemini 4 Argon through the same samples with one fixed random seed. Opus finishes first of the four at 76.28, with Astra at 75.99, Sol at 70.91 and Argon at 66.86.

Key points

Four models now have scores from one harness: overall spans 66.86 to 76.28, instruction following 85.89 to 99.35, spatial reasoning 69.38 to 82.50, and general manipulation only 38.79 to 46.70.
Specialist policies still take the top two places on the combined table — Paimon at 0.7754 and HiDream-O1-Embodied at 0.7692 against Opus at 0.7628 — but a Table III footnote says external entries “retain their respective training and execution settings” and did not run through Agent2Act.
Image quality is the harshest of five perturbations: Sol’s instruction-following score falls from 0.96 to 0.18 (Δ0.77) while Opus goes from 0.97 to 0.76 (Δ0.20), both measured against each model’s own instruction-following baseline, not its overall score.

Background

Earlier evaluations of frontier models as robot policies mostly tested one model on one team’s own scaffolding. The three prior studies the report cites all appeared in September. RoboDojo ran all 42 of its tasks, with Astra and GPT-5.5 at the official 50 episodes per task and DeepSeek-Flash at 10; Astra’s 2,100 trials produced 22.48% average success and a score of 28.97, against 0.88% for GPT-5.5 and 1.92% for DeepSeek-Flash under the same post-processing. Galbot split the question into six domains and got 48% success for π0.5 hybrid control on a RoboDojo subset. RoboICL lifted Astra’s zero-shot progress by 20 to 27 points across 30 tasks for an overall 50.64. Those numbers come from different task sets under different harnesses and cannot be laid side by side.

This site covered the opposite question on October 18, when Manifold’s R²SI took the RoboDojo agent column by wrapping the same GPT-6 Astra in a different outer loop. Here the loop is fixed and the model changes. For context on one of the subjects, Anthropic shipped Claude Opus 5.5 on September 23 as a model for long-running coding agents and knowledge work, with general capability and API pricing and no robot-control evaluation at all.

The facts

The protocol is stated tightly. Every model runs once per case; within a track all four see identical samples, the same fixed random seed and the same Agent2Act harness. A task score is the mean of its substep scores, a track score is the mean of its task scores, and tasks are weighted equally rather than by case count. All four tracks use the benchmark’s original instructions with no supplementary task prompts, and the general robot-control instructions and interface descriptions are identical across models.

Table I, in percent, ordered by overall:

Model	Overall	Instruction	Spatial	Manipulation	Robustness
Claude Opus 5.5	76.28	96.60	82.50	46.70	79.30
GPT-6 Astra	75.99	99.35	78.33	45.86	80.42
GPT-6.1 Sol	70.91	95.50	77.71	40.01	70.40
Gemini 4 Argon	66.86	85.89	69.38	38.79	73.36

Table III places the four on RoboColiseum’s official overall leaderboard beside policy models, on a 0–1 scale:

Rank	Model	Overall	Instruction	Spatial	Manipulation	Robustness
1	Paimon	0.7754	0.8961	0.6791	0.6613	0.8653
2	HiDream-O1-Embodied	0.7692	0.8913	0.7437	0.6442	0.7978
3	Claude Opus 5.5	0.7628	0.9660	0.8250	0.4670	0.7930
4	GPT-6 Astra	0.7599	0.9935	0.7833	0.4586	0.8042
5	GPT-6.1 Sol	0.7091	0.9550	0.7771	0.4001	0.7040
6	DM0.5	0.7076	0.8444	0.6146	0.6370	0.7344
7	Gemini 4 Argon	0.6686	0.8589	0.6938	0.3879	0.7336
8	GigaBrain-0.7	0.6504	0.8166	0.4729	0.6320	0.6800

The task count has to be added up from the three tables in §III-C: Table IV lists 10 instruction-following tasks, Table V 8 spatial-reasoning tasks and Table VI 10 manipulation tasks, 28 in all, with the five perturbation types in Table VII on top. The “78 high-fidelity tasks” that circulates in Chinese coverage does not appear in the paper’s text or tables; 78 shows up only inside decimals such as 78.33 and 0.78. Twenty-eight tasks plus five perturbations is the only figure you can count.

Cost is reported as relative ordering with no absolute numbers. Opus has the highest overall score among the four and the lowest total token use; Astra and Sol are close, with Sol slightly higher; Argon consumes the most. At reference API prices the report estimates Sol reaches a score close to Astra’s at substantially lower inference cost, and Argon’s estimated cost lands slightly above Sol’s.

The capability profile

The gap between tracks is the report’s steadiest finding. All four models score above 85.89 on instruction following and above 69 on spatial reasoning, and all four fall below 46.70 on general manipulation. The report’s own phrasing is that strong zero-shot instruction following “has not yet translated into reliable complex physical manipulation.”

Task by task, the weak spots cluster where continuous contact adjustment is required. In manipulation, clean desktop runs 0.05 to 0.20 across the four, continuous sorting 0.00 to 0.07, pour workpiece 0.10 to 0.24, and pot holding splits 0.60 for Opus, 0.41 for Astra, 0.23 for Sol and 0.00 for Argon. In spatial reasoning, bowl stacking is 0.00 for every model because repeated failed grasps at the bowl rim exhaust the step budget. In instruction following, most of the spread comes from straighten object, which requires picking the object up and rotating the wrist: Argon scores 0.60 there against 0.83 to 0.98 for the others.

Case studies cover Astra only. It coordinates base, torso and arm to dispose of trash, retrieve a cup from a cabinet, open a door and push a chair; it stores chess pieces on all four embodiments (ARX, AgileX, G1, G2); it walks to a kitchen and brings a can of Sprite to a dining table. The failures are equally specific: dexterous-hand bowl stacking still fails after finger and palm geometry are added, phone charging and Ethernet insertion both grasp the object without aligning it, and a first missed coin placement is never recovered. The real-robot work connects Astra to an ARX Lift2 with three RealSense D405 cameras and judges outcomes by human observation: with depth, towel and garment folding in half succeed, while multi-stage garment folding does not.

Robustness

The robustness table is the easiest one to misread. Table VII compares each model against its own instruction-following baseline (0.99 Astra, 0.96 Sol, 0.97 Opus, 0.86 Argon), and the perturbed set is derived from the instruction-following test set with one dimension changed at a time, so a 0.18 is not an overall score collapsing to 18.

Image quality, which includes random frame loss across the head and two wrist cameras, lens occlusion and several noise types, does the most damage: Sol drops from 0.96 to 0.18 (Δ0.77), Astra from 0.99 to 0.68 (Δ0.32), Argon from 0.86 to 0.60 (Δ0.26) and Opus from 0.97 to 0.76 (Δ0.20, the best score under this condition). The other four perturbations are milder: background and lighting costs 0.08 to 0.16, camera position 0.10 to 0.20, instruction phrasing 0.09 to 0.18 and initial pose 0.06 to 0.20. Opus varies least across the five, and Argon’s overall robustness of 0.7336 beats Sol’s 0.7040 despite the lower baseline.

Harness ablations

The report ablates seven harness components using Astra alone: six on the 10 instruction-following tasks and one on four block-in-box layouts. The full system succeeds on 9 of 10, averaging 16.7 planning decisions, 32.50 minutes and 118.8 issued steps.

Removing execution-deviation feedback drops success to 8 of 10, raises decisions to 35.6 and pushes mean time to 58.26 minutes (+79.3%). Removing gripper geometry and contact guidance is the worst of the six: success falls to 7 of 10, decisions rise to 51.6 (+209.0%), time to 68.26 minutes (+110.0%). Softening IK validation instead lifts success to 10 of 10 at 46.9 decisions (+180.8%). Switching to direct joint control pushes issued steps to 254.3 (+114.1%) while cutting mean time to 19.62 minutes (−39.6%). Removing history takes the four layouts from 100% success to 50%, decisions from 27.5 to 66.5 and time from 64.65 to 131.81 minutes (+103.9%); both failures pick the block up and then repeat the same placement action.

Part of any model’s score, in other words, belongs to the harness. The report’s conclusion says the same thing about deployment: inference cost and response latency remain the main barriers.

Caveats in the comparison

Three problems weaken the cross-model framing.

Argon ran at medium reasoning effort. The report says Astra, Sol and Opus used high, and that Argon used medium because responses through the API channel used here were too slow at high for large-scale batch evaluation, then adds that “Comparisons involving Argon therefore reflect differences in both the model and its reasoning setting.” The 9.42-point gap to Opus mixes model and setting.

The external policy entries did not use Agent2Act. Paimon, HiDream-O1-Embodied, DM0.5 and GigaBrain-0.7 come from the official leaderboard, and the footnote says external entries “retain their respective training and execution settings”; the light-blue rows mark only the four multimodal models evaluated through Agent2Act. The manipulation gap — 0.3879 to 0.4670 for the LLMs against 0.6320 to 0.6613 for the policies — is what the table reports, but one side ran through Agent2Act and the other kept its own settings, so it is a shared table rather than a shared exam. What observation and action interfaces the external entries used, the report does not say.

The report evaluates the benchmark owner’s own leaderboard, and the four model results were produced by the report’s authors; the paper mentions no independent reproduction.

Chinese coverage adds two claims that the paper does not support. A Xinzhiyuan article republished by Wallstreetcn on October 20 advertises “78 high-fidelity tasks” and a simulation-to-real correlation of 89.5% with differences under 10%; neither figure can be found in the paper’s text or tables. That article does report the four image-quality drops correctly, at 0.20, 0.26, 0.32 and 0.77, but omits that the baseline is instruction following.

Our take

For teams building embodied systems, the useful reading is per track rather than per total. As a high-level planner, Opus and Astra are effectively tied: 0.29 points of overall score, from single runs, does not establish which is better. As a policy that emits actions directly, all four general models sit below 0.47 on manipulation while the policies on the same table run 0.6320 to 0.6613 — the weakest policy, GigaBrain-0.7 at 0.6320, beats the best general model, Opus at 0.4670, by 0.165, and the whole spread across the four general models on that track is 0.0791. The practical shape today is a frontier LLM as the decision and correction layer, with a learned policy doing the execution.

Three honest limits. Each model runs once with no reported variance, so the 0.29-point Opus–Astra separation cannot be called significant. Argon’s downgraded setting, the external policies’ retained settings and the self-run leaderboard all cut against the “unified comparison” pitch. And the ablations cover Astra only; another model may not gain the same amount from the same harness components.

What is missing is equally plain: no absolute token counts or costs, only an ordering; no way to check the 78-task figure; and real-robot results for one model judged by eye, with no simulation-to-real correlation number behind them.

What to watch

RoboColiseum runs as a standing leaderboard, and the report closes by saying it will keep adding multimodal models and agent systems while improving the evaluation in the same loop. The checkable next step is the leaderboard itself: new submissions will move Table III, but only a second model put through the same ablations will show whether the harness findings generalize. For the other side of the same gap, this site’s note on OpenAI’s 19 robotics job listings covers a lab hiring for actuators and manufacturing while this report measures how far models still are from execution.