SkeLAM trains a robot world model from ~10 hours of data, then transfers human-hand motion zero-shot

SkeLAM is a skeleton-grounded latent action model for robot world models, built by a six-person team from Harbin Institute of Technology (Shenzhen), NUS Show Lab and Ping An Technology. The paper went up on arXiv on October 22, 2026 (2610.31474, cs.RO). Starting from the official pretrained weights of the Cosmos-Predict2.5-2B video world model, it trains on roughly 10 hours of robot interaction data — the hours figure is from the project page — without the extra large-scale interaction pretraining its strongest baseline DreamDojo used, and takes first place on 7 of 8 DROID metrics. It can also turn a UniSkill human-hand demonstration video into robot-arm rollout frames without changing a single parameter. Jiqizhixin’s Chinese explainer on October 27 is currently the main writeup of the paper in either language.

Key points

• Against the strongest baseline DreamDojo, SkeLAM cuts end-effector trajectory error (ADE) from 51.11 to 41.15 pixels (−19.5%) and LPIPS from 0.1473 to 0.1310 (−11.1%), leading on all four visual-fidelity metrics.
• The one metric it loses, direction F1, is 0.5465 against Ctrl-World’s 0.5888 and OSCAR’s 0.5810 — numbers the project page lists itself.
• The zero-shot transfer rests on qualitative evidence: two UniSkill examples (pushing a towel down, pulling a tissue up). The project page states they “do not establish task completion or broad transfer performance,” and the paper’s own appendix reads: “The two examples assess visible motion correspondence, not task completion or reliable transfer across arbitrary viewpoints and embodiments.”

Background

Latent action models learn dynamics from unlabeled video: encode the visual change between two frames as a proxy action, and a world model predicts the next step without ground-truth robot action labels. The cost is entanglement. One latent has to carry both the robot’s own motion and scene responses — objects being nudged, cloth deforming — so the action conditioning drifts away from what a real controller sends. OSCAR is the prior skeleton-supervised entry on this line, and SkeLAM reuses its skeleton preprocessing.

Data efficiency has become the shared theme of this autumn’s robot-learning results. The H-JEPA paper co-authored by Yann LeCun went up on arXiv on October 5, also learning a world model from video, lifting Visual AntMaze success from 18% to 73%. The RoboColiseum four-model evaluation and R²SI topping the RoboDojo agent board measured how far general models sit from real execution. SkeLAM squeezes a different variable: how much robot data an action-conditioned world model actually needs.

How the method works

SkeLAM splits the latent action of every video transition into two 32-dimensional vectors: a kinematic component supervised by the robot skeleton, and a residual component that keeps the remaining visual dynamics. The skeleton signal follows OSCAR’s protocol — pre-rendered skeleton images, processed through the same kinematics, projection and rasterization pipeline.

Three mechanisms tie skeleton to pixels. Cross-modal swap reconstruction replaces the RGB kinematic latent with the posterior mean of the paired skeleton latent, keeps the residual, and demands that the same decoder reconstruct the same next frame; the skeleton branch stays frozen and gradients flow only through the residual. Branch dropout answers a deployment problem: residual latents are derived from future frames that do not exist at inference time. Training therefore drops whole branches at random — four combinations of keeping or dropping the kinematic and residual inputs — so the model learns to predict from the current frame plus kinematics alone, and the residual is zeroed at inference. Finally, a lightweight MLP adapter (7→256→256→32) maps 7-dimensional robot state deltas into the kinematic latent space: first 2K updates training only the adapter with the world model fixed, then 30K action-conditioned generation updates in which the paper fine-tunes the full world-model backbone (Table 5 lists Full DiT as trainable). The RGB encoder is what stays frozen throughout.

Results and where they stop

The main table runs 11 baselines on 400 DROID test clips (1 observed frame plus 12 predicted, at 480×640) and reports 8 metrics: PSNR, SSIM, LPIPS and MSE for visual fidelity, plus ADE, direction F1, magnitude L1 and magnitude correlation for motion. SkeLAM ranks first on 7. The main methods (full table on the project page):

Method	Control	PSNR↑	SSIM↑	LPIPS↓	ADE↓ (px)	Direction F1↑
TesserAct	Text reference	15.08	0.7400	0.2561	83.76	0.2950
IRASim	Action embedding	11.55	0.4137	0.5993	67.11	0.2616
Ctrl-World	Action embedding	16.34	0.5910	0.4029	42.96	0.5888
Cosmos3	Action embedding	16.98	0.7630	0.2186	67.23	0.3278
OSCAR	Skeleton	12.57	0.4471	0.4788	43.74	0.5810
DreamDojo†	Robot action	18.91	0.8057	0.1473	51.11	0.4989
CD-LAM†	Action to latent	18.34	0.7910	0.1586	53.90	0.4903
SkeLAM	Action to kinematic latent	19.35	0.8340	0.1310	41.15	0.5465

The dagger marks methods with large-scale interaction pretraining — DreamDojo and CD-LAM carry it, SkeLAM does not. So the headline comparison against DreamDojo is one where the winner used less robot data.

Direction F1 is the miss: 0.5465 against 0.5888 for Ctrl-World and 0.5810 for OSCAR. OSCAR reads skeletons the same way, which suggests the skeleton signal still has room for better uses.

Beyond world modeling, the paper tests SkeLAM as a scorer for robot policies: 7 policies, 15 RoboArena sessions, 105 episodes (9 real-world successes, 96 failures). Predicted success rate reaches 63.02% balanced accuracy with 33.33% success recall, and success-rate MAE drops from 7.62 to 4.76 percentage points against the Cosmos-Predict2.5 baseline. This section runs on recorded-action replay; the paper marks closed-loop policy evaluation as future work.

Zero-shot transfer across embodiments

The transfer experiments use human-hand videos from UniSkill. With all parameters frozen, the RGB encoder extracts latent action sequences from a human demonstration, bypasses the action adapter, and feeds them straight into the world model, which generates rollouts starting from the robot’s own initial observation. Two examples are shown: a hand pushing a towel downward produces an arm reaching down; a hand pulling a tissue upward produces an upward arm motion. DreamDojo, run for comparison, shows little end-effector displacement in both.

On these two examples, the paper says “these results provide qualitative evidence that human-derived latent actions can guide robot motion despite differences in embodiment and scene appearance,” and the appendix reads: “The two examples assess visible motion correspondence, not task completion or reliable transfer across arbitrary viewpoints and embodiments.” The project page puts it more bluntly: the examples “do not establish task completion or broad transfer performance.” There are no success rates and no tests from other viewpoints or on other embodiments; the “local gripper errors remain” caveat belongs to the world-model qualitative results (the Figure 3 discussion), and the transfer examples carry no such note.

Our take

For teams blocked on robot-data budgets, the persuasive part is the combination: ~10 hours of data, starting from the official Cosmos-Predict2.5-2B weights, still tops the visual-fidelity and trajectory-error rankings — with none of the extra large-scale interaction pretraining the daggered baselines used. Grounding the action latent in the robot’s own kinematics is a recipe worth borrowing for any action-conditioned world model — skeleton supervision and branch dropout are both described precisely enough to re-implement. One detail to keep straight: the backbone is not frozen during training — the pipeline runs latent-action learning, then adapter-only updates, then 30K full-backbone fine-tuning updates; the RGB encoder is what stays frozen. Three reasons to hold off: no code release (the project page links only the paper, videos and BibTeX); the zero-shot claim rests on two qualitative examples; and the “~10 hours” figure comes from the project page, while the paper counts clips (60,000 for stage one, 45,358 per stage after) without giving the conversion. Until the authors ship code, nobody can reproduce either number.

Two boundaries on the coverage. Jiqizhixin’s October 27 explainer introduces the work as coming from “Harbin Institute of Technology (Shenzhen), NUS Show Lab and other institutions,” and its author bios add that Yuxin Jiang is a PhD student at NUS and A*STAR; the two Ping An Technology authors — Jianzong Wang, and Jiayi Lin who is also affiliated with HIT (Shenzhen) — appear only in the paper and on the project page. In Chinese, Jiqizhixin’s piece is the main report, with Sohu and Sina Finance reposting it the same day; no independent English coverage exists that we could find. Every number in this entry has been checked against the paper’s HTML version, the project-page table and the Jiqizhixin text.