Can a general-purpose language model, with no robotics training, perceive a scene, track a robot's state and issue actions that work in the physical world? On 9 July 2026 Anthropic's Frontier Red Team published a benchmark study of that question. The page is titled "How Claude performs on robotics tasks"; its heading reads "Claude plays robotics", and the suite is called Embody. The post is by Shmuel Berman, Michael Ilie, Jia Deng and Daniel Freeman (byline). Its central claim is that a model's robotics score depends "heavily on how they are connected to the robot", that is, on the control interface, and that newer models are improving quickly but unevenly.
The study matters for two reasons the post gives. Robots are a route by which a model's actions reach the physical world, which Anthropic tracks for safety. And the post argues that evaluations which test a model without the tools a deployed system would have will understate its capability.
The version of record is the anthropic.com post, read on 13 November 2026: a long web post (about 9,000 words of page text by this note's count, including a technical appendix) with 26 image charts that print most of their values. No paper or arXiv version was found. The appendix says the code "once released, will be in github.com/safety-research/embody"; that address returned "not found" on 13 November 2026. The post has no numbered sections or figures, so locations below are section headings and chart titles.
What the study does
Bodies, tasks and interfaces
The suite covers three areas (introduction and "Summary of findings"):
- Classic control: toy reinforcement-learning problems in the MuJoCo simulator, namely balancing an inverted pendulum, driving a hopper forward, and a new pinball-like task, TwinFlipper, in which the model works two flippers to keep a ball in the air. The authors added TwinFlipper because the classic tasks appear often in pretraining data; "none of the models have seen it before" ("Simple settings").
- Locomotion: a simulated Unitree G1 humanoid (29 degrees of freedom) and Unitree Go2 quadruped (12 degrees of freedom), asked to stand up from a collapsed pose, keep balance from an upright start, and, for the Go2, walk forward. A separate suite of eleven navigation tasks uses the Go2 through a pretrained gait policy (below). The Go2 is the robot used in Project Fetch: Phase two, which the post links.
- Manipulation: a fixed-base, 7-degree-of-freedom Franka Panda arm in kitchen-style scenes from the LIBERO benchmark, with tasks such as "put the plate on the stove" ("Low-level manipulation").
Each body was driven through one or more of four control interfaces, which the post orders by abstraction ("Simple settings"):
- Direct control: the model chooses low-level actions (torques, forces, or for the arm, seven-dimensional end-effector motions) at each step.
- Programmatic (code) control: the model writes a Python controller that maps observations to actions, then runs it.
- Policy control: the model issues high-level commands to a pretrained policy. For locomotion this is a joystick-style gait policy that takes velocity commands; for manipulation it is a vision-language-action (VLA) model, a pretrained network that maps camera images and an instruction to arm motions (MolmoAct in every experiment). The language model can accept, edit or replace each action the VLA proposes.
- Reinforcement-learning supervision: the model writes a reward function, policy network and training schedule, trains the policy with PPO on a GPU, and deploys it (appendix, "Reinforcement learning details"). Training sessions were capped at 1.5 hours for classic control and 4 hours for the G1 and Go2.
Pausing the simulator
For direct and code control of the classic tasks and legged robots, the simulator was paused while the model produced its next action, so the results are, in the authors' words, an approximate "upper bound" on direct-control performance ("Simple settings"). The post puts the gap at roughly 83 Hz needed for real-time legged control against about 0.2 to 0.4 Hz for current non-reasoning inference, "roughly two orders of magnitude" ("Low-level locomotion"). The arm scenes were not paused, since a fixed arm has no balance constraint (appendix, "Latency").
Models, prompts and trial counts
The appendix lists twelve models from five providers: Claude Opus 4, 4.1, 4.5, 4.6 and 4.7 and Claude Mythos Preview through the Claude Agent SDK (with its built-in tools disabled), and GPT-5.4, GPT-5.1, Gemini 3.1 Pro Preview, Gemini 2.5 Pro, Kimi K2.6 and Qwen 3.6+ through OpenRouter. Not every model appears in every chart: the headline composite shows eight, and the high-level locomotion chart adds Claude Sonnet 4, Sonnet 4.5, Sonnet 4.6 and Haiku 4.5, which the appendix table does not list. Grok 4.1 Fast appears only in a real-world vignette.
Reasoning settings differed by family: newer Claude models used adaptive reasoning, older Claude models fixed budgets, and other models "the high reasoning setting equivalent" (appendix). Prompts are described below under "The prompts and instructions used".
Most cells (one model on one setting) ran 35 trials; some Mythos Preview and Opus 4.7 reruns ran 50; LIBERO-40 runs ran 200 (40 tasks times 5 seeds); the LIBERO tool ablations ran 50 (10 tasks times 5 seeds); the three novel VLA tasks ran 36 (3 tasks times 12 seeds); and the high-level locomotion suite ran 100 per model and condition (appendix, "Trial counts"). Some charts show error bars (the locomotion composite gives bootstrap 95% intervals); others do not.
Scoring in simulation is automatic. The physical-robot runs on a real Go2 are reported as vignettes, with no trial counts: "we could not achieve high N count trials due to the serial nature of work in the real world" ("Real-world vignettes").
What it finds
The headline composite
The opening chart, "Per-model embodiment score by interface", gives a composite in which each of the four interfaces contributes up to 0.25 (a perfect total is 1.0), averaged over every body and task except high-level locomotion. Values printed on the chart: Claude Opus 4 0.115, Opus 4.1 0.137, Opus 4.5 0.302, Opus 4.6 0.351, Opus 4.7 0.332, Mythos Preview 0.389, GPT-5.4 0.251 and Gemini 3.1 0.209. The best model reaches under 40% of the maximum, and Opus 4.7 scores slightly below Opus 4.6. The chart does not show intervals.
Classic control
Across the classic tasks the post reports "consistent generational improvement": Opus 4.5 and 4.6 beat Opus 4 and 4.1 on almost every task except direct-control TwinFlipper, where all models do poorly, and hopper velocity, "our noisiest task" ("Simple settings"). The authors attribute most of the gain to adapting after seeing earlier outcomes: average first-try scores are similar across models, with Opus 4 and 4.1 "very slightly" ahead on first tries, and later models pull ahead on retries (chart "Generational gains on classic control come from retries, not first attempts"). Mythos Preview is the exception, with more robust first attempts on Pendulum. When models trained their own RL policy, nearly all did worse than with a hand-written controller; GPT-5.4 was the only model that consistently learned a competent TwinFlipper policy, despite weak results on the other interfaces.
Low-level locomotion
Direct control of the Go2 was hard for every model. Under code control, Opus 4.6, Opus 4.7 and Mythos Preview balanced the Go2 for "nearly two full seconds"; Gemini 3.1 and GPT-5.4 wrote similarly strong controllers but lagged far behind under direct control ("Low-level locomotion"). No model stood the G1 humanoid up from a collapsed pose even once, though balancing an already-standing G1 improved from Opus 4 to Opus 4.7. Given four hours to train locomotion policies, GPT-5.4 and Mythos Preview trained the most competent ones. The authors add their own caveats: if the Go2's starting position is randomized to include lying on its back, Opus 4.6 "is unable to stand it up even once", and environments were reset between balancing attempts, with only a few models balancing robustly on the first attempt.
Low-level manipulation
Under direct control on the 40-task LIBERO set (200 trials per model), success rates were 5.5% for Mythos Preview, 3.5% for Opus 4.6, 1.5% for Opus 4.7, 1.0% for Gemini 3.1, 0.5% for Opus 4.5 and Kimi K2.6, and 0% for the other six models (chart "Direct LLM Control on 40-Task LIBERO", which prints the values with error bars). Newer models reached, touched and grasped the target more often before failing (chart "Per-subgoal progress on LIBERO under direct control"). Mythos Preview touched and grasped less than Opus 4.6 but completed more tasks, which the post attributes to Opus 4.6 making "more mistakes and adjustments". The authors conclude that full task success "is still rare".
High-level locomotion with a pretrained gait
With a pretrained gait policy, the Go2 suite has eleven tasks, from walking to a table marked with a blue X to mazes, silently corrupted commands (drift_detection) and planning a whole route from a map with no camera (oneshot_course) ("High-level locomotion"). Composite scores (0 to 100, at least 100 trials per task, bootstrap 95% intervals) range from 11 (Haiku 4.5, 20k thinking budget) to 54 (Mythos Preview, adaptive-max reasoning) (chart "High-Level Locomotion: Go2 + Pretrained Gait Policy"). The post sees two jumps, from Opus 4.1 to Opus 4.5 and from Opus 4.7 to Mythos Preview, with Opus 4.5 to 4.7 on a plateau. It says the plateau hides task-level shifts: comparing each model at its best setting, Opus 4.7 dropped on invisible_walls (3% against Opus 4.6's 15%) and gained 24 points on turn_correction and 11 on return_home. The authors' takeaway is that models complete simple navigation but "reliably fail" tasks needing sustained spatial bookkeeping or open-loop plans.
High-level manipulation with a VLA
On LIBERO-40, MolmoAct alone succeeded 86% of the time (±2.5 points). Supervised by a language model, success was 76% with Opus 4.6, 58% with Opus 4.5, 51% with Opus 4.7, 34% with Mythos Preview, 30% with Kimi K2.6, 25% with GPT-5.1, 12% with Opus 4.1, 11% with GPT-5.4, 7% with Opus 4, 6% with Qwen 3.6+, 4% with Gemini 3.1 and 0% with Gemini 2.5 (chart "VLA+LLM on 40-task LIBERO"). Every model did worse than the VLA alone. Mythos Preview, the strongest model in most other tests, scored below Opus 4.5 and 4.6 here because it "overrides the VLA more often than is warranted" ("High-level manipulation"). The post measures a follow rate, the share of steps where the model passes on the VLA's full seven-dimensional action unchanged; Claude models followed the VLA more than GPT-5.4 and Gemini 3.1, and Opus 4.5 and 4.6 followed most often on LIBERO-40.
To separate deference from judgment, the team made three new LIBERO-like tasks on which MolmoAct alone succeeds in none of its baseline trials (36 trials per model). Here Opus 4.5, 4.6 and 4.7 deferred to the VLA much less, while Opus 4 and 4.1 followed it at about the rate they did when it was competent; the post calls their behavior "largely indiscriminate". Success on the novel tasks was 8.3% for Mythos Preview, 2.8% for Opus 4.7 and zero for the other six models charted; grasp rates ranged from 2.8% (Opus 4.1) to 42% (Opus 4.6) (chart "LLM+MolmoAct: all 10 LIBERO-goal tasks vs 3 novel VLA tasks"). The text says Opus 4.5, Opus 4.6 and Gemini 3.1 "outperform MolmoAct alone" on these tasks. Their charted success rate is zero, the same as MolmoAct's. The post does not name the metric; this note infers that the comparison rests on progress or grasp rate, and no chart shows MolmoAct's own progress or grasp rate on the novel tasks. The opening summary's statement that "on tasks the VLA cannot do alone the strongest models already provide net uplift" rests on these 36-trial cells.
What limits performance
The post tests several explanations ("What's the bottleneck?"):
- Perception aids. Depth maps, segmentation overlays, a center crosshair and a depth heatmap were roughly neutral, the heatmap mildly hurting stronger models. A third-person chase camera was model-dependent: -3.6 points for Opus 4.6, +5.8 for Opus 4.7 and +10.7 for Mythos Preview on the locomotion composite. Two aids that give orientation helped every model: a movable cursor on the gripper camera that reports the object and distance at a point (Mythos Preview's success on a 10-task subset rose from 6% to 32%, chart "Vision-tool ablation"), and a compass giving the robot's heading in degrees as text. The authors read this as models needing "better orientation, not a different view of the scene".
- Images against text. Replacing images with scene descriptions written by Gemini 3.1 helped older Claude models and, in the text's words, made Opus 4.6 and 4.7 "perform slightly worse", which the post takes to mean that newer models get spatial information from pixels that is lost in text. The chart's printed values (average best progress on the 10-task manipulation subset, chart "Can talking with a visualizer compensate for vision?") show larger drops than "slightly" suggests for Opus 4.6: 35% with images against 21% with descriptions; Opus 4.7 fell from 22% to 16%, GPT-5.4 from 17% to 6% and Gemini 3.1 from 20% to 15%, while Opus 4, 4.1 and 4.5 went from 18%, 17% and 19% to 18%, 19% and 23%.
- Reasoning budget. Reasoning "had little effect on most of our results, with many differences falling within standard error". Newer models regressed on classic control with more reasoning; GPT-5.4 alone gained significantly on the code-control locomotion tests (chart "Effect of reasoning on locomotion"); on the high-level locomotion composite, Opus 4.6 stayed within 37.8 to 40.4 across the no-reasoning, 20k-budget and adaptive-max settings (its adaptive-low score, 34 on the chart, falls outside that band; the post calls adaptive-low "the one consistent loser"), while Mythos Preview spanned 40.2 (adaptive-low) to 54.1 (adaptive-max) ("Does reasoning help?").
- Learning from experience. Keeping only the first 10 turns plus the last 12 or 6 turns of a LIBERO episode changed little for most models; only Opus 4.6 showed a statistically significant drop (3.5% to 0% in both truncated conditions, chart "Context-Window Management: LIBERO-40 Success"), and Mythos Preview stayed between 5.0% and 6.0%. The authors conclude that adaptation is "mostly short-range". On the map-only course, a few practice runs helped on an easy L-shaped hallway, with Mythos Preview strong after one example; on a harder course neither Mythos Preview nor Opus 4.7 completed a trial even with twenty practice runs.
Real-robot vignettes
On a physical Go2, Opus 4.6 with the crosshair overlay corrected its heading in a hallway, but in one run judged a trash can to be out of the way because it was left of the crosshair, walked into it and dragged it "for a couple of meters" before the team stopped the robot ("Vignettes: Vision tools on the physical Go2"). In a real version of the blue-X task, models often stopped short of the one-meter goal, and Grok 4.1 Fast chased the table's reflection in a glass door until stopped. Every model failed an informal test of walking one loop of the office hallway by vision alone ("Real-world vignettes"). These are anecdotes without counts.
The prompts and instructions used
The study's text inputs are task prompts, not principles. Each interface in this robotics study used one fixed template, filled at trial time with task details such as joint count and force limits; the authors state, "We did not tune prompts separately for each model" (appendix, "Prompts"). The post does not print the templates. One aid, the compass, is text: the robot's heading in degrees appended to each frame (appendix, "Vision tools"). The post does not vary the wording of any instruction.
Limitations and open questions
The authors state these limits:
- Direct-control results come from a paused simulator and are a best case, not real-time performance ("Simple settings"; appendix, "Latency").
- More complex locomotion tasks and starting conditions "were generally beyond the reach of even frontier models" and were dropped; randomized starts defeat Opus 4.6 ("Low-level locomotion").
- Real-world trials were few ("Real-world vignettes").
- No formal latency study was run (appendix, "Latency").
Beyond the authors' list:
- The study is a developer's evaluation of its own models alongside others', designed, run and scored by the same team, with no outside replication. The code was not public on 13 November 2026.
- Reasoning settings were not matched across providers, and other providers' models were reached through OpenRouter with provider-specific workarounds (appendix), so cross-provider comparisons carry those differences.
- The composite weights each interface equally and leaves out high-level locomotion; the post gives no sensitivity check for other weightings.
- The headline composite prints no intervals, although most other charts do, and several claims of difference ("significant", "within standard error") are not tied to a stated test.
Research questions
- Do the rankings survive real-time control? All direct-control results come from a paused simulator. Running the same cells at real-time rates, or with a fixed inference budget per simulated second, would show how much of the ranking reflects speed rather than competence.
- Why do stronger models supervise a VLA worse on familiar tasks? Mythos Preview overrode MolmoAct too often. Logging each override and whether the VLA's original action would have succeeded would show whether the cost comes from a few systematic errors or many small edits.
- Is the net uplift on novel tasks robust? The claim rests on three tasks and 36 trials per model. More novel tasks, with MolmoAct's own progress and grasp rates reported alongside, would test it.
- What does the composite hide? Opus 4.7 scores below Opus 4.6 overall but gains on some tasks. Reporting per-task results with intervals, and the composite under different interface weights, would show whether the generational ordering is stable.
- Would orientation aids carry over to the physical robot? The compass and cursor helped every model in simulation. Repeating the hallway loop on the physical Go2 with a compass, over enough trials to count, would test the transfer.
- How sensitive are the results to the prompt template? One untuned template was used per interface. Comparing several templates on a few cells would show how much prompt wording moves the scores.
How it connects
- Project Fetch. The post links Project Fetch: Phase two (June 2026) for the Go2 robot, the quadruped of Project Fetch: Can Claude train a robot dog? (November 2025). Phase two measured Claude working alone on the Fetch tasks with a physical robot; this post moves to many bodies in simulation.
- Drones. Andon Labs' Drone-Bench paper, released with Project Pilot: Can AI models fly drones? (July 2026), cites this post as finding that "capability depends heavily on interface abstraction" and says it took that lesson into its design. The Project Pilot post itself does not link this one.
- Weapons and targeting. This note connects: Measuring AI capabilities in intelligence targeting and conventional weapons (September 2026), also from the Frontier Red Team, evaluates models writing guidance code for simulated drones.
References
- Berman, S., Ilie, M., Deng, J., & Freeman, D. (2026, July 9). How Claude performs on robotics tasks [page heading: Claude plays robotics]. Anthropic. https://www.anthropic.com/research/claude-plays-robotics record
- Ilie, M., Freeman, C. D., & Troy, K. K. (2026, June 18). Project Fetch: Phase two. Anthropic. https://www.anthropic.com/research/project-fetch-phase-two record · 3 pages
- Sharrock, C., et al. (2026, July 24). Drone-Bench: Tracking simple drone surveillance capabilities of frontier models. Andon Labs. https://andonlabs.com/docs/Drone_Bench.pdf record · 2 pages