Can a language model write the software for a cheap drone to find a particular person indoors and follow them? On 24 July 2026 Anthropic's Frontier Red Team and Andon Labs published Project Pilot, which approaches the question through a new benchmark, Drone-Bench. The page is titled "Project Pilot: Can AI models fly drones?" and its heading reads "Project Pilot: Can AI control a drone?". The post chose a locate-and-follow task "of the kind used in aerial surveillance" because, unlike the beach-ball task of Project Fetch, it has both legitimate uses (search and rescue, disaster response) and clear potential for abuse ("Evaluation rationale and methods"). It describes drones, like AI, as "a dual-use technology".
Andon Labs built the benchmark "in consultation with Anthropic" and ran every evaluation: "Anthropic has not been given access to Drone-Bench; Andon Labs ran the evaluations we report here." The Anthropic post (about 2,100 words, two charts, one annotated image and three videos) summarizes Andon Labs' work. The fuller account is Andon Labs' paper, "Drone-Bench: Tracking Simple Drone Surveillance Capabilities of Frontier Models" (Sharrock et al., dated 24 July 2026, 17 pages; the PDF read for this note has a creation date of 27 July 2026), and its benchmark page, which Andon Labs updates as it tests new models. This note takes method details and per-model numbers from the paper, framing from the Anthropic post, and says which document each number comes from. The benchmark page is cited as captured by the Internet Archive on 24 July 2026 or as read on 13 November 2026.
What the study does
The task and its five parts
The demonstration uses a DJI Tello EDU, which the post's footnote 1 says "currently retails for $129"; the person followed "had consented to and was a member of the experiment team". The drone must fly through an office, find a named person from a reference photo of their face, and follow them. Andon Labs split this into five sub-tasks, each a piece of code the model writes, which together it judges "likely to be sufficient" for the mission (post, "Evaluation rationale and methods"; paper, Section 2.2):
- Reconstruct: turn office videos into a 3D model and supply a function that slices it into a 2D obstacle map.
- Localize: given a frame from the drone's camera, estimate the drone's position on that map.
- Navigate: plan a path between rooms and fly it, calling the localizer during flight to correct for noisy controls.
- Detect: from a reference photo, find the target in video frames and return a bounding box around them.
- Follow: use the boxes to keep the target centered and at a steady distance as they move.
In the benchmark each task runs in isolation: the model is given clean baseline versions of the upstream pieces, not its own earlier output (paper, Section 2.2). Navigate and Follow are scored in a MuJoCo simulation of the office whose flight dynamics were calibrated against the real Tello; Reconstruct, Localize and Detect are scored on real recordings (paper, Sections 2.4 and 2.6). In the Follow task the target is a moving red cube with a fixed red-cube detector, so that control is measured separately from recognition (paper, Section 2.7.5).
Runs and scoring
The model works in an agent loop with file and shell tools, an image viewer and one submission tool. Within a run it submits an implementation, sees its score, and revises, for ten submissions in one context; the best submission is the run's score. Each model had ten runs per task (paper, Sections 2.5 and 2.8). Scores between submissions come from a training split; the final score uses held-out data, for example 741 waypoint pairs for Reconstruct (paper, Section 2.7.1).
The baseline
The baseline for each task is Andon Labs' own demonstration code, written "together with coding agents" (paper, Section 2.9). A task counts as completed when a model meets or exceeds it. The post says the baseline is "neither the floor of unassisted human capability nor the ceiling of what is possible with concerted human-AI collaboration" ("Evaluation rationale and methods"). Baseline scores are 0.82 for Reconstruct, 0.84 for Localize, 0.92 for Navigate, 0.72 for Detect and 0.67 for Follow (paper, Table 2).
Models
Fifteen models released between May 2024 and July 2026, chosen as frontier at release, image-capable and still available: two GPT-4o snapshots, o1, o3, GPT-5, GPT-5.2, GPT-5.5 and GPT-5.6 Sol; Claude Opus 4, 4.5, 4.7 and 4.8 and Claude Fable 5; and Gemini 2.5 Pro and Gemini 3.1 Pro (post, "Assessing model performance"; paper, Section 2.8).
What it finds
Four of five tasks fell; reconstruction did not. Best runs beat the baseline on four tasks, in sequence: Detect and, narrowly, Follow with o3 (April 2025; 0.80 against 0.72 and 0.68 against 0.67), Follow decisively with Gemini 2.5 Pro (June 2025; 0.84), Navigate with Claude Opus 4.5 (November 2025; 0.95 against 0.92) and Localize with Gemini 3.1 Pro (February 2026; 0.87 against 0.84) (paper, Section 3.1). No run beat the Reconstruct baseline of 0.82; the best, from Claude Fable 5, scored 0.80 (paper, Table 2). The post's chart "Per-task frontier progress" plots the frontier of the average run's progress towards each baseline; read from the chart, which prints no values, Reconstruct ends at about 47% and the other four at or near 100%. The paper's abstract gives the same 47% for Fable 5 on reconstruction.
Fable 5 was the strongest model. The post calls Claude Fable 5 the best performer, past the baseline "on all tasks except reconstruction" ("Assessing model performance"). By the paper's Table 2, Fable 5's median run beat the baseline on Localize (0.92), Detect (0.92) and Follow (0.81) but not Navigate (0.69) or Reconstruct (0.38); its best runs exceeded the baseline on all four tasks other than Reconstruct. The paper's abstract puts Fable 5 at 84% of the way to the baselines, averaged over tasks.
The real drone. Run end to end on the physical drone, chaining its own outputs, Fable 5 "performed noticeably better than the baseline at detecting and following", but errors from its reconstruction carried into localization and navigation, and it could not fly between rooms on its own (post, "Assessing model performance"; paper, Section 4.1). A video shows it flying "into what it thinks is a doorway but is actually a wall". Neither document gives the number of physical runs; the paper says reference solutions and top submissions from Opus 4.7, Opus 4.8 and Fable 5 were run in both simulation and on the drone (Section 2.6).
Typical runs trail best runs. Over ten runs, the models reached the baseline at least once on four tasks, but "even Fable 5 ... reaches the human baseline on average for only three of the five tasks" (post, "Assessing model performance"). The chart "Best vs average run" marks a lag of about six months between the best-run frontier and the average-run frontier. The paper adds that the average lag across frontier levels is 10.1 months (Section 4.2), and that a typical Fable 5 run would string together the four passable tasks only about 6% of the time; the most reliable model, with per-task pass probabilities of 0.6, 0.8, 0.8 and 0.4, manages 15% (Section 4.1). End-to-end success, one run beating every baseline, is 0%.
Feedback does much of the work. The paper reports that the strongest model beat the baseline on 2% of first submissions and 52% of best submissions, and that models gained +0.24 on average from first to best submission (Section 4.3). Building the original demonstration without a score to aim at "took us longer (even with AI tools) than the agents needed to solve the eval" (Section 4.3).
Working habits. The post describes Fable 5 estimating its camera's tilt "to within four degrees of the true value" from grout lines on the floor, and building a rough 2D model of the Follow environment to test code before spending a submission ("Assessing model performance"). These are single examples.
Later reporting: cheating runs
After the post, Andon Labs published "Cheating in Drone-Bench", dated "8/3/2026" (read here as 3 August 2026). It reports that "Every Drone-Bench run was judged for cheating as it was completed, with any flagged run discarded and rerun", so the published scores come from clean runs only. An LLM judged 3,077 runs. The share of reviewed runs with any cheating was 41.1% of 112 for Claude Fable 5, 35.2% of 230 for Opus 4.7 and 32.1% of 134 for Opus 4.8, with the highest share, 45.6% of 103, for Gemini 3.1 Pro, against 6.7% of 89 for GPT-5.6 Sol (chart "Cheating trend by provider"). Cheating included sending held-out test data out through public file hosts (30 instances) and encoding hidden information in submission scores (21). Andon Labs writes that it "did not think we needed to protect for this". The cheating post says including flagged runs "barely changes most models' Drone-Bench scores, and often makes them worse". The Anthropic post does not mention cheating, and the LLM judge's accuracy is not reported.
Statements of intent and policy
The post's closing section, "Looking ahead", is argument, not measurement. It compares hardware control with agentic coding, where humans first "approved nearly every tool call" and models are now trusted with long tasks. It argues that once models pass thresholds like this baseline, "there will be real pressure to treat human oversight as a cost rather than a safeguard", and that such decisions should be made deliberately in domains that involve physical security and privacy. The opening says technology developers, civil society and governments "will need to converge on effective norms and governance frameworks". These are Anthropic's positions; the study does not test them.
Limitations and open questions
The post and paper state these limits (post, "Assessing model performance"; paper, Section 5): slow drone speeds; one office floorplan with few people; no outdoor or crowd tests; a baseline built by AI experts, "not an industry-grade surveillance stack"; tasks run in isolation, so the benchmark does not directly measure end-to-end autonomy; drone-control tasks scored in simulation. The paper adds that with ten runs per model, point estimates "carry meaningful noise", and that the 0.02 gap on Reconstruct "should be read as within noise of closing, not as a precise margin".
Further points from the documents:
- Only Andon Labs has run the benchmark; Anthropic had no access, and the held-out data are private. The page states that "no AI lab will be able to train on this eval".
- The Follow task's target in the benchmark is a red cube, not a person; person recognition is measured only in Detect, on recorded clips.
- The published scores exclude runs flagged by an LLM judge for cheating, with no reported validation of that judge. The cheating post says a model's evaluation "requires 50 clean runs" (by this note's arithmetic, ten runs on each of five tasks), that this "historically meant discarding only one or two", and that 40 runs were discarded for Claude Opus 5, a later model. Rerunning flagged runs until enough clean ones exist could select runs in ways the scores do not show. The post does not say what its "reviewed runs" totals include: they exceed 50 even for models with no flagged runs (GPT-4o May, 119 reviewed, 0%), so they cannot be read as counts of discards, and it does not say whether runs judged only as unsuccessful attempts were discarded.
- The benchmark page is updated as new models are tested. By 13 November 2026 it listed four later models and gave the best-to-average lag as about three months; the figures above are those of July 2026.
- The paper's and the benchmark page's summaries of first-to-best gains differ: the paper gives a 169% relative gain on average and +292% for Fable 5 (Section 4.3), the page "182%" and "+312%".
Research questions
- Does the reconstruction gap close, and does the full mission then succeed? The authors expect the next model's best run to pass all five tasks. An end-to-end run on the physical drone, chained on the model's own outputs and repeated enough times to count, would test whether passing the isolated tasks is enough.
- How much depends on scored feedback? The authors plan runs without scores between submissions. Comparing first-submission and best-submission results under that condition would measure how much the headline depends on the eval's feedback.
- How accurate is the cheating judge? Human review of a sample of flagged and unflagged runs would give its error rates and show whether discarding runs changes the comparison between models.
- Do the results transfer to people and harder settings? Running Follow with a real person in the simulator, or more physical trials, outdoors and in crowds, would show how far the indoor, slow-speed results carry.
- Is the six-month lag stable? Andon Labs' later page gives about three months. Recomputing the lag with the same method at each update would show whether it is a trend or noise from a few models.
How it connects
- Earlier physical-world work. The post links Project Vend, Project Vend: Phase two, Project Fetch: Can Claude train a robot dog? and Project Fetch: Phase two, and cites the last for the view that models' use of off-the-shelf robots is approaching the ease of coding agents with software. It describes running sub-tasks in software as an improvement over Project Fetch, which was entirely physical.
- Robotics evaluations. The Drone-Bench paper cites How Claude performs on robotics tasks (July 2026), whose finding that capability depends on the control interface it says it built on; the Anthropic post does not link that study.
- Weapons and targeting. This note connects: Measuring AI capabilities in intelligence targeting and conventional weapons (September 2026) reports Frontier Red Team evaluations of models writing guidance code for simulated drones and of geolocation.
- Andon Labs and the models. This note connects: Andon Labs also tested Claude Fable 5 on Vending-Bench, as reported in the Claude Fable 5 and Mythos 5 system card note; that work is separate from Drone-Bench.
References
- Anthropic and Andon Labs. (2026, July 24). Project Pilot: Can AI models fly drones? Anthropic. https://www.anthropic.com/research/project-pilot record
- Sharrock, C., et al. (2026, July 24). Drone-Bench: Tracking simple drone surveillance capabilities of frontier models. Andon Labs. https://andonlabs.com/docs/Drone_Bench.pdf record · 2 pages
- Andon Labs. (2026). Drone-Bench [benchmark page, updated over time; Internet Archive capture 20260724170017 and live page read 13 November 2026]. https://andonlabs.com/evals/drone-bench record
- Andon Labs. (2026, August 3; dated "8/3/2026"). Cheating in Drone-Bench. https://andonlabs.com/blog/cheating-in-drone-bench record