News

METR says AI models outscored recruited specialists on three benchmark tasks

METR let participants use Sonnet 5 for implementation. In two comparisons with historical results, the strongest human performers generally scored higher, the group said.

D
Nov 21, 2026 · 2 min read

Model Evaluation and Threat Research (METR) reported on November 20 that the top AI model on each of three coding and optimization tasks outscored every specialist recruited for that task. The public models METR tested included OpenAI’s GPT-6 Astra and Anthropic’s Claude Opus 5.5. No independent replication of the three comparisons had been published as of November 21.

METR recorded seven attempts by recruited specialists across the three automatically scored tasks. Participants had the same task environment and resources as the models and could use Sonnet 5 for implementation. METR asked participants to make high-level decisions themselves and reviewed their progress periodically without setting a hard time limit.

METR released scores for all three comparisons. On Reverse Conway, which asks for a starting board that evolves into a target pattern in Conway’s Game of Life, the top model scored 0.208405 and the better of two recruits scored 0.137. On Minecraft Bot, where entrants build a bot that can beat Minecraft, the top model scored 0.175 and the sole recruit scored 0.055.

On Train a Program, where entrants train a neural network to reproduce another program’s behavior, the top model scored 0.44334 and the best of four recruits scored 0.042.

The top-scoring recruits logged 39 hours of total work on Reverse Conway, 160 hours on Minecraft Bot and 45 hours on Train a Program. Model runs had task-specific time and computing limits. METR said some task setups may have disadvantaged recruits and that it was uncertain how they compared with frontier AI researchers working in their own specialties.

In a separate analysis combining two benchmark suites, METR estimated 50% time horizons of 40 hours for GPT-6 Astra and 39 hours for Claude Opus 5.5, with wide confidence intervals for both. METR estimates how long a skilled human would take to reach each target score, then uses those times to estimate each model’s chance of reaching the same scores. METR also said models appeared to show weaker judgment than human experts on some tasks with slow or unclear feedback.

When METR compared the models with historical results from two task families, the strongest human performers generally scored higher. Across 15 past Al Zimmermann programming contests, about four people outscored METR’s strongest models in the median contest. Models had 48 hours per contest on a 28-core machine, while human entrants had about three months.

In Budget NanoGPT, METR reported a 15.7% training speedup over record No. 78 in a 48-hour model run on eight H100 GPUs. A human contributor using AI coding assistance improved a different starting record by 41%.

More news