Notes from Việt Nam on law, technology, languages, history, culture and travel.

Twelve Percent of the Job: SemiAnalysis's GDPx Sends Frontier Agents Into Eleven High-Value Domains

On Monday 2 November 2026, SemiAnalysis, the semiconductor and AI research firm run by Dylan Patel, published GDPx, a benchmark it describes as measuring “agentic capabilities across the most valuable domains”. The write-up is signed by Max Kan, Andrew Megalaa and Patel. Its headline line is blunt: “GDPx is a true frontier-level benchmark: even the best model scores just 12%.”

The same day, OpenAI published LifeVal, an evaluation of how well its models handle everyday errands such as enrolling a child in kindergarten or planning meals for a diabetic. On the sample tasks OpenAI shows, its latest models score above 90%. Two benchmarks released within hours of each other describe the same generation of models as nearly done and as barely started. Both are right. The difference is the task, and that is what makes GDPx worth reading closely.

What GDPx is

Version 1.0 has 206 tasks across 11 domains, built with 14 partner startups that sell reinforcement-learning environments and data to AI labs. The domains and their partners, as listed in the write-up:

  • Chip design: Aria, Arcverex, Ergodex
  • Cybersecurity: Arithmetic
  • Biology: Latch Bio
  • Finance: Living Context
  • Legal: Chakra
  • Quant trading: Third Moment
  • Recursive self-improvement (RSI): Extrapolate
  • Hardware design: Beautiful Machines, with Diode Computers on PCB tasks
  • Materials science: ValenceForge, Emulated
  • Alignment: Fleet
  • Software engineering: Datacurve

The domain list is a deliberate bet, not a census. SemiAnalysis writes that “it’s a mistake to build a benchmark for general agentic capabilities based on share of GDP or the SOC/O*NET occupational taxonomy because AI diffusion won’t be equally distributed and not all domains will need frontier intelligence.” So government, real estate and insurance are left out on purpose, and chip design, cybersecurity and biology are in because the authors expect those to be “most accelerated by AI in the next 1-2 years”.

What separates GDPx from a question-and-answer benchmark is that every task is an environment. The finance tasks are seeded by a real private-equity transaction and come with roughly 4,000 anonymised files; the model has to find what matters. The cybersecurity tasks are “based on real zero days discovered by Arithmetic’s security engineers”. Hardware tasks “include live sessions of Siemens NX and test the model output on real AWS FPGA instances”. Materials-science tasks put the model inside simulated laboratories. Sandboxes run on Modal, and Fireworks served inference for the open-weight models.

The harness rules matter for anyone trying to reproduce the numbers. Each model ran in its vendor’s own agent harness where one exists: Claude Code for Opus 5.5, Codex for GPT-6 Astra, Antigravity for Gemini 4 Carbon, and OpenCode for everything else. The default time limit is 8 hours per task. The headline figure is pass@1 averaged over three attempts, all 11 domains are weighted equally, and within a domain each partner is weighted equally. Grading is “a mix of binary and partial grading depending on the domain”; the hardware tasks, for instance, combine deterministic verifiers with an LLM judge scoring against expert-written rubrics.

The quality-control story is the part I found most interesting. SemiAnalysis says it approached more than 30 environment companies, and only kept partners whose tasks passed its automated QA (checking consistency, completeness, cheating and unfair verifiers) on the first try, because “any QA system can be gamed given sufficient iteration”. When it ran the same QA over Terminal Bench 4.0, a widely cited coding benchmark, it “flags nearly 50% of tasks as broken”. That claim is SemiAnalysis’s own and unaudited, but if it holds even approximately it says something uncomfortable about the leaderboards everyone quotes.

The leaderboard

The public table shows 13 of the 20 models evaluated. Score is the average across all 11 domains; cost is the average per task.

# Model GDPx Cost per task Output tokens
1 Claude Opus 5.5 12.0% $51.41 548K
2 Gemini 4 Carbon 10.9% $16.29 594K
3 GPT-6 Astra 9.6% $53.99 275K
4 Muse Wonder Full 9.0% no public price 212K
5 Claude Fable 5.1 8.5% $67.16 418K
6 GPT 6.1 Sol 8.4% $9.80 352K
7 Claude Sonnet 5.5 8.2% $45.27 967K
8 Gemini 4 Argon 7.3% $12.79 359K
9 Muse Wonder 6.8% $15.63 215K
10 DeepSeek 4.1 Flash 1017 5.4% $0.94 246K
11 GPT-6 Luna 5.1% $0.92 438K
12 Grok 4.7 3.9% $16.34 145K
13 GLM 5.3 2.7% $11.79 212K

A few things stand out. Gemini 4 Carbon, which this blog covered last week when it took first place on the Artificial Analysis index, is described here as “a real improvement from Google and the only model in the same ballpark as Opus and Astra”, and it gets there at about a third of Opus’s cost per task. The two models that opened September’s price war land mid-table. DeepSeek 4.1 Flash 1017 and GPT-6 Luna both come in under a dollar per task, which is roughly 50 times cheaper than Opus for a little under half the score. SemiAnalysis sums up the cost-versus-capability chart in one line: “Luna and DeepSeek fight for the low end, Opus wins the high end, and OpenAI claims the middle.”

The per-domain results are published as images, so I can only report them as the authors describe them. Opus 5.5 “wins in quant trading, hardware design, and chip design”. Gemini 4 Carbon “is the best bio model (likely due to weaker safety classifiers) and edges out the win in software engineering”. GPT-6 Astra “wins convincingly for RSI and is also the best model for materials science”. Meta’s Muse Wonder Full is “the best available cyber model today by a meaningful margin”. And then the sentence that should temper every “AI will replace analysts” pitch of 2026: “Our finance and legal tasks are so challenging that basically every model scores exactly 0.”

Safety classifiers scramble the ranking

The authors flag one confound themselves: “certain models like Opus 5.5 and GPT-6 Astra have sandbagged overall scores due to their safety classifiers.” In cybersecurity, biology and alignment, those models sometimes refuse rather than attempt, and a refusal is graded as a fail. SemiAnalysis therefore publishes a second chart with those three domains removed. That chart is also an image; OfficeChai reads it as GPT-6 Astra edging ahead of Opus 5.5 by a tenth of a point, with Gemini 4 Carbon dropping from second to fifth, and Emergent reports the same reversal at the top.

Read that alongside the note that Carbon is “the best bio model (likely due to weaker safety classifiers)”. Google gated Carbon to vetted cyber defenders precisely because of its offensive capability, yet on this benchmark its looser classifiers are a scoring advantage. The practical lesson is that for agents working in biology or security, the refusal policy is part of the product, and a leaderboard that cannot separate “could not” from “would not” is measuring policy as much as capability.

Why environments, and why now

GDPx is explicitly a response to OpenAI’s GDPval, released in September 2025. SemiAnalysis’s critique is worth quoting because it applies to most “economic value” benchmarks: “Every GDPval task is precisely specified and non-interactive,” with the necessary context “spoon-fed to the model in the prompt”; “human expert inter-rater agreement was just 71% on the gold subset and their automated grader was even worse”; and the lack of simulated environments is “the single biggest issue with GDPval”. In GDPx the model has to discover the context, use real tools and satisfy a verifier.

The 12% figure also lines up with another frontier-style result this blog covered in October, where Epoch AI’s InnovationEval gave agents 3,000 GPU-hours to reinvent a post-training method and the best in-scope run recovered 15% of the human gains. The emerging picture from independent evaluators is consistent: on long, open-ended, verifier-graded expert work, current agents complete a small fraction and fail most of the rest, while on well-specified everyday tasks they are close to ceiling. LifeVal and GDPx are two ends of the same curve.

For developers, the actionable findings are less about the ranking than about the columns next to it. The cost per task at an eight-hour budget is a number most teams have never measured for their own agents. The harness is part of the model: SemiAnalysis ran each lab’s own, which means the scores already include the lab’s scaffolding, retries and tool wiring. And capability is spiky by domain, so the question “which model is best” has a different answer in chip design, biology and RSI. Picking per workload, not per leaderboard, is the only reading of this table that holds up.

What to treat with caution

  • No human baseline. 12% is a score against task verifiers, not against what a domain expert would achieve in eight hours. We do not know whether the verifiers are calibrated to expert performance.
  • Small samples in places. RSI has 10 tasks. The hardware write-up describes 12 rollouts across four models with none achieving “a verified full pass”. Three attempts per task is thin for pass@1 averages, and I found no variance figures in the readable text.
  • Seven models are missing. The table says 13 of 20; the other seven are not named. The adjusted leaderboard exists only as an image, so the “Astra by 0.1 point” ordering rests on secondary readings.
  • The partners sell environments to labs. Every task set comes from a startup whose business is selling exactly this kind of task to AI companies. That does not make the tasks bad, and SemiAnalysis’s first-try QA rule is a sensible filter, but it does make GDPx partly a product showcase, and it raises the usual contamination question, which the write-up does not address as far as I could read.
  • It is a living benchmark. SemiAnalysis says it will release new versions every three to six months. That keeps it hard, and it also means a score today will not be comparable to a score next spring.
  • No independent replication yet. All numbers come from SemiAnalysis’s own runs; secondary coverage so far repeats them.

None of this is a reason to dismiss the result. It is a reason to read “12%” as what it is: a snapshot from one evaluator, using one set of vendor-built environments, that says the best agents of late 2026 can complete about one in eight long expert tasks in the fields where their makers most expect to be paid. That is both much more than two years ago and much less than the marketing implies.

Sources