AntflyAntfly
Back to The Weekly Recall

The harness moved the numbers, not the model

The scaffolding around the model produced this week's biggest measured gains, which makes the model the swappable part and price the deciding one

A CRT agent uses a long lever and triangular fulcrum to lift a heavy computing module.

The scaffolding around the model moved the numbers more than the model did this week. Microsoft Research released Webwright, a framework that wraps a model in a terminal and a set of tools and points it at a browser, and reported 60.1 on Odyssey, a web-agent benchmark, against 33.5 for the same model without the wrapper. Anthropic shipped Opus 4.8 to a warm reception and an immediate argument about price. A couple of weeks ago that wrapper layer was mostly a curiosity with its first vendors showing up; now it is where the measured gains are, which makes the model the swappable part. AMD and at least one serving startup spent the same week going after token prices from below. Nobody is arguing much about quality any more, so they argue about the bill instead. Renewals will show it before benchmarks do.

Top Stories#

A harness nearly doubled a benchmark score without changing the model. Microsoft Research released Webwright, a terminal-native framework for web agents, reporting 60.1 on Odyssey against 33.5 for the base GPT-5.4 running underneath it. Same model, same weights, same price per token. What changed is the loop around it: which tools it can call, what environment it works in, how results come back. A jump that size normally arrives with a new model generation and a new price tier attached. This time you just download it. If the result holds up when somebody other than the authors runs the evaluation, the interesting engineering has moved out of the model and into everything wrapped around it.

Opus 4.8 arrived to a good reception and an argument about price. Anthropic introduced the model and the early reads were strong, including a review arguing the version number undersold it. The complaint that keeps surfacing is not about capability. I mean, sure, the model is better; nobody is arguing the model is not better. The argument is about what a serious workload costs at the top of the lineup, and that concern now runs through most of the public developer conversation about the lab. Which puts the argument in an odd place: teams like the model fine, and still have to walk the renewal past finance.

The attack on token price is coming from hardware and serving at the same time. AMD claimed leadership on tokens per dollar with a Ryzen AI Halo developer box at $3,999, priced against Nvidia's Spark. Meanwhile, kog.ai reported 3,000 tokens per second per request on standard datacenter GPUs, which is a claim about serving software rather than silicon, on the ordinary accelerators already sitting in racks. Neither one takes quality away from the frontier labs, but both shorten the distance to a cheaper answer for the many workloads that never needed the frontier in the first place.

  • Introducing dynamic workflows in Claude Code (claude.com): harness features shipping as product, which is the same shift Webwright is measuring.
  • Building self-improving tax agents with Codex (openai.com): an agent wired into a loop that revises its own procedures, in a domain where being wrong is costly and auditable.
  • How the community trained Gemma to "Think" with Tunix and TPUs (developers.googleblog.com): reasoning training carried out by volunteers on borrowed accelerators, which is a stranger sentence than it first appears.