Review capacity is the constraint, not model capability
The week's proposals all moved checking after the fact, to sampled trajectories and runtime enforcement, because agent output now exceeds what anyone will read before it lands
The checking all moved after the fact this week. A post named the constraint plainly: agents can open pull requests nobody can review, and the prescription is to stop producing changes in that shape in the first place. A paper proposed sampling and triaging agent trajectories, meaning look hard at a slice of the runs instead of all of them, and an essay argued that frontier safety is shifting to runtime, to checks that fire while an agent is working rather than tests that run before release. We wrote a couple of weeks ago that the agent loop had finally acquired a debugger, which was good news right up to the obvious follow-up question: who reads the recording? This week's answer is that somebody reads some of it. The scarce thing now is somebody with time to read what the agent did. The tell will be whether a team can say its sampling rate out loud, and still say the same number a quarter later.
Top Stories
Agents are writing more change than anyone will review. Jon Udell's post treats unreviewable pull requests as a self inflicted problem with an obvious remedy: constrain the agent to changes a person can actually hold in their head. The interesting part is what the remedy concedes. A long running loop can generate work faster than the humans reviewing it, so more throughput moves the bottleneck rather than removing it. Teams that measure agent output in merged lines are measuring the wrong end of the pipe, because the cost landed on whoever has to read it.
A paper proposes triage for agent trajectories. Signals describes sampling agent interactions and triaging them rather than reviewing the full set, which is the quality assurance pattern support and operations teams have run for a long time: listen to a portion of the calls, escalate the bad ones, do not pretend to listen to all of them. Borrowing that pattern is an admission worth saying out loud. Complete review of agent behavior is off the table, so the design question becomes which slice gets looked at and on what signal. That is a tractable problem, and a different one from making the agent correct.
Safety work is moving from evaluation to runtime. An essay argues that frontier safety engineering is diverging toward runtime enforcement and away from pre-release testing. The reasoning is mechanical rather than ideological. Pre-release evaluation covers behaviors somebody thought to write a test for, and an agent with tools composes situations nobody enumerated. Checks that run during execution catch what the tests could not. Sure, but those checks run on infrastructure the operator owns, which quietly moves part of the safety burden from the lab to whoever deployed the thing.
Model cards are doing work that release notes do not. Google published the Gemma 4 model card, the document that states what a model was measured on and how it scored, in the same week that OpenAI's simple-evals suite sat in public view, including the browsing benchmark labs like to put on stage when they announce something new. Which chart a lab leads with tells you what it was aiming at. A card lets a reader work out what a release was optimized for, and a public harness lets a buyer rerun the tests against work that resembles their own. Database changelogs could stand to steal the format.
Top Links
- AI Cost Structure Shift: Foundation Labs, Frontier Models, and Open Models (linkedin.com): three forces reshaping what inference costs and who captures the difference.
- Leanstral 1.5 (docs.mistral.ai): a new model card from Mistral, worth reading for which numbers it leads with.
- Deep News (yage.ai): a curated feed of AI engineering writing, including the runtime safety piece above.