Efficiency keeps landing at somebody else's layer

Every efficiency win this week landed one layer below the model, in the tier somebody else operates. vLLM, the open-source engine many teams use to serve models, shipped a cache store for agentic workloads. Thinking Machines argued low-rank finetuning gives up less than its reputation suggests, and Nous Research reported faster pre-training. All of it makes serving cheaper for whoever runs the fleet. The counterweight arrived the same week: someone metered local inference on Apple Silicon against renting the same tokens, counted the electricity, and the hardware lost. The cheap layer is again the one teams do not operate, which is what we wrote about last week when the coding assistant was quietly picking everyone's database. Agent harness frameworks are libraries today; the week one starts sending invoices, it is the same story again.

Top Stories#

Serving infrastructure absorbed the week's real efficiency gains. vLLM published a cache store built with Mooncake. The point is mundane and large: a long agent session keeps re-reading its own history, and parking that cached state outside the serving process means it stops paying for that history twice. It followed with a study of TurboQuant, a way of squeezing model weights into smaller numbers, weighing what that costs in accuracy against what it buys in speed. Neither is a model release, though both move the per-token bill more than model releases do.

Owning the hardware stopped being the cheap option. Someone metered offline LLM energy use on Apple Silicon against renting the same tokens. Once electricity is on the bill, the machine on the desk loses. Meanwhile, a walkthrough of what a GGUF file actually contains, GGUF being the format local models ship in, catalogs what it leaves out: mostly the metadata a tool would need to know what a model is without loading it. Local inference keeps privacy, and it keeps working on a plane. The cost argument just got harder.

The training side got cheaper at both ends. Thinking Machines published LoRA Without Regret. LoRA is the trick of training a small set of extra weights and bolting them onto a finished model instead of retraining it: cheap, and mildly disreputable. It holds up against full finetuning more often than practitioners assume, the post argues. At the other end, Nous Research released token superposition training, reporting pre-training up to 2.5x faster from 270M to 10B parameters. Cheaper finetuning lets more teams specialize a model, and faster pre-training lets them afford a second try.

The agent harness is turning into a layer with vendors. The harness is the code wrapped around the model: the loop that calls it, the tools it reaches, the state between turns. Flue is a framework for exactly that, and Google announced Genkit middleware for hardening agentic apps, the same territory from the platform side. But the reason it needs hardening turned up the same week, when a LinkedIn user hid a prompt injection in his bio and had recruiting bots addressing him as my lord. Nobody sanitizes a bio field, until a bot reads one as instructions.

Top Stories#

Top Links#