The savings showed up on either side of the model
This week's engineering attacked the cost of putting text into a model and holding what it already read, which is where teams go once long context runs into either the model's limit or the invoice
The week's engineering went after the cost on either side of the model call. Baseten, one of the companies that hosts models for everybody else, described compacting the KV cache inside a single forward pass, and a separate write-up argued for a small parsing loop that runs before retrieval. We noticed a few weeks ago that the improvements were arriving outside the weights, in the plumbing rather than in the models. They have now split cleanly to either side of the context window: cheaper to put text in, cheaper to hold what the model already read. Everyone did it to get the bill down. Pouring everything into a long context is still the default first attempt at an agent, and teams give it up when they meet the model's limit or the monthly bill, whichever lands first. The savings are real, and they accrue in the serving layer and the pipeline, which a team either operates or rents. Whether any of it reaches a customer's bill is a separate question, and the next round of price sheets will say.
Top Stories
Cache compaction moved inside the forward pass. Baseten published Still, an amortized approach to KV cache compaction that does the work in one pass through the model instead of a separate step. The KV cache is the model's working memory of everything it has already read in a session, and it grows every turn, so a long conversation pays for it in expensive memory well before anyone notices it in the token count. Picture a desk nobody ever clears: fine on Monday, unusable by Thursday. Doing the compaction in line means the tidying happens while the model is already working, without a second trip over the context. It is a serving-layer win, which is the part worth noting: it lands for whoever runs the inference, and reaches everybody else only if that operator passes it along.
A small loop is landing in front of retrieval. A write-up on loop engineering for question parsing makes the case for a cheap, tight loop that reshapes a question before anything gets fetched. Retrieval here means going and finding the documents a model should read before it answers, and the loop's job is to work out what was actually asked first. The mechanism is unglamorous, which is rather the point. Most bad answers start as a bad query, and repairing the query costs a fraction of what re-reading the corpus costs. It is the same instinct that puts an index in front of a scan. Teams that treat the question as given, then spend on context to compensate, are paying at the most expensive stage in the chain.
Open weights are improving faster than their packaging. Open weights means the model file is downloadable and anyone can run it, and the sizes there are moving faster than the leaderboards: GLM 5.2 lands under a trillion parameters against Kimi 3's 2.8 trillion, and DeepSeek V4 Flash holds up at roughly a third of GLM's size again. But the artifacts underneath are still fragile. A Gemma 4 file, re-uploaded with duplicate vocabulary entries, broke inference for anyone not on the most common loader, quietly, under a name that had not changed. Open weights are a supply chain now, and it is one where a re-upload can change what a pinned reference actually resolves to. Which is fine, presumably, until the week it is not.
Top Links
- gigatoken (github.com): language model tokenization at gigabytes per second, aimed at the stage almost nobody used to measure.
- Open-weight AI is having its Kubernetes moment. Let's not ruin it. (tobi.knaup.me): an argument that the open-weight ecosystem is standing at the fork container orchestration once stood at, with the same chance of taking the bad branch.