The machinery went public, the data did not
The expensive parts of the serving stack were published this week, which leaves access to the customer's data and the capacity to check model output as the two things still scarce
The parts of the stack that usually stay private were published this week. Moonshot, the lab behind the Kimi models, released an expert-parallelism library and attention kernels alongside its new model, and a separate project put out a distributed platform for running agent environments at scale. These are the pieces that make serving a big model affordable, and they used to be the house secret. Keeping them closed was never much of an advantage, it turns out, and fine-tuning pipelines are going open for roughly the same reason. What stays scarce is access to the customer's data, which is where the platforms already make their money by hosting it and taking an upcharge on the tokens, and the capacity to check what the models produce. We argued a few weeks ago that review capacity, not model capability, was the constraint. This week the tooling for it started to arrive, and so did its first failure mode. The tell is what a lab still keeps closed, and the list is getting short.
Top Stories
A lab published its serving stack alongside the model. Moonshot, the lab behind the Kimi models, released MoonEP, an expert-parallelism library that balances load using redundant experts and FlashKDA, kernels for its delta attention variant, while AgentEnv arrived as a distributed platform for running agent environments at scale. These are the pieces that normally stay in-house, because they are where the operating cost lives. A model of this shape is really a committee of specialists with only a few consulted per token, so the job is routing each token to the right specialist without stranding half the hardware, then running the attention math fast enough for the parameter count to mean anything. Handing that out lowers the cost of serving a large model for people who will never run this particular one. It also narrows what a lab keeps to itself, and the remainder is mostly data and customers.
Stacked pull requests are now a supported workflow. GitHub moved stacked pull requests into public preview, so a chain of dependent changes can be reviewed one small piece at a time rather than as one wall of diff. The technique is old news to anyone who has kept a patch queue by hand. It is the volume that pushed it into the product: generation is cheap and reading is not, which leaves the size of the unit as about the only lever anyone still controls. Smaller units also hand a reviewing agent a scope it can plausibly hold at once, which matters more than a workflow change usually does.
A review agent will eventually tell you the code is clean. A talk on kernels, reinforcement learning and reward hacking in agents lays out the general shape: an agent optimizes the signal it is scored on rather than the thing that signal was standing in for. Long review sessions show the domestic version of it. Let one agent keep re-reviewing findings it has already fixed, and its context fills with resolved issues until it concludes the work is done, while a fresh session over the same code tends to surface a dozen more. So somebody has to keep starting fresh sessions, or the review quietly stops finding things. Finding broken code also appears to be far easier than writing it, which is where the hours are going.
Top Links
- codex-security (github.com): SDKs and a CLI for Codex Security, which puts checking the generated code on the same invoice that generated it.
- Agenda (softwareyoucanlove.ca): this year's program, with a track leaning heavily on systems languages and Zig in particular.
- David Heinemeier Hansson (DHH), Co-founder of 37signals (open.spotify.com): an interview that is at its best on omakase, the case for handing people a chosen stack instead of a menu.