AntflyAntfly
Back to The Weekly Recall

The useful models this week were the small ones

The week's practical results came from models small enough to run beside the data, not from token generation at the frontier

A CRT agent places a compact computing module beside files in a small cabinet.

The most useful models this week were the small ones. Fastino Labs open-sourced a 300M-parameter safety classifier reported to match or beat models 23 to 90 times its size. Pioneer got 44x more throughput out of one L4, a mid-range datacenter GPU, by stacking small add-on models on a shared base. Someone indexed a year of video on a five-year-old laptop. None of that is token generation, which is what the industry mostly means by inference now. Last week the efficiency gains all landed in the tier somebody else operates; this week they landed on hardware teams already own. The frontier kept launching anyway. The next round of token bills will say whether the small tier is a trend or a hobby.

Top Stories#

A 300M-parameter model is now competitive at safety moderation. Fastino Labs open-sourced GLiGuard, a classifier that decides whether content breaks a policy, with reported accuracy matching or beating models 23 to 90 times larger. The paper calls the method schema-conditioned classification. In plainer terms: instead of asking a big general model to read the rules and judge, the small model gets the policy's label set as part of its input. Moderation is an expensive per-call habit in a production stack, because every message pays for it. It appears to have stopped needing a frontier model.

Adapters are becoming the unit of deployment. An adapter is a small stack of extra weights trained on somebody else's base model: cheap to make, cheap to swap. Pioneer published the 33rd adapter problem, an account of getting 44x more throughput from one L4 by changing how many adapters share a GPU, plus a companion post on the agent built on top. The pattern recurs in extraction and embedding work: what ships as a new model is often an adapter merged into a base someone else paid to train. That makes supporting the next popular model nearly free, and hands the leverage to whoever owns the base.

Local indexing got cheap enough to be boring. Someone indexed a year of video with Gemma on an M1 Max, a laptop already several generations old, which not long ago meant a cluster and a budget line. Meanwhile, token costs came up at a dinner of IT leaders, usually the moment a line item becomes a budget conversation. Both point the same way: some work is worth moving off the meter, and finance has started asking which work that is.

Anthropic bought an SDK generator, which is a distribution move. The company acquired Stainless, whose product turns an API specification into client libraries, the packages developers install to call a service. As tooling it is a modest deal. What Anthropic gets out of it is a cheap guarantee that a usable client exists in whatever language a team already writes, which is how a model provider ends up inside environments it never had to sell into. Whoever ships the easiest client wins the default, and that happens at the packaging layer, which nobody puts on a slide.

  • Introducing Gemini Omni (blog.google): Google's newest model, launched the same week the small-model results landed.
  • Cohere's Command A (venturebeat.com): lossless quantization plus citations as a model-native feature, which leaves open what corpus they point at.
  • GLiNER-Multiv2.1 (huggingface.co): a playground for the multilingual entity extractor much of this week's work descends from.
  • I tried a new 8B local LLM (xda-developers.com): a writeup arguing the design, not the parameter count, is the shift worth noticing.
  • Project Glasswing: An initial update (anthropic.com): an early progress report published as an update rather than a result, rarer than it should be.