Back to The Weekly Recall

October's new models are so good we had to move the goal posts (again)

October's model releases, Artificial Analysis's revised index, Anthropic's IPO filing, Supabase Compute, and AI research.

A square-eyed agent pushes a freestanding goal frame forward.

A model leaderboard is a fairly simple thing. You give the models the same jobs, count what they finish, and rank the results. Then the models get better at the jobs, some of the questions become public, and people start asking whether the exam still measures the thing they care about. This week Artificial Analysis changed its Intelligence Index: it added private coding and scientific research tasks, revised three other tests, and now gets 60% of the score from private test sets or held-out answers. It also introduced a separate Progress Index for comparing models over time, which is useful when the current-model exam keeps changing.

October, meanwhile, supplied some great models. Basically every lab released a new one: Mistral, Anthropic, Meta, Google, NVIDIA, MiniMax, Moonshot, and OpenAI.

Then, a day after the index changed, Google announced Gemini 4 Carbon, which took first place, though Google pulled a Glasswing and is keeping Carbon in an invite-only preview.

The new index also created more distance between the leading models and the open weight models. Lower scores don't mean that the open models are getting worse, but they're not getting better at the same rate that the leading models are. The models you can self-host today are better than ever, but the models you can rent continue to widen their lead.

Top Stories#

Anthropic's S-1 reinforces both optimists and pessimists. Before an IPO, a company files an S-1 to describe its business and why people should buy its shares. Anthropic's filing shows first-half revenue rising from $1.2 billion to $16.3 billion in a year, while compute and infrastructure expense fell from 201% of revenue to 55%. That is a good chart for the optimist. The compute bill itself rose from $2.3 billion to $8.9 billion, and Anthropic lost $693 million from operations, which gives the pessimist a chart too.

Supabase put long-running compute beside Postgres. Its private alpha can run services and jobs in the same project as the database, including an embedding backfill. Calls from inference code to Postgres should take less time when the two are nearby. You still have to deploy the model runtime, keep track of the backfill, and recover from a failed job. Supabase says services may restart at any time, so long jobs must save their progress to Postgres. The alpha is waitlisted in us-west-2 and has no GPU, autoscaling, or persistent disk.

Anthropic's internal Claude model combined software bugs into working attacks. A single bug is often only the first step; to get from a web page to a computer, an attacker may have to link several. Anthropic says its unreleased model assembled five such chains across browsers and operating systems, including one that escaped Chrome's sandbox. Google has patched 62 related Chrome vulnerabilities, according to Anthropic. The model remains internal, so this is a demonstration of what Anthropic can do, rather than a tool anyone can download.

OpenAI is putting outside researchers between its science claims and the victory lap. It reports results in mathematics, computer science, and physics, including a claimed disproof of a longstanding idea about computing problems. The math and computer science materials are public. An external group will review the physics result before its materials are released, and OpenAI is funding open examination of the computer science claim. After last week's dispute over hundreds of AI-generated proofs, getting other researchers into the loop seems like a useful part of the announcement.

  • Voxtral 3 (mistral.ai): Mistral's new speech models cover transcription, generated speech, and live voice in public preview.
  • Timestral (mistral.ai): Mistral released time-series forecasting models small enough to run on CPUs.
  • Promptable transcription (huggingface.co): NVIDIA's open-weight speech model can take instructions and context along with the recording.
  • Genie 4 (deepmind.google): Google's world-model preview generates interactive scenes with native audio, for trusted testers first.

The week in AI, once a week.