Back to Rowan Copley

· Lab

Multimodal decision models and very expensive piles of PDFs

Rowan Copley
Founding Engineer

What would it take to index a massive corpus of PDFs? Like, for example, the entirety of the Epstein files? This is a question I found myself asking recently as we're scaling Antfly's ability to ingest multimodal data.

Like, here's a table I had Claude make that estimates the total cost of extracting the Epstein files' text on various services:

Service / modePublished USD per 1,000 pagesCost for 3.5M pages1
AWS Textract — text OCR$1.50 first 1M/month; $0.60 thereafter$3,000
AWS Textract — Tables, including Layout and OCR$15 first 1M/month; $10 thereafter$40,000
AWS Textract — Forms + Tables$65 first 1M/month; $50 thereafter$190,000
Azure Document Intelligence — Read$1.50 first 1M/month; $0.60 thereafter$3,000
Azure — Layout$10$35,000
LlamaParse — Fast$1.25$4,375
LlamaParse — Cost-effective$3.75$13,125
LlamaParse — Agentic$12.50$43,750
LlamaParse — Agentic Plus$56.25$196,875
Google Document AI — Enterprise OCR$1.50 through 5M/month; $0.60 thereafter$5,250
Google Document AI — Layout Parser$10$35,000
Mistral OCR 4.1 — standard$4$14,000
Mistral OCR 4.1 — asynchronous batch$2, after 50% batch discount$7,000
Unstructured — public PAYG$15$52,500

Sources:

That's a gigantic range, mostly dominated by what kind of workload you're doing. I would not consider any option cheap. Some of the really expensive options are specialized, like for tables, so we'd in theory be able to cut down the cost by routing the tables to the expensive model and the rest to the cheaper one. Some of these providers do have their own "smart routing" that purports to do that for us. And most of the Epstein documents won't need to be put through OCR anyway because they'll have text you can extract.

But it's still an unnervingly large range that we could end up spending, and anyway, we need to be able to run everything locally when we do this with Antfly.

Take this page from a 1989 paper on backpropagation: Page from a 1989 backpropagation paper showing a neural network architecture diagram

Here's what a modern OCR model will do with that page: OCR output that replaces the neural network diagram with a short caption

(You can try the demo out yourself here)

That's LightOnOCR-3-4B; there's also a 0.8B model which is nearly as good in our tests (and much faster). You might notice that it actually just captions the image, losing all the detail about what's happening in the diagram. It's possible to pass in a prompt to the model and exract out much more of the text, though that can start getting you unpredictable results (and in my limited test runs, the model started segmenting images into regions and getting hyper-specific about what was in them without being asked to do that). You might also notice that the OCR model turned "8 1%" into 81% and "19 4% into 194%. It's an old scan and the 'i's and 'j's are missing their dots, but it does illustrate that there's a ceiling to quality for these models. If you want better you'll need a bigger model. GPT-6-Luna, for example, transcribed it correctly for ¢0.38, which would put the total cost of our Epstein workload at $13,000 or so.

Depending on the page's complexity it can take double-digit seconds to process a single page. With a corpus that's in the millions of pages, that's over a year with serial processing or weeks to months if we batch, depending on what we're running this on. Most pages don't need to be OCR'd because they already have text that can be extracted and we just need to do cheap heuristic checks to see if it has been poorly extracted (or the text itself is the output of an ancient and bad OCR model). So you end up with a pipeline that routes PDF pages based on a number of decisions in order to avoid always choosing the most compute-expensive option.

That got me interested in seeing if we could use multimodal decision models to spot check the results, now that Jev and a whole bunch of similar models (and locally-runnable!) exist. Are they better than basic (and free) heuristics? We're not even going to use an OCR model if we think we got all the text from a page in the first place since it's so much faster not to.

Decision model OCR quality benchmark comparing AUROC, latency, and cost

Some observations:

  • Honestly nothing is a slam-dunk, even Jev as the best-performing model has a 20% chance to misclassify a page with this configuration
  • I made a number of kinda arbitrary decisions without trying everything, so it's possible there's better performance to be had
  • But still, anything that's not Jev or Clef is barely better than chance
  • Clef Omni took almost as long as some pages took to go through the OCR model itself, but way less than the median OCR time, and it's basically as good as Jev

We don't need any of these models to be perfect, we just need to know we're generally routing the hard pages to the more expensive models but saving on compute and/or money by getting the rest of the pages done the cheap way. Most of us probably aren't processing PDF corpuses the size of the Epstein files routinely (though if you are, get in touch!). But you're probably not going to one-shot the whole thing either; you're going to need to iterate and reducing the cost of running through the PDFs also means reducing the cost of a failed run. It's not surprising that the models can do this, but it's nice to know they're not half bad.

Footnotes#

  1. The DOJ reported nearly 3.5 million pages as of January 30, 2026, including prior releases. CBS News later counted about 2.7 million pages still publicly available after takedowns. ↩