AntflyAntfly

Inference Components and Providers

The kinds of inference Antfly runs, where each plugs in, and which providers serve each kind

Antfly turns raw data into searchable, ranked, structured knowledge with a small set of inference components. Each component is a role, such as "embed this text" or "read this page". A provider is who runs it: the built-in Antfly runtime or a service you connect. Every component is configured the same way, as a named entry with a provider, a model, and provider-specific options such as credentials:

{
  "embedders": {
    "docs": { "provider": "openai", "model": "text-embedding-3-small", "api_key": "${secret:openai.api_key}" }
  },
  "readers": {
    "pages": { "provider": "antfly", "model": "lightonai/LightOnOCR-3-0.8B" }
  }
}

Indexes, enrichments, and queries refer to these entries by name or carry the same object inline. Credentials stay in ${secret:...} references or provider environment variables; see Secrets management.

Providers by Component#

ProviderEmbedderRerankerChunkerReaderTranscriberSpeechGeneratorExtractorDeciderWeb search
Antfly (antfly)✓✓✓✓✓✓✓✓
Anthropic (anthropic)✓✓
Apple (apple)✓✓✓
AWS Bedrock (bedrock)✓✓✓
Brave (brave)✓
ChatGPT plan (chatgpt)✓
Cloudflare Workers AI (cloudflare)✓✓✓✓✓✓✓✓
Cohere (cohere)✓✓
ElevenLabs (elevenlabs)✓
Exa (exa)✓
Gemini API (gemini)✓✓✓
Google Vertex AI (vertex)✓✓✓✓✓✓✓✓
Jev (TypeSafe) (jev)✓
Linkup (linkup)✓
Ollama (ollama)✓✓
OpenAI (openai)✓✓✓✓✓✓✓
OpenRouter (openrouter)✓✓
Pioneer (pioneer)✓
Serper (serper)✓
Tavily (tavily)✓
You.com (you)✓

The table comes from the provider enums in the API specification. Inputs depend on the provider and often on the model; each provider page lists what its adapter sends.

Components#

Embedder#

Turns text, images, or audio into vectors. Configured under embedders and referenced by embeddings indexes, which embed documents as they are written, and by semantic queries, which embed the query. Antfly marks each request as a query or a document for models that distinguish the two. Direct endpoint: /ai/v1/embed. See Multimodal Embeddings.

Reranker#

Rescores search candidates against the query with a cross-encoder. Configured under rerankers and applied as a query processor after retrieval. Direct endpoint: /ai/v1/rerank. See Tune Hybrid Search.

Chunker#

Splits long text into passages before embedding, keeping offsets back to the source. Configured under chunkers and used by embedding and artifact indexes. Chunking runs in Antfly; any embedder can embed the chunks. Direct endpoint: /ai/v1/chunk.

Reader#

Reads text from page images and documents: OCR and document parsing. Configured under readers, used by the reader enrichment producer and by document_extraction for scanned pages. Direct endpoint: /ai/v1/read. LightOnOCR-3 is the recommended local reader.

Transcriber#

Turns recorded audio into timed text, optionally with speakers. Configured under transcribers and used by the transcriber enrichment producer. Direct endpoint: /ai/v1/transcribe. See Index Recorded Calls and Podcasts.

Speech Synthesizer#

Turns text into audio. Configured under text_to_speech and called through /ai/v1/synthesize.

Generator#

Produces text from a prompt and optional media: RAG answers, agents, summaries, and generated enrichments. Configured under generators and used by the generator enrichment producer, retrieval agents, and generated deciders and extractors. Generators can be chained so a request falls back to the next provider. Direct endpoints: /ai/v1/generate and /ai/v1/chat/completions. See the Named inference gateway for chains, sessions, and tools, and Provider rate limits for quotas.

Server-side agents can call function tools through the Antfly, OpenAI, OpenRouter, Ollama, Gemini, Vertex AI, Anthropic, and Bedrock generators. Choose a model that supports function calling; the adapter does not make a model tool-capable. An embedded Antfly runtime must expose its JSON generation callback to use tools, because text-only callbacks cannot carry tool calls and results.

Extractor#

Pulls entities, relations, and structured records out of text. Configured under extractors and used by the extractor enrichment producer and graph indexes. OpenAI, Gemini, Vertex AI, Anthropic, and Bedrock Claude models extract directly: set the extractor's provider and model, and Antfly runs the model with a constrained output schema. The generated form ("method": "generated") adds instructions, output mode, probabilities, and batching, and works with any qualified generator. Direct endpoint: /ai/v1/extract.

Decider#

Answers typed questions about a document or row, such as a yes/no verdict, a choice, or a score. Configured under deciders and used by the decision enrichment producer and the ai_decide SQL functions. A decider can be a dedicated decision model, a generator with constrained output, or embedding similarity against example categories. See the typed decisions reference.

Fetches live results from the web for retrieval agents and deep research. Configured as a web_search connection. See Web search providers.

Choosing Local or Hosted#

Local models keep data on your machines and have no per-call cost; hosted providers give you larger models without managing hardware. You can mix them per component, for example local embedding and OCR with a hosted generator. The Antfly provider page links the recommended local models, and Model Downloads covers installing them.

Next Steps#