Antfly inference

Antfly inference is Antfly's local ML inference service for embeddings, chunking, reranking, and text generation from text, image, and audio prompts. It supports optimized ONNX and native GGUF execution without external API dependencies.

Features#

• Embedding Generation - Text and multimodal (CLIP) embedding models
• Text Chunking - Semantic chunking with ONNX models or fixed-size fallback
• Reranking - Relevance re-scoring for search results
• Local Generation - Text, image, and audio prompts with supported GGUF models
• Privacy - Data never leaves your infrastructure

Model Management#

List Available Models#

View all available models in the registry:

antfly inference list --remote

Download Models#

Models are pulled from HuggingFace by their full owner/name reference. A bare name without an owner is rejected.

antfly inference pull hf:BAAI/bge-small-en-v1.5 --tasks embed
antfly inference pull hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF:gguf:Q8_0 --tasks rerank
antfly inference pull hf:ggml-org/gemma-4-E4B-it-GGUF:gguf:Q4_0 --tasks generate --projector auto

Use --tasks to record what the model is for, which determines where it is discovered and which endpoints will serve it.

--projector auto is the default. Gemma 4's unified projector enables image and audio inputs and is recorded in the generated model manifest. Use --projector none for a text-only installation.

Verified starting points:

• Embedders: BAAI/bge-small-en-v1.5, antflydb/clipclap (text + image + audio)
• Rerankers: ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF:gguf:Q8_0
• Generators: ggml-org/gemma-4-E4B-it-GGUF — see Supported models
• Transcribers: openai/whisper-tiny
• Readers: antflydb/Florence-2-base
• Extractors: antflydb/gliner2-base-v1:gguf:Q4_K

List Local Models#

View models you've downloaded:

antfly inference list

Model Variants#

Models support different quantization variants:

Append the variant to the reference to select one:

antfly inference pull hf:BAAI/bge-small-en-v1.5:i8 --tasks embed

Model Storage#

Models are stored in ~/.antfly/inference/models/ by default, under their HuggingFace owner and repository name. Explicit variants use distinct internal leaf names so quantizations cannot be mixed accidentally. Antfly inference auto-discovers these models, as well as legacy role-specific directories, when Antfly starts in standalone mode. Use --models-dir to override the default location.

Traditional ML predictors use the parallel ~/.antfly/inference/ml/ directory. Use --ml-dir to override it. Neither inference asset directory follows the database --data-dir implicitly.

For text generation, only a specific set of model architectures is supported. See Supported models before pulling a generator.

Configuration#

Antfly inference models can be used in index and query configurations:

Embedder Configuration#

The API uses one EmbedderProvider and one canonical EmbedderConfig across inference features. Index creation accepts the purpose-specific IndexEmbedderConfig subset (antfly, ollama, openai, or bedrock), so generated clients cannot advertise providers that the indexing runtime cannot execute.

{
  "provider": "antfly",
  "model": "BAAI/bge-small-en-v1.5"
}

Chunker Configuration#

fixed_bert and fixed_bpe are built in and need no download.

{
  "provider": "antfly",
  "model": "fixed_bert",
  "target_tokens": 512,
  "overlap_tokens": 50
}

Reranker Configuration#

Reranking supports antfly, cohere, and vertex. Ollama reranking is not advertised because Antfly does not implement a compatible Ollama rerank API.

{
  "provider": "antfly",
  "model": "ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF:gguf:Q8_0",
  "field": "body"
}

Troubleshooting an enrichment backlog#

Inspect an embeddings index while enrichment is pending:

curl http://localhost:8080/db/v1/tables/TABLE/indexes/INDEX

The enrichment_runtime object reports sequence progress, retry and failure counters, active batch information, and worker lifecycle state. worker_started says whether the background worker is running. stalled is the narrower condition where work is pending but no worker is running and no retry or terminal failure already explains the backlog.

For exact keys that have reached Antfly's durable artifact or index repair queues, use the bounded operator endpoint:

curl -X POST http://localhost:8080/db/v1/tables/TABLE/repair/issues \
  -H 'Content-Type: application/json' \
  -d '{"target":"artifact","limit":100}'

This repair endpoint does not enumerate every document in the pending enrichment tail. If pending work is reported but no durable repair issue is listed, use the runtime counters and server logs to distinguish a worker lifecycle problem from retrying inference.

API Reference#

See the Inference API documentation for details on the HTTP endpoints.