Antfly Inference

Antfly Inference is the model runtime inside Antfly. It runs embedding, chunking, reranking, extraction, transcription, reading, and generation models (text generation from text, image, and audio prompts) in the same process as the database, so index enrichers and query processors call it without a network hop and nothing leaves the machine. It executes model artifacts through native (CPU/Metal/CUDA), ONNX, and ortgenai backends without external API dependencies. The decision it puts in front of you is which model to pull for each job; the Local Model Compatibility page holds the verified list, and this page covers how the runtime finds, stores, and serves them.

Pull a model, then embed a sentence through the running engine:

antfly inference pull antflydb/clipclap:gguf:Q4_K

curl -X POST http://127.0.0.1:8080/ai/v1/embed \
  -H "Content-Type: application/json" \
  -d '{"model":"antflydb/clipclap","input":["a photograph of a lighthouse"]}'

The response is a 512-dimension vector in the OpenAI embeddings shape ({"object":"list","data":[{"embedding":[...]}]}).

Where It Listens#

In standalone mode the runtime is served on the API port under /ai/v1, beside /db/v1. Run as its own process with antfly inference run, it listens on 127.0.0.1:8090 by default, and a database points at it with the inference.api_url config key. External providers (Ollama, OpenAI, Bedrock) plug into the same embedder, reranker, and generator interfaces, so an index config names a provider and a model and never cares which process answers.

Configuring a Separate Inference Process#

antfly inference run --config inference.json reads admission settings and the model settings models_dir, ml_dir, max_loaded_models, and preload:

{
  "inference": {
    "models_dir": "/models",
    "ml_dir": "/ml",
    "max_loaded_models": 3,
    "preload": [
      {"kind": "embedder", "name": "BAAI/bge-small-en-v1.5", "backend": "native"}
    ]
  }
}

Pull the referenced models into that directory before starting the process. The operator's flat config spelling (those four fields at the top level) is also accepted. If both spellings exist, the nested inference value wins for each field, including an explicit empty preload list. No api_url is required to start the server; configure the listener with --host and --port.

Explicit CLI options override config regardless of argument order. Supplying one or more --preload-model flags replaces the config preload list rather than appending to it. For an exact match of kind, model reference (including artifact variant), and backend, config-only residency_mode and memory_budget_mb policies are retained; unrelated config models are not loaded. Duplicate config entries matching the same CLI selection are rejected as ambiguous. A model-count limit of zero disables that limit. When paths are absent from both CLI and config, the usual environment/home defaults apply. Preload entries retain their backend, artifact format/quantization, and optional residency/memory-budget choices. An unqualified preload name with format/quantization is expanded to an explicit artifact reference (for example, owner/model:gguf:Q4_K) before resolution. Conflicts between preferences and already-qualified references are rejected rather than silently selecting another installed artifact.

The runtime reads this config directly. The operator additionally emits equivalent --models-dir, --ml-dir, --max-loaded-models, and one --preload-model flag per preload entry, so a pod still running an older runtime binary that predates config-file model policies keeps working during a rolling upgrade. The operator resolves nested model overrides into the flat config spelling before emitting JSON or CLI arguments, writes default model kinds (generator when omitted in spec.config.preload) and normalized artifact references into both outputs, and preserves remaining non-model nested settings alongside the client URL field required by the legacy schema; admission settings and flat per-model policies remain intact.

For operator-generated eager preloads (the default loading strategy), specify a recognized task such as tasks: [embed] or tasks: [generate] on each model. The warm CLI needs a model kind; an omitted or unrecognized task hint does not imply a generator. The operator reports a configuration validation failure before creating or updating the workload when that kind is ambiguous. Alternatively, supply spec.config.preload with an explicit kind, or use lazy/bounded loading to discover the model task at request time. Per-model eager strategy overrides have the same requirement.

Empty or null optional backend, format, and quantization fields in operator preload configuration are treated as unspecified in both JSON and CLI output.

Pulling Models#

Models come from HuggingFace by their full owner/name reference; a bare name without an owner is rejected. The hf: prefix is accepted.

antfly inference pull hf:BAAI/bge-small-en-v1.5 --tasks embed
antfly inference pull hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF:gguf:Q8_0 --tasks rerank
antfly inference pull hf:ggml-org/gemma-4-E4B-it-GGUF:gguf:Q4_0 --tasks generate --projector auto

--tasks records what the model is for, which decides where it is discovered and which endpoints serve it. Text generation supports a specific set of decoder architectures, so check Local Model Compatibility before pulling a generator.

--projector auto is the default. Gemma 4's unified projector enables image and audio inputs and is recorded in the generated model manifest. Use --projector none for a text-only installation.

Verified starting points, one per job:

See what is installed:

antfly inference list

The full registry, with descriptions and sizes, is the models browser.

Variants#

A variant appended to the reference selects an artifact format or quantization. The advertised variants are gguf, gguf:<quant> (for example gguf:Q4_K), onnx, hybrid, and safetensors; i8 and f32 are still recognized and select among the ONNX artifacts:

antfly inference pull hf:BAAI/bge-small-en-v1.5:i8 --tasks embed

With no variant the pull is auto. Keep one variant per model directory: a second variant pulled beside the first leaves both in place, and different parts of a model can then resolve to different variants. If a model misbehaves after a second pull, delete the directory and pull once.

Model Storage#

Models live under $ANTFLY_INFERENCE_MODELS_DIR, else ~/.antfly/inference/models; --models-dir overrides both. A variant-less pull installs flat under the HuggingFace owner and repository name:

~/.antfly/inference/models/
  BAAI/bge-small-en-v1.5/

An explicit variant gets its own distinct leaf directory beside the plain one, so quantizations cannot be mixed accidentally. Older per-task subdirectories (embedders/, chunkers/, rerankers/, generators/) are still discovered. Antfly Inference loads what it finds there when standalone mode starts.

Traditional ML predictors use the parallel ~/.antfly/inference/ml/ directory; --ml-dir overrides it. Neither inference asset directory follows the database --data-dir implicitly.

Naming a Model in an Index or Query#

The provider name is antfly. The same three shapes appear in table, index, and query configs:

{ "provider": "antfly", "model": "BAAI/bge-small-en-v1.5" }

The API uses one EmbedderProvider and one canonical EmbedderConfig across inference features. Index creation accepts the purpose-specific IndexEmbedderConfig subset (antfly, ollama, openai, or bedrock), so generated clients cannot advertise providers that the indexing runtime cannot execute.

{ "provider": "antfly", "model": "ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF:gguf:Q8_0", "field": "body" }

Reranking supports antfly, cohere, and vertex. Ollama reranking is not advertised because Antfly does not implement a compatible Ollama rerank API.

{ "provider": "antfly", "model": "fixed_bert", "text": { "target_tokens": 512, "overlap_tokens": 50 } }

The chunkers fixed_bert and fixed_bpe are built in and need no download. Chunk sizing nests under text, and target_tokens must be larger than overlap_tokens or the request fails with InvalidChunkOverlap.

When Enrichment Falls Behind#

An embeddings index reports its own backlog. Inspect it while enrichment is pending:

curl http://127.0.0.1:8080/db/v1/tables/TABLE/indexes/INDEX

The enrichment_runtime object carries sequence progress, retry and failure counters, the active batch, and worker lifecycle state. worker_started says whether the background worker is running. stalled is the narrower condition: work is pending, no retry or terminal failure already explains the backlog, and either no worker is running or the active batch has passed its deadline without progress. stall_reason says which (worker_missing, model_loading, embedding_overdue, or publishing_overdue).

For keys that have reached the durable repair queues, ask the bounded operator endpoint:

curl -X POST http://127.0.0.1:8080/db/v1/tables/TABLE/repair/issues \
  -H 'Content-Type: application/json' \
  -d '{"target":"artifact","limit":100}'

It lists durable repair issues, not every document in the pending tail. Pending work with no listed issue means the cause is a worker lifecycle problem or inference that is still retrying; the runtime counters and server logs tell those apart.

Use Agent Skills#

Everything above is also encoded in the Antfly skill, so a coding agent can execute this guide for you:

npx skills add antflydb/antfly-skills

Then prompt it with the outcome, for example "Set up local embeddings and reranking with Antfly Inference and wire them into my table's index", and use this page to judge the result.

Next Steps#

• Local Model Compatibility: which artifacts load, which are rejected, and why
• Multimodal Embeddings: the ClipClap embedder on an image index
• Tune Hybrid Search: the reranker in a query pipeline
• Inference API reference: the HTTP surface