Antfly Inference

Antfly Inference is the model runtime inside Antfly. It runs embedding, chunking, reranking, extraction, decision, transcription, reading, and generation models (text generation from text, image, and audio prompts) on the same machine as the database, with no external API and nothing leaving the host. It executes model artifacts through the native backend (CPU, Metal on macOS, CUDA in CUDA builds), plus ONNX Runtime and ortgenai in builds compiled with ONNX support. The decision it puts in front of you is which model to pull for each job; the Local Model Compatibility page holds the verified list, and this page covers how the runtime finds, stores, and serves them.

Pull a model, then embed a sentence through the running engine:

antfly inference pull antflydb/clipclap:gguf:Q4_K

curl -X POST http://127.0.0.1:8080/ai/v1/embed \
  -H "Content-Type: application/json" \
  -d '{"model":"antflydb/clipclap","input":["a photograph of a lighthouse"]}'

The response is a 512-dimension vector in the OpenAI embeddings shape ({"object":"list","data":[{"embedding":[...]}]}).

Where It Listens#

In standalone mode the runtime is served on the API port under /ai/v1, beside /db/v1. Run as its own process with antfly inference run, it listens on 127.0.0.1:8090 by default, and a database points at it with the inference.api_url config key. External providers (Ollama, OpenAI, Bedrock) plug into the same embedder, reranker, and generator interfaces, so an index config names a provider and a model and never cares which process answers.

In a CPU-only build, standalone runs models inside the database process. A build that includes a GPU or driver-backed backend (Metal, which is on by default on macOS, or CUDA, ONNX Runtime, or PJRT) starts a supervised child worker (antfly inference _worker) at startup and sends all embedded inference to it, CPU models included. A driver call cannot be interrupted safely, so a wedged or crashed worker is killed and respawned without taking the database down.

Setting inference.api_url disables embedded inference. It is not a fallback. Standalone then keeps no embedded provider, loads no preloads, and serves no /ai/v1 routes on its own port, and every provider: antfly call goes to that URL. antfly inference run refuses a non-loopback --host unless you also pass --allow-insecure-public-bind, because the listener has no built-in auth or TLS.

Configuring a Separate Inference Process#

antfly inference run --config inference.json reads admission settings and the model settings models_dir, ml_dir, max_loaded_models, preload, keep_alive, prompt_cache, and kernel_jit. keep_alive is the same duration string standalone uses (default 5m; "0" disables idle unloading); the operator's flat integer-millisecond keep_alive_ms is also accepted, and if both are set, keep_alive wins. prompt_cache takes the same fields as standalone's (enabled, mode, max_bytes_mb, min_tokens, ttl_ms); the prompt KV cache stays on by default when prompt_cache is omitted entirely, matching prior behavior. kernel_jit takes the same fields as standalone's (mode, cache_dir, max_cache_bytes_mb, preload_budget_ms); the --kernel-jit-* flags and ANTFLY_INFERENCE_KERNEL_JIT_MODE still override it:

{
  "inference": {
    "models_dir": "/models",
    "ml_dir": "/ml",
    "max_loaded_models": 3,
    "preload": [
      {"kind": "embedder", "name": "BAAI/bge-small-en-v1.5", "backend": "native"}
    ]
  }
}

Pull the referenced models into that directory before starting the process. The operator's flat config spelling (those fields at the top level, including keep_alive_ms) is also accepted. If both spellings exist, the nested inference value wins for each field, including an explicit empty preload list. No api_url is required to start the server; configure the listener with --host and --port.

Explicit CLI options override config regardless of argument order. Supplying one or more --preload-model flags replaces the config preload list rather than appending to it. For an exact match of kind, model reference (including artifact variant), and backend, config-only residency_mode and memory_budget_mb policies are retained; unrelated config models are not loaded. Duplicate config entries matching the same CLI selection are rejected as ambiguous. A model-count limit of zero disables that limit. When paths are absent from both CLI and config, the usual environment/home defaults apply. Preload entries retain their backend, artifact format/quantization, and optional residency/memory-budget choices. An unqualified preload name with format/quantization is expanded to an explicit artifact reference (for example, owner/model:gguf:Q4_K) before resolution. Conflicts between preferences and already-qualified references are rejected rather than silently selecting another installed artifact.

The runtime reads this config directly. The operator additionally emits equivalent --models-dir, --ml-dir, --max-loaded-models, and one --preload-model flag per preload entry, so a pod still running an older runtime binary that predates config-file model policies keeps working during a rolling upgrade. The operator resolves nested model overrides into the flat config spelling before emitting JSON or CLI arguments, writes default model kinds (generator when omitted in spec.config.preload) and normalized artifact references into both outputs, and preserves remaining non-model nested settings alongside the client URL field required by the legacy schema; admission settings and flat per-model policies remain intact.

For operator-generated eager preloads (the default loading strategy), specify a recognized task such as tasks: [embed] or tasks: [generate] on each model. The warm CLI needs a model kind; an omitted or unrecognized task hint does not imply a generator. The operator reports a configuration validation failure before creating or updating the workload when that kind is ambiguous. Alternatively, supply spec.config.preload with an explicit kind, or use lazy/bounded loading to discover the model task at request time. Per-model eager strategy overrides have the same requirement.

Empty or null optional backend, format, and quantization fields in operator preload configuration are treated as unspecified in both JSON and CLI output.

Pulling Models#

Models come from Hugging Face by their full owner/name reference. Pull rejects all bare names and friendly aliases. The hf: prefix is accepted. Append @revision to request a branch, tag, or commit; omit it to use main. Each pull resolves one immutable snapshot for all files. See Model Downloads for format selection and revision examples.

antfly inference pull hf:BAAI/bge-small-en-v1.5 --tasks embed
antfly inference pull hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF:gguf:Q8_0 --tasks rerank
antfly inference pull hf:ggml-org/gemma-4-E4B-it-GGUF:gguf:Q4_0 --tasks generate --projector auto

Pull publishes the model atomically, and the running process discovers it on the next model-list or inference request. No restart or refresh call is needed. The Antfarm dashboard refreshes its model list about every 30 seconds, so allow up to a minute for a newly pulled model to appear there.

--tasks records what the model is for, which decides where it is discovered and which endpoints serve it. Text generation supports a specific set of decoder architectures, so check Local Model Compatibility before pulling a generator.

--projector auto is the default. Gemma 4's unified projector enables image and audio inputs and is recorded in the generated model manifest. Use --projector none for a text-only installation.

On Apple silicon the Gemma 4 audio encoder runs on Metal end to end: the log-mel front end is two BLAS matrix products, and from the convolutional subsampler through the conformer blocks (clamped projections, chunked local attention with relative position bias, gated depthwise convolution) to the output projection every op executes as a device kernel inside one command buffer. The projector's tensors are served from the session's resident copies and the projector file is parsed once per model rather than per request. A 2.5-second clip encodes in about 50 ms on an M-series laptop, and the audio prompt tokens then flow through the same framed prefill as text. Set TERMITE_SERVER_GENERATE_TIMING=1 to log the multimodal_prepare, prefill and decode split for each request, and TERMITE_GEMMA4_AUDIO_METAL_PROFILE=1 to print the encoder's per-stage GPU time for each clip.

Verified starting points, one per job:

See what is installed:

antfly inference list

The full registry, with descriptions and sizes, is the models browser.

Variants#

A variant appended to the reference selects an artifact format or quantization. The advertised variants are gguf, gguf:<quant> (for example gguf:Q4_K), onnx, hybrid, and safetensors; i8 and f32 are still recognized and select among the ONNX artifacts:

antfly inference pull hf:BAAI/bge-small-en-v1.5:i8 --tasks embed

With no variant the pull is auto. Keep one variant per model directory: a second variant pulled beside the first leaves both in place, and different parts of a model can then resolve to different variants. If a model misbehaves after a second pull, delete the directory and pull once.

Model Storage#

Models live under $ANTFLY_INFERENCE_MODELS_DIR, else ~/.antfly/inference/models; --models-dir overrides both. A variant-less pull installs flat under the HuggingFace owner and repository name:

~/.antfly/inference/models/
  BAAI/bge-small-en-v1.5/

An explicit variant gets its own distinct leaf directory beside the plain one, so quantizations cannot be mixed accidentally. Older per-task subdirectories (embedders/, chunkers/, rerankers/, generators/) are still discovered. Antfly Inference discovers what it finds there. A model loads on its first request unless it is listed in preload or --preload-model.

Traditional ML predictors use the parallel ~/.antfly/inference/ml/ directory; --ml-dir overrides it. Neither inference asset directory follows the database --data-dir implicitly.

Naming a Model in an Index or Query#

The provider name is antfly. The same three shapes appear in table, index, and query configs:

{ "provider": "antfly", "model": "BAAI/bge-small-en-v1.5" }

The API uses one EmbedderProvider and one canonical EmbedderConfig across inference features. Index creation accepts the purpose-specific IndexEmbedderConfig subset (antfly, ollama, openai, openrouter, cohere, gemini, vertex, or bedrock), so generated clients cannot advertise providers that the indexing runtime cannot execute.

{ "provider": "antfly", "model": "ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF:gguf:Q8_0", "field": "body" }

Reranking supports antfly, cohere, and vertex. Ollama reranking is not advertised because Antfly does not implement a compatible Ollama rerank API.

{ "provider": "antfly", "model": "fixed_bert", "text": { "target_tokens": 512, "overlap_tokens": 50 } }

The chunkers fixed_bert and fixed_bpe are built in and need no download. Chunk sizing nests under text, and target_tokens must be larger than overlap_tokens or the request fails with InvalidChunkOverlap.

Capacity, Admission, and Memory#

• Request admission. admission.inference.max_concurrent_requests (default 32; --max-concurrent-requests on antfly inference run; 0 disables the limit) bounds concurrent inference requests. The same value sizes a weighted work budget, so a few heavy requests can exhaust capacity before the count is reached. Standalone shares one budget across direct enrichment and query calls, /ai/v1, and /ml/v1. Requests are never queued. When capacity is exhausted, HTTP callers get 503 with Retry-After: 1 and a JSON body carrying retryable: true.
• Model residency. At most max_loaded_models models stay loaded (default 10; 0 means unlimited), and the least recently used idle model is evicted to make room. Idle models unload after inference.keep_alive (default 5m; "0" disables idle unloading); antfly inference run also accepts the operator's flat keep_alive_ms spelling (integer milliseconds; 0 disables idle unloading), and keep_alive wins if both are set. Concurrent requests for the same unloaded model share one load.
• Memory. Model loads and requests reserve memory before they allocate. The reservation comes from one process envelope shared with storage and caches. Set the envelope with --process-memory-budget-mb or ANTFLY_PROCESS_MEMORY_BUDGET_MB on antfly standalone or antfly inference run. Otherwise the cgroup limit is used, and host memory without a cgroup limit. In Kubernetes Burstable pods, set it below the container limit. A request that can never fit returns MODEL_RESOURCE_LIMIT. Temporary pressure returns a retryable MODEL_RESOURCE_BUSY once idle models have been evicted.
• Query priority. Background enrichment yields to query-time work in the same database process. Embedding backfill pauses (up to 5 s) while a query-time embed is running, and background generation pauses while an interactive generate or chat request is in flight. Backfill that receives 503 retries with backoff.

When to Split Inference Out#

Embedded inference suits a single machine and models that fit comfortably beside the database's caches. Run antfly inference run (or an operator InferencePool) and point inference.api_url at it when:

• models are large enough to crowd the database's memory envelope
• you want to scale or restart GPU capacity independently of storage
• several database nodes should share one pool of loaded models

Query-over-backfill priority applies within one database process. Admission and memory limits apply within one inference process.

When Enrichment Falls Behind#

An embeddings index reports its own backlog. Inspect it while enrichment is pending:

curl http://127.0.0.1:8080/db/v1/tables/TABLE/indexes/INDEX

The enrichment_runtime object carries sequence progress, retry and failure counters, the active batch, and worker lifecycle state. worker_started says whether the background worker is running. stalled is the narrower condition: work is pending, no retry or terminal failure already explains the backlog, and either no worker is running or the active batch has passed its deadline without progress. stall_reason says which (worker_missing, model_loading, embedding_overdue, or publishing_overdue).

For keys that have reached the durable repair queues, ask the bounded operator endpoint:

curl -X POST http://127.0.0.1:8080/db/v1/tables/TABLE/repair/issues \
  -H 'Content-Type: application/json' \
  -d '{"target":"artifact","limit":100}'

It lists durable repair issues, not every document in the pending tail. Pending work with no listed issue means the cause is a worker lifecycle problem or inference that is still retrying; the runtime counters and server logs tell those apart.

Use Agent Skills#

Everything above is also encoded in the Antfly skill, so a coding agent can execute this guide for you:

npx skills add antflydb/antfly-skills

Then prompt it with the outcome, for example "Set up local embeddings and reranking with Antfly Inference and wire them into my table's index", and use this page to judge the result.

Next Steps#

• Local Model Compatibility: which artifacts load, which are rejected, and why

• Multimodal Embeddings: the ClipClap embedder on an image index

• Tune Hybrid Search: the reranker in a query pipeline

• Inference API reference: the HTTP surface

• Typed Decisions: named choice, ordinal score, and Boolean questions through HTTP and C, Rust, Go, Python, or TypeScript embedded inference