Local model compatibility

Antfly is a database first and an inference engine second. It enables the model runtime paths needed to turn unstructured data into searchable, ranked, structured knowledge, without claiming compatibility with every artifact published under a model-family name.

Every discovered model reports compatibility in /ai/v1/models. Compatibility is a property of the actual artifact and this Antfly build, not a certification label attached to a repository name. Antfly checks the bundle contract, architecture runtime, required sidecars, GGUF tensor names and encodings, available backends, and resource admission before loading weights.

Compatibility states#

Resource fit is separate from compatibility. Model weights are admitted against stable host and accelerator limits before load; KV-cache and scratch estimates join the same shared reservation ledger for each request. A request that can never fit returns MODEL_RESOURCE_LIMIT; temporary contention returns MODEL_RESOURCE_BUSY. A one-shot live memory-pressure check protects the process without making the published compatibility state flap with momentary system load.

Automatic backend selection may fall back to another compatible backend (for example, from Metal to native when a quant is not implemented on Metal). An explicitly requested backend is strict and never silently falls back.

Start here#

Pull models by their full owner/name reference:

antfly inference pull hf:BAAI/bge-small-en-v1.5 --tasks embed
antfly inference pull hf:nomic-ai/nomic-embed-text-v1.5 --tasks embed
antfly inference pull hf:BAAI/bge-m3 --tasks embed
antfly inference pull hf:Qwen/Qwen3-Embedding-0.6B-GGUF:q8-0-bundle-v1
antfly inference pull hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF:gguf:Q8_0 --tasks rerank
antfly inference pull hf:antflydb/clipclap:gguf:Q4_K --tasks embed
antfly inference pull hf:ggml-org/gemma-4-E4B-it-GGUF:gguf:Q4_0 --tasks generate --projector auto

Gemma 4's automatically selected unified projector enables text, image, and audio generation inputs. Pass --projector none when only text generation is needed.

The Qwen3 0.6B text reranker is publicly downloadable as a single GGUF. It uses the checkpoint's fixed text prompt, final active token, and yes/no score head. This is separate from the Qwen3-VL 2B multimodal reranker, whose Metal serving route requires its exact qualified managed bundle; community VL GGUFs cannot be substituted for that bundle solely because they share a model name or quant.

Antfly treats retrieval roles as model input semantics, not model-name aliases. Semantic searches automatically use RETRIEVAL_QUERY; index and artifact writes use RETRIEVAL_DOCUMENT. Known asymmetric models load a declarative embedding_profile from their trusted bundle or Sentence Transformers metadata. A model that declares task-sensitive inputs but has no complete query/document profile is rejected before its weights are admitted.

Provider-specific overrides are rarely needed. When they are, keep them under the embedder's retrieval object:

{
  "provider": "bedrock",
  "model": "cohere.embed-v4:0",
  "request_format": "cohere_v4",
  "region": "us-east-1",
  "retrieval": {
    "query_input_type": "search_query",
    "document_input_type": "search_document"
  }
}

The older flat query_input_type, document_input_type, and query_instruction fields remain read-compatible but are deprecated.

Thoroughly tested artifacts (documentation certification)#

These particular artifacts were checked for a meaningful result rather than merely a successful response. This is the certification matrix for operators, maintained in the documentation rather than baked into runtime policy. A certified artifact must still pass the same compatibility and resource-fit checks as every other artifact.

BAAI/bge-m3 resolves to the official immutable safetensors revision 84790c1a606f60d06c6932e4ecdd174b466d84ac, because the repository's current main revision no longer publishes model.safetensors. The managed receipt and install-directory identity retain the full safetensors@<commit> variant. The local embeddings API exposes BGE-M3's normalized 1024-dimensional dense vector; its sparse lexical weights and ColBERT token vectors require separate response contracts and are not exposed by the dense embeddings endpoint.

Generation#

Text generation works with the decoder artifacts listed above. Multimodal generation is also available when the bundle contains a compatible Antfly Gemma 3 projector or supported Gemma 4 CLIP projector: Gemma 3 accepts images, while Gemma 4 projector metadata can enable images, audio, or both. The projector is opened only when a request contains media, so the same bundle continues to serve text-only requests without projector execution.

Gemma 4 26B-A4B support is intentionally narrower than dense Gemma 4. The GGUF must match the qualified 30-layer, 128-expert, top-8 Q4_0 layout. Metal supports bounded or resident placement. CUDA requires a 16 GiB resident A4B envelope and an SM89 GPU. Unsupported geometry, streamed CUDA, or unavailable kernels are rejected at load time rather than falling back to host MoE execution. Server deployments must also leave transient host/combined admission headroom for the encoded GGUF during CUDA upload; only the 16 GiB CUDA envelope remains leased after load.

CUDA uses a bounded multithreaded pinned-host load pipeline by default. Clean GGUF pages remain kernel-reclaimable and shared across rolling workers unless --a4b-drop-host-cache-after-load is selected. Production images can install a versioned $MODEL/a4b-cuda-pack-v2 created with antfly-inference a4b-pack; use a4b-pack --verify after copying it and --a4b-prepared-pack required to fail closed if that artifact is absent or stale. The default auto policy warns and uses the canonical GGUF when an optional pack is unusable. This is deliberately a full-model residency contract, not an ngl-style layer split. Cache retention and server startup_strategy: "prefetch" are deployment tools rather than a latency guarantee; hosts must have enough reclaimable memory for their checkpoint and loader working set, and should measure replacement-worker load time on the actual storage topology.

Known incompatible artifacts and families#

Rejected with HTTP 400 before any load. Some of these do not merely fail — they would otherwise crash the server, so blocking them protects every other caller.

A known-incompatible generation request returns HTTP 400:

{
  "error": "INCOMPATIBLE_MODEL",
  "message": "the current decoder path is known to be missing, unsafe, or to produce unusable output"
}

An artifact whose compatibility cannot be determined instead returns UNKNOWN_MODEL_COMPATIBILITY. To try it deliberately:

antfly inference run --allow-unknown-models

The flag does not bypass INCOMPATIBLE_MODEL or resource limits.

Licensing#

Babelscape/rebel-large is CC-BY-NC-SA — non-commercial only, and about 3 GB. Its relation-output parser has unit coverage, but the actual BART inference path is incompatible with 0.2.0. It is not part of any default bundle and must not be deployed commercially. antflydb/gliner2-base-v1 is the recommended extraction model.

Known caveats#

Llama 3.x chat templates. Llama 3.x ships a Jinja chat template Antfly's template engine cannot yet parse. The model loads and generates correctly, but chat requests fall back to raw prompting, which produces continuation-style output instead of a reply. Affected models report "chat_template": false in /ai/v1/models.

Small Gemma 3 models over the chat endpoint. gemma-3-1b answers correctly and stops cleanly through antfly inference generate, but can fail to stop through /ai/v1/chat/completions. Prefer gemma-3-4b or larger for chat.

Gemma 4 artifacts and quants differ. The current ggml-org/gemma-4-E4B-it-GGUF repository publishes BF16, Q4_0, and Q8_0 decoder weights; it does not publish Q4_K_M. Pull the exact Q4_0 variant shown above. Its automatically selected Q8 projector supports both image and audio inputs. Google QAT Q4_0 and tested third-party Q4_1/Q4_K_M variants also produce a clean final channel after the GGUF-over-sidecar structural fix. A tested UD-Q4_K_XL contains IQ4_XS; Metal rejects that quant before generation and automatic selection falls back to native. Keep one quant per directory rather than mixing artifacts from different variants.

Mixed-variant model directories. Pulling a second variant into a directory that already holds another leaves both in place, and different parts of a model can then resolve to different variants. ClipClap in particular fails this way. Keep one variant per directory; if in doubt, delete and re-pull.

Chunk overlap. /chunk returns InvalidChunkOverlap when target_tokens is smaller than overlap_tokens. Set both together when using small chunks.

Not covered here#

Hosted generation providers (OpenAI and Gemini/Vertex), Ollama, and Antfly's generator endpoint are configured separately and are not part of this local-model matrix.

Checking a model before running it#

smoke --inspect-only reports what Antfly makes of a bundle without loading weights:

antfly inference smoke ~/.antfly/inference/models/unsloth/Qwen3-1.7B-GGUF "hi" --inspect-only

gguf path=.../Qwen3-1.7B-UD-Q4_K_XL.gguf architecture=qwen3 tensors=310 metadata=32
  gpt family=qwen3 layers=28 heads=16 kv_heads=8 hidden=2048 head_dim=128 ...
  unsupported_tensor_types=none
  missing_required_tensors=none

missing_required_tensors with a non-zero count means the architecture was recognised but its tensor names could not be mapped — the signature of an unsupported or fused layout.