Choose the right ingestion shape#

Antfly supports three explicit multimodal workflows:

1. Native multimodal embeddings use the antfly index provider with a locally installed model such as antflydb/clipclap. A template turns a URL into a media part with remoteMedia, and the model embeds that part directly.
2. Application-managed descriptions run OCR, captioning, or transcription before the write. Store the resulting text in a document field and use any managed text embedder supported for index creation.
3. Application-managed vectors create an external: true embeddings index and write vectors through _embeddings. This is the escape hatch for a model or provider that Antfly's managed indexing runtime does not execute.

EmbeddingsIndexConfig deliberately has no summarizer field. Keeping summarization outside the index definition avoids an implicit second provider call on every write and makes retries, cost, and durable intermediate output observable. It also prevents an index configuration from parsing successfully and failing only after ingestion starts.

Native image and audio embeddings#

ClipClap embeds text, images, and audio into one vector space. Install it once:

antfly inference pull antflydb/clipclap:gguf:Q4_K

Restart antfly standalone if the running inference service does not discover the newly installed model.

Define the source field#

Link annotations make the remote-media intent visible in the table schema and available to tooling. The index template still controls when the URL is fetched.

antfly table create --table product_catalog \
  --schema '{
    "document_schemas": {
      "product": {
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "description": {"type": "string"},
            "image_url": {
              "type": "string",
              "x-antfly-types": ["link"]
            }
          }
        }
      }
    }
  }'

Create the managed index#

antfly index create --table product_catalog \
  --index visual_search \
  --type embeddings \
  --coverage-policy partial \
  --template '{{#if image_url}}{{remoteMedia url=image_url}}{{/if}}' \
  --embedder '{
    "provider": "antfly",
    "model": "antflydb/clipclap"
  }' \
  --distance-metric cosine

partial treats documents without an image as intentional skips while still requiring every source document to reach a durable outcome. Antfly probes the model during index creation and records its dimension, so omit dimension unless you want creation to verify an expected value.

For remote URLs, configure remote_content with the narrowest required host or object-store allowlist. Remote access is denied by default; private and special IP addresses remain blocked unless explicitly enabled. Data URIs do not require network access, but they are still subject to request and decoded-media limits.

Wait for existing documents to finish indexing before querying:

antfly index wait --table product_catalog \
  --index visual_search \
  --until searchable-artifacts=1 \
  --timeout 20m

Search with text#

Because ClipClap shares a vector space across modalities, ordinary text can retrieve matching images:

antfly query --table product_catalog \
  --semantic-search "brown leather jacket with zipper" \
  --indexes visual_search \
  --fields "name,description,image_url" \
  --limit 10

Search with an image URL#

Use embedding_template to turn the query value into a media part using the same embedding model as the index:

curl -X POST http://127.0.0.1:8080/db/v1/query \
  -H "Content-Type: application/json" \
  -d '{
    "table": "product_catalog",
    "semantic_search": "https://images.example.com/query.jpg",
    "embedding_template": "{{remoteMedia url=this}}",
    "indexes": ["visual_search"],
    "fields": ["name", "description", "image_url"],
    "limit": 10
  }'

For an inline data URI, use {{media url=this}} instead. Avoid logging or persisting large data URIs in query histories.

Precompute descriptions when text retrieval is the goal#

If users search for detailed attributes that a joint embedding model does not capture well, generate a caption or transcript in the application and store it as a normal field:

{
  "_type": "product",
  "name": "Vintage Leather Jacket",
  "image_url": "https://images.example.com/jacket.jpg",
  "media_description": "Brown leather motorcycle jacket with an asymmetric zipper"
}

Then index only the durable text:

antfly index create --table product_catalog \
  --index media_description_search \
  --type embeddings \
  --field media_description \
  --embedder '{
    "provider": "openai",
    "model": "text-embedding-3-small"
  }'

This shape is efficient for repeated rebuilds: captioning runs once, the description can be inspected or corrected, and changing the embedding model does not repeat vision inference.

Bring your own vectors#

Use an external index when the application already calls Gemini, Vertex, Cohere, OpenRouter, or another embedding service that is not executable by managed index creation:

antfly index create --table product_catalog \
  --index application_vectors \
  --type embeddings \
  --external \
  --dimension 768 \
  --distance-metric cosine

Write each vector with the document under the index's _embeddings entry. The application owns provider retries, batching, and vector-space consistency; Antfly owns storage and retrieval. Do not mix vectors from different models or model revisions in one index.

Managed index creation accepts antfly, ollama, openai, and bedrock. Provider names exposed by broader embedding APIs are not automatically valid for managed indexing.

Durable PDF ingestion#

Template helpers are request-time transformations; they do not persist PDF pages or make OCR output independently reprocessable. Use a document_extraction asset producer for durable document ingestion:

{
  "name": "document_units_v1",
  "kind": "asset",
  "field": "url",
  "content_type": "application/json",
  "producer_json": "{\"type\":\"document_extraction\",\"config\":{}}"
}

The built-in parser targets unencrypted, born-digital PDFs. Pages with no native text use the local antflydb/Florence-2-base reader by default. Install it with:

antfly inference pull antflydb/Florence-2-base

If the model is absent, the page records a failed OCR outcome instead of silently disappearing. Install the model and explicitly reprocess the artifact to retry. Set ocr_fallback to false or ocr.enabled to false to disable OCR. For formats outside the built-in parser, use the docsaf example.

remotePDF remains a compatibility helper for template-time extraction, but it is deprecated. Prefer durable extraction for production workflows.

Operational guidance#

• Keep remote_content allowlists narrow and use dedicated read-only object storage credentials.
• Use partial coverage when media fields are optional; use strict when every source document must produce a vector.
• Store durable captions, transcripts, or extracted pages when they are costly to reproduce.
• Use cosine for ClipClap and other models trained for cosine similarity. The database default remains l2_squared, so select the model's metric explicitly when it differs.
• Monitor index status and wait for the required coverage before serving a new index to users.