Overview#

AntflyDB supports multimodal embeddings, allowing you to process and search not just text, but also images, PDFs, and other remote content. This is achieved through:

1. Schema annotations that mark fields as links to remote content
2. Template-based processing using Handlebars helpers to fetch and process remote content
3. Vision-language models that can understand and describe visual content before generating embeddings

How It Works#

1. Schema-Based Link Annotations#

Mark fields in your table schema as links using the x-antfly-types extension:

{
  "properties": {
    "title": {
      "type": "string"
    },
    "image_url": {
      "type": "string",
      "x-antfly-types": ["link"]
    },
    "pdf_url": {
      "type": "string",
      "x-antfly-types": ["link"]
    }
  }
}

Fields marked as link type will be automatically processed during indexing. Supported URL schemes:

• HTTP/HTTPS URLs: http:// or https://
• S3 URLs: s3://
• File URLs: file://

2. Template Helpers for Remote Content#

AntflyDB provides Handlebars helpers to process remote content in index templates:

• {{remoteMedia url="..."}} - Downloads media and prepares it as model inference input
• {{remotePDF url="..."}} - Deprecated. Downloads and extracts text from born-digital PDFs
• {{remoteText url="..."}} - Downloads and preserves text content (HTML, markdown, etc.)

These helpers automatically:

• Download the content with security limits (100MB max, 30s timeout)
• Block private IPs for security
• Process images (resize, convert to data URIs)
• Extract text from PDFs
• Handle errors gracefully

Template helpers run while a template is rendered. They do not persist PDF page units or make OCR output independently reprocessable. For durable PDF ingestion, configure a document_extraction asset producer on the URL field:

{
  "name": "document_units_v1",
  "kind": "asset",
  "field": "url",
  "content_type": "application/json",
  "producer_json": "{\"type\":\"document_extraction\",\"config\":{}}"
}

document_extraction downloads the URL server-side and persists one unit per PDF page. Pages with zero native text are rendered at 150 DPI and sent to the local antflydb/Florence-2-base reader by default. Install it explicitly with antfly inference pull antflydb/Florence-2-base; Antfly does not download it automatically. If it is absent, the page is stored with extraction_status: "failed_ocr", ocr_used: false, and a ModelNotFound warning. Install the model and explicitly reprocess the artifact to retry. Set ocr_fallback to false or ocr.enabled to false to disable this fallback.

The built-in parser targets unencrypted, born-digital PDFs. For a broader file-ingestion workflow, including formats the built-in parser does not support, use the docsaf example.

Creating Multimodal Indexes#

To create an index that processes images or other remote content, you need to:

1. Define a schema with link-annotated fields
2. Create an index with a template that uses remote helpers
3. Configure a summarizer (for vision models) and embedder

Example: Image Search with Schema Annotations#

First, create a table with a schema that marks the image field as a link:

antfly table create --table product_catalog \
  --schema '{
    "document_schemas": {
      "product": {
        "schema": {
          "properties": {
            "name": {
              "type": "string"
            },
            "image_url": {
              "type": "string",
              "x-antfly-types": ["link"]
            },
            "description": {
              "type": "string"
            }
          }
        }
      }
    }
  }'

Then create an index with a template that processes the image:

antfly index create --table product_catalog \
  --index visual_search \
  --type embeddings \
  --coverage-policy partial \
  --template '{{name}} {{description}} {{remoteMedia url=image_url}}' \
  --dimension 384 \
  --embedder '{
    "provider": "ollama",
    "model": "all-minilm",
    "url": "http://localhost:11434"
  }' \
  --summarizer '{
    "provider": "ollama",
    "model": "llava",
    "url": "http://localhost:11434"
  }'

Because image fields are commonly optional, partial treats missing images as intentional skips while still waiting for every product to settle. Before running semantic image searches, wait for the index explicitly:

antfly index wait --table product_catalog \
  --index visual_search \
  --until complete \
  --timeout 20m

In this configuration:

• Schema annotation: x-antfly-types: ["link"] tells AntflyDB to process this field as a remote link
• Template: {{remoteMedia url=image_url}} downloads the image and converts it to a format the vision model can process
• --summarizer: LLaVA vision model analyzes the image and generates a description
• --embedder: all-minilm creates searchable embeddings from the combined text and image description

Legacy example: PDF Document Search with remotePDF#

This compatibility example uses the deprecated remotePDF helper. Prefer document_extraction for durable ingestion.

antfly table create --table documents \
  --schema '{
    "document_schemas": {
      "paper": {
        "schema": {
          "properties": {
            "title": {"type": "string"},
            "pdf_url": {
              "type": "string",
              "x-antfly-types": ["link"]
            }
          }
        }
      }
    }
  }'

antfly index create --table documents \
  --index pdf_content \
  --type embeddings \
  --template '{{title}} {{remotePDF url=pdf_url}}' \
  --dimension 384 \
  --embedder '{
    "provider": "ollama",
    "model": "all-minilm",
    "url": "http://localhost:11434"
  }'

Using Native Multimodal Embedders#

Some models like Gemini support native multimodal embeddings. The template still uses the helpers, but the model processes both text and images directly:

antfly index create --table product_catalog \
  --index gemini_visual \
  --type embeddings \
  --template '{{name}} {{remoteMedia url=image_url}}' \
  --dimension 768 \
  --embedder '{
    "provider": "gemini",
    "model": "text-embedding-004"
  }'

How It Works#

When you insert a document with link-annotated fields:

1. Schema Detection: AntflyDB identifies fields marked with x-antfly-types: ["link"] in the schema
2. Link Processing: During indexing, the template is rendered with document data
3. Remote Content Fetching: Template helpers (like {{remoteMedia}}) download and process the remote content
4. Summarization (if configured): Vision models analyze images and generate textual descriptions
5. Embedding Generation: The processed content (text + image descriptions) is converted to vector embeddings
6. Indexing: The embeddings are stored in the vector index for similarity search

Complete Example: Building an Image Search System#

Step 1: Create the table with schema#

antfly table create --table product_catalog \
  --schema '{
    "document_schemas": {
      "product": {
        "schema": {
          "properties": {
            "name": {"type": "string"},
            "description": {"type": "string"},
            "image_url": {
              "type": "string",
              "x-antfly-types": ["link"]
            },
            "price": {"type": "number"},
            "category": {"type": "string"}
          },
          "required": ["name", "image_url"]
        }
      }
    }
  }'

Step 2: Create the multimodal index#

antfly index create --table product_catalog \
  --index visual_search \
  --type embeddings \
  --template '{{name}} {{description}} {{remoteMedia url=image_url}}' \
  --dimension 384 \
  --embedder '{
    "provider": "ollama",
    "model": "all-minilm",
    "url": "http://localhost:11434"
  }' \
  --summarizer '{
    "provider": "ollama",
    "model": "llava",
    "url": "http://localhost:11434"
  }'

Step 3: Insert products with images#

antfly insert --table product_catalog \
  --data '{
    "_type": "product",
    "id": "SKU-001",
    "name": "Vintage Leather Jacket",
    "description": "Classic style with modern comfort",
    "image_url": "https://store.example.com/images/leather-jacket.jpg",
    "price": 299.99,
    "category": "clothing"
  }'

Step 4: Search for similar products#

antfly query --table product_catalog \
  --semantic-search "brown leather jacket with zipper" \
  --indexes visual_search \
  --limit 10

Best Practices#

1. Use Schema Annotations:

   • Always mark link fields with x-antfly-types: ["link"] for automatic processing
   • Define document schemas for type safety and validation
   • Use nested schemas for complex document structures (tested and supported)

2. Template Design:

   • Combine text fields and remote content in templates: {{name}} {{remoteMedia url=image_url}}
   • For PDFs, use document_extraction; {{remotePDF}} is deprecated compatibility behavior
   • Use {{remoteText}} for HTML articles or other text content
   • Templates work with or without schemas for backward compatibility

3. Model Selection:

   • Use vision-language models like LLaVA for detailed image understanding
   • Use Gemini for native multimodal support
   • Consider model size vs. accuracy tradeoffs
   • Local models (Ollama) are better for high-volume processing

4. Security and Performance:

   • Remote content is automatically limited to 100MB max download size
   • 30-second timeout prevents hanging on slow servers
   • Private IPs are blocked for security
   • Images are automatically resized (max 2048px dimension)
   • Failed downloads are handled gracefully without blocking indexing

5. Error Handling:

   • Missing or broken links won't prevent document indexing
   • Custom prompts can be used for specialized summarization tasks
   • Use the _type field to identify document schemas

Supported Content Types#

Images (via {{remoteMedia}}):

• JPEG/JPG
• PNG
• WebP
• Returns Genkit dotprompt media directive for vision models

PDFs (via deprecated {{remotePDF}}):

• Extracts text content from PDF documents
• Optional output="markdown" parameter for formatted output
• Returns plain text (not a directive)

Text Content (via {{remoteText}}):

• HTML articles
• Markdown files
• Plain text
• Preserves content as-is

Advanced Features#

Custom Summarization Prompts#

You can customize how content is summarized using the WithSummarizePrompt option:

customPrompt := `{{this}}

Summarize the above in exactly 5 words.`

summaries, err := summarizer.SummarizeRenderedDocs(ctx, rendered,
    WithSummarizePrompt(customPrompt))

The {{this}} placeholder represents the rendered document content.

Nested Link Fields#

Link fields work with nested document structures:

{
  "metadata": {
    "properties": {
      "thumbnail": {
        "type": "string",
        "x-antfly-types": ["link"]
      }
    }
  }
}

Supported URL Schemes#

• http:// and https:// - Web resources
• s3:// - AWS S3 objects
• file:// - Local filesystem (with security restrictions)

Future Enhancements#

AntflyDB's multimodal capabilities are continuously expanding. Planned features include:

• Audio file support with speech-to-text
• Video frame extraction and indexing
• Support for additional embedding models like ImageBind
• Additional template helpers for specialized content types

Multimodal Search Queries#

In addition to indexing multimodal content, AntflyDB supports multimodal search queries. This allows you to search using images, PDFs, or other content types - not just text.

Using embedding_template for Query-Time Processing#

The embedding_template field in query requests lets you specify how the semantic_search value should be processed before embedding. The template has access to this which contains the semantic_search string.

Available helpers:

• {{remoteMedia url=this}} - Fetches and embeds remote images
• {{remotePDF url=this}} - Deprecated. Fetches and extracts content from PDFs
• {{remoteText url=this}} - Fetches remote text content
• {{media url=this}} - Embeds inline data URIs (base64 images)

Example: Search by Image URL#

Search for similar products using an image URL:

curl -X POST http://localhost:8080/v1/query \
  -H "Content-Type: application/json" \
  -d '{
    "table": "product_catalog",
    "semantic_search": "https://example.com/my-image.jpg",
    "embedding_template": "{{remoteMedia url=this}}",
    "indexes": ["visual_search"],
    "limit": 10
  }'

Example: Search by Base64 Image with Vertex AI#

For native multimodal embedding without summarization, use Google Vertex AI's multimodal embedding model with base64-encoded images:

1. Create an index with Vertex multimodal embedder:

antfly index create --table product_catalog \
  --index vertex_multimodal \
  --type embeddings \
  --template '{{name}} {{media url=image_url}}' \
  --dimension 1408 \
  --embedder '{
    "provider": "vertex",
    "model": "multimodalembedding@001",
    "project": "your-gcp-project",
    "location": "us-central1"
  }'

2. Search using a base64-encoded image:

# Encode your search image to base64
IMAGE_BASE64=$(base64 -w 0 search-image.jpg)

curl -X POST http://localhost:8080/v1/query \
  -H "Content-Type: application/json" \
  -d '{
    "table": "product_catalog",
    "semantic_search": "data:image/jpeg;base64,'$IMAGE_BASE64'",
    "embedding_template": "{{media url=this}}",
    "indexes": ["vertex_multimodal"],
    "limit": 10
  }'

Using the Go SDK:

import (
    "encoding/base64"
    "io"
    "os"

    antfly "github.com/antflydb/antfly/go/pkg/sdk"
)

// Read and encode the image
imageFile, _ := os.Open("search-image.jpg")
imageData, _ := io.ReadAll(imageFile)
base64Image := base64.StdEncoding.EncodeToString(imageData)
dataURI := "data:image/jpeg;base64," + base64Image

// Search using the image
results, err := client.Query(ctx, antfly.QueryRequest{
    Table:             "product_catalog",
    SemanticSearch:    dataURI,
    EmbeddingTemplate: "{{media url=this}}",
    Indexes:           []string{"vertex_multimodal"},
    Limit:             10,
})

Using the TypeScript SDK:

import { readFileSync } from 'fs';

// Read and encode the image
const imageData = readFileSync('search-image.jpg');
const base64Image = imageData.toString('base64');
const dataURI = `data:image/jpeg;base64,${base64Image}`;

// Search using the image
const results = await client.query({
  table: 'product_catalog',
  semantic_search: dataURI,
  embedding_template: '{{media url=this}}',
  indexes: ['vertex_multimodal'],
  limit: 10,
});

Legacy example: Search by PDF Content#

The deprecated remotePDF helper remains available for compatibility in query templates:

curl -X POST http://localhost:8080/v1/query \
  -H "Content-Type: application/json" \
  -d '{
    "table": "documents",
    "semantic_search": "https://example.com/reference-paper.pdf",
    "embedding_template": "{{remotePDF url=this}}",
    "indexes": ["pdf_content"],
    "limit": 10
  }'

Combining Text and Multimodal Content#

You can mix text with multimodal content in your search:

curl -X POST http://localhost:8080/v1/query \
  -H "Content-Type: application/json" \
  -d '{
    "table": "product_catalog",
    "semantic_search": "https://example.com/red-dress.jpg",
    "embedding_template": "Find products similar to this image: {{remoteMedia url=this}}",
    "indexes": ["visual_search"],
    "limit": 10
  }'

Supported Multimodal Embedding Providers#

For providers that don't natively support multimodal embeddings, use the --summarizer option to convert images to text descriptions before embedding.