clip-vit-base-patch32

About CLIP#

CLIP (Contrastive Language-Image Pre-training) is a multimodal model developed by OpenAI that learns visual concepts from natural language supervision. It creates a joint embedding space where both text and images can be represented as vectors, enabling powerful cross-modal search and retrieval.

This model uses the ViT-B/32 architecture (Vision Transformer base with 32x32 patches).

CLIP embeddings allow you to search images using natural language queries, or find similar images based on visual content - all through the same vector space.

Capabilities#

Text Embeddings: Convert text descriptions into vectors
Image Embeddings: Convert images into vectors in the same space as text
Cross-Modal Retrieval: Search images with text queries, or text with images
Zero-Shot Classification: Classify images into categories without training

Use Cases#

Use Case	Description
Image Search	Find images using natural language queries like "a sunset over mountains"
Visual Similarity	Find images that look similar to a reference image
Content Moderation	Detect inappropriate content by comparing against text descriptions
Multi-modal RAG	Build retrieval-augmented generation systems with both text and images

Antfly Inference API Usage#

Text Embedding#

Generate embeddings for text descriptions:

curl -X POST "http://localhost:8080/ai/v1/embed" \
-H "Content-Type: application/json" \
-d '{
"model": "clip-vit-base-patch32",
"input": [
  {"type": "text", "text": "a photo of a cat"},
  {"type": "text", "text": "a dog playing in the park"}
]
}'

import requests

response = requests.post(
  "http://localhost:8080/ai/v1/embed",
  json={
      "model": "clip-vit-base-patch32",
      "input": [
          {"type": "text", "text": "a photo of a cat"},
          {"type": "text", "text": "a dog playing in the park"}
      ]
  }
)

embeddings = response.json()["embeddings"]

const response = await fetch("http://localhost:8080/ai/v1/embed", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
  model: "clip-vit-base-patch32",
  input: [
    { type: "text", text: "a photo of a cat" },
    { type: "text", text: "a dog playing in the park" }
  ]
})
});

const { embeddings } = await response.json();

Image Embedding (Remote URL)#

Generate embeddings for images from URLs:

curl -X POST "http://localhost:8080/ai/v1/embed" \
-H "Content-Type: application/json" \
-d '{
"model": "clip-vit-base-patch32",
"input": [
  {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}}
]
}'

import requests

response = requests.post(
  "http://localhost:8080/ai/v1/embed",
  json={
      "model": "clip-vit-base-patch32",
      "input": [
          {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}}
      ]
  }
)

embeddings = response.json()["embeddings"]

const response = await fetch("http://localhost:8080/ai/v1/embed", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
  model: "clip-vit-base-patch32",
  input: [
    { type: "image_url", image_url: { url: "https://example.com/cat.jpg" } }
  ]
})
});

const { embeddings } = await response.json();

Image Embedding (Base64)#

Generate embeddings for images using base64 data URIs:

import requests
import base64

# Read and encode an image
with open("cat.jpg", "rb") as f:
  image_data = base64.b64encode(f.read()).decode()

response = requests.post(
  "http://localhost:8080/ai/v1/embed",
  json={
      "model": "clip-vit-base-patch32",
      "input": [
          {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}}
      ]
  }
)

embeddings = response.json()["embeddings"]

import { readFileSync } from "fs";

// Read and encode an image
const imageData = readFileSync("cat.jpg").toString("base64");

const response = await fetch("http://localhost:8080/ai/v1/embed", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
  model: "clip-vit-base-patch32",
  input: [
    { type: "image_url", image_url: { url: `data:image/jpeg;base64,${imageData}` } }
  ]
})
});

const { embeddings } = await response.json();

Mixed Text and Images#

Embed both text and images in a single request - useful for building cross-modal search indexes:

import requests

response = requests.post(
  "http://localhost:8080/ai/v1/embed",
  json={
      "model": "clip-vit-base-patch32",
      "input": [
          {"type": "text", "text": "a photo of a cat"},
          {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}},
          {"type": "text", "text": "a dog playing fetch"},
          {"type": "image_url", "image_url": {"url": "https://example.com/dog.jpg"}}
      ]
  }
)

# Returns 4 embeddings in the same order as input
embeddings = response.json()["embeddings"]

Supported Input Formats#

CLIP in Antfly Inference supports multiple ways to provide images:

Format	Example	Description
HTTP/HTTPS	`https://example.com/image.jpg`	Remote images from the web
Data URI	`data:image/png;base64,...`	Base64-encoded images
Local file	`file:///path/to/image.jpg`	Local filesystem paths
S3	`s3://endpoint/bucket/key`	Images stored in S3-compatible storage

Remote URL fetching includes SSRF protection - private IP ranges are blocked by default.

Limitations#

CLIP was designed as a research model. Review these limitations before production deployment.

Based on OpenAI's model card:

Fine-grained classification: Struggles with subtle distinctions (e.g., car models, bird species)
Object counting: Limited ability to count objects in images
Abstract concepts: May not handle novel or abstract visual concepts well
Language bias: Primarily trained on English text - limited multilingual support
Data bias: Training data skews toward developed nations and younger demographics
Not for surveillance: Should not be used for facial recognition or surveillance applications

Antfly Configuration#

To use CLIP as the default embedder in your Antfly configuration:

embedder:
  provider: antfly
  model: clip-vit-base-patch32

For multimodal tables that store images:

tables:
  images:
    indexes:
      embedding:
        type: embedding
        embedder:
          provider: antfly
          model: clip-vit-base-patch32
        fields:
          - image_url
          - description

Tag	Status
latest	Available
v1.0.0	Latest

About CLIP#

Capabilities#

Use Cases#

Antfly Inference API Usage#

Text Embedding#

Image Embedding (Remote URL)#

Image Embedding (Base64)#

Mixed Text and Images#

Supported Input Formats#

Limitations#

Antfly Configuration#

Versions

Usage

Pull the model

Start Antfly Inference

Use in configuration

Available variants (smaller/faster)