clap-htsat-unfused

About CLAP#

CLAP (Contrastive Language-Audio Pre-training) is a multimodal model developed by LAION-AI that learns audio concepts from natural language supervision. It creates a joint embedding space where both text and audio can be represented as vectors, enabling powerful cross-modal search and retrieval.

This model uses the HTSAT-unfused architecture (Hierarchical Token-Semantic Audio Transformer without feature fusion).

CLAP embeddings allow you to search audio using natural language queries, or find similar audio based on sound content - all through the same vector space.

Capabilities#

Text Embeddings: Convert text descriptions into vectors
Audio Embeddings: Convert audio clips into vectors in the same space as text
Cross-Modal Retrieval: Search audio with text queries, or text with audio
Zero-Shot Classification: Classify sounds into categories without training

Use Cases#

Use Case	Description
Audio Search	Find audio clips using natural language queries like "a dog barking in the rain"
Music Discovery	Search music libraries by describing mood, instruments, or genre
Sound Classification	Categorize environmental sounds, speech, or music without labeled data
Audio RAG	Build retrieval-augmented generation systems with audio content

Antfly Inference API Usage#

Text Embedding#

Generate embeddings for text descriptions:

curl -X POST "http://localhost:8080/ai/v1/embed" \
-H "Content-Type: application/json" \
-d '{
"model": "clap-htsat-unfused",
"input": [
  {"type": "text", "text": "a dog barking loudly"},
  {"type": "text", "text": "jazz piano playing softly"}
]
}'

import requests

response = requests.post(
  "http://localhost:8080/ai/v1/embed",
  json={
      "model": "clap-htsat-unfused",
      "input": [
          {"type": "text", "text": "a dog barking loudly"},
          {"type": "text", "text": "jazz piano playing softly"}
      ]
  }
)

embeddings = response.json()["embeddings"]

const response = await fetch("http://localhost:8080/ai/v1/embed", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
  model: "clap-htsat-unfused",
  input: [
    { type: "text", text: "a dog barking loudly" },
    { type: "text", text: "jazz piano playing softly" }
  ]
})
});

const { embeddings } = await response.json();

Audio Embedding (Remote URL)#

Generate embeddings for audio from URLs:

curl -X POST "http://localhost:8080/ai/v1/embed" \
-H "Content-Type: application/json" \
-d '{
"model": "clap-htsat-unfused",
"input": [
  {"type": "audio_url", "audio_url": {"url": "https://example.com/dog-bark.wav"}}
]
}'

import requests

response = requests.post(
  "http://localhost:8080/ai/v1/embed",
  json={
      "model": "clap-htsat-unfused",
      "input": [
          {"type": "audio_url", "audio_url": {"url": "https://example.com/dog-bark.wav"}}
      ]
  }
)

embeddings = response.json()["embeddings"]

const response = await fetch("http://localhost:8080/ai/v1/embed", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
  model: "clap-htsat-unfused",
  input: [
    { type: "audio_url", audio_url: { url: "https://example.com/dog-bark.wav" } }
  ]
})
});

const { embeddings } = await response.json();

Audio Embedding (Base64)#

Generate embeddings for audio using base64 data URIs:

import requests
import base64

# Read and encode an audio file
with open("dog-bark.wav", "rb") as f:
  audio_data = base64.b64encode(f.read()).decode()

response = requests.post(
  "http://localhost:8080/ai/v1/embed",
  json={
      "model": "clap-htsat-unfused",
      "input": [
          {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_data}"}}
      ]
  }
)

embeddings = response.json()["embeddings"]

import { readFileSync } from "fs";

// Read and encode an audio file
const audioData = readFileSync("dog-bark.wav").toString("base64");

const response = await fetch("http://localhost:8080/ai/v1/embed", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
  model: "clap-htsat-unfused",
  input: [
    { type: "audio_url", audio_url: { url: `data:audio/wav;base64,${audioData}` } }
  ]
})
});

const { embeddings } = await response.json();

Mixed Text and Audio#

Embed both text and audio in a single request - useful for building cross-modal search indexes:

import requests

response = requests.post(
  "http://localhost:8080/ai/v1/embed",
  json={
      "model": "clap-htsat-unfused",
      "input": [
          {"type": "text", "text": "a dog barking loudly"},
          {"type": "audio_url", "audio_url": {"url": "https://example.com/dog-bark.wav"}},
          {"type": "text", "text": "birds chirping at dawn"},
          {"type": "audio_url", "audio_url": {"url": "https://example.com/birds.wav"}}
      ]
  }
)

# Returns 4 embeddings in the same order as input
embeddings = response.json()["embeddings"]

Supported Audio Formats#

CLAP in Antfly Inference supports multiple ways to provide audio:

Format	Example	Description
HTTP/HTTPS	`https://example.com/audio.wav`	Remote audio from the web
Data URI	`data:audio/wav;base64,...`	Base64-encoded audio
Local file	`file:///path/to/audio.wav`	Local filesystem paths
S3	`s3://endpoint/bucket/key`	Audio stored in S3-compatible storage

Supported audio codecs: WAV, MP3, FLAC, OGG.

Remote URL fetching includes SSRF protection - private IP ranges are blocked by default.

Limitations#

Review these limitations before production deployment.

Based on the LAION CLAP paper:

Audio duration: Optimized for clips up to ~10 seconds; longer audio is truncated
Language bias: Primarily trained on English text descriptions
Training data bias: Skewed toward common environmental sounds and music; may underperform on niche or domain-specific audio
Fine-grained audio: May struggle with subtle distinctions between similar sounds (e.g., different engine types)
Speech content: Captures acoustic properties of speech rather than linguistic content

Antfly Configuration#

To use CLAP as the default embedder in your Antfly configuration:

embedder:
  provider: antfly
  model: clap-htsat-unfused

For tables that store audio content:

tables:
  audio:
    indexes:
      embedding:
        type: embedding
        embedder:
          provider: antfly
          model: clap-htsat-unfused
        fields:
          - audio_url
          - description

Tag	Status
latest	Available
v1.0.0	Latest

About CLAP#

Capabilities#

Use Cases#

Antfly Inference API Usage#

Text Embedding#

Audio Embedding (Remote URL)#

Audio Embedding (Base64)#

Mixed Text and Audio#

Supported Audio Formats#

Limitations#

Antfly Configuration#

Versions

Usage

Pull the model

Start Antfly Inference

Use in configuration