Index Recorded Calls and Podcasts

The Result#

Each document in the table points at a recording. Antfly transcribes it once, stores the transcript as a durable artifact with per-phrase timing, cuts it into chunks that carry _start_time_ms and _end_time_ms, and indexes those chunks. A search hit is a passage plus the second it was spoken, so a client can deep-link into the player. The hit below is the first of responses[0].hits.hits, trimmed.

curl -X POST http://127.0.0.1:8080/db/v1/tables/calls/query \
  -H "Content-Type: application/json" \
  -d '{"full_text_search":{"field":"text","match":"billing migration"},"hierarchy":{},"limit":3}'

{
  "_id": "af1:chunk:Y2FsbC0yMDI2LTA5LTEy:ZG9jdW1lbnRfY2h1bmtzX3Yx:0:unit:YXVkaW86MDAwMDAx",
  "_score": 2.1465,
  "hierarchy": {"level": "chunk", "parent_doc_key": "call-2026-09-12", "parent_unit_id": "audio:000001", "...": "..."},
  "_source": {
    "_parent_doc_key": "call-2026-09-12",
    "_parent_unit_id": "audio:000001",
    "_chunk_id": 0,
    "_start_offset": 0,
    "_end_offset": 256,
    "_start_time_ms": 0,
    "_end_time_ms": 20525,
    "text": "Okay, let's get started. First item is the launch date. Marketing wants the 12th, but engineering is asking for one more week to finish the billing migration. I think we should hold the date and ship billing behind a feature flag. Second item, the customer"
  }
}

Before You Start#

Antfly running in standalone mode with a Whisper checkpoint pulled:

antfly inference pull openai/whisper-small --tasks transcribe
antfly inference list

whisper-small is the smallest checkpoint that is reliably accurate on meeting audio; whisper-tiny is faster and fine for clean dictation but drops words in cross-talk. Any Whisper checkpoint on the Local Model Compatibility page works, and the openai and vertex providers work without a local model.

Recordings arrive as http://, https://, s3://, or data: URLs. The runtime decodes WAV, FLAC, MP3, AAC-LC in ADTS/M4A/MP4 (iPhone and macOS voice memos, most exported meeting recordings), ALAC, and Opus or Vorbis in Ogg or WebM/Matroska (browser recorders, Zoom, OBS) natively; the Opus decoder reproduces the libopus reference output on the RFC 8251 test vectors, the Vorbis decoder matches libvorbis on the Xiph test vectors, and the AAC decoder matches ffmpeg on the checked-in corpus. AIFF, CAF, and AU containers are read as well. A WebM or Matroska recording keeps its timeline: if the recorder was paused, or its audio track starts after its video, that silence is preserved, so segment offsets still match where a player would seek. A recording served as video (video/webm, video/mp4, a .mkv or .mov file) is transcribed from its audio track, so a screen recording or a Zoom export needs no conversion first. A recording may be up to 128 MiB by default (about one hour of 128 kbps MP3), and length is not otherwise limited: audio over 30 seconds is transcribed in windows cut at pauses, with each window conditioned on the previous one.

Build It#

1. Create the Table With a Transcriber Enrichment#

The transcriber shorthand on an asset enrichment declares that the field holds a recording URL and names the speech-to-text provider. A chunk enrichment consumes the transcript artifact, and the full-text index indexes the chunks:

curl -X POST http://127.0.0.1:8080/db/v1/tables/calls \
  -H "Content-Type: application/json" \
  -d '{
    "num_shards": 1,
    "indexes": {
      "call_text": {
        "type": "full_text",
        "field": "text",
        "artifact_name": "document_chunks_v1",
        "enrichments": [
          {
            "name": "document_units_v1",
            "kind": "asset",
            "field": "recording_url",
            "transcriber": {"provider": "antfly", "model": "openai/whisper-small", "language_code": "en"}
          },
          {
            "name": "document_chunks_v1",
            "kind": "chunk",
            "field": "text",
            "source_artifact_name": "document_units_v1",
            "chunk_size": 256,
            "full_text_index": true
          }
        ]
      }
    }
  }'

transcriber expands to a document_extraction producer whose audio route calls the provider; the stored index config shows the expanded producer_json. Omit language_code to let Whisper detect the language from the first window. Fields that a remote provider needs (api_key, api_url, project_id) go in the same object; see the provider notes after the steps. A table that already uses document_extraction for PDFs can add "transcription": {"enabled": true, "config": {...}} to that producer's config instead, and audio documents take the transcription route while PDFs keep theirs.

2. Write a Recording#

Write the document with the URL; the transcript is produced by the enrichment worker after the write returns. Remote downloads follow the remote_content allowlist, which blocks private and loopback addresses by default, so a local file is best sent as a data:audio/mpeg;base64,... URL. sync_level: "enrichments" makes the write wait until the transcript artifact exists, which is convenient for a script and too slow for a bulk import.

curl -X POST http://127.0.0.1:8080/db/v1/tables/calls/batch \
  -H "Content-Type: application/json" \
  -d '{"inserts":{"call-2026-09-12":{"title":"Launch sync","recording_url":"https://recordings.example.com/2026-09-12-launch-sync.mp3"}},"sync_level":"enrichments"}'

Read the artifact manifest back to confirm the recording took the audio route and the transcript was cut into chunks:

curl -s "http://127.0.0.1:8080/db/v1/tables/calls/documents/call-2026-09-12/artifacts/document_units_v1"

{"document_id": "call-2026-09-12", "artifact_name": "document_units_v1", "route_type": "audio", "unit_count": 1, "chunk_count": 3, "merge_status": "converged", "...": "..."}

A route_type of error with last_error_code names what went wrong; PrivateIpBlocked means the URL pointed at an address the remote_content policy refuses. The unit record behind the manifest stores the transcript text, the provider's transcript_confidence when it reports one, and provenance.transcript_spans, one entry per sentence with char_start, char_end, start_ms, and end_ms. Phrase boundaries come from the model's own timestamps; the sentence splits inside a phrase are interpolated from word lengths, so a chunk's time is exact at phrase edges and approximate within a phrase. The transcript is stored once and only recomputed when the recording bytes or the transcriber config change.

3. Search With Timestamps#

Chunk hits include the timing fields, so a result can open the recording at the right second:

curl -X POST http://127.0.0.1:8080/db/v1/tables/calls/query \
  -H "Content-Type: application/json" \
  -d '{"full_text_search":{"field":"text","match":"hiring"},"hierarchy":{},"limit":3}'

{"_chunk_id": 1, "_start_offset": 256, "_end_offset": 512, "_start_time_ms": 14390, "_end_time_ms": 32418, "text": " from the pilot reported that search results were slow on large tables. We traced it to a missing index on the created a column, and the fix has already merged. Last thing, hiring. We have two offers out for the platform team and both candidates are decidi"}

The chunk is cut on a character boundary, so it can start or end mid-word; its time range runs from the first sentence it overlaps (14390) to the end of the last (32418), which is what a player needs. hierarchy: {} returns chunk-level hits with their _source; without it the query returns the parent document. For semantic search, add an embedding enrichment over document_chunks_v1 and an embeddings index exactly as in Artifact Indexes; the chunk fields are the same.

Provider Notes#

timestamps defaults to true. Setting it to false tells vertex not to request word offsets, which leaves the transcript as one untimed unit; the antfly and openai routes always return their phrase timing.

Local Speaker Labels#

With "diarization": true on the antfly provider (or on /ai/v1/transcribe), each phrase is labelled SPEAKER_00, SPEAKER_01, ... in order of first appearance, and the response lists the speakers found. The labels come from a speaker verification model run in-process: 3 s windows every 1.5 s over the spoken parts of the recording are embedded with 3D-Speaker's CAM++, the embeddings are clustered by voice similarity, each word takes the speaker of the window nearest to it, and a phrase whose words change speaker is split at the change. So a turn change that Whisper folded into one phrase still comes out as two segments. Pull the model once:

antfly inference pull csukuangfj/speaker-embedding-models:3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx

Without it, a diarization request fails with SPEAKER_MODEL_UNAVAILABLE rather than silently returning an unlabelled transcript. Diarization adds roughly a tenth of a second per phrase on CPU. A transcript produced as text (rather than JSON) keeps the turns as SPEAKER_00: ... lines; the JSON form carries speaker on each segment and a speakers list. Labels are per recording: SPEAKER_00 in one file is not the same person as SPEAKER_00 in another.

As an enrichment, the labels are durable. Each transcript phrase keeps the speaker who said it, and every chunk cut from a phrase carries _speaker alongside _start_time_ms and _end_time_ms, so a search hit says who was talking. A chunk that straddles a turn carries no label rather than crediting the wrong person. Providers that name speakers their own way, such as Vertex, are renumbered into the same SPEAKER_NN order.

Tradeoffs#

The decision is how the transcript is chunked, because a chunk is what search returns and what an embedding represents. chunk_size counts characters of transcript for the default chunker; 256 is about fifteen seconds of speech and gives one idea per hit with a tight time range, while 1024 keeps a whole topic together for an answer agent at the cost of a range that spans a minute or more. Chunks are cut on text boundaries, not silence, so a chunk can straddle a topic change; the timing on each chunk spans from the first sentence it overlaps to the last, which is what a player needs even when the cut is imperfect.

The other decision is where transcription runs. The local antfly provider keeps audio on the machine and costs decoder time: whisper-small transcribes a minute of audio in about nine seconds on an M-series laptop and whisper-tiny in about one, so a one hour call costs a worker about nine minutes with small and about a minute with tiny. Local transcription overlaps two recordings per worker: the model runs one at a time because its state is shared, but one recording's decoding and feature extraction proceed while the model is busy with the other. The openai and vertex routes fan out up to eight recordings per batch. A slow first pass is paid once per recording.

Use Agent Skills#

Everything above is also encoded in the Antfly skill, so a coding agent can execute this guide for you:

npx skills add antflydb/antfly-skills

Then prompt it with the outcome, for example "Create an Antfly table for our recorded customer calls that transcribes each recording with Whisper and lets me search for a phrase and jump to that moment in the recording", and use this page to judge the result.

Next Steps#

• Build Voice Dictation for the request-time /ai/v1/transcribe and /ai/v1/dictate endpoints, including streaming sessions.
• Artifact Indexes to add chunk embeddings and hybrid search over the transcripts.
• Document Engine for how artifacts, units, and hierarchy queries fit together.