Resumable Media Enrichment
Plan media once and produce independent embeddings, transcripts, and supervised predictions.
Plan and Output Identity
Model-aware /chunk requests name their boundary model explicitly. Only that model, the source version, and the chunk policy determine the plan and parent chunk IDs. Each named output carries its own model identity, revision, and recipe ID. Updating an independent output preserves the parent plan; changing the boundary model or policy creates a new plan.
{
"input": {
"type": "source",
"uri": "https://recordings.example.com/call.mp4",
"mime_type": "video/mp4",
"version": "\"immutable-object-etag\"",
"version_kind": "strong_etag",
"length": 10485760
},
"outputs": {
"boundary_model": "embedding",
"targets": [
{"name": "embedding", "kind": "embedding", "model": "google/embeddinggemma-2:bf16-safetensors-bundle-v1", "dimensions": 768},
{"name": "speech", "kind": "transcribe", "model": "openai/whisper-small", "diarize": true, "required": false}
]
},
"embedding": {"model": "google/embeddinggemma-2:bf16-safetensors-bundle-v1", "dimensions": 768, "allow_partial": true},
"config": {"max_chunks": 64},
"allow_partial": true,
"stream": true
}
The legacy embedding option remains supported and corresponds to the output named embedding. With named outputs, the explicit boundary wins. A transcribe-only request omits embedding, names its transcription target as boundary_model, and uses the transcriber's 30-second planning limit. Smaller consumers split a parent window into results keyed by (parent ordinal, sub_index).
Use model_identity to pin a deployed model generation when reproducibility across model-alias updates is required. Source references require a strong ETag or S3 version ID. Inline base64 media and plain text use the same client and output contract.
Durable Embedded Enrichment
Enrichment runs in bounded pages and commits each page with its cursor, so a restart or a failed output resumes where it stopped instead of decoding the whole source again. Each output retries independently, and healthy outputs stay published while another is repaired. Changing an output's model reprocesses only that output; changing the boundary model or chunk policy creates a new plan.
Declare the shared chunk producer once. A dense enrichment consumes its primary vectors. Each independent output that must be stored has an asset enrichment:
[
{
"name": "media_windows",
"kind": "chunk",
"field": "recording",
"chunker": {
"provider": "antfly",
"embedding": {"model": "google/embeddinggemma-2:bf16-safetensors-bundle-v1", "dimensions": 768},
"outputs": {
"boundary_model": "embedding",
"targets": [
{"name": "embedding", "kind": "embedding", "model": "google/embeddinggemma-2:bf16-safetensors-bundle-v1", "dimensions": 768},
{"name": "speech", "kind": "transcribe", "model": "openai/whisper-small", "required": false}
]
}
}
},
{
"name": "media_vectors",
"kind": "embedding",
"field": "text",
"source_artifact_name": "media_windows",
"expected_dims": 768
},
{
"name": "transcript",
"kind": "asset",
"source_artifact_name": "media_windows",
"producer": {"type": "media_output", "config": {"output_name": "speech"}}
},
{
"name": "transcript_chunks",
"kind": "chunk",
"field": "text",
"source_artifact_name": "transcript",
"chunk_size": 512,
"full_text_index": true
}
]
For standalone indexes, the source field may contain a plain HTTP, HTTPS, S3, or GCS URL. The enrichment runtime fetches it under remote_content policy and passes the response MIME type and bytes to the model-aware chunker. Related named-output consumers share the prepared source without another download or media copy. Ordinary passage strings remain text inputs.
Use a versioned source object for large media that should stay on the range-based streaming path. HTTP sources in that form must honor byte ranges and the supplied strong ETag; an origin that only supports full downloads can use a plain URL within the configured download limit. A media_output asset must name an output declared by its upstream model-aware chunk producer.
These are entries in an index's enrichments list. The embeddings index's embedder must match the primary embedding model, task, and dimensions; its source is media_vectors. A full-text index can select transcript_chunks as its source. Model-aware chunk rows always retain descriptors; store_chunks additionally retains their payloads. Output payloads are stored separately from chunk recipes.
A media_output asset has object: "media.output.list", its boundary plan_id, source_fingerprint, and source_version, per-parent output results, searchable text, and transcript_spans. Cached hydration checks that source identity before reusing any output payload. Spans retain UTF-8 byte offsets, exact nanosecond phrase clocks, parent ordinal, sub-index, and speaker index with its scope. Text chunks cut from that asset retain _media.transcript_spans with offsets relative to the text chunk, plus _media.start_ns and _media.end_ns. Cutting through a phrase preserves that phrase's timing envelope; it does not invent word timestamps. Transcript stamping validates offset ordering and uses a single sweep for ordered spans and cuts. Imported unordered offsets retain the general scan path. New diarized outputs and projected transcript spans carry speaker_scope: "source": the same voice keeps its index across parent chunks and output sub-windows for this source and diarization recipe. Older receipts without that marker retain their parent/sub-window scope. Go SDK speaker indices are presence-aware: index 0 identifies the first voice, while an omitted index remains unassigned.
For a new fused media pipeline, replace the recording's transcriber/document_extraction enrichment with the named transcription output and media_output asset above. Text consumers read that asset. This removes the separate extraction decode from the new pipeline. The older transcription shorthand remains available for existing provider configurations and PDF/document extraction workflows.
Speaker Diarization and Turn Boundaries
A named diarize output runs the speaker embedding model without speech recognition. Its speaker_spans contain original-source start_ns, end_ns, and an optional speaker_index. Silence, ambiguous voices, and voices beyond the configured speaker limit remain unassigned. Speaker numbers follow first appearance within the recording; they do not identify people across recordings.
Pull the speaker model once:
antfly inference pull csukuangfj/speaker-embedding-models:3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx
This /ai/v1/chunk request produces audio chunks split at detected speaker transitions, with a maximum duration of two seconds:
{
"input": {"type": "media", "mime_type": "audio/wav", "data": "<base64>"},
"outputs": {
"boundary_model": "voices",
"targets": [{
"name": "voices",
"kind": "diarize",
"model": "csukuangfj/speaker-embedding-models:3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx",
"diarization": {"similarity_threshold": 0.45, "max_speakers": 64}
}]
},
"policy": {"audio_speaker_turns": true, "window_ns": 2000000000}
}
The selected boundary target must enable diarization. A transcription boundary can use diarize: true or a diarization object; an audio embedding boundary can set diarization: {} to select the default speaker model for turn boundaries while retaining its own model limits. Use a separate diarize output for annotations with ordinary model windows. Turn boundaries currently require zero overlap_ns and zero audio_pause_ns. Long turns split further to satisfy model limits. Unassigned intervals remain part of the source coverage. Homogeneous attributed chunks carry descriptor.coordinates.speaker_index.
Adding a diarization output preserves existing parent IDs. Selecting audio_speaker_turns changes the plan, which includes the boundary speaker model's SHA-256 revision and clustering settings. Each diarized output also hashes those dependencies into its own recipe_id. diarization.model_identity optionally pins the speaker file, including when it accompanies a transcription target. On a transcription target, diarization.model selects the companion speaker model; on a diarize target, use the target's model.
The analysis pass reads three-second windows on a 1.5-second grid, so turn boundaries are approximate at that resolution. Transcription phrases split at word timestamps when words change speaker; text with no usable word alignment stays unattributed when it spans a turn.
Failures and Status
A required output must cover every ordinal. A terminal required failure is retained as failed coverage and repair debt. An optional failure makes its output asset degraded while healthy vectors remain searchable. The asset reports complete, failed, and terminal parent-chunk counts and preserves the provider error on each failed result.
Runtime status exposes cumulative output_counts for embedding, transcription, and classification. They describe accepted output coverage, not every inference retry or the current row count.
Zero-Shot Labels from Stored Vectors
A vector_labels asset consumes a dense embedding enrichment, rather than media. For example:
{
"name": "scene_labels",
"kind": "asset",
"source_artifact_name": "media_vectors",
"producer": {
"type": "vector_labels",
"config": {
"labels": [
{"label": "meeting", "prompt": "people speaking in a meeting"},
{"label": "music", "prompt": "music playing"}
],
"top_k": 2,
"min_score": 0.1
}
}
}
Changing the vocabulary rescores stored vectors and never fetches or decodes media. Scores are similarities, not calibrated probabilities.
Supervised Media Outputs
A target with kind: "classify" names a supervised classifier model. Its vocabulary and preprocessing come from the model bundle, not caller label prompts. The bundle declares media_classifier in its manifest, alongside its classification capability, task, and id2label vocabulary:
{
"media_classifier": {
"input": "waveform",
"score_transform": "softmax",
"sample_rate": 16000,
"normalize_waveform": true,
"max_audio_ns": 10000000000
}
}
Waveform classifiers take one float32 input_values tensor and return float32 logits. Pixel classifiers take pixel_values and also declare width, height, mean, std_dev, and resample in the profile. Scores use the declared softmax or sigmoid transform.
Serverless
Serverless deployments run the same media pass. Chunk descriptors, vectors, independent outputs, and transcript spans are stored in the derived document; unfinished outputs keep the document pending and retry up to three times per output recipe, and terminal failures remain visible in media_outputs.
Checkpoint, cache, and repair internals are in the media pass runtime notes.
Composed queries
A query template such as astronaut voice recording {{media url=this}} is one ordered input group: the text and media produce one query vector. Index queries use the model's retrieval-query task and the request's cancellation and deadline controls. They do not require enabling the public inference endpoint. Independent media items used for index backfill remain separate batch rows.