AntflyAntfly

Inference API#

Antfly inference is a local inference server for GGUF, safetensors, and ONNX models (ONNX graphs are parsed natively, without ONNX Runtime), running on CPU, Metal, or CUDA.

Named inference gateway#

Database nodes also serve a gateway at /ai/v1 independently of local model availability. Select configured targets explicitly with canonical names such as generators/support, deciders/triage, embedders/dense, and readers/ocr. Bare names resolve within the endpoint task; a collision with a local model fails explicitly. Use local/owner/model for an explicit local artifact or connections/name/models/owner/model for an Antfly runtime connection.

Named targets cover generation and chains, typed decisions, embeddings, reranking, chunking, OCR, extraction, rewriting, transcription, speech synthesis and traditional ML. Existing task envelopes and feature schemas are preserved. OpenAI-compatible chat adapters forward tool history, supported structured output, usage and genuine incremental SSE.

Backend feature declarations in inference_capabilities are intersected with adapter support. Unknown support cannot qualify a required feature. Chains validate every candidate before execution, use at most eight total attempts and share one deadline/cancellation chain. Fallback requires an explicit condition on the failed link and is restricted to transient upstream responses. Timeouts require on_timeout; refusals, invalid input, authentication failures and committed responses never switch models. Embedding and decision cross-model failover are disabled.

Session handles are credential-bound and pinned to a physical connection. Multi-replica deployments must preserve gateway/proxy affinity using the returned X-Antfly-Inference-Affinity handle. Unknown sessions fail closed. Task capabilities are independent of catalog freshness and availability; offline upstreams do not remove configured aliases. Request IDs and per-attempt telemetry contain target identities and actual usage only.

What is Antfly inference#

Antfly inference provides local ML inference with an OpenAI-compatible API:

  • Embedding Generation: OpenAI-compatible text embedding endpoints, with dense or sparse embedding payloads
  • Text Chunking: Fixed token-count chunking for text, with fixed time windows for audio
  • Reranking: Relevance re-scoring for search results
  • Extraction: Schema-driven entity, relation, classification, and structured extraction
  • Text Rewriting: Transform text using Seq2Seq models (question generation, query generation, etc.)

Download the latest release at https://antfly.io/docs/downloads

When to Use Antfly inference#

Antfly inference can run standalone or as part of an Antfly cluster:

  • Local model inference without external API dependencies
  • OpenAI-compatible embedding payloads served at /ai/v1/embed and /ai/v1/embeddings
  • Text chunking for RAG pipelines
  • Relevance reranking for improved search quality
  • Centralized model serving across distributed nodes
  • Privacy-preserving ML inference (data never leaves your infrastructure)

Authentication#

The standalone inference listener speaks plaintext HTTP and does not provide built-in authentication. It refuses non-loopback binds by default. Keep the loopback default, or put it behind a trusted proxy that terminates TLS and authenticates every request; only then use --allow-insecure-public-bind to opt into a non-loopback listener. When authentication is enabled, the unified Antfly server requires a valid principal with inference/* read permission for /ai/v1 and /ml/v1. Global */* read or admin grants also satisfy this requirement through the existing wildcard policy.

Features#

Embedding Generation#

  • Models: Embedding models auto-discovered from {models_dir}
  • API: OpenAI-compatible payloads served at /ai/v1/embed and /ai/v1/embeddings
  • Response Shape: Each data[i].embedding is either a dense float vector or a sparse {indices, values} object

Text Chunking#

  • Models: Built-in fixed chunker (token count for text, time windows for audio); no model download needed

Reranking#

  • Model Discovery: Auto-discovers reranker models from {models_dir}
  • Quantization: Automatically uses quantized models if available
  • Input: Pre-rendered text prompts (client handles field extraction)

Create embeddings (alias of /embeddings)#

POST/embed

Alias of /ai/v1/embeddings.

Accepts the OpenAI embeddings request shape and returns the same OpenAI-compatible response envelope. For sparse-capable models, data[i].embedding is a sparse vector object instead of a dense float array. Dense image inputs are header-validated and admitted against the aggregate decoded-pixel budget before model loading. Remote URL byte potential is reserved before fetch; inline sources use their actual encoded size.

Parameters#

NameTypeLocationRequiredDescription
AcceptstringheaderNoSet to application/vnd.antfly.numeric.v1 to receive the values as a binary
frame instead of JSON, which avoids serializing every float as text. Any
other value, or none, returns the JSON body.

Request Body#

Example:

{
  "model_identity": "string",
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/embed" \
  -H "Content-Type: application/json" \
  -d '{
  "model_identity": "string",
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}'

Responses#

{
  "backend": "string",
  "model_identity": "string",
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [
        0
      ],
      "index": 0
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "total_tokens": 0
  },
  "errors": [
    {
      "index": 0,
      "code": "string",
      "message": "string",
      "stage": "parse",
      "retryable": true,
      "status": 0,
      "retry_after_ms": 0
    }
  ],
  "summary": {
    "total": 0,
    "succeeded": 0,
    "failed": 0
  }
}

Chunk content with optional model-aware embeddings#

POST/chunk

Splits content into chunks. Without embedding, the fixed chunker and legacy response remain unchanged. With embedding, the resolved target model constrains boundaries for text, PDF pages, images or explicit tiles, composited GIF frames, audio windows and consecutive video windows.

Each model-aware result pairs source coordinates and an optional vector or error. include_content independently controls payload inclusion; video uses the original source and sampling recipe. execute: false plans without executing the embedding model. stream: true delivers bounded NDJSON records followed by terminal coverage. A cap, gap or lost terminal record must not be treated as complete source coverage. Continuations bind the original source version and semantic plan; retries may deterministically recompute and must deduplicate chunk IDs.

Models#

Fixed Chunking#

  • Token-count splitting with overlap for text; fixed time windows for audio
  • Use model="fixed" (aliases fixed_bert, fixed_bpe, and fixed-bert-tokenizer are accepted)
  • Fast and deterministic
  • Other model names are rejected as unsupported

Request Body#

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/chunk"

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "chunk",
      "index": 0,
      "id": 0,
      "text": "This is the first chunk...",
      "start_char": 0,
      "end_char": 100,
      "mime_type": "text/plain"
    },
    {
      "object": "chunk",
      "index": 1,
      "id": 1,
      "text": "This is the second chunk...",
      "start_char": 90,
      "end_char": 190,
      "mime_type": "text/plain"
    }
  ],
  "model": "fixed",
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 0,
    "total_tokens": 12
  },
  "cache_hit": false
}

Rerank documents by relevance#

POST/rerank

Re-scores documents by relevance to a text query. Returns one score per document, in request order.

Each entry in documents is either a string or an array of content parts, in the same format that generation and embedding use: text parts, image_url parts, and inline media parts with an image/* MIME type. The client renders document fields or templates to text before calling this endpoint.

Text-only documents work with any reranker. Documents with images require a model that supports them: a ColQwen-style late-interaction reranker (manifest capability colqwen or multimodal_late_interaction) or a Qwen3-VL reranker bundled with its GGUF vision projector. Otherwise the request is rejected with a 400. Within a request that contains images, documents without images are scored by the model's text scorer.

Models#

  • Models are auto-discovered from models_dir/rerankers/
  • Cross-encoder rerankers are supported through the text scorer
  • Late-interaction text rerankers such as ColBERT can opt in with model_manifest.json capability late_interaction or colbert
  • Automatically prefers quantized variants if available

Remote image URLs are fetched subject to the configured content security policy. Image headers and aggregate decoded pixels are admitted before the model loads.

Parameters#

NameTypeLocationRequiredDescription
AcceptstringheaderNoSet to application/vnd.antfly.numeric.v1 to receive the values as a binary
frame instead of JSON, which avoids serializing every float as text. Any
other value, or none, returns the JSON body.

Request Body#

Example:

{
  "model": "BAAI/bge-reranker-v2-m3",
  "query": "machine learning applications",
  "documents": [
    "Introduction to machine learning...",
    [
      {
        "type": "text",
        "text": "Quarterly invoice"
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "data:image/png;base64,iVBORw0KGgo..."
        }
      }
    ]
  ],
  "prompts": [
    "string"
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/rerank" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "query": "machine learning applications",
  "documents": [
    "Introduction to machine learning...",
    [
      {
        "type": "text",
        "text": "Quarterly invoice"
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "data:image/png;base64,iVBORw0KGgo..."
        }
      }
    ]
  ],
  "prompts": [
    "string"
  ]
}'

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "rerank.score",
      "index": 0,
      "score": 0
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

Generate text using LLM (OpenAI-compatible)#

POST/generate

Generates text using local LLM models (e.g., Gemma 3). Fully compatible with the OpenAI Chat Completions API.

Models#

Models are auto-discovered from models_dir/generators/ at startup. Use the /ai/v1/models endpoint to list available models.

Streaming#

Set stream: true to receive Server-Sent Events (SSE) with incremental token deltas. Each event contains a ChatCompletionChunk object. A successful stream ends with data: [DONE]. If generation or stream writing fails after the HTTP response starts, the server instead emits an event: error frame whose data is a plain-text error message, then closes the stream without a [DONE] frame.

Input Format#

Uses OpenAI-compatible chat format with messages array containing role (system, user, assistant) and content. Set stream: true for streaming responses.

Downloaded and inline encoded media is limited cumulatively across the request to the lower of 100 MiB, configured max_download_size_bytes, and—when admission.inference.max_concurrent_requests is positive—16 MiB times that capacity. A zero configured download limit disables nonempty media. Remote URL byte potential is reserved before fetch; inline sources reserve their actual encoded size without adding it to the existing request-body reservation. Accepted image headers are then validated and decoded source pixels are admitted at a conservative 16 bytes per pixel against the lower of 512 MiB or 16 MiB times a positive admission.inference.max_concurrent_requests; a zero concurrency setting still uses the finite 512 MiB ceiling. max_image_dimension limits each source edge. Malformed images return 400, while dimension or aggregate excess returns 413 before model loading. Initial capacity admission occurs before media fetch, so an overloaded server returns 503 without fetching content.

Request Body#

Example:

{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "logprobs": true,
  "top_logprobs": 1,
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/generate" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "logprobs": true,
  "top_logprobs": 1,
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}'

Responses#

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1704123456,
  "model": "string",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": null,
        "content": "string",
        "reasoning_content": "string",
        "tool_calls": [
          null
        ]
      },
      "finish_reason": "stop",
      "logprobs": {}
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  },
  "speculation": {
    "policy": "string",
    "calibration": "string",
    "decision": "string",
    "disabled_reason": "string"
  }
}

Generate text for a synchronous batch of requests#

POST/generate/batch

Runs multiple non-streaming generation requests as one synchronous batch. Compatible native requests for the same model are executed through the native batched KV decoder; unsupported per-item options are returned as per-item errors without failing sibling requests.

A syntactically valid batch envelope returns HTTP 200 even when some or all items fail. Clients must inspect each item's mutually exclusive response and error fields, plus summary, rather than treating HTTP 200 as success for every item. Fatal service or envelope failures still use non-2xx responses.

This endpoint implements the synchronous form only. Future async durable batching will use the same request item shape with mode: async.

Batch generation accepts bounded image and audio content parts. Media is validated and admitted before model dispatch; models without compatible multimodal capabilities return an independent per-item error without failing sibling requests.

Request Body#

Example:

{
  "mode": "sync",
  "requests": [
    {
      "custom_id": "string",
      "body": {
        "model": "google/gemma-3-1b-it",
        "messages": [
          {
            "role": null,
            "content": null,
            "tool_calls": [
              null
            ],
            "tool_call_id": "string"
          }
        ],
        "logprobs": true,
        "top_logprobs": 1,
        "max_tokens": 256,
        "temperature": 0.7,
        "top_p": 0,
        "top_k": 0,
        "stream": true,
        "stream_options": {
          "include_usage": true
        },
        "chat_template_kwargs": {
          "enable_thinking": true
        },
        "ignore_eos": true,
        "tools": [
          {
            "type": "function",
            "function": {
              "name": "get_weather",
              "description": "Get the current weather in a location",
              "parameters": {
                "type": "object",
                "properties": {
                  "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                  }
                },
                "required": [
                  "location"
                ]
              },
              "strict": true
            }
          }
        ],
        "min_p": 0,
        "repetition_penalty": 0,
        "frequency_penalty": -2,
        "presence_penalty": -2,
        "response_format": {
          "type": "text",
          "json_schema": {
            "name": "string",
            "strict": true,
            "schema": {}
          }
        },
        "grammar": "string",
        "draft_model": "string",
        "speculative_k": 1,
        "speculation_policy": "auto",
        "speculation_calibration": "none",
        "cache_dtype": "f16",
        "cache_compaction_ratio": 0,
        "prompt_cache_key": "string",
        "prompt_cache": true,
        "backend": "auto",
        "mode": "eager",
        "compiled_target": "partitioned",
        "tool_choice": "auto"
      }
    }
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/generate/batch" \
  -H "Content-Type: application/json" \
  -d '{
  "mode": "sync",
  "requests": [
    {
      "custom_id": "string",
      "body": {
        "model": "google/gemma-3-1b-it",
        "messages": [
          {
            "role": null,
            "content": null,
            "tool_calls": [
              null
            ],
            "tool_call_id": "string"
          }
        ],
        "logprobs": true,
        "top_logprobs": 1,
        "max_tokens": 256,
        "temperature": 0.7,
        "top_p": 0,
        "top_k": 0,
        "stream": true,
        "stream_options": {
          "include_usage": true
        },
        "chat_template_kwargs": {
          "enable_thinking": true
        },
        "ignore_eos": true,
        "tools": [
          {
            "type": "function",
            "function": {
              "name": "get_weather",
              "description": "Get the current weather in a location",
              "parameters": {
                "type": "object",
                "properties": {
                  "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                  }
                },
                "required": [
                  "location"
                ]
              },
              "strict": true
            }
          }
        ],
        "min_p": 0,
        "repetition_penalty": 0,
        "frequency_penalty": -2,
        "presence_penalty": -2,
        "response_format": {
          "type": "text",
          "json_schema": {
            "name": "string",
            "strict": true,
            "schema": {}
          }
        },
        "grammar": "string",
        "draft_model": "string",
        "speculative_k": 1,
        "speculation_policy": "auto",
        "speculation_calibration": "none",
        "cache_dtype": "f16",
        "cache_compaction_ratio": 0,
        "prompt_cache_key": "string",
        "prompt_cache": true,
        "backend": "auto",
        "mode": "eager",
        "compiled_target": "partitioned",
        "tool_choice": "auto"
      }
    }
  ]
}'

Responses#

{
  "object": "generate.batch",
  "data": [
    {
      "custom_id": "string",
      "index": 0,
      "response": {
        "id": "chatcmpl-abc123",
        "object": "chat.completion",
        "created": 1704123456,
        "model": "string",
        "choices": [
          {
            "index": 0,
            "message": {
              "role": null,
              "content": "string",
              "reasoning_content": "string",
              "tool_calls": [
                null
              ]
            },
            "finish_reason": "stop",
            "logprobs": {}
          }
        ],
        "usage": {
          "prompt_tokens": 0,
          "completion_tokens": 0,
          "total_tokens": 0,
          "cached_prompt_tokens": 0
        },
        "speculation": {
          "policy": "string",
          "calibration": "string",
          "decision": "string",
          "disabled_reason": "string"
        }
      },
      "error": {
        "code": "string",
        "message": "string",
        "retryable": true,
        "retry_after_ms": 1
      }
    }
  ],
  "summary": {
    "total": 0,
    "succeeded": 0,
    "failed": 0
  },
  "execution": {
    "requested_items": 0,
    "native_batches": 0,
    "native_items": 0,
    "serial_items": 0,
    "rejected_items": 0,
    "fallback_items": 0,
    "fallback_reason": "string"
  }
}

OpenAI Chat Completions endpoint#

POST/chat/completions

OpenAI-compatible chat completions path for SDKs that call /chat/completions relative to the configured base URL.

Request Body#

Example:

{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "logprobs": true,
  "top_logprobs": 1,
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "logprobs": true,
  "top_logprobs": 1,
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}'

Responses#

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1704123456,
  "model": "string",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": null,
        "content": "string",
        "reasoning_content": "string",
        "tool_calls": [
          null
        ]
      },
      "finish_reason": "stop",
      "logprobs": {}
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  },
  "speculation": {
    "policy": "string",
    "calibration": "string",
    "decision": "string",
    "disabled_reason": "string"
  }
}

Rewrite text using Seq2Seq models#

POST/rewrite

Rewrite/transform text using Seq2Seq models (T5, FLAN-T5, BART, etc.).

Models#

  • Models are auto-discovered from models_dir/rewriters/
  • Seq2Seq models have encoder.onnx, decoder-init.onnx, and decoder.onnx files
  • Compatible with LMQG question generation models

Use Cases#

  • Question Generation: Generate questions from answer-context pairs
  • Query Generation: Generate search queries from documents
  • Paraphrasing: Rewrite text in different words
  • Translation: Translate text between languages

Request Body#

Example:

{
  "model": "lmqg/flan-t5-small-squad-qg",
  "inputs": [
    "Translate to German: Hello, how are you?"
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/rewrite" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "lmqg/flan-t5-small-squad-qg",
  "inputs": [
    "Translate to German: Hello, how are you?"
  ]
}'

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "rewrite",
      "index": 0,
      "texts": [
        "string"
      ]
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

Read text from images (OCR/document understanding)#

POST/read

Extracts text from images using Vision2Seq models like TrOCR, Donut, Florence-2, or Pix2Struct.

Models#

Models are auto-discovered from models_dir/readers/ at startup. Use the /ai/v1/models endpoint to list available models.

  • TrOCR: Pure OCR for printed/handwritten text
  • Donut: Document understanding with structured output (receipts, forms)
  • Florence-2: Multi-task vision model (OCR, captioning, VQA)
  • Pix2Struct: Visual question answering with natural-language prompts
  • Moondream: Decoder-only vision-language reader that can return text plus optional flattened fields

Task Prompts#

Some models support task prompts for different extraction modes:

  • Donut CORD: <s_cord-v2> for receipt parsing
  • Donut DocVQA: <s_docvqa><s_question>...</s_question><s_answer> for visual QA
  • Florence-2 OCR: <OCR> for text extraction
  • Florence-2 Caption: <CAPTION> for image description
  • Pix2Struct: natural-language questions like What type of document is this?
  • Moondream: natural-language prompts like Describe this image.

Image admission reserves the effective downloaded-byte ceiling at 16 MiB per weighted capacity unit and at least one unit per two declared images. The effective ceiling is the minimum of the configured per-image limit times image count, the read-batch limit (256 MiB by default), and—when admission is bounded—16 MiB times admission.inference.max_concurrent_requests. Admission happens before model resolution or download.

After download, image headers are validated before model loading. Decoded source pixels are admitted at a conservative 16 bytes per pixel against the lower of 512 MiB or 16 MiB times a positive admission.inference.max_concurrent_requests; a zero concurrency setting still uses the finite 512 MiB ceiling. The reservation grows atomically from downloaded-byte admission before inference. max_image_dimension limits each source edge; malformed images return 400 and dimension or aggregate excess returns 413. The same decoded-pixel policy covers generate/chat, dense embedding, multimodal reranking, image /extract, and the embedded read, extract, and dense embedding APIs. Batch generation accepts bounded image and audio media parts, applies the same aggregate encoded-byte and decoded-image admission before model execution, and rejects malformed or unsupported parts before dispatch.

Request Body#

Example:

{
  "model": "microsoft/trocr-base-printed",
  "images": [
    {
      "url": "data:image/png;base64,iVBORw0KGgo..."
    }
  ],
  "prompt": "What type of document is this?",
  "max_tokens": 256
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/read" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "microsoft/trocr-base-printed",
  "images": [
    {
      "url": "data:image/png;base64,iVBORw0KGgo..."
    }
  ],
  "prompt": "What type of document is this?",
  "max_tokens": 256
}'

Responses#

{
  "object": "list",
  "data": [
    {
      "text": "Invoice Total: $123.45",
      "fields": {
        "menu.nm": "Coffee",
        "menu.price": "$3.50",
        "total": "$123.45"
      },
      "regions": [
        {
          "text": "string",
          "bbox": [
            0
          ],
          "confidence": 0,
          "label": "string"
        }
      ]
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  },
  "execution": {
    "requested_items": 0,
    "native_batches": 0,
    "native_items": 0,
    "serial_items": 0,
    "rejected_items": 0,
    "fallback_items": 0,
    "fallback_reason": "string"
  }
}

Transcribe audio to text (speech-to-text)#

POST/transcribe

Transcribes audio to text using Speech2Seq models like Whisper, Wav2Vec2, or HuBERT.

Models#

Models are auto-discovered from models_dir/transcribers/ at startup. Use the /api/models endpoint to list available models.

  • Whisper: OpenAI's Whisper models (multilingual, automatic language detection)
  • Wav2Vec2: Facebook's Wav2Vec 2.0 models (English-focused)
  • HuBERT: Facebook's HuBERT models (self-supervised)

Audio Input#

Audio data should be base64-encoded. Supported formats depend on the model:

  • WAV (recommended - raw PCM)
  • MP3
  • FLAC
  • M4A/AAC

Request Body#

Example:

{
  "model": "openai/whisper-tiny",
  "audio": "string",
  "language": "en",
  "timestamps": true,
  "diarization": true
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/transcribe" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "openai/whisper-tiny",
  "audio": "string",
  "language": "en",
  "timestamps": true,
  "diarization": true
}'

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "transcription",
      "index": 0,
      "text": "Hello, how are you today?",
      "language": "en",
      "duration_ms": 0,
      "segments": [
        {
          "text": "string",
          "start_ms": 0,
          "end_ms": 0,
          "words": [
            {
              "word": "string",
              "start_ms": 0,
              "end_ms": 0
            }
          ],
          "speaker": "string"
        }
      ],
      "speakers": [
        "string"
      ]
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

Dictate speech into clean written text#

POST/dictate

Push-to-talk dictation. Transcribes one recorded clip with a Whisper transcriber, then rewrites the transcript as clean written text with a generator model: fillers, false starts, and repeated words are removed, punctuation and paragraphing are added, and preferred spellings from dictionary are applied. Clips longer than the 30 s Whisper window are transcribed in windows cut at the quietest pause near the boundary.

Set cleanup_model to the generator that rewrites the transcript. Without it, or with style: verbatim, the response carries the raw transcript and no generation runs.

The framed attachment transport is accepted: send the JSON as the envelope metadata with "audio": "attachment:0" and the clip as the single attachment.

With stream: true the response is Server-Sent Events. The stream emits one dictation.transcript event as soon as transcription finishes, then dictation.delta events with cleaned-text tokens, then dictation.completed with the full cleaned text, then [DONE].

{
  "model": "openai/whisper-tiny",
  "cleanup_model": "ggml-org/gemma-4-E4B-it-GGUF",
  "audio": "UklGRi...",
  "dictionary": ["Antfly", "Colony"],
  "context": "reply in a Slack thread",
  "stream": true
}

Request Body#

Example:

{
  "model": "openai/whisper-tiny",
  "audio": "string",
  "language": "en",
  "timestamps": true,
  "cleanup_model": "ggml-org/gemma-4-E4B-it-GGUF",
  "style": "clean",
  "dictionary": [
    "string"
  ],
  "context": "string",
  "instructions": "string",
  "transcript_prompt": "string",
  "vad": {
    "model": "onnx-community/silero-vad",
    "silero_threshold": 0,
    "threshold": 0,
    "min_speech_ms": 0,
    "min_silence_ms": 1,
    "speech_pad_ms": 0
  },
  "audio_context": "full",
  "stream": true,
  "max_tokens": 1
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/dictate" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "openai/whisper-tiny",
  "audio": "string",
  "language": "en",
  "timestamps": true,
  "cleanup_model": "ggml-org/gemma-4-E4B-it-GGUF",
  "style": "clean",
  "dictionary": [
    "string"
  ],
  "context": "string",
  "instructions": "string",
  "transcript_prompt": "string",
  "vad": {
    "model": "onnx-community/silero-vad",
    "silero_threshold": 0,
    "threshold": 0,
    "min_speech_ms": 0,
    "min_silence_ms": 1,
    "speech_pad_ms": 0
  },
  "audio_context": "full",
  "stream": true,
  "max_tokens": 1
}'

Responses#

{
  "object": "dictation",
  "id": "string",
  "created": 0,
  "model": "string",
  "cleanup_model": "string",
  "transcript": {
    "text": "string",
    "language": "string",
    "duration_ms": 0,
    "segments": [
      {
        "text": "string",
        "start_ms": 0,
        "end_ms": 0,
        "words": [
          {
            "word": "string",
            "start_ms": 0,
            "end_ms": 0
          }
        ],
        "speaker": "string"
      }
    ]
  },
  "text": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

Open a streaming transcription session#

POST/transcription/sessions

Creates a server-side session that accepts audio in chunks and returns transcript events as speech is endpointed. Append audio with POST /transcription/sessions/{session_id}/audio; each append runs voice activity detection over the buffered audio and returns the events it produced:

  • partial: the open speech segment decoded again. stable_text is the word prefix that agreed with the previous hypothesis and can be rendered as committed text.
  • final: a segment closed by vad.min_silence_ms of silence, by max_segment_ms of continuous speech, or by commit: true.

Sessions expire after ttl_seconds without appends and are closed with DELETE.

Request Body#

Example:

{
  "model": "openai/whisper-tiny",
  "language": "string",
  "vad": {
    "model": "onnx-community/silero-vad",
    "silero_threshold": 0,
    "threshold": 0,
    "min_speech_ms": 0,
    "min_silence_ms": 1,
    "speech_pad_ms": 0
  },
  "audio_context": "full",
  "partial_interval_ms": 1,
  "max_segment_ms": 1,
  "emit_partials": true,
  "dictionary": [
    "string"
  ],
  "transcript_prompt": "string",
  "ttl_seconds": 1
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/transcription/sessions" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "openai/whisper-tiny",
  "language": "string",
  "vad": {
    "model": "onnx-community/silero-vad",
    "silero_threshold": 0,
    "threshold": 0,
    "min_speech_ms": 0,
    "min_silence_ms": 1,
    "speech_pad_ms": 0
  },
  "audio_context": "full",
  "partial_interval_ms": 1,
  "max_segment_ms": 1,
  "emit_partials": true,
  "dictionary": [
    "string"
  ],
  "transcript_prompt": "string",
  "ttl_seconds": 1
}'

Responses#

{
  "object": "transcription.session",
  "id": "string",
  "model": "string",
  "language": "string",
  "created": 0,
  "expires_at": 0,
  "buffered_ms": 0,
  "total_ms": 0,
  "finals": 0,
  "partials": 0
}

Inspect a streaming transcription session#

GET/transcription/sessions/{session_id}

Parameters#

NameTypeLocationRequiredDescription
session_idstringpathYes

Code Examples#

curl -X GET "http://localhost:8080/ai/v1/transcription/sessions/{session_id}"

Responses#

{
  "object": "transcription.session",
  "id": "string",
  "model": "string",
  "language": "string",
  "created": 0,
  "expires_at": 0,
  "buffered_ms": 0,
  "total_ms": 0,
  "finals": 0,
  "partials": 0
}

Close a streaming transcription session#

DELETE/transcription/sessions/{session_id}

Discards buffered audio that has not been committed.

Parameters#

NameTypeLocationRequiredDescription
session_idstringpathYes

Code Examples#

curl -X DELETE "http://localhost:8080/ai/v1/transcription/sessions/{session_id}"

Responses#

{
  "object": "transcription.session.deleted",
  "id": "string",
  "deleted": true
}

Subscribe to a session's transcript events#

GET/transcription/sessions/{session_id}/events

Long-lived Server-Sent Events stream that pushes every partial and final event the session produces, whether they came from POST .../audio appends or a POST .../stream upload. Clients that append from one connection and render from another use this instead of reading the append responses. A ping is sent after 15 s of silence. The stream ends with session.closed and [DONE] when the session is deleted or expires.

Parameters#

NameTypeLocationRequiredDescription
session_idstringpathYes

Code Examples#

curl -X GET "http://localhost:8080/ai/v1/transcription/sessions/{session_id}/events"

Responses#

No response body

Stream raw audio into a session and receive events as they occur#

POST/transcription/sessions/{session_id}/stream

Full-duplex transcription over one request. The request body is raw little-endian mono PCM (format selects 16-bit or float32 samples at sample_rate), sent as it is captured. The server decodes as chunks arrive and writes transcription.event messages on the response while the upload continues. At end of body, buffered speech is finalized when commit is true (the default).

Over HTTP/2 the body is read incrementally. Over HTTP/1.1 the body is read after it has fully arrived, so use /audio appends there for live results. Appends to the same session are refused with 409 while a stream is open.

Parameters#

NameTypeLocationRequiredDescription
session_idstringpathYes
formatstring (pcm16, pcm_f32)queryNoRaw sample format. Default pcm16.
sample_rateintegerqueryNoSample rate of the raw stream. Default 16000.
commitbooleanqueryNoFinalize open speech at end of body. Default true.

Request Body#

Example:

"string"

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/transcription/sessions/{session_id}/stream?format=value&sample_rate=value&commit=value"

Responses#

No response body

Append audio to a streaming transcription session#

POST/transcription/sessions/{session_id}/audio

Appends one chunk of audio and runs endpointing and decoding over the session buffer. The response lists the events produced by this append, in order. Appends to one session must be sequential; a concurrent append is rejected with 409.

audio is base64. With format: auto (default) the bytes are a container the runtime can decode (WAV, Opus, MP3, FLAC, ...). With format: pcm16 or pcm_f32 the bytes are raw little-endian mono samples at sample_rate, which lets a client send microphone frames without re-encoding. Chunks of 250 ms to 1 s balance latency and decoder work.

commit: true finalizes buffered speech even without trailing silence. It may be sent without audio to flush at the end of a recording.

The framed attachment transport is accepted: send the JSON as the envelope metadata with "audio": "attachment:0" and the bytes as the single attachment.

Parameters#

NameTypeLocationRequiredDescription
session_idstringpathYes

Request Body#

Example:

{
  "audio": "string",
  "format": "auto",
  "sample_rate": 8000,
  "commit": true
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/transcription/sessions/{session_id}/audio" \
  -H "Content-Type: application/json" \
  -d '{
  "audio": "string",
  "format": "auto",
  "sample_rate": 8000,
  "commit": true
}'

Responses#

{
  "object": "list",
  "session_id": "string",
  "model": "string",
  "data": [
    {
      "object": "transcription.event",
      "type": "partial",
      "sequence": 0,
      "text": "string",
      "stable_text": "string",
      "start_ms": 0,
      "end_ms": 0,
      "language": "string",
      "words": [
        {
          "word": "string",
          "start_ms": 0,
          "end_ms": 0
        }
      ]
    }
  ],
  "buffered_ms": 0,
  "total_ms": 0
}

Answer named decision questions#

POST/decisions

Execute text or multimodal evidence through named deciders, or an ordered batch of evidence items. Hosted typed deciders and qualified constrained generators accept text and images per batch item. Generated native models must support both images and constrained output. Native trained and embedding deciders require text. Embedding similarity supports choice and multi_choice; trained decision models support choice, score and predicate. Answer variants distinguish trained probabilities from raw cosine similarities.

Request Body#

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/decisions"

Responses#

No response body

Extract entities, relations, classifications, and structures#

POST/extract

Schema-driven extraction over shared AI content parts. This is the canonical public API for named entity recognition, relation extraction, text/document classification, token classification, and structured document extraction.

Set schema_version: 2 for strict mixed-task schemas, span attributes, constrained classification, typed records and JointIE. Each input may replace the shared schema/options. Offsets default to half-open UTF-8 bytes, and text content parts are joined by a single newline. Inputs are validated atomically. Unsupported features and long documents are rejected explicitly; windowing requires runtime support.

Image-backed extraction uses the same byte-reserving and image-count-weighted admission policy as /read, before model resolution or download. Text-only extraction consumes one admission unit.

Request Body#

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/extract"

Responses#

No response body

List model instances#

GET/models/instances

Inspect this inference server's shared model cache without loading models or extending idle eviction deadlines. Each incarnation appears once, even when it supports multiple tasks.

Code Examples#

curl -X GET "http://localhost:8080/ai/v1/models/instances"

Responses#

{
  "object": "list",
  "data": [
    {
      "id": "string",
      "model": "string",
      "tasks": [
        "generator"
      ],
      "state": "loading",
      "active_handles": 0,
      "startup_managed": true,
      "unloadable": true,
      "unload_blocked_reason": "loading",
      "backend": "string",
      "residency_mode": "backend_managed",
      "idle_ms": 0
    }
  ]
}

Load a model instance#

POST/models/instances

Prepare the instance used by the requested task. Reuses an existing compatible instance. Normal idle eviction still applies; loading does not pin weights or guarantee device residency.

Request Body#

Example:

{
  "model": "string",
  "task": "generator"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/models/instances" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "string",
  "task": "generator"
}'

Responses#

{
  "id": "string",
  "model": "string",
  "tasks": [
    "generator"
  ],
  "state": "loading",
  "active_handles": 0,
  "startup_managed": true,
  "unloadable": true,
  "unload_blocked_reason": "loading",
  "backend": "string",
  "residency_mode": "backend_managed",
  "idle_ms": 0
}

Unload a model instance#

DELETE/models/instances/{instance_id}

Evict one exact incarnation from the shared server cache. A stale ID is an idempotent miss and never evicts a replacement. Loading, busy, and startup-managed instances cannot be unloaded.

Parameters#

NameTypeLocationRequiredDescription
instance_idstringpathYes

Code Examples#

curl -X DELETE "http://localhost:8080/ai/v1/models/instances/{instance_id}"

Responses#

{
  "outcome": "unloaded"
}

List available models#

GET/models

Returns lists of available embedding, chunking, reranking, generator, extractor, rewriter, reader, and transcriber models.

Embedders#

  • Embedding models from models_dir
  • Quantized variants have -i8 suffix

Chunkers#

  • Always "fixed" (built-in); chunking does not load models from models_dir

Rerankers#

  • Native or ONNX rerankers from models_dir/rerankers/
  • model_manifest.json capabilities can mark late-interaction text rerankers (late_interaction, colbert)
  • Empty if no models configured

Generators#

  • LLM models from models_dir/generators/
  • Empty if no models configured

Extractors#

  • Extraction-capable models from the managed model registry
  • Includes GLiNER models for zero-shot entity and relation extraction

Rewriters#

  • Seq2Seq models from models_dir/rewriters/
  • T5, FLAN-T5, BART, and LMQG question generation models

Readers#

  • Vision2Seq models from models_dir/readers/
  • TrOCR, Donut, Florence-2 for OCR and document understanding

Transcribers#

  • Speech2Seq models from models_dir/transcribers/
  • Whisper, Wav2Vec2, HuBERT for speech-to-text

Models are discovered at service startup and cached.

Code Examples#

curl -X GET "http://localhost:8080/ai/v1/models"

Responses#

{
  "public_inference_enabled": true,
  "object": "list",
  "data": [
    {}
  ],
  "allow_downloads": true,
  "backends": {
    "native": true,
    "metal": true,
    "cuda": false,
    "xla": false,
    "wasm": false
  },
  "chunkers": {},
  "rerankers": {},
  "embedders": {},
  "extractors": {},
  "deciders": {},
  "generators": {},
  "rewriters": {},
  "readers": {},
  "transcribers": {}
}

Create embeddings (OpenAI-compatible)#

POST/embeddings

OpenAI-compatible embeddings endpoint. Accepts the same request format as OpenAI's /v1/embeddings API, served here under /ai/v1/embeddings. For sparse-capable models, each data item still uses the embedding field, but its value is a sparse vector object instead of a dense float array. Dense image inputs are header-validated and admitted against the aggregate decoded-pixel budget before model loading. Remote URL byte potential is reserved before fetch; inline sources use their actual encoded size. Use this endpoint for drop-in compatibility with OpenAI SDKs.

Parameters#

NameTypeLocationRequiredDescription
AcceptstringheaderNoSet to application/vnd.antfly.numeric.v1 to receive the values as a binary
frame instead of JSON, which avoids serializing every float as text. Any
other value, or none, returns the JSON body.

Request Body#

Example:

{
  "model_identity": "string",
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/embeddings" \
  -H "Content-Type: application/json" \
  -d '{
  "model_identity": "string",
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}'

Responses#

{
  "backend": "string",
  "model_identity": "string",
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [
        0
      ],
      "index": 0
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "total_tokens": 0
  },
  "errors": [
    {
      "index": 0,
      "code": "string",
      "message": "string",
      "stage": "parse",
      "retryable": true,
      "status": 0,
      "retry_after_ms": 0
    }
  ],
  "summary": {
    "total": 0,
    "succeeded": 0,
    "failed": 0
  }
}

List Traditional ML predictors#

GET/predictors

Returns the Traditional ML predictor catalog for /ml/v1/predict. Predictors are loaded from <ml_dir>/<name>/ and exposed separately from the AI model catalog.

Code Examples#

curl -X GET "http://localhost:8080/ai/v1/predictors"

Responses#

{
  "public_inference_enabled": true,
  "object": "list",
  "predictors": {}
}

Run a traditional ML predictor#

POST/predict

Run a tabular predictor (tree ensemble, linear, or SVM) on a batch of feature vectors. Models are loaded from <ml_dir>/<name>/ and identified by name. Use /ml/v1/models for the list of available predictors and their feature schemas.

Request Body#

Example:

{
  "model": "string",
  "input": [
    [
      0
    ]
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/predict" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "string",
  "input": [
    [
      0
    ]
  ]
}'

Responses#

{
  "model": "string",
  "task": "regression",
  "predictions": [
    [
      0
    ]
  ]
}