Inference API#

Antfly inference is a local inference server for ONNX-based ML models.

What is Antfly inference#

Antfly inference provides local ML inference with an OpenAI-compatible API:

Embedding Generation: OpenAI-compatible text embedding endpoints, with dense or sparse embedding payloads
Text Chunking: Semantic chunking with ONNX models or fixed-size fallback
Reranking: Relevance re-scoring for search results
Extraction: Schema-driven entity, relation, classification, and structured extraction
Text Rewriting: Transform text using Seq2Seq models (question generation, query generation, etc.)

Download the latest release at https://antfly.io/docs/downloads

When to Use Antfly inference#

Antfly inference can run standalone or as part of an Antfly cluster:

Local ONNX model inference without external API dependencies
OpenAI-compatible embedding payloads served at /ai/v1/embed and /ai/v1/embeddings
Semantic text chunking for RAG pipelines
Relevance reranking for improved search quality
Centralized model serving across distributed nodes
Privacy-preserving ML inference (data never leaves your infrastructure)

Authentication#

The standalone inference listener speaks plaintext HTTP and does not provide built-in authentication. It refuses non-loopback binds by default. Keep the loopback default, or put it behind a trusted proxy that terminates TLS and authenticates every request; only then use --allow-insecure-public-bind to opt into a non-loopback listener. When authentication is enabled, the unified Antfly server requires a valid principal with inference/* read permission for /ai/v1 and /ml/v1. Global */* read or admin grants also satisfy this requirement through the existing wildcard policy.

Features#

Embedding Generation#

Models: ONNX models auto-discovered from {models_dir}/embedders/
API: OpenAI-compatible payloads served at /ai/v1/embed and /ai/v1/embeddings
Response Shape: Each data[i].embedding is either a dense float vector or a sparse {indices, values} object

Text Chunking#

Models: Fixed-size chunking (always available) + ONNX models
Model Discovery: Auto-discovers models from {models_dir}/chunkers/
Caching: 2-minute TTL memory cache
Fallback: Falls back to fixed chunking if model fails

Reranking#

Model Discovery: Auto-discovers ONNX models from {models_dir}/rerankers/
Quantization: Automatically uses quantized models if available
Input: Pre-rendered text prompts (client handles field extraction)

Create embeddings (alias of `/embeddings`)#

POST/embed

Alias of /ai/v1/embeddings.

Accepts the OpenAI embeddings request shape and returns the same OpenAI-compatible response envelope. For sparse-capable models, data[i].embedding is a sparse vector object instead of a dense float array. Dense image inputs are header-validated and admitted against the aggregate decoded-pixel budget before model loading. Remote URL byte potential is reserved before fetch; inline sources use their actual encoded size.

Request Body#

Example:

{
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/embed" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}'

const response = await fetch("http://localhost:8080/ai/v1/embed", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "string",
    "input": "string",
    "encoding_format": "float",
    "dimensions": 0,
    "task_type": "RETRIEVAL_QUERY",
    "instruction": "string",
    "input_type": "search_query",
    "error_policy": "fail_fast"
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/embed", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "string",
    "input": "string",
    "encoding_format": "float",
    "dimensions": 0,
    "task_type": "RETRIEVAL_QUERY",
    "instruction": "string",
    "input_type": "search_query",
    "error_policy": "fail_fast"
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/embed",
    json={
        "model": "string",
        "input": "string",
        "encoding_format": "float",
        "dimensions": 0,
        "task_type": "RETRIEVAL_QUERY",
        "instruction": "string",
        "input_type": "search_query",
        "error_policy": "fail_fast"
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/embed", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [
        0
      ],
      "index": 0
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "total_tokens": 0
  },
  "errors": [
    {
      "index": 0,
      "code": "string",
      "message": "string",
      "stage": "parse",
      "retryable": true,
      "status": 0,
      "retry_after_ms": 0
    }
  ],
  "summary": {
    "total": 0,
    "succeeded": 0,
    "failed": 0
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Chunk text into smaller segments#

POST/chunk

Splits text into smaller chunks using semantic or fixed-size chunking models.

Models#

Fixed Chunking (always available)#

Simple token-based splitting with overlap
Use model="fixed"
Fast and deterministic

ONNX Models#

Semantic chunking based on content similarity
Models auto-discovered from models_dir/chunkers/
Falls back to fixed chunking if model fails

Caching#

Results are cached in memory for 2 minutes. Cache key includes both config and text content.

Request Body#

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/chunk"

const response = await fetch("http://localhost:8080/ai/v1/chunk", {
  method: "POST"
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/chunk", {
  method: "POST"
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post("http://localhost:8080/ai/v1/chunk")

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/chunk", nil)

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "chunk",
      "index": 0,
      "id": 0,
      "text": "This is the first chunk...",
      "start_char": 0,
      "end_char": 100,
      "mime_type": "text/plain"
    },
    {
      "object": "chunk",
      "index": 1,
      "id": 1,
      "text": "This is the second chunk...",
      "start_char": 90,
      "end_char": 190,
      "mime_type": "text/plain"
    }
  ],
  "model": "fixed",
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 0,
    "total_tokens": 12
  },
  "cache_hit": false
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Rerank multimodal documents by relevance#

POST/rerank_multimodal

Re-scores multimodal documents based on relevance to a text query.

This endpoint accepts the same content-part image conventions as generation and embedding. Text-only requests can be served immediately. Image-bearing requests reserve the stable contract for native ColQwen-style late-interaction reranking as that encoder lands. Image-bearing requests already run native Zig image preprocessing and grid preparation. Remote URL byte potential is reserved before fetch, and image headers plus aggregate decoded pixels are admitted before model loading.

Request Body#

Example:

{
  "model": "vidore/colqwen2-v1.0",
  "query": "invoice total due date",
  "documents": [
    {
      "id": "string",
      "content": null
    }
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/rerank_multimodal" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "vidore/colqwen2-v1.0",
  "query": "invoice total due date",
  "documents": [
    {
      "id": "string",
      "content": null
    }
  ]
}'

const response = await fetch("http://localhost:8080/ai/v1/rerank_multimodal", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "vidore/colqwen2-v1.0",
    "query": "invoice total due date",
    "documents": [
      {
        "id": "string",
        "content": null
      }
    ]
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/rerank_multimodal", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "vidore/colqwen2-v1.0",
    "query": "invoice total due date",
    "documents": [
      {
        "id": "string",
        "content": null
      }
    ]
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/rerank_multimodal",
    json={
        "model": "vidore/colqwen2-v1.0",
        "query": "invoice total due date",
        "documents": [
            {
                "id": "string",
                "content": null
            }
        ]
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "vidore/colqwen2-v1.0",
  "query": "invoice total due date",
  "documents": [
    {
      "id": "string",
      "content": null
    }
  ]
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/rerank_multimodal", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "rerank.score",
      "index": 0,
      "score": 0
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Rerank prompts by relevance#

POST/rerank

Re-scores pre-rendered text prompts based on relevance to a query using native or ONNX reranking models.

Client Responsibilities#

The client must:

Extract relevant fields from documents
Render any templates
Send pre-rendered text strings as prompts

This design keeps inference stateless and allows clients to customize rendering logic.

Models#

Models are auto-discovered from models_dir/rerankers/
Cross-encoder rerankers are supported through the existing text scorer
Late-interaction text rerankers such as ColBERT can opt in with model_manifest.json capability late_interaction or colbert
Supports quantized models (model_quantized.onnx)
Automatically prefers quantized variants if available

This endpoint is still text-only. Real ColQwen-style multimodal reranking requires a future request shape that carries page images or image-derived embeddings.

For document-based reranking with field extraction, use the client-side lib/reranking package which handles rendering before calling this endpoint.

Request Body#

Example:

{
  "model": "BAAI/bge-reranker-v2-m3",
  "query": "machine learning applications",
  "prompts": [
    "Introduction to machine learning...",
    "Deep learning fundamentals..."
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/rerank" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "query": "machine learning applications",
  "prompts": [
    "Introduction to machine learning...",
    "Deep learning fundamentals..."
  ]
}'

const response = await fetch("http://localhost:8080/ai/v1/rerank", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "BAAI/bge-reranker-v2-m3",
    "query": "machine learning applications",
    "prompts": [
      "Introduction to machine learning...",
      "Deep learning fundamentals..."
    ]
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/rerank", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "BAAI/bge-reranker-v2-m3",
    "query": "machine learning applications",
    "prompts": [
      "Introduction to machine learning...",
      "Deep learning fundamentals..."
    ]
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/rerank",
    json={
        "model": "BAAI/bge-reranker-v2-m3",
        "query": "machine learning applications",
        "prompts": [
            "Introduction to machine learning...",
            "Deep learning fundamentals..."
        ]
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "BAAI/bge-reranker-v2-m3",
  "query": "machine learning applications",
  "prompts": [
    "Introduction to machine learning...",
    "Deep learning fundamentals..."
  ]
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/rerank", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "rerank.score",
      "index": 0,
      "score": 0
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Generate text using LLM (OpenAI-compatible)#

POST/generate

Generates text using local LLM models (e.g., Gemma 3). Fully compatible with the OpenAI Chat Completions API.

Models#

Models are auto-discovered from models_dir/generators/ at startup. Use the /ai/v1/models endpoint to list available models.

Streaming#

Set stream: true to receive Server-Sent Events (SSE) with incremental token deltas. Each event contains a ChatCompletionChunk object. A successful stream ends with data: [DONE]. If generation or stream writing fails after the HTTP response starts, the server instead emits an event: error frame whose data is a plain-text error message, then closes the stream without a [DONE] frame.

Input Format#

Uses OpenAI-compatible chat format with messages array containing role (system, user, assistant) and content. Set stream: true for streaming responses.

Downloaded and inline encoded media is limited cumulatively across the request to the lower of 100 MiB, configured max_download_size_bytes, and—when admission.inference.max_concurrent_requests is positive—16 MiB times that capacity. A zero configured download limit disables nonempty media. Remote URL byte potential is reserved before fetch; inline sources reserve their actual encoded size without adding it to the existing request-body reservation. Accepted image headers are then validated and decoded source pixels are admitted at a conservative 16 bytes per pixel against the lower of 512 MiB or 16 MiB times a positive admission.inference.max_concurrent_requests; a zero concurrency setting still uses the finite 512 MiB ceiling. max_image_dimension limits each source edge. Malformed images return 400, while dimension or aggregate excess returns 413 before model loading. Initial capacity admission occurs before media fetch, so an overloaded server returns 503 without fetching content.

Request Body#

Example:

{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/generate" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}'

const response = await fetch("http://localhost:8080/ai/v1/generate", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "google/gemma-3-1b-it",
    "messages": [
      {
        "role": null,
        "content": null,
        "tool_calls": [
          null
        ],
        "tool_call_id": "string"
      }
    ],
    "max_tokens": 256,
    "temperature": 0.7,
    "top_p": 0,
    "top_k": 0,
    "stream": true,
    "stream_options": {
      "include_usage": true
    },
    "chat_template_kwargs": {
      "enable_thinking": true
    },
    "ignore_eos": true,
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather in a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              }
            },
            "required": [
              "location"
            ]
          },
          "strict": true
        }
      }
    ],
    "min_p": 0,
    "repetition_penalty": 0,
    "frequency_penalty": -2,
    "presence_penalty": -2,
    "response_format": {
      "type": "text",
      "json_schema": {
        "name": "string",
        "strict": true,
        "schema": {}
      }
    },
    "grammar": "string",
    "draft_model": "string",
    "speculative_k": 1,
    "speculation_policy": "auto",
    "speculation_calibration": "none",
    "cache_dtype": "f16",
    "cache_compaction_ratio": 0,
    "prompt_cache_key": "string",
    "prompt_cache": true,
    "backend": "auto",
    "mode": "eager",
    "compiled_target": "partitioned",
    "tool_choice": "auto"
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/generate", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "google/gemma-3-1b-it",
    "messages": [
      {
        "role": null,
        "content": null,
        "tool_calls": [
          null
        ],
        "tool_call_id": "string"
      }
    ],
    "max_tokens": 256,
    "temperature": 0.7,
    "top_p": 0,
    "top_k": 0,
    "stream": true,
    "stream_options": {
      "include_usage": true
    },
    "chat_template_kwargs": {
      "enable_thinking": true
    },
    "ignore_eos": true,
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather in a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              }
            },
            "required": [
              "location"
            ]
          },
          "strict": true
        }
      }
    ],
    "min_p": 0,
    "repetition_penalty": 0,
    "frequency_penalty": -2,
    "presence_penalty": -2,
    "response_format": {
      "type": "text",
      "json_schema": {
        "name": "string",
        "strict": true,
        "schema": {}
      }
    },
    "grammar": "string",
    "draft_model": "string",
    "speculative_k": 1,
    "speculation_policy": "auto",
    "speculation_calibration": "none",
    "cache_dtype": "f16",
    "cache_compaction_ratio": 0,
    "prompt_cache_key": "string",
    "prompt_cache": true,
    "backend": "auto",
    "mode": "eager",
    "compiled_target": "partitioned",
    "tool_choice": "auto"
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/generate",
    json={
        "model": "google/gemma-3-1b-it",
        "messages": [
            {
                "role": null,
                "content": null,
                "tool_calls": [
                    null
                ],
                "tool_call_id": "string"
            }
        ],
        "max_tokens": 256,
        "temperature": 0.7,
        "top_p": 0,
        "top_k": 0,
        "stream": true,
        "stream_options": {
            "include_usage": true
        },
        "chat_template_kwargs": {
            "enable_thinking": true
        },
        "ignore_eos": true,
        "tools": [
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get the current weather in a location",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {
                                "type": "string",
                                "description": "The city and state, e.g. San Francisco, CA"
                            }
                        },
                        "required": [
                            "location"
                        ]
                    },
                    "strict": true
                }
            }
        ],
        "min_p": 0,
        "repetition_penalty": 0,
        "frequency_penalty": -2,
        "presence_penalty": -2,
        "response_format": {
            "type": "text",
            "json_schema": {
                "name": "string",
                "strict": true,
                "schema": {}
            }
        },
        "grammar": "string",
        "draft_model": "string",
        "speculative_k": 1,
        "speculation_policy": "auto",
        "speculation_calibration": "none",
        "cache_dtype": "f16",
        "cache_compaction_ratio": 0,
        "prompt_cache_key": "string",
        "prompt_cache": true,
        "backend": "auto",
        "mode": "eager",
        "compiled_target": "partitioned",
        "tool_choice": "auto"
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/generate", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1704123456,
  "model": "string",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": null,
        "content": "string",
        "reasoning_content": "string",
        "tool_calls": [
          null
        ]
      },
      "finish_reason": "stop",
      "logprobs": {}
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  },
  "speculation": {
    "policy": "string",
    "calibration": "string",
    "decision": "string",
    "disabled_reason": "string"
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Generate text for a synchronous batch of requests#

POST/generate/batch

Runs multiple non-streaming generation requests as one synchronous batch. Compatible native requests for the same model are executed through the native batched KV decoder; unsupported per-item options are returned as per-item errors without failing sibling requests.

A syntactically valid batch envelope returns HTTP 200 even when some or all items fail. Clients must inspect each item's mutually exclusive response and error fields, plus summary, rather than treating HTTP 200 as success for every item. Fatal service or envelope failures still use non-2xx responses.

This endpoint implements the synchronous form only. Future async durable batching will use the same request item shape with mode: async.

Batch generation is text-only. Image or other multimodal content parts are rejected per item as UNSUPPORTED_MULTIMODAL before media fetch or model loading; other item failures remain independently reported.

Request Body#

Example:

{
  "mode": "sync",
  "requests": [
    {
      "custom_id": "string",
      "body": {
        "model": "google/gemma-3-1b-it",
        "messages": [
          {
            "role": null,
            "content": null,
            "tool_calls": [
              null
            ],
            "tool_call_id": "string"
          }
        ],
        "max_tokens": 256,
        "temperature": 0.7,
        "top_p": 0,
        "top_k": 0,
        "stream": true,
        "stream_options": {
          "include_usage": true
        },
        "chat_template_kwargs": {
          "enable_thinking": true
        },
        "ignore_eos": true,
        "tools": [
          {
            "type": "function",
            "function": {
              "name": "get_weather",
              "description": "Get the current weather in a location",
              "parameters": {
                "type": "object",
                "properties": {
                  "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                  }
                },
                "required": [
                  "location"
                ]
              },
              "strict": true
            }
          }
        ],
        "min_p": 0,
        "repetition_penalty": 0,
        "frequency_penalty": -2,
        "presence_penalty": -2,
        "response_format": {
          "type": "text",
          "json_schema": {
            "name": "string",
            "strict": true,
            "schema": {}
          }
        },
        "grammar": "string",
        "draft_model": "string",
        "speculative_k": 1,
        "speculation_policy": "auto",
        "speculation_calibration": "none",
        "cache_dtype": "f16",
        "cache_compaction_ratio": 0,
        "prompt_cache_key": "string",
        "prompt_cache": true,
        "backend": "auto",
        "mode": "eager",
        "compiled_target": "partitioned",
        "tool_choice": "auto"
      }
    }
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/generate/batch" \
  -H "Content-Type: application/json" \
  -d '{
  "mode": "sync",
  "requests": [
    {
      "custom_id": "string",
      "body": {
        "model": "google/gemma-3-1b-it",
        "messages": [
          {
            "role": null,
            "content": null,
            "tool_calls": [
              null
            ],
            "tool_call_id": "string"
          }
        ],
        "max_tokens": 256,
        "temperature": 0.7,
        "top_p": 0,
        "top_k": 0,
        "stream": true,
        "stream_options": {
          "include_usage": true
        },
        "chat_template_kwargs": {
          "enable_thinking": true
        },
        "ignore_eos": true,
        "tools": [
          {
            "type": "function",
            "function": {
              "name": "get_weather",
              "description": "Get the current weather in a location",
              "parameters": {
                "type": "object",
                "properties": {
                  "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                  }
                },
                "required": [
                  "location"
                ]
              },
              "strict": true
            }
          }
        ],
        "min_p": 0,
        "repetition_penalty": 0,
        "frequency_penalty": -2,
        "presence_penalty": -2,
        "response_format": {
          "type": "text",
          "json_schema": {
            "name": "string",
            "strict": true,
            "schema": {}
          }
        },
        "grammar": "string",
        "draft_model": "string",
        "speculative_k": 1,
        "speculation_policy": "auto",
        "speculation_calibration": "none",
        "cache_dtype": "f16",
        "cache_compaction_ratio": 0,
        "prompt_cache_key": "string",
        "prompt_cache": true,
        "backend": "auto",
        "mode": "eager",
        "compiled_target": "partitioned",
        "tool_choice": "auto"
      }
    }
  ]
}'

const response = await fetch("http://localhost:8080/ai/v1/generate/batch", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "mode": "sync",
    "requests": [
      {
        "custom_id": "string",
        "body": {
          "model": "google/gemma-3-1b-it",
          "messages": [
            {
              "role": null,
              "content": null,
              "tool_calls": [
                null
              ],
              "tool_call_id": "string"
            }
          ],
          "max_tokens": 256,
          "temperature": 0.7,
          "top_p": 0,
          "top_k": 0,
          "stream": true,
          "stream_options": {
            "include_usage": true
          },
          "chat_template_kwargs": {
            "enable_thinking": true
          },
          "ignore_eos": true,
          "tools": [
            {
              "type": "function",
              "function": {
                "name": "get_weather",
                "description": "Get the current weather in a location",
                "parameters": {
                  "type": "object",
                  "properties": {
                    "location": {
                      "type": "string",
                      "description": "The city and state, e.g. San Francisco, CA"
                    }
                  },
                  "required": [
                    "location"
                  ]
                },
                "strict": true
              }
            }
          ],
          "min_p": 0,
          "repetition_penalty": 0,
          "frequency_penalty": -2,
          "presence_penalty": -2,
          "response_format": {
            "type": "text",
            "json_schema": {
              "name": "string",
              "strict": true,
              "schema": {}
            }
          },
          "grammar": "string",
          "draft_model": "string",
          "speculative_k": 1,
          "speculation_policy": "auto",
          "speculation_calibration": "none",
          "cache_dtype": "f16",
          "cache_compaction_ratio": 0,
          "prompt_cache_key": "string",
          "prompt_cache": true,
          "backend": "auto",
          "mode": "eager",
          "compiled_target": "partitioned",
          "tool_choice": "auto"
        }
      }
    ]
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/generate/batch", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "mode": "sync",
    "requests": [
      {
        "custom_id": "string",
        "body": {
          "model": "google/gemma-3-1b-it",
          "messages": [
            {
              "role": null,
              "content": null,
              "tool_calls": [
                null
              ],
              "tool_call_id": "string"
            }
          ],
          "max_tokens": 256,
          "temperature": 0.7,
          "top_p": 0,
          "top_k": 0,
          "stream": true,
          "stream_options": {
            "include_usage": true
          },
          "chat_template_kwargs": {
            "enable_thinking": true
          },
          "ignore_eos": true,
          "tools": [
            {
              "type": "function",
              "function": {
                "name": "get_weather",
                "description": "Get the current weather in a location",
                "parameters": {
                  "type": "object",
                  "properties": {
                    "location": {
                      "type": "string",
                      "description": "The city and state, e.g. San Francisco, CA"
                    }
                  },
                  "required": [
                    "location"
                  ]
                },
                "strict": true
              }
            }
          ],
          "min_p": 0,
          "repetition_penalty": 0,
          "frequency_penalty": -2,
          "presence_penalty": -2,
          "response_format": {
            "type": "text",
            "json_schema": {
              "name": "string",
              "strict": true,
              "schema": {}
            }
          },
          "grammar": "string",
          "draft_model": "string",
          "speculative_k": 1,
          "speculation_policy": "auto",
          "speculation_calibration": "none",
          "cache_dtype": "f16",
          "cache_compaction_ratio": 0,
          "prompt_cache_key": "string",
          "prompt_cache": true,
          "backend": "auto",
          "mode": "eager",
          "compiled_target": "partitioned",
          "tool_choice": "auto"
        }
      }
    ]
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/generate/batch",
    json={
        "mode": "sync",
        "requests": [
            {
                "custom_id": "string",
                "body": {
                    "model": "google/gemma-3-1b-it",
                    "messages": [
                        {
                            "role": null,
                            "content": null,
                            "tool_calls": [
                                null
                            ],
                            "tool_call_id": "string"
                        }
                    ],
                    "max_tokens": 256,
                    "temperature": 0.7,
                    "top_p": 0,
                    "top_k": 0,
                    "stream": true,
                    "stream_options": {
                        "include_usage": true
                    },
                    "chat_template_kwargs": {
                        "enable_thinking": true
                    },
                    "ignore_eos": true,
                    "tools": [
                        {
                            "type": "function",
                            "function": {
                                "name": "get_weather",
                                "description": "Get the current weather in a location",
                                "parameters": {
                                    "type": "object",
                                    "properties": {
                                        "location": {
                                            "type": "string",
                                            "description": "The city and state, e.g. San Francisco, CA"
                                        }
                                    },
                                    "required": [
                                        "location"
                                    ]
                                },
                                "strict": true
                            }
                        }
                    ],
                    "min_p": 0,
                    "repetition_penalty": 0,
                    "frequency_penalty": -2,
                    "presence_penalty": -2,
                    "response_format": {
                        "type": "text",
                        "json_schema": {
                            "name": "string",
                            "strict": true,
                            "schema": {}
                        }
                    },
                    "grammar": "string",
                    "draft_model": "string",
                    "speculative_k": 1,
                    "speculation_policy": "auto",
                    "speculation_calibration": "none",
                    "cache_dtype": "f16",
                    "cache_compaction_ratio": 0,
                    "prompt_cache_key": "string",
                    "prompt_cache": true,
                    "backend": "auto",
                    "mode": "eager",
                    "compiled_target": "partitioned",
                    "tool_choice": "auto"
                }
            }
        ]
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "mode": "sync",
  "requests": [
    {
      "custom_id": "string",
      "body": {
        "model": "google/gemma-3-1b-it",
        "messages": [
          {
            "role": null,
            "content": null,
            "tool_calls": [
              null
            ],
            "tool_call_id": "string"
          }
        ],
        "max_tokens": 256,
        "temperature": 0.7,
        "top_p": 0,
        "top_k": 0,
        "stream": true,
        "stream_options": {
          "include_usage": true
        },
        "chat_template_kwargs": {
          "enable_thinking": true
        },
        "ignore_eos": true,
        "tools": [
          {
            "type": "function",
            "function": {
              "name": "get_weather",
              "description": "Get the current weather in a location",
              "parameters": {
                "type": "object",
                "properties": {
                  "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                  }
                },
                "required": [
                  "location"
                ]
              },
              "strict": true
            }
          }
        ],
        "min_p": 0,
        "repetition_penalty": 0,
        "frequency_penalty": -2,
        "presence_penalty": -2,
        "response_format": {
          "type": "text",
          "json_schema": {
            "name": "string",
            "strict": true,
            "schema": {}
          }
        },
        "grammar": "string",
        "draft_model": "string",
        "speculative_k": 1,
        "speculation_policy": "auto",
        "speculation_calibration": "none",
        "cache_dtype": "f16",
        "cache_compaction_ratio": 0,
        "prompt_cache_key": "string",
        "prompt_cache": true,
        "backend": "auto",
        "mode": "eager",
        "compiled_target": "partitioned",
        "tool_choice": "auto"
      }
    }
  ]
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/generate/batch", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "generate.batch",
  "data": [
    {
      "custom_id": "string",
      "index": 0,
      "response": {
        "id": "chatcmpl-abc123",
        "object": "chat.completion",
        "created": 1704123456,
        "model": "string",
        "choices": [
          {
            "index": 0,
            "message": {
              "role": null,
              "content": "string",
              "reasoning_content": "string",
              "tool_calls": [
                null
              ]
            },
            "finish_reason": "stop",
            "logprobs": {}
          }
        ],
        "usage": {
          "prompt_tokens": 0,
          "completion_tokens": 0,
          "total_tokens": 0,
          "cached_prompt_tokens": 0
        },
        "speculation": {
          "policy": "string",
          "calibration": "string",
          "decision": "string",
          "disabled_reason": "string"
        }
      },
      "error": {
        "code": "string",
        "message": "string",
        "retryable": true,
        "retry_after_ms": 1
      }
    }
  ],
  "summary": {
    "total": 0,
    "succeeded": 0,
    "failed": 0
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

OpenAI Chat Completions endpoint#

POST/chat/completions

OpenAI-compatible chat completions path for SDKs that call /chat/completions relative to the configured base URL.

Request Body#

Example:

{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}'

const response = await fetch("http://localhost:8080/ai/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "google/gemma-3-1b-it",
    "messages": [
      {
        "role": null,
        "content": null,
        "tool_calls": [
          null
        ],
        "tool_call_id": "string"
      }
    ],
    "max_tokens": 256,
    "temperature": 0.7,
    "top_p": 0,
    "top_k": 0,
    "stream": true,
    "stream_options": {
      "include_usage": true
    },
    "chat_template_kwargs": {
      "enable_thinking": true
    },
    "ignore_eos": true,
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather in a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              }
            },
            "required": [
              "location"
            ]
          },
          "strict": true
        }
      }
    ],
    "min_p": 0,
    "repetition_penalty": 0,
    "frequency_penalty": -2,
    "presence_penalty": -2,
    "response_format": {
      "type": "text",
      "json_schema": {
        "name": "string",
        "strict": true,
        "schema": {}
      }
    },
    "grammar": "string",
    "draft_model": "string",
    "speculative_k": 1,
    "speculation_policy": "auto",
    "speculation_calibration": "none",
    "cache_dtype": "f16",
    "cache_compaction_ratio": 0,
    "prompt_cache_key": "string",
    "prompt_cache": true,
    "backend": "auto",
    "mode": "eager",
    "compiled_target": "partitioned",
    "tool_choice": "auto"
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "google/gemma-3-1b-it",
    "messages": [
      {
        "role": null,
        "content": null,
        "tool_calls": [
          null
        ],
        "tool_call_id": "string"
      }
    ],
    "max_tokens": 256,
    "temperature": 0.7,
    "top_p": 0,
    "top_k": 0,
    "stream": true,
    "stream_options": {
      "include_usage": true
    },
    "chat_template_kwargs": {
      "enable_thinking": true
    },
    "ignore_eos": true,
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather in a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              }
            },
            "required": [
              "location"
            ]
          },
          "strict": true
        }
      }
    ],
    "min_p": 0,
    "repetition_penalty": 0,
    "frequency_penalty": -2,
    "presence_penalty": -2,
    "response_format": {
      "type": "text",
      "json_schema": {
        "name": "string",
        "strict": true,
        "schema": {}
      }
    },
    "grammar": "string",
    "draft_model": "string",
    "speculative_k": 1,
    "speculation_policy": "auto",
    "speculation_calibration": "none",
    "cache_dtype": "f16",
    "cache_compaction_ratio": 0,
    "prompt_cache_key": "string",
    "prompt_cache": true,
    "backend": "auto",
    "mode": "eager",
    "compiled_target": "partitioned",
    "tool_choice": "auto"
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/chat/completions",
    json={
        "model": "google/gemma-3-1b-it",
        "messages": [
            {
                "role": null,
                "content": null,
                "tool_calls": [
                    null
                ],
                "tool_call_id": "string"
            }
        ],
        "max_tokens": 256,
        "temperature": 0.7,
        "top_p": 0,
        "top_k": 0,
        "stream": true,
        "stream_options": {
            "include_usage": true
        },
        "chat_template_kwargs": {
            "enable_thinking": true
        },
        "ignore_eos": true,
        "tools": [
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get the current weather in a location",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {
                                "type": "string",
                                "description": "The city and state, e.g. San Francisco, CA"
                            }
                        },
                        "required": [
                            "location"
                        ]
                    },
                    "strict": true
                }
            }
        ],
        "min_p": 0,
        "repetition_penalty": 0,
        "frequency_penalty": -2,
        "presence_penalty": -2,
        "response_format": {
            "type": "text",
            "json_schema": {
                "name": "string",
                "strict": true,
                "schema": {}
            }
        },
        "grammar": "string",
        "draft_model": "string",
        "speculative_k": 1,
        "speculation_policy": "auto",
        "speculation_calibration": "none",
        "cache_dtype": "f16",
        "cache_compaction_ratio": 0,
        "prompt_cache_key": "string",
        "prompt_cache": true,
        "backend": "auto",
        "mode": "eager",
        "compiled_target": "partitioned",
        "tool_choice": "auto"
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "google/gemma-3-1b-it",
  "messages": [
    {
      "role": null,
      "content": null,
      "tool_calls": [
        null
      ],
      "tool_call_id": "string"
    }
  ],
  "max_tokens": 256,
  "temperature": 0.7,
  "top_p": 0,
  "top_k": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "ignore_eos": true,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            }
          },
          "required": [
            "location"
          ]
        },
        "strict": true
      }
    }
  ],
  "min_p": 0,
  "repetition_penalty": 0,
  "frequency_penalty": -2,
  "presence_penalty": -2,
  "response_format": {
    "type": "text",
    "json_schema": {
      "name": "string",
      "strict": true,
      "schema": {}
    }
  },
  "grammar": "string",
  "draft_model": "string",
  "speculative_k": 1,
  "speculation_policy": "auto",
  "speculation_calibration": "none",
  "cache_dtype": "f16",
  "cache_compaction_ratio": 0,
  "prompt_cache_key": "string",
  "prompt_cache": true,
  "backend": "auto",
  "mode": "eager",
  "compiled_target": "partitioned",
  "tool_choice": "auto"
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/chat/completions", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1704123456,
  "model": "string",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": null,
        "content": "string",
        "reasoning_content": "string",
        "tool_calls": [
          null
        ]
      },
      "finish_reason": "stop",
      "logprobs": {}
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  },
  "speculation": {
    "policy": "string",
    "calibration": "string",
    "decision": "string",
    "disabled_reason": "string"
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Rewrite text using Seq2Seq models#

POST/rewrite

Rewrite/transform text using Seq2Seq models (T5, FLAN-T5, BART, etc.).

Models#

Models are auto-discovered from models_dir/rewriters/
Seq2Seq models have encoder.onnx, decoder-init.onnx, and decoder.onnx files
Compatible with LMQG question generation models

Use Cases#

Question Generation: Generate questions from answer-context pairs
Query Generation: Generate search queries from documents
Paraphrasing: Rewrite text in different words
Translation: Translate text between languages

Request Body#

Example:

{
  "model": "lmqg/flan-t5-small-squad-qg",
  "inputs": [
    "Translate to German: Hello, how are you?"
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/rewrite" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "lmqg/flan-t5-small-squad-qg",
  "inputs": [
    "Translate to German: Hello, how are you?"
  ]
}'

const response = await fetch("http://localhost:8080/ai/v1/rewrite", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "lmqg/flan-t5-small-squad-qg",
    "inputs": [
      "Translate to German: Hello, how are you?"
    ]
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/rewrite", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "lmqg/flan-t5-small-squad-qg",
    "inputs": [
      "Translate to German: Hello, how are you?"
    ]
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/rewrite",
    json={
        "model": "lmqg/flan-t5-small-squad-qg",
        "inputs": [
            "Translate to German: Hello, how are you?"
        ]
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "lmqg/flan-t5-small-squad-qg",
  "inputs": [
    "Translate to German: Hello, how are you?"
  ]
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/rewrite", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "rewrite",
      "index": 0,
      "texts": [
        "string"
      ]
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Read text from images (OCR/document understanding)#

POST/read

Extracts text from images using Vision2Seq models like TrOCR, Donut, Florence-2, or Pix2Struct.

Models#

Models are auto-discovered from models_dir/readers/ at startup. Use the /ai/v1/models endpoint to list available models.

TrOCR: Pure OCR for printed/handwritten text
Donut: Document understanding with structured output (receipts, forms)
Florence-2: Multi-task vision model (OCR, captioning, VQA)
Pix2Struct: Visual question answering with natural-language prompts
Moondream: Decoder-only vision-language reader that can return text plus optional flattened fields

Task Prompts#

Some models support task prompts for different extraction modes:

Donut CORD: <s_cord-v2> for receipt parsing
Donut DocVQA: <s_docvqa><s_question>...</s_question><s_answer> for visual QA
Florence-2 OCR: <OCR> for text extraction
Florence-2 Caption: <CAPTION> for image description
Pix2Struct: natural-language questions like What type of document is this?
Moondream: natural-language prompts like Describe this image.

Image admission reserves the effective downloaded-byte ceiling at 16 MiB per weighted capacity unit and at least one unit per two declared images. The effective ceiling is the minimum of the configured per-image limit times image count, the read-batch limit (256 MiB by default), and—when admission is bounded—16 MiB times admission.inference.max_concurrent_requests. Admission happens before model resolution or download.

After download, image headers are validated before model loading. Decoded source pixels are admitted at a conservative 16 bytes per pixel against the lower of 512 MiB or 16 MiB times a positive admission.inference.max_concurrent_requests; a zero concurrency setting still uses the finite 512 MiB ceiling. The reservation grows atomically from downloaded-byte admission before inference. max_image_dimension limits each source edge; malformed images return 400 and dimension or aggregate excess returns 413. The same decoded-pixel policy covers generate/chat, dense embedding, multimodal reranking, image /extract, and the embedded read, extract, and dense embedding APIs. Batch generation rejects multimodal content before media fetch.

Request Body#

Example:

{
  "model": "microsoft/trocr-base-printed",
  "images": [
    {
      "url": "data:image/png;base64,iVBORw0KGgo..."
    }
  ],
  "prompt": "What type of document is this?",
  "max_tokens": 256
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/read" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "microsoft/trocr-base-printed",
  "images": [
    {
      "url": "data:image/png;base64,iVBORw0KGgo..."
    }
  ],
  "prompt": "What type of document is this?",
  "max_tokens": 256
}'

const response = await fetch("http://localhost:8080/ai/v1/read", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "microsoft/trocr-base-printed",
    "images": [
      {
        "url": "data:image/png;base64,iVBORw0KGgo..."
      }
    ],
    "prompt": "What type of document is this?",
    "max_tokens": 256
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/read", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "microsoft/trocr-base-printed",
    "images": [
      {
        "url": "data:image/png;base64,iVBORw0KGgo..."
      }
    ],
    "prompt": "What type of document is this?",
    "max_tokens": 256
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/read",
    json={
        "model": "microsoft/trocr-base-printed",
        "images": [
            {
                "url": "data:image/png;base64,iVBORw0KGgo..."
            }
        ],
        "prompt": "What type of document is this?",
        "max_tokens": 256
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "microsoft/trocr-base-printed",
  "images": [
    {
      "url": "data:image/png;base64,iVBORw0KGgo..."
    }
  ],
  "prompt": "What type of document is this?",
  "max_tokens": 256
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/read", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {
      "text": "Invoice Total: $123.45",
      "fields": {
        "menu.nm": "Coffee",
        "menu.price": "$3.50",
        "total": "$123.45"
      },
      "regions": [
        {
          "text": "string",
          "bbox": [
            0
          ],
          "confidence": 0,
          "label": "string"
        }
      ]
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Transcribe audio to text (speech-to-text)#

POST/transcribe

Transcribes audio to text using Speech2Seq models like Whisper, Wav2Vec2, or HuBERT.

Models#

Models are auto-discovered from models_dir/transcribers/ at startup. Use the /api/models endpoint to list available models.

Whisper: OpenAI's Whisper models (multilingual, automatic language detection)
Wav2Vec2: Facebook's Wav2Vec 2.0 models (English-focused)
HuBERT: Facebook's HuBERT models (self-supervised)

Audio Input#

Audio data should be base64-encoded. Supported formats depend on the model:

WAV (recommended - raw PCM)
MP3
FLAC
M4A/AAC

Request Body#

Example:

{
  "model": "openai/whisper-tiny",
  "audio": "string",
  "language": "en"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/transcribe" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "openai/whisper-tiny",
  "audio": "string",
  "language": "en"
}'

const response = await fetch("http://localhost:8080/ai/v1/transcribe", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "openai/whisper-tiny",
    "audio": "string",
    "language": "en"
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/transcribe", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "openai/whisper-tiny",
    "audio": "string",
    "language": "en"
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/transcribe",
    json={
        "model": "openai/whisper-tiny",
        "audio": "string",
        "language": "en"
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "openai/whisper-tiny",
  "audio": "string",
  "language": "en"
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/transcribe", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "transcription",
      "index": 0,
      "text": "Hello, how are you today?",
      "language": "en"
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cached_prompt_tokens": 0
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Extract entities, relations, classifications, and structures#

POST/extract

Schema-driven extraction over shared AI content parts. This is the canonical public API for named entity recognition, relation extraction, text/document classification, token classification, and structured document extraction.

Image-backed extraction uses the same byte-reserving and image-count-weighted admission policy as /read, before model resolution or download. Text-only extraction consumes one admission unit.

Request Body#

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/extract"

const response = await fetch("http://localhost:8080/ai/v1/extract", {
  method: "POST"
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/extract", {
  method: "POST"
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post("http://localhost:8080/ai/v1/extract")

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/extract", nil)

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

No response body

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

List available models#

GET/models

Returns lists of available embedding, chunking, reranking, generator, extractor, rewriter, reader, and transcriber models.

Embedders#

ONNX models from models_dir/embedders/
Quantized variants have -i8 suffix

Chunkers#

Always includes "fixed" (built-in)
Plus any ONNX models from models_dir/chunkers/

Rerankers#

Native or ONNX rerankers from models_dir/rerankers/
model_manifest.json capabilities can mark late-interaction text rerankers (late_interaction, colbert)
Empty if no models configured

Generators#

LLM models from models_dir/generators/
Empty if no models configured

Extractors#

Extraction-capable models from the managed model registry
Includes GLiNER models for zero-shot entity and relation extraction

Rewriters#

Seq2Seq models from models_dir/rewriters/
T5, FLAN-T5, BART, and LMQG question generation models

Readers#

Vision2Seq models from models_dir/readers/
TrOCR, Donut, Florence-2 for OCR and document understanding

Transcribers#

Speech2Seq models from models_dir/transcribers/
Whisper, Wav2Vec2, HuBERT for speech-to-text

Models are discovered at service startup and cached.

Code Examples#

curl -X GET "http://localhost:8080/ai/v1/models"

const response = await fetch("http://localhost:8080/ai/v1/models", {
  method: "GET"
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/models", {
  method: "GET"
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.get("http://localhost:8080/ai/v1/models")

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    req, _ := http.NewRequest("GET", "http://localhost:8080/ai/v1/models", nil)

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {}
  ],
  "allow_downloads": true,
  "backends": {
    "native": true,
    "onnx": false,
    "metal": true,
    "cuda": false,
    "xla": false,
    "wasm": false
  },
  "chunkers": {},
  "rerankers": {},
  "classifiers": {},
  "embedders": {},
  "extractors": {},
  "generators": {},
  "rewriters": {},
  "readers": {},
  "transcribers": {}
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Create embeddings (OpenAI-compatible)#

POST/embeddings

OpenAI-compatible embeddings endpoint. Accepts the same request format as OpenAI's /v1/embeddings API, served here under /ai/v1/embeddings. For sparse-capable models, each data item still uses the embedding field, but its value is a sparse vector object instead of a dense float array. Dense image inputs are header-validated and admitted against the aggregate decoded-pixel budget before model loading. Remote URL byte potential is reserved before fetch; inline sources use their actual encoded size. Use this endpoint for drop-in compatibility with OpenAI SDKs.

Request Body#

Example:

{
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/embeddings" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}'

const response = await fetch("http://localhost:8080/ai/v1/embeddings", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "string",
    "input": "string",
    "encoding_format": "float",
    "dimensions": 0,
    "task_type": "RETRIEVAL_QUERY",
    "instruction": "string",
    "input_type": "search_query",
    "error_policy": "fail_fast"
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/embeddings", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "string",
    "input": "string",
    "encoding_format": "float",
    "dimensions": 0,
    "task_type": "RETRIEVAL_QUERY",
    "instruction": "string",
    "input_type": "search_query",
    "error_policy": "fail_fast"
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/embeddings",
    json={
        "model": "string",
        "input": "string",
        "encoding_format": "float",
        "dimensions": 0,
        "task_type": "RETRIEVAL_QUERY",
        "instruction": "string",
        "input_type": "search_query",
        "error_policy": "fail_fast"
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "string",
  "input": "string",
  "encoding_format": "float",
  "dimensions": 0,
  "task_type": "RETRIEVAL_QUERY",
  "instruction": "string",
  "input_type": "search_query",
  "error_policy": "fail_fast"
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/embeddings", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [
        0
      ],
      "index": 0
    }
  ],
  "model": "string",
  "usage": {
    "prompt_tokens": 0,
    "total_tokens": 0
  },
  "errors": [
    {
      "index": 0,
      "code": "string",
      "message": "string",
      "stage": "parse",
      "retryable": true,
      "status": 0,
      "retry_after_ms": 0
    }
  ],
  "summary": {
    "total": 0,
    "succeeded": 0,
    "failed": 0
  }
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

List Traditional ML predictors#

GET/predictors

Returns the Traditional ML predictor catalog for /ml/v1/predict. Predictors are loaded from <ml_dir>/<name>/ and exposed separately from the AI model catalog.

Code Examples#

curl -X GET "http://localhost:8080/ai/v1/predictors"

const response = await fetch("http://localhost:8080/ai/v1/predictors", {
  method: "GET"
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/predictors", {
  method: "GET"
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.get("http://localhost:8080/ai/v1/predictors")

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    req, _ := http.NewRequest("GET", "http://localhost:8080/ai/v1/predictors", nil)

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "object": "list",
  "predictors": {}
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Run a traditional ML predictor#

POST/predict

Run a tabular predictor (tree ensemble, linear, or SVM) on a batch of feature vectors. Models are loaded from <ml_dir>/<name>/ and identified by name. Use /ml/v1/models for the list of available predictors and their feature schemas.

Request Body#

Example:

{
  "model": "string",
  "input": [
    [
      0
    ]
  ]
}

Code Examples#

curl -X POST "http://localhost:8080/ai/v1/predict" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "string",
  "input": [
    [
      0
    ]
  ]
}'

const response = await fetch("http://localhost:8080/ai/v1/predict", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "string",
    "input": [
      [
        0
      ]
    ]
  })
});

const data = await response.json();

fetch("http://localhost:8080/ai/v1/predict", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "model": "string",
    "input": [
      [
        0
      ]
    ]
  })
})
  .then(response => response.json())
  .then(data => console.log(data));

import requests

response = requests.post(
    "http://localhost:8080/ai/v1/predict",
    json={
        "model": "string",
        "input": [
            [
                0
            ]
        ]
    }
)

data = response.json()

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func main() {
    body := []byte(`{
  "model": "string",
  "input": [
    [
      0
    ]
  ]
}`)
    req, _ := http.NewRequest("POST", "http://localhost:8080/ai/v1/predict", bytes.NewBuffer(body))
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, _ := client.Do(req)
    defer resp.Body.Close()
}

Responses#

{
  "model": "string",
  "task": "regression",
  "predictions": [
    [
      0
    ]
  ]
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

{
  "error": "string",
  "message": "string",
  "reason": "inference_capacity",
  "retryable": true,
  "retry_after_ms": 0
}

Inference API#

What is Antfly inference#

When to Use Antfly inference#

Authentication#

Features#

Embedding Generation#

Text Chunking#

Reranking#

Create embeddings (alias of /embeddings)#

Request Body#

Code Examples#

Responses#

Chunk text into smaller segments#

Models#

Fixed Chunking (always available)#

ONNX Models#

Caching#

Request Body#

Code Examples#

Responses#

Rerank multimodal documents by relevance#

Request Body#

Code Examples#

Responses#

Rerank prompts by relevance#

Client Responsibilities#

Models#

Request Body#

Code Examples#

Responses#

Generate text using LLM (OpenAI-compatible)#

Models#

Streaming#

Input Format#

Request Body#

Code Examples#

Responses#

Generate text for a synchronous batch of requests#

Request Body#

Code Examples#

Responses#

OpenAI Chat Completions endpoint#

Request Body#

Code Examples#

Responses#

Rewrite text using Seq2Seq models#

Models#

Use Cases#

Request Body#

Code Examples#

Responses#

Read text from images (OCR/document understanding)#

Models#

Task Prompts#

Request Body#

Code Examples#

Responses#

Transcribe audio to text (speech-to-text)#

Models#

Audio Input#

Request Body#

Code Examples#

Responses#

Extract entities, relations, classifications, and structures#

Request Body#

Code Examples#

Responses#

List available models#

Embedders#

Chunkers#

Rerankers#

Generators#

Extractors#

Rewriters#

Readers#

Transcribers#

Code Examples#

Responses#

Create embeddings (OpenAI-compatible)#

Request Body#

Create embeddings (alias of `/embeddings`)#