Inference, built in.
The ML engine inside Antfly
Antfly Inference runs the models your search needs — embedders, rerankers, chunkers — directly inside Antfly. No external APIs. No per-request billing. No configuration.
Replace the toolchain
Running local AI usually means stitching together separate model servers, Python pipelines, API keys, and custom code. Antfly Inference replaces the entire ML toolchain and runs inside the database.
- Built inZero dependencies
Embedders, rerankers, and chunkers run inside Antfly. No cloud API calls, no per-request billing, no data leaving your infrastructure.
- SimpleOllama-like workflow
Pull, serve, use. The model garden handles discovery and management — if you've used Ollama, you already know it.
- Predictable costFixed, not per-request
Cloud embedding APIs charge per request and costs spiral at scale. Antfly Inference runs on your hardware at a fixed cost.
- Private by defaultData stays on your infra
Your data never leaves your infrastructure. No third-party processing, no retention policies to review.
- ONNX-optimizedFast on CPU or GPU
Models are optimized for fast inference on available hardware. No CUDA toolkit required for CPU inference.
- Always availableYour uptime, not theirs
When inference is someone else's API, their outage is your outage. Antfly Inference runs on your infrastructure.
Up and running in under a minute
1# Install Antfly Inference
2brew install antflydb/taps/antfly
3
4# Pull embedding and reranker models
5antfly inference pull BAAI/bge-small-en-v1.5
6antfly inference pull mixedbread-ai/mxbai-rerank-base-v1
7
8# Start serving
9antfly inference run
10
11# Test with curl
12curl http://localhost:8080/ai/v1/embed \
13 -d '{"model": "BAAI/bge-small-en-v1.5", "input": "Hello, world!"}'Ready for more? Read the full getting-started guide