Architecture
How Antfly Inference works
From model loading through multi-backend inference & caching, to distributed Kubernetes deployments with model-aware routing.
Inference Pipeline
API requests flow through backpressure control into lazy-loaded model registries. The session manager selects the fastest available backend—ONNX Runtime, XLA, or pure Go—then runs tokenization, inference, and decoding with two-tier caching and singleflight deduplication.
Loading diagram…
Ready to get started?
Run Antfly Inference locally in one command or deploy distributed inference pools with the Kubernetes operator.