AntflyAntfly

Antfly Inference on Kubernetes#

The Antfly operator provides Kubernetes-native management for Antfly Inference inference pools with autoscaling, GPU/TPU support, and intelligent traffic routing.

Overview#

The unified Antfly operator automates the deployment and management of Antfly Inference inference servers on Kubernetes:

  • Autoscaling - Scale inference pools based on request load
  • GPU/TPU Support - Schedule workloads on accelerated hardware
  • Traffic Routing - Route requests to optimal model instances
  • Health Monitoring - Automatic health checks and recovery
  • Model Versioning - Manage multiple model versions

Installation#

Prerequisites#

  • Kubernetes 1.24+
  • kubectl configured with cluster access

Install with kubectl#

kubectl apply -f https://antfly.io/antfly-operator-install.yaml

Quick Start#

Create an Inference Pool#

apiVersion: antfly.io/v1alpha1
kind: Antfly InferencePool
metadata:
  name: embeddings
  namespace: default
spec:
  models:
    preload:
      - name: BAAI/bge-small-en-v1.5
  replicas:
    min: 2
    max: 2
  resources:
    limits:
      memory: 4Gi
      cpu: 2

Apply the configuration:

kubectl apply -f inference-pool.yaml

Check Status#

kubectl get inferencepool embeddings

Configuration#

Antfly InferencePool Spec#

FieldTypeDescription
modelstringModel name to serve (e.g., BAAI/bge-small-en-v1.5)
replicasintNumber of replicas
resourcesResourceRequirementsCPU/memory/GPU limits
autoscalingAutoscalingSpecAutoscaling configuration

Autoscaling#

Enable autoscaling based on request queue depth:

spec:
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 10
    targetQueueDepth: 5

GPU Support#

Schedule on GPU nodes:

spec:
  resources:
    limits:
      nvidia.com/gpu: 1
  nodeSelector:
    accelerator: nvidia-tesla-t4

Architecture#

Antfly Inference on Kubernetes consists of:

  1. Controller - The Antfly operator watches Antfly InferencePool resources and reconciles state
  2. Proxy - Routes requests to available model instances
  3. Metrics Server - Collects metrics for autoscaling decisions

Troubleshooting#

Common Issues#

Pods stuck in Pending

  • Check node resources with kubectl describe node
  • Verify GPU drivers are installed if using GPU resources

Model not loading

  • Check pod logs: kubectl logs -l app=antfly_inference
  • Verify model name is correct

High latency

  • Enable autoscaling to handle load spikes
  • Consider using quantized models for faster inference

Next Steps#