Architecture

Overview#

Antfly is one runtime that owns storage, indexing, inference, and background work. Documents, full-text segments, vector postings, graph edges, and the models that chunk, embed, rerank, and extract all live in one process tree, so a deployment needs no external queue, cache tier, vector store, or model server. Native libraries (GPU drivers, an optional libwasmtime) plug into Antfly's runtime rather than the other way round. The server, the storage engine it shares with embedded use, and the inference runtime are written in Zig under zig/; the background for that choice is in Search-and-Inference, From First Principles.

The rest of this page describes what ships today and labels work in progress as such. Each section links the design doc that is authoritative for it.

Deployment Modes and Storage Engines#

Topology and persistence are independent choices. A deployment mode answers which processes run; a storage engine answers where durable state lives. Lite is a storage engine, not a topology.

Switching a running deployment between engines is a migration, never a config toggle. Fresh local, single-shard standalone tables without replication keep source embeddings in a separate vector store instead of the primary LSM. Design: STORAGE.md, LITE.md, VECTOR_STORE.md, SERVERLESS.md. The concept page for the single-file engine is Antfly Lite.

Data Model#

Tables, shards, and Raft groups. A table is split by key range into num_shards shards. In a distributed deployment the metadata nodes run one Raft group for cluster topology and schemas, and every shard runs its own Raft group with replication_factor replicas (default 3, maximum 5). Shards move, replicate, and fail independently. Raft messages and snapshots travel between nodes over HTTP. Lite has no Raft, and serverless tables allocate no data Raft groups. Design: raft/RAFT.md, DB.md.

Table profiles. Tables are document-first by default: each document is a compressed JSON value with optional, soft schema, and every index is derived from it. A storage_mode: relational table is the second profile on the same engine. Its schema is required and closed, rows are stored as typed packed rows, and declared scalar properties become typed columns so predicates, sorts, and aggregates can be served columnar. A json column keeps document-style indexing where it is wanted. Both profiles use the same shards, Raft, indexes, and enrichments. Design: RELATIONAL.md, SYSTEM_CATALOG.md; the Document Engine page covers mappings and queries on document tables.

Lakes and foreign sources. Lake tables query Parquet and Iceberg data in object storage in place, with Antfly-native full-text, vector, sparse, graph, and algebraic indexes built as sidecars. Lake tables and SQL are in progress (#145). Two features connect external PostgreSQL databases today. Query-time foreign_sources read or join external tables per request, and CDC (replication_sources) streams changes over logical replication into ordinary Antfly tables. Design: LAKES.md, REMOTE_TABLE_SERVING.md, CDC.md; the CDC replication guide walks through a pipeline.

Write Path#

A batch write follows a document-first contract:

1. Parse once. Each document is parsed and stripped of DB-owned special fields (_embeddings, _edges), and its text, vector, and graph directives are derived from that one parsed form.
2. Commit the base document. The base state commits through the shard's Raft log. Single-shard batches are atomic within the shard. Batches that span shards use a distributed transaction (see Consistency and Transactions).
3. Append to the derived-work log. One batch-shaped record goes into a sequence-numbered, idempotent log.
4. Advance per-index watermarks. Each index worker applies the log at its own pace and publishes a durable checkpoint saying which base sequence it serves. The log truncates at the minimum watermark. Rebuilding from the document store is a repair fallback, not the replay path.

The per-request sync_level picks how far a write gets before the response returns:

Enrichments and artifacts. Enrichments are managed producers of durable artifacts, and they come in three kinds. A chunk enrichment splits text, audio, or images into windows; the fixed chunker is the default. An embedding enrichment produces dense or sparse vectors. An asset enrichment produces any other derived payload: summaries, entity extraction, classification, decisions, OCR, and transcripts. The document_extraction asset producer turns PDFs into durable page units, falling back to local OCR for pages without native text, and it handles audio transcription. Enrichments can chain: a chunk enrichment can read an asset artifact, and an embedding enrichment can read chunks. Artifacts belong to the enrichment, not to an index, and they live inside their document's key range. As a result, one chunking pass can feed several indexes, and deleting a document cleans up its artifacts with a bounded scan. Indexes consume artifacts. An artifact-backed index can union up to 64 artifact streams, so whole-document and chunk vectors can sit in one embeddings index. Only the active shard leader runs enrichment work, and coverage accounting keeps a failed or lagging producer from reporting as ready. Design: DB.md: Write Contract, DB.md: Derived Work Log, ENRICHMENTS.md, ARTIFACTS.md; the guide is Artifact Indexes.

Indexes#

Index types form a closed set, and one table can carry any combination of them:

• full_text: BM25 over Lucene-style segments with FST term dictionaries and roaring-bitmap postings. Analyzers include Snowball stemmers. Queries cover phrase, fuzzy, wildcard, regex, numeric, and geo, plus highlighting and faceting through aggregations (terms, range, histogram, date_histogram, geo). Search visibility waits for the segment write, not compaction; merges run in the background. Design: FULL_TEXT.md.
• embeddings, dense: an HBC (hierarchical balanced clustering) index. Search runs a beam over a centroid tree (branching factor 16, search width 32), scans the closest leaf postings, then does a boundary rerank. That rerank exact-scores only the candidates whose RaBitQ error bounds overlap the cutoff. Leaves store 1-bit-per-dimension residual codes against the leaf centroid, and full-precision vectors are kept for rerank. Large indexes route through a flat exact-centroid directory instead of the tree. The only public tuning knob is search_effort (0 to 1, default 0.5). HBC applies SPFresh-style local maintenance: inserts touch one posting, and leaves split and merge locally. Routine writes never trigger a full rebuild; shard splits rebuild child indexes with a bulk builder. Distance kernels use portable @Vector SIMD with AVX2/AVX-512 and NEON paths. Design: HBC.md, SPFRESH.md, DENSE_INDEXING_LIFECYCLE.md.
• embeddings, sparse (sparse: true): a SPLADE learned-sparse inverted index.
• graph: edges come from the _edges field or from extraction artifacts. graph_queries run traversals, shortest and k-shortest paths, and MATCH patterns, and they can start from another result set such as $full_text_results. Published graph metrics (pagerank, degree, eigenvector, hits_authority, hits_hub) are maintained in the background and can be queried during rebuilds. Design: GRAPH.md, GRAPH_METRICS.md; guide: Graph Navigation.
• algebraic: typed fields derived from the schema, plus declarative exact aggregate recipes that the engine materializes. Eligible queries reuse a complete, snapshot-bound materialization instead of scanning. Design: ALGEBRAIC.md.
• relational: composite ordered keys on relational tables, each with its own direction, null placement, and collation. Indexes build asynchronously, and indexed queries require complete coverage. Design: RELATIONAL.md.

Fusion and query processors. A query can combine full-text, dense, sparse, and graph result sets. merge_config fuses them with Reciprocal Rank Fusion (default, rank_constant 60) or Relative Score Fusion, and per-index weights apply to both. Query processors then run in order. The reranker is a cross-encoder, or a late-interaction model (ColBERT- or ColQwen-style MaxSim); late interaction is a reranker, not an index type. graph_metric_rerank reorders by a published graph metric, pruner drops low-quality tails after fusion and reranking and before paging, and document_renderer formats results. Design: DB.md: Search Contract, RERANKING.md; the guide is Tune Hybrid Search.

Query Surfaces#

• HTTP JSON API under /db/v1, generated from the OpenAPI specs in specs/openapi/. Go, TypeScript, and Python SDKs wrap generated clients; the Rust SDK is generated.
• SQL and the PostgreSQL wire protocol (in progress): SQL is lowered to native typed plans rather than executed as SQL text, and an opt-in pgwire listener serves psql-compatible clients. Design: SQL extraction, pgwire.
• Native agents: POST /agents/retrieval (retrieval and streaming answers over SSE), POST /agents/query-builder (natural language to an Antfly query), and POST /agents/research, which also runs as durable /agents/research/jobs (Deep Research). Design: A2A.md, DEEP_RESEARCH.md.
• MCP and A2A: an MCP server at /mcp/v1, and an A2A adapter at /a2a with /.well-known/agent-card.json. Both delegate to the same operations as HTTP, so auth and validation stay in one place. Design: MCP.md, A2A.md.
• C API and embedded bindings: libantfly is the stable C ABI for local databases and for database-free inference. Go, Python, Rust, and TypeScript bindings sit on it under the embedded package name. Design: CAPI.md, embedded bindings.
• WASM extensions: a Postgres-style extension lifecycle under /extensions/v1, whose WASM handlers run in a sandbox with fuel and memory limits. Today only wasm: MCP-tool handlers execute, and they need an external libwasmtime that Antfly loads at runtime (ANTFLY_WASMTIME_LIB). Design: EXTENSIONS.md.

Inference#

Antfly Inference runs embedding, chunking, reranking, extraction, classification, decision, OCR, transcription, and generation models. It loads GGUF, safetensors, and ONNX directly; ONNX graphs are parsed and run on Antfly's own kernels, with no ONNX Runtime. Backends are CPU, Metal, and CUDA, with PJRT/XLA and WASM/WebGPU builds experimental. The runtime serves /ai/v1 beside /db/v1.

Standalone runs inference in-process, and managed enrichments call it directly instead of looping back through HTTP. Builds with a GPU or driver-backed backend (Metal, CUDA, PJRT) delegate all inference to a supervised antfly inference _worker child, so a wedged driver call is killed and respawned without taking the database down. Setting inference.api_url turns embedded inference off and sends provider: antfly calls to a separate antfly inference run process; on Kubernetes, the operator manages those as InferencePools. External providers implement the same interfaces. Embedders are Ollama, OpenAI, OpenRouter, Cohere, Gemini, Vertex AI, and AWS Bedrock, and rerankers and generators have their own provider lists. Design: STANDALONE.md, inference ROADMAP.md, ONNX.md.

Consistency and Transactions#

• Raft per shard. Writes commit through the owning shard's Raft log, and each shard has its own leader. Only the active leader runs reconciliation, enrichment, and other background mutation work.
• Cross-shard writes. These use coordinator-driven two-phase commit over write intents. The coordinator shard writes a transaction record and participants write intents that stay invisible until resolution. A recovery loop finishes resolution after a coordinator failure.
• Optimistic concurrency. POST /transactions/commit is stateless OCC: the client submits read versions (from X-Antfly-Version) with its writes, and a changed version aborts with 409. Server-managed sessions under /transactions/begin add staged reads and writes, plus savepoints.
• Timestamps. A committed key's visible version is the transaction's commit timestamp, not its intent timestamp. Batch predicates and transactional predicates compare against the same versioning model.

Design: DB.md: Transaction / HLC Contract, AntflyTransaction.tla.

Runtime and Resources#

Each node owns one BackendRuntime, shared by metadata, data, and every DB opened on the node. It owns bounded std.Io executor lanes (general, Raft inbound and outbound, public API, inference, control plane), a durable job lane with per-owner close and drain, and shared storage I/O pools. Subsystems do not spawn private threads. Derived replay, LSM flush and compaction, text merge, enrichment replay, TTL cleanup, and transaction recovery all run as typed work on this runtime. Background execution is an optimization over the inline path, not a separate durability model. That is why the same code runs threaded in a server, inline in tests and embedded use, and under deterministic simulation.

Memory is governed per process. ResourceManager owns the physical budget shared by storage caches, indexing, and embedded inference. ModelManager decides model residency and eviction. An AdmissionController mirrors inference leases into the ResourceManager. The budget comes from --process-memory-budget-mb (or ANTFLY_PROCESS_MEMORY_BUDGET_MB), clamped by a finite cgroup limit, with host memory as a development fallback. Under pressure, optional cache admission is sampled and then stopped. Inference admission fails fast with 503 and Retry-After rather than queueing. Design: MANAGERS.md, DB.md: Runtime Ownership.

Availability and Operations#

• Backups and restore. Table and cluster backups go to local disk, S3, or GCS as content-addressed manifests and blobs, and incremental capture is supported. There is no point-in-time recovery or WAL archive; restore picks one backup manifest. Design: BACKUPS.md, HOT_STANDBY.md.
• Shard split and merge. Automatic split and merge on max_shard_size_bytes (default 64 MiB) is opt-in: set the top-level node config disable_shard_alloc: false. Design: DB.md: Shard Split Contract, SCALING.md.
• Autoscaling. The Kubernetes operator can scale data nodes on CPU and memory utilization within configured bounds, scaling down one node at a time. It is opt-in. InferencePools scale through a HorizontalPodAutoscaler. Design: OPERATOR.md.

Upgrades#

Upgrading from v0.2 to v0.3 is a stop-all upgrade. Take a backup, stop every metadata, data, and standby node, and start the same v0.3 build on all of them: metadata nodes first, then data nodes, then standbys. The first open advances the on-disk format, so rolling back means restoring the stopped copy with the old binary. The procedure is in Upgrading v0.2 servers. Rolling mixed-version upgrades are the contract for releases after v0.3. For a standalone deployment, hot-standby switchover is the online upgrade path.

Verification#

• VOPR deterministic simulation. A virtual std.Io runs unmodified production code under seeded scheduling, network, clock, crash, and resource faults. Scenarios cover metadata, Raft, transactions, storage, standby, derived workflows, backup, and a full deployment-shaped cluster, and each promoted scenario must replay byte for byte. Design: VOPR.md.
• TLA+. Specs cover Raft (forked from etcd's), 2PC with OCC, shard split, snapshot transfer, and the LSM lifecycle. Trace validation replays Raft and transaction executions against their models. Design: specs/tla/README.md.
• Chaos and end to end. Bounded chaos campaigns cover Raft restarts over HTTP, LSM compaction, and hot-standby crashes and partitions, and a Python end-to-end suite exercises the public API and SDKs. Design: TESTING.md.

Further Reading#

The full index of design documents is in zig/ROADMAP.md. For terms used on this page, see the Glossary.