Document Engine

Overview#

Antfly's document engine is one query system over several physical indexes. A table stores JSON documents, schema mappings describe how fields become queryable structures, and the planner chooses the exact or approximate execution path that matches the request.

The important pieces are:

• Documents are the source payload returned to clients.
• Mappings declare field types such as text, keyword, number, date, boolean, embedding, and geo_point.
• Full-text indexes execute BM25 and phrase-style searches over analyzed text fields.
• Vector indexes execute semantic nearest-neighbor search over embeddings.
• Typed field values power exact structured filters, aggregations, and field sort without parsing stored JSON on the hot path.
• Graph indexes materialize relationships and run traversal or pattern queries over document ids.
• The query planner composes those structures, applies filters and exclusions, sorts or ranks the eligible set, and returns one page.

Stored JSON remains the document payload. It is not the production search or sort index.

For indexes that union generated document, chunk, embedding, or relationship streams, see Artifact-Backed Indexes.

Schema And Field Capabilities#

Antfly can accept schemaless documents, but production query behavior is best when fields that participate in filters, sort, and graph traversals have schema mappings. Mappings give the planner a stable type model and let Antfly build the right physical sections during indexing.

Use text for analyzed search and an exact scalar mapping for exact filters and sort. A common pattern is to expose both a searchable text field and a keyword subfield:

{
  "version": 1,
  "default_type": "doc",
  "document_schemas": {
    "doc": {
      "schema": {
        "type": "object",
        "additionalProperties": true,
        "properties": {
          "title": {
            "type": "string",
            "x-antfly-field": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "sortable": true
                }
              }
            }
          },
          "created_at": {
            "type": "string",
            "format": "date-time",
            "x-antfly-field": {
              "type": "date",
              "sortable": true
            }
          },
          "price": {
            "type": "number",
            "x-antfly-field": {
              "type": "number",
              "sortable": true
            }
          },
          "published": {
            "type": "boolean",
            "x-antfly-field": {
              "type": "boolean",
              "sortable": true
            }
          },
          "body": {
            "type": "string",
            "x-antfly-field": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}

sortable is the user-facing declaration. Antfly derives the internal typed field-value structures needed for exact sort and filters. Do not configure doc_values directly.

SDKs expose DocumentFieldMapping and DocumentSubfieldMapping for the x-antfly-field shape above. TemplateFieldMapping is intentionally narrower: it represents one dynamic-template output and does not accept fields.

The compact x-antfly-types shorthand remains supported for simple and multi-field search mappings, but does not implicitly enable exact sort. Use x-antfly-field as the detailed form when a field needs physical options such as sortable: true. This intentionally avoids adding doc-value storage and build cost to every scalar field. The mapping change rebuilds physical coverage. Once the write or rebuild visibility barrier completes, clients may issue order_by directly: Antfly validates a cold field on first use and fails closed if physical coverage is incomplete. Status reads never warm columns, so a safe but not-yet-validated field can remain declared until that first sort.

Explicit mappings are exact, table-wide physical declarations. Present values must be encodable by the mapping—for example, a date mapping accepts an RFC 3339/date string or a non-negative nanosecond timestamp—and invalid writes are rejected before indexing. An allOf may contribute an exact mapping. Every alternative of an anyOf or oneOf must contribute the same normalized mapping; conditional or dynamically named mappings are rejected because they cannot guarantee one stable physical type for the dotted path.

Field capabilities report what the planner can safely do with a concrete field:

• query_modes describes modes such as full_text, exact, range, geo, and autocomplete.
• sortable says the field is declared sortable.
• sort_lifecycle_state reports cached physical evidence. queryable and accelerated are already validated; declared or indexed can be validated by the first exact-sort request. unsupported is not sortable.
• index_sort_position and index_sort_order identify fields participating in the table's physical index sort.

Query Pipeline#

A query request can combine several inputs:

• query or full_text_search supplies the scoring text query.
• semantic_search supplies the vector query.
• filter_query is an AND condition over the main query.
• exclusion_query removes matching documents.
• graph_queries run graph traversals or patterns alongside document search.
• order_by, search_after, and search_before control exact ordered pages.
• reranker and pruner can post-process the selected search results.
• fields controls the response projection.

The planner normalizes the request into one execution shape. Full-text and semantic search can both produce candidates, filters and exclusions narrow the eligible set, graph searches can add relationship results, and sort or score ordering decides which page is returned.

{
  "table": "articles",
  "full_text_search": {
    "query": "body:database"
  },
  "filter_query": {
    "query": "+published:true +created_at:>2026-01-01"
  },
  "order_by": [
    {"field": "created_at", "desc": true}
  ],
  "fields": ["title", "created_at"],
  "limit": 20,
  "profile": true
}

Graph Queries#

graph_queries is an operation-keyed DSL. Each named query contains exactly one of traverse, match, shortest_path, or k_shortest_paths. A traversal defaults to one hop, and stored documents are returned only when the operation enables include_documents.

Traversal, MATCH edges, shortest paths, and K-shortest paths all use the same stored-edge direction: out by default, in for reverse expansion, and both for undirected expansion. Store one physical relationship and select both when its query semantics are undirected; storing a reciprocal edge creates a second physical relationship and can therefore create a second binding or path.

Every canonical result is self-describing through kind: MATCH returns bindings or aggregates, traversal returns nodes, and shortest-path operations return paths. Each paths[] item owns one authoritative path and, when requested, its terminal document; clients never need to align a parallel terminal-node array with the path array.

Use $query_results to seed a graph operation from the final ranked result page, regardless of whether retrieval was lexical, vector, or fused:

{
  "full_text_search": {"query": "database"},
  "graph_queries": {
    "citations": {
      "index": "citation_graph",
      "traverse": {
        "start": {"result_ref": "$query_results", "limit": 20},
        "edge_types": ["cites"],
        "max_depth": 1,
        "include_documents": true,
        "fields": ["title"]
      }
    }
  }
}

Traversal and path outputs can seed another graph operation with $graph_results.<query-name>. A path result contributes the endpoint node of each returned path. MATCH produces rows with multiple aliases, so a downstream selector must also name the returned binding:

{"result_ref": "$graph_results.authors_and_posts", "binding": "post", "limit": 100}

Omit limit only when the referenced result is complete. Antfly rejects unbounded references to truncated result pages rather than returning a partial answer. Table-qualified identities use {"key":"…","table":"…"}; omitting table, or explicitly naming the queried table, has the same identity semantics.

Canonical paths preserve that identity on both nodes and edges. Each path edge uses typed from and to endpoints and is ordered with the node list: edges[i] traverses from nodes[i] to nodes[i + 1]. The deprecated graph_searches response keeps its original unqualified source and target edge keys during the compatibility window.

match expresses branched conjunctive patterns and can return rows or exact named counts. Node filters are non-scoring stored-document predicates; they do not run full-text analyzers. Range predicates use explicit numeric_range or term_range wrappers, or date_range with RFC 3339 bounds, so their types remain unambiguous in every SDK. Date bounds are normalized into Antfly's unsigned Unix-nanosecond domain (1970-01-01 through 2554-07-21 UTC).

{
  "graph_queries": {
    "authored_replies": {
      "index": "social_graph",
      "match": {
        "anchor": "author",
        "nodes": {
          "author": {"filter": {"term": "person", "path": "/kind"}},
          "post": {"table": "messages"},
          "reply": {
            "table": "messages",
            "filter": {
              "numeric_range": {"path": "/score", "min": 0.8}
            }
          }
        },
        "edges": [
          {"from": "author", "to": "post", "types": ["AUTHORED"]},
          {"from": "reply", "to": "post", "types": ["REPLIES_TO"]}
        ],
        "where": {
          "not_equal": {
            "left": {"alias": "author"},
            "right": {"alias": "reply"}
          }
        }
      },
      "return": {
        "aggregates": {
          "matches": {"count": "*"},
          "distinct_replies": {"count": "reply", "distinct": true}
        }
      }
    }
  }
}

Optional groups are correlated left-outer patterns and run in array order. Each group must connect to an alias from the required pattern or an earlier optional group. A group with no match retains one row and exposes each alias introduced by that group as null. where.not_exists is the anti-join form: its edges may refer only to aliases already visible at that point and do not introduce aliases.

For example, this forms authored-post bindings with optional replies. The anti-join excludes bindings where the author also has a BLOCKED relationship to that post:

{
  "graph_queries": {
    "post_activity": {
      "index": "social_graph",
      "match": {
        "anchor": "author",
        "nodes": {
          "author": {},
          "post": {"table": "messages"}
        },
        "edges": [
          {"from": "author", "to": "post", "types": ["AUTHORED"]}
        ],
        "where": {
          "not_exists": {
            "edges": [
              {"from": "author", "to": "post", "types": ["BLOCKED"]}
            ]
          }
        },
        "optional": [
          {
            "nodes": {"reply": {"table": "messages"}},
            "edges": [
              {"from": "reply", "to": "post", "types": ["REPLIES_TO"]}
            ]
          }
        ]
      },
      "return": {
        "aggregates": {
          "rows": {"count": "*"},
          "posts": {"count": "post", "distinct": true},
          "replies": {"count": "reply"}
        }
      }
    }
  }
}

count(*) includes the null-extended row produced when a post has no reply. count(alias) ignores rows where that alias is null, matching SQL count semantics. If several replies match one post, the row and reply counts observe each resulting binding; use distinct: true on an alias count when unique identities are required, as the posts aggregate does above.

Use CountGraphRows and CountGraphAlias in Go, countGraphRows and countGraphAlias in TypeScript, or count_graph_rows and count_graph_alias in Python to construct validated count expressions. The alias helpers reject reserved or unsafe graph identifiers before a request is sent.

Exact aggregation streams bindings without materializing a bounded row page. Row returns expose each alias as a compact {key, table?, document?} binding. Traversal-only metrics such as depth, distance, and path are intentionally absent because a node in a branched match has no single canonical traversal distance. Request a traversal or path operation when those metrics are the desired result. match.anchor names the alias enumerated from the query table. Omit a node's table for that same table; declare table on every cross-table alias. The declaration is part of the alias's identity and lets the planner safely expand a relationship in reverse without guessing its source table. Reached aliases retain their table-qualified identities, and a binding from a different table never satisfies the declaration. Anchor identities are read in stable snapshot-pinned pages, independently for each named graph operation. A request may contain at most eight named match operations. Put multiple counts over one pattern in a single match return object so they share one complete anchor scan and one set of bindings. For cross-table multi-hop MATCH patterns, declare an alias at each table boundary and connect those aliases with single-hop edges. Antfly fails closed when exact execution would otherwise need to reverse a variable-length edge through unnamed intermediate nodes whose source tables cannot be proven. Aliases and aggregate names are limited to 128 Unicode code points. A binding projection explicitly lists between one and 64 unique aliases; list every declared alias when the complete row is required. These bounds keep admission predictable and let the matcher compile projection membership once per operation. Fixed single-hop relationships preserve physical self-loops, so distinct aliases may bind the same node unless where.not_equal excludes that binding. Variable- length relationships remain node-simple except for explicit repeated-alias cycle closure. MATCH edge direction is relative to from and defaults to out. Use direction: "in" to traverse a stored relationship in reverse, or direction: "both" to treat one stored relationship as undirected. A physical self-loop still contributes only one relationship binding under both. Top-level full-text, semantic, and filter clauses shape the retrieval result only; they do not implicitly narrow MATCH. Put source constraints on the node named by match.anchor. Row-level authorization still applies to every anchor and reached node. An ids filter, or a disjunction containing only ids filters, resolves through the primary identity index and needs no secondary index. Other fields used by anchor or authorization filters must have native index coverage because exact anchors are cursor-scanned in _id order. Antfly returns graph_anchor_filter_requires_index when it cannot prove that complete scan; index the referenced fields rather than relying on stored-source fallback. Graph document filters address stored values with RFC 6901 JSON Pointers. For example, use /kind for a top-level field and /author/name for a nested field; escape ~ as ~0 and / within a key as ~1.

All graph expansion work is budgeted across the whole request, including every named operation and every K-shortest-path spur search. Traversal and path operations also retain their explicit per-operation depth and result bounds. The matcher also enforces a byte ceiling on retained frontier ancestry, in addition to its node, edge, and intermediate-state limits, so long identities cannot bypass the memory invariant. If Antfly cannot enumerate the complete authorized anchor relation or exhausts that budget, the request fails instead of returning a partial count. Exact count(distinct alias) sets also share a request-scoped identity and byte budget. Exhaustion returns graph_distinct_budget_exceeded; narrow the anchor predicate or use a non-distinct aggregate.

The deprecated graph_searches request field remains a wire-compatibility adapter for the v0.2 shape. Its operation names remain opaque legacy map keys; the canonical GraphIdentifier policy applies only to graph_queries. The request-wide 64-operation safety limit applies to both dialects. New clients should generate only graph_queries. A successful stateful request that uses graph_searches includes the RFC 9745 deprecation date Deprecation: @1787702400 (2026-08-26 UTC), and Antfly emits a stable low-cardinality acceptance log without table names, operation names, or query contents. The compatibility adapter must not be removed until fleet telemetry shows that legacy traffic has drained; serverless Antfly and MemoryAF remain canonical-only and do not emit legacy responses.

Full-Text And Vector Search#

Full-text search uses analyzed text fields and returns BM25-style relevance. Use it when exact terms, phrases, field scoping, and lexical constraints matter.

Vector search ranks by semantic similarity. Use it when natural-language meaning matters more than exact wording. Hybrid search runs both and fuses the ranked sets, commonly with Reciprocal Rank Fusion.

When a request includes semantic_search, the primary result order is semantic score or hybrid relevance. Exact field sorting is intentionally separate because approximate vector top-k is not the same as "all matching documents sorted by field." If a request would need approximate overfetch and rerank to pretend to be exact field sort, Antfly should reject it instead of returning a misleading page.

Filters And Exclusions#

Filters are query clauses that must match in addition to the main query. Exclusions remove documents after candidate selection. For scalar fields with native mappings, Antfly can evaluate term, range, boolean, and date filters from native field structures instead of parsing stored JSON.

This matters for sort because exact sorted pages need an exact eligible set. A broad filter with an index_sort-compatible order can be served by scanning sorted segments. A selective filter often works better as candidate-first doc-values top-N. Unsupported filter and sort combinations should fail closed with a stable 422 reason rather than falling back to an unbounded stored-source scan.

Sort And Cursor Pagination#

Use order_by for exact sorted pages over _id or mapped scalar fields whose capability is queryable or accelerated. Antfly appends _id ascending as a deterministic tie-breaker when it is omitted.

The response includes _sort values for each hit. Pass those values back unchanged as search_after for the next page or search_before for the previous page:

{
  "table": "articles",
  "filter_query": {
    "query": "published:true"
  },
  "order_by": [
    {"field": "created_at", "desc": true}
  ],
  "search_after": ["2026-07-01T00:00:00Z", "doc:article:1842"],
  "limit": 20
}

Cursor values preserve JSON scalar types. Strings remain strings, numbers remain numbers, and booleans remain booleans. Nulls, arrays, objects, and non-finite numbers are rejected because they cannot form replayable typed cursor tuples.

Use cursor pagination for large or changing result sets. Offset pagination is useful for shallow UI pages, but it requires the engine to count past skipped results and is less stable when documents change between requests.

Physical Index Sort#

index_sort is an optional acceleration path for one dominant order. It stores new segments in that physical tuple order and preserves the order during merges. When a broad query asks for the same order, the planner can use sorted segment seek instead of collecting all candidates and sorting them.

There is only one physical order per index generation. Arbitrary order_by fields still use typed field values and top-N collection. Changing index_sort is a layout change; existing generations need backfill, compaction, or reindex before the accelerated path can be considered fully covered.

Rejections And Diagnostics#

Exact public query behavior should fail closed. A sorted request can return 422 when the engine cannot execute the requested order exactly through a native path. Common reasons include:

• unmapped sort field
• analyzed text field used directly for sort
• non-scalar, geo, embedding, blob, object, or array sort field
• field not yet queryable or accelerated
• missing typed field coverage in older segments
• invalid search_after or search_before tuple
• semantic/vector candidate source used for exact field sort
• count-only query combined with ordered pagination

Set profile: true while diagnosing. The stable public sort profile reports the selected plan, source, candidate source, selection reason, exactness, candidate counts, source-load behavior, and rejection reasons. Lower-level executor counters belong in logs, traces, benchmarks, or explicit debug surfaces rather than normal SDK responses.