Document Engine

How one query composes full-text, vector, filters, graph traversal, exact sort, and cursor pagination over a table's physical indexes

A table stores JSON documents. Schema mappings describe how each field becomes a queryable structure, and the planner picks the exact or approximate execution path that matches the request. The decision this page is most often read for: when a query needs an exact sorted page, which fields can serve it, and why the planner refuses rather than guesses.

The pieces the planner composes:

Documents are the source payload returned to clients. Stored JSON is never the production search or sort index.
Mappings declare field types such as text, keyword, number, date, boolean, embedding, and geo_point.
Full-text indexes execute BM25 and phrase-style searches over analyzed text fields.
Vector indexes execute semantic nearest-neighbor search over embeddings.
Typed field values power exact structured filters, aggregations, and field sort without parsing stored JSON on the hot path.
Graph indexes materialize relationships and run traversal or pattern queries over document ids.

Indexes that union generated document, chunk, embedding, or relationship streams are covered in Artifact-Backed Indexes.

Mappings Decide What a Field Can Do#

Antfly accepts schemaless documents, but fields that take part in filters, sort, and graph traversals behave best with a mapping. A mapping gives the planner a stable type and lets indexing build the right physical sections.

Use text for analyzed search and an exact scalar mapping for exact filters and sort. The common shape is a searchable text field with a keyword subfield:

{
  "default_type": "doc",
  "document_schemas": {
    "doc": {
      "schema": {
        "type": "object",
        "additionalProperties": true,
        "properties": {
          "title": {
            "type": "string",
            "x-antfly-field": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "sortable": true
                }
              }
            }
          },
          "created_at": {
            "type": "string",
            "format": "date-time",
            "x-antfly-field": {
              "type": "date",
              "sortable": true
            }
          },
          "price": {
            "type": "number",
            "x-antfly-field": {
              "type": "number",
              "sortable": true
            }
          },
          "published": {
            "type": "boolean",
            "x-antfly-field": {
              "type": "boolean",
              "sortable": true
            }
          },
          "body": {
            "type": "string",
            "x-antfly-field": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}

sortable is the whole declaration. Antfly derives the typed field-value structures that exact sort and filters need; there is no doc_values to configure. The SDKs expose DocumentFieldMapping and DocumentSubfieldMapping for this x-antfly-field shape. TemplateFieldMapping is deliberately narrower: it represents one dynamic-template output and does not accept fields.

The compact x-antfly-types shorthand still works for simple and multi-field search mappings, but it never enables exact sort on its own. Sort needs the detailed x-antfly-field form with sortable: true, so that doc-value storage and build cost land only on the scalar fields that asked for them. A mapping change rebuilds physical coverage. Once the write or rebuild visibility barrier completes, clients can issue order_by directly: Antfly validates a cold field on first use and fails closed if physical coverage is incomplete. Status reads never warm columns, so a safe but not-yet-validated field can remain declared until that first sort.

Explicit mappings are exact, table-wide physical declarations. Present values must be encodable by the mapping (a date mapping accepts an RFC 3339 date string or a non-negative nanosecond timestamp), and invalid writes are rejected before indexing. An allOf may contribute an exact mapping. Every alternative of an anyOf or oneOf must contribute the same normalized mapping; conditional or dynamically named mappings are rejected because they cannot guarantee one stable physical type for the dotted path.

Field capabilities report what the planner can safely do with a concrete field:

query_modes lists modes such as full_text, exact, range, geo, and autocomplete.
sortable says the field is declared sortable.
sort_lifecycle_state reports cached physical evidence. queryable and accelerated are already validated; declared or indexed can be validated by the first exact-sort request; unsupported is not sortable.
index_sort_position and index_sort_order identify fields in the table's physical index sort.

One Request, One Execution Shape#

A query request combines several inputs:

query or full_text_search supplies the scoring text query.
semantic_search supplies the vector query.
filter_query is an AND condition over the main query.
exclusion_query removes matching documents.
graph_queries run graph traversals or patterns alongside document search.
order_by, search_after, and search_before control exact ordered pages.
reranker and pruner post-process the selected results.
fields controls the response projection.

The planner normalizes the request into one execution shape. Full-text and semantic search both produce candidates, filters and exclusions narrow the eligible set, graph searches add relationship results, and sort or score ordering decides which page comes back.

{
  "table": "articles",
  "full_text_search": {
    "query": "body:database"
  },
  "filter_query": {
    "query": "+published:true +created_at:[2026-01-01T00:00:00Z TO *]"
  },
  "order_by": [
    {"field": "created_at", "desc": true}
  ],
  "fields": ["title", "created_at"],
  "limit": 20,
  "profile": true
}

Scoring Order Versus Exact Order#

Full-text search uses analyzed text fields and returns BM25-style relevance. Use it when exact terms, phrases, field scoping, and lexical constraints matter. Vector search ranks by semantic similarity; use it when meaning matters more than wording. Hybrid search runs both and fuses the ranked sets, most often with Reciprocal Rank Fusion.

When a request includes semantic_search, the primary order is semantic score or hybrid relevance. Exact field sorting stays separate on purpose: approximate vector top-k is not "all matching documents sorted by field". A request that could only be served by approximate overfetch and rerank pretending to be exact sort is rejected instead of answered with a misleading page.

Filters Shape the Eligible Set#

Filters are clauses that must match in addition to the main query. Exclusions remove documents after candidate selection. For scalar fields with native mappings, term, range, boolean, and date filters are evaluated from native field structures rather than parsed JSON.

This is what makes exact sort possible: an exact sorted page needs an exact eligible set. A broad filter with an index_sort-compatible order is served by scanning sorted segments. A selective filter usually works better as candidate-first doc-values top-N. A filter and sort combination the planner cannot serve exactly fails closed with a stable 422 reason rather than falling back to an unbounded stored-source scan.

Graph Queries#

graph_queries is an operation-keyed DSL. Each named query contains exactly one of traverse, match, shortest_path, or k_shortest_paths. A traversal defaults to one hop, and stored documents come back only when the operation enables include_documents.

Traversal, MATCH edges, shortest paths, and K-shortest paths all use the same stored-edge direction: out by default, in for reverse expansion, and both for undirected expansion. Store one physical relationship and select both when its semantics are undirected; a reciprocal edge is a second physical relationship and can produce a second binding or path.

Every result is self-describing through kind: MATCH returns bindings or aggregates, traversal returns nodes, and shortest-path operations return paths. Each paths[] item owns one authoritative path and, when requested, its terminal document, so clients never align a parallel terminal-node array with the path array.

Seeding From Search Results#

$query_results seeds a graph operation from the final ranked result page, whether retrieval was lexical, vector, or fused:

{
  "full_text_search": {"query": "database"},
  "graph_queries": {
    "citations": {
      "index": "citation_graph",
      "traverse": {
        "start": {"result_ref": "$query_results", "limit": 20},
        "edge_types": ["cites"],
        "max_depth": 1,
        "include_documents": true,
        "fields": ["title"]
      }
    }
  }
}

Traversal and path outputs can seed another operation with $graph_results.<query-name>. A path result contributes the endpoint node of each returned path. MATCH produces rows with several aliases, so a downstream selector also names the binding:

{"result_ref": "$graph_results.authors_and_posts", "binding": "post", "limit": 100}

Omit limit only when the referenced result is complete. Antfly rejects unbounded references to truncated result pages rather than returning a partial answer. Table-qualified identities use {"key":"…","table":"…"}; omitting table, or naming the queried table explicitly, has the same identity semantics.

Canonical paths keep that identity on nodes and edges. Each path edge uses typed from and to endpoints and is ordered with the node list: edges[i] traverses from nodes[i] to nodes[i + 1]. The deprecated graph_searches response keeps its original unqualified source and target edge keys during the compatibility window.

MATCH Patterns#

match expresses branched conjunctive patterns and returns rows or exact named counts. Node filters are non-scoring stored-document predicates; they do not run full-text analyzers. Range predicates use explicit numeric_range or term_range wrappers, or date_range with RFC 3339 bounds, so their types stay unambiguous in every SDK. Date bounds normalize into Antfly's unsigned Unix-nanosecond domain (1970-01-01 through 2554-07-21 UTC).

{
  "graph_queries": {
    "authored_replies": {
      "index": "social_graph",
      "match": {
        "anchor": "author",
        "nodes": {
          "author": {"filter": {"term": "person", "path": "/kind"}},
          "post": {"table": "messages"},
          "reply": {
            "table": "messages",
            "filter": {
              "numeric_range": {"path": "/score", "min": 0.8}
            }
          }
        },
        "edges": [
          {"from": "author", "to": "post", "types": ["AUTHORED"]},
          {"from": "reply", "to": "post", "types": ["REPLIES_TO"]}
        ],
        "where": {
          "not_equal": {
            "left": {"alias": "author"},
            "right": {"alias": "reply"}
          }
        }
      },
      "return": {
        "aggregates": {
          "matches": {"count": "*"},
          "distinct_replies": {"count": "reply", "distinct": true}
        }
      }
    }
  }
}

match.anchor names the alias enumerated from the query table. Omit a node's table for that same table; declare table on every cross-table alias. The declaration is part of the alias's identity and lets the planner expand a relationship in reverse without guessing its source table. Reached aliases keep their table-qualified identities, and a binding from a different table never satisfies the declaration. For cross-table multi-hop patterns, declare an alias at each table boundary and connect them with single-hop edges; Antfly fails closed when exact execution would need to reverse a variable-length edge through unnamed intermediate nodes whose source tables cannot be proven.

MATCH edge direction is relative to from and defaults to out. direction: "in" traverses a stored relationship in reverse; direction: "both" treats one stored relationship as undirected, and a physical self-loop still contributes only one relationship binding under both. Fixed single-hop relationships preserve physical self-loops, so distinct aliases may bind the same node unless where.not_equal excludes it. Variable-length relationships stay node-simple except for explicit repeated-alias cycle closure.

Top-level full-text, semantic, and filter clauses shape the retrieval result only; they do not narrow MATCH. Put source constraints on the node named by match.anchor. Row-level authorization still applies to every anchor and reached node. Anchor identities are read in stable snapshot-pinned pages, independently for each named operation. An ids filter, or a disjunction containing only ids filters, resolves through the primary identity index and needs no secondary index. Other fields used by anchor or authorization filters must have native index coverage, because exact anchors are cursor-scanned in _id order; Antfly returns graph_anchor_filter_requires_index when it cannot prove that complete scan. Index the referenced fields rather than relying on stored-source fallback.

Graph document filters address stored values with RFC 6901 JSON Pointers: /kind for a top-level field, /author/name for a nested one; escape ~ as ~0 and / within a key as ~1.

Optional Groups and Anti-Joins#

Optional groups are correlated left-outer patterns and run in array order. Each group must connect to an alias from the required pattern or an earlier optional group. A group with no match keeps one row and exposes each alias it introduced as null. where.not_exists is the anti-join form: its edges may refer only to aliases already visible at that point and introduce none.

This forms authored-post bindings with optional replies, and excludes bindings where the author also has a BLOCKED relationship to that post:

{
  "graph_queries": {
    "post_activity": {
      "index": "social_graph",
      "match": {
        "anchor": "author",
        "nodes": {
          "author": {},
          "post": {"table": "messages"}
        },
        "edges": [
          {"from": "author", "to": "post", "types": ["AUTHORED"]}
        ],
        "where": {
          "not_exists": {
            "edges": [
              {"from": "author", "to": "post", "types": ["BLOCKED"]}
            ]
          }
        },
        "optional": [
          {
            "nodes": {"reply": {"table": "messages"}},
            "edges": [
              {"from": "reply", "to": "post", "types": ["REPLIES_TO"]}
            ]
          }
        ]
      },
      "return": {
        "aggregates": {
          "rows": {"count": "*"},
          "posts": {"count": "post", "distinct": true},
          "replies": {"count": "reply"}
        }
      }
    }
  }
}

Counts and Rows#

count(*) includes the null-extended row produced when a post has no reply. count(alias) ignores rows where that alias is null, matching SQL count semantics. If several replies match one post, the row and reply counts observe each resulting binding; use distinct: true on an alias count when unique identities are required, as the posts aggregate does above. Put multiple counts over one pattern in a single match return object so they share one complete anchor scan and one set of bindings. A request may contain at most eight named match operations.

Use CountGraphRows and CountGraphAlias in Go, countGraphRows and countGraphAlias in TypeScript, or count_graph_rows and count_graph_alias in Python to build validated count expressions; the alias helpers reject reserved or unsafe graph identifiers before a request is sent.

Exact aggregation streams bindings without materializing a bounded row page. Row returns expose each alias as a compact {key, table?, document?} binding. A binding projection lists between one and 64 unique aliases; list every declared alias when the complete row is required. Aliases and aggregate names are limited to 128 Unicode code points. Traversal-only metrics such as depth, distance, and path are absent from MATCH results, because a node in a branched match has no single canonical traversal distance; request a traversal or path operation when those are the result you want.

Budgets#

All graph expansion work is budgeted across the whole request, including every named operation and every K-shortest-path spur search. Traversal and path operations also keep their explicit per-operation depth and result bounds. The matcher enforces a byte ceiling on retained frontier ancestry in addition to its node, edge, and intermediate-state limits, so long identities cannot bypass the memory invariant. If Antfly cannot enumerate the complete authorized anchor relation or exhausts the budget, the request fails instead of returning a partial count. Exact count(distinct alias) sets share a request-scoped identity and byte budget; exhaustion returns graph_distinct_budget_exceeded, and the fix is to narrow the anchor predicate or use a non-distinct aggregate.

The Deprecated `graph_searches` Dialect#

graph_searches remains a wire-compatibility adapter for the v0.2 shape. Its operation names stay opaque legacy map keys; the canonical GraphIdentifier policy applies only to graph_queries. The request-wide 64-operation safety limit applies to both dialects. New clients generate only graph_queries. A successful stateful request that uses graph_searches includes the RFC 9745 deprecation date Deprecation: @1787702400 (2026-08-26 UTC), and Antfly emits a stable low-cardinality acceptance log without table names, operation names, or query contents. The adapter stays until fleet telemetry shows legacy traffic has drained; serverless Antfly and MemoryAF are canonical-only and do not emit legacy responses.

Exact Sort and Cursor Pagination#

order_by produces exact sorted pages over _id or mapped scalar fields whose capability is queryable or accelerated. Antfly appends _id ascending as a deterministic tie-breaker when it is omitted.

Each hit carries _sort values. Pass them back unchanged as search_after for the next page or search_before for the previous one:

{
  "table": "articles",
  "filter_query": {
    "query": "published:true"
  },
  "order_by": [
    {"field": "created_at", "desc": true}
  ],
  "search_after": ["2026-07-01T00:00:00Z", "doc:article:1842"],
  "limit": 20
}

Cursor values preserve JSON scalar types: strings stay strings, numbers stay numbers, booleans stay booleans. Nulls, arrays, objects, and non-finite numbers are rejected because they cannot form replayable typed cursor tuples.

Use cursors for large or changing result sets. Offset pagination suits shallow UI pages, but it makes the engine count past skipped results and shifts when documents change between requests.

Physical Index Sort#

index_sort is an optional acceleration path for one dominant order. New segments are stored in that physical tuple order and the order survives merges. When a broad query asks for the same order, the planner uses sorted segment seek instead of collecting every candidate and sorting.

There is one physical order per index generation. Other order_by fields still use typed field values and top-N collection. Changing index_sort is a layout change; existing generations need backfill, compaction, or reindex before the accelerated path counts as fully covered.

When a Sorted Query Returns 422#

Exact query behavior fails closed. A sorted request returns 422 when the engine cannot execute the requested order exactly through a native path:

unmapped sort field
analyzed text field used directly for sort
non-scalar, geo, embedding, blob, object, or array sort field
field not yet queryable or accelerated
missing typed field coverage in older segments
invalid search_after or search_before tuple
semantic or vector candidate source used for exact field sort
count-only query combined with ordered pagination

Set profile: true while diagnosing. The public sort profile reports the selected plan, source, candidate source, selection reason, exactness, candidate counts, source-load behavior, and rejection reasons. Lower-level executor counters live in logs, traces, and benchmarks, not in SDK responses.

Use Agent Skills#

Everything above is also encoded in the Antfly skill, so a coding agent can execute this guide for you:

npx skills add antflydb/antfly-skills

Then prompt it with the outcome, for example "Map created_at as sortable on my articles table and page through published articles newest-first with search_after", and use this page to judge the result.

Next Steps#

Tune Hybrid Search: fusion, reranking, and pruning for relevance-ordered queries
Artifact-Backed Indexes: indexes over generated chunk, embedding, and relationship streams
Build a Support Answer Agent: the query pipeline feeding a generated answer