Artifact-Backed Indexes

How one full-text, vector, or graph index unions the records that enrichments generate, and how to tell when it is ready

An artifact-backed index consumes the records that named enrichments produce, rather than the source documents themselves. Its sources array unions up to 64 streams into one full-text, embeddings, or graph index, and each member keeps the stable identity (artifact, key), so records from different streams never collide even when their keys match. The decision this page is read for: when to search whole documents and their extracted chunks as one index, and how to know when that index is actually serving all of them.

Confirm the deployment supports it first:

curl http://127.0.0.1:8080/db/v1/status

index_capabilities.artifact_sources must be true. Embedded, standalone, and distributed deployments support artifact sources. Serverless rejects them and does not emulate older artifact-index behavior.

Document and Chunk Vectors in One Index#

One document-level embedding captures the whole; the chunks that document extraction produces are more precise. Indexing both as members of one embeddings index closes that gap. Both embedding enrichments must share compatible dimensions and a compatible vector space.

{
  "type": "embeddings",
  "dimension": 768,
  "embedder": { "provider": "antfly", "model": "BAAI/bge-base-en-v1.5" },
  "sources": [
    { "artifact": "document_dense_v1" },
    { "artifact": "document_chunk_dense_v1" }
  ],
  "enrichments": [
    {
      "name": "document_chunks_v1",
      "kind": "chunk",
      "field": "semantic_content",
      "chunk_size": 512
    },
    {
      "name": "document_dense_v1",
      "kind": "embedding",
      "field": "semantic_content",
      "expected_dims": 768,
      "vector_space": "searchaf:v1"
    },
    {
      "name": "document_chunk_dense_v1",
      "kind": "embedding",
      "field": "text",
      "source_artifact_name": "document_chunks_v1",
      "expected_dims": 768,
      "vector_space": "searchaf:v1"
    }
  ]
}

Create it at POST /db/v1/tables/{table}/indexes/document_vectors. vector_space is an application-owned compatibility identifier: every source in a multi-source index must either use the same explicit identifier or use semantically identical producer configurations, and explicit and implicit modes cannot be mixed.

Query document_vectors through the normal semantic search API. Raw member results keep their artifact identity. When the UI needs one source document with its matching chunks nested beneath it, use hierarchy grouping and request the documented top-level field projection.

Full-Text Sources With Different Shapes#

A top-level field is inherited by every source that does not override it. A source-local field lets one index union artifact records with different JSON shapes:

{
  "type": "full_text",
  "field": "text",
  "sources": [
    { "artifact": "document_units_v1", "field": "summary" },
    { "artifact": "document_chunks_v1" }
  ]
}

document_units_v1 indexes summary; document_chunks_v1 inherits text. If neither level names a field, Antfly uses the default text projection. Missing, null, and non-text values add no postings.

To query a named full-text index, set full_text_index together with full_text_search or a scoring text clause in the recursive query AST:

{
  "full_text_index": "document_text",
  "full_text_search": { "query": "retrieval augmented generation" },
  "limit": 10
}

Filter-only and exclusion-only query trees are not scoring text queries and receive HTTP 422 when combined with full_text_index.

Graph Sources and Edge Ownership#

Graph sources are ordered. When two sources materialize the same edge identity, the earlier source wins. Mapping (path, format, nodes, edge, and context) belongs on each source, because streams can use different record shapes. The optional source object is the one-source convenience form and is mutually exclusive with sources.

Every materialized relationship belongs to the document that produced its artifact. By default that document is also the logical source endpoint, and nodes.target maps the target. An optional nodes.source template maps a different logical source; it requires an edge.edge_id mapping. Ownership still belongs to the producing document, so replay, deletion, and range transfers follow the fact even when neither endpoint is that document.

Without edge_id, identity is (index, source, type, target): repeated writes update the same relationship. With edge_id, identity also includes the producing document and the supplied ID. Two facts with the same endpoints and type therefore survive independently. A repeated explicit identity is an idempotent upsert. IDs are scoped to that identity, not globally unique, and a UUID stored only in metadata does not affect identity. Use a stable producer ID; random IDs generated during replay and array positions that change when input reorders are unsuitable.

Project searchable facts onto arbitrary endpoints#

This index keeps each fact as a normal document and projects its relations field into the graph:

{
  "type": "graph",
  "source": {
    "artifact": "fact_relations_v1",
    "path": "$",
    "nodes": {
      "source": "{{ _item.source }}",
      "target": "{{ _item.target }}"
    },
    "edge": {
      "edge_id": "{{ _doc.key }}",
      "type": "RELATES_TO",
      "weight": "{{ _item.confidence }}"
    }
  },
  "artifact": {
    "name": "fact_relations_v1",
    "kind": "asset",
    "source": { "type": "field", "value": "relations" },
    "content_type": "application/json"
  }
}

Insert these documents under keys fact:works and fact:founded:

{
  "fact": "Alice works at Acme",
  "relations": [{ "source": "alice", "target": "acme", "confidence": 0.9 }]
}

{
  "fact": "Alice co-founded Acme",
  "relations": [{ "source": "alice", "target": "acme", "confidence": 0.8 }]
}

Both relationships have type RELATES_TO, so type filters continue to work. Their edge_id values distinguish the facts; owner_document in edge and path results identifies the producing document when it differs from the logical source. Fetch that document from the graph index's table for its text, embeddings, provenance, and validity fields. Endpoint hydration continues to return entity documents. For several same-pair relationships produced by one document, map a per-item UUID instead of _doc.key.

Updating a fact's artifact replaces its previous relationship projection, including retiring relationships whose endpoints changed. Deleting the fact or removing its source artifact removes its relationships. Updating or deleting an endpoint document does not delete independently owned fact relationships; endpoints can exist without stored entity documents. Deleting one of several facts leaves the pair connected until its last relationship disappears. Ordered source precedence applies to the same complete relationship identity within one owner.

Endpoint cleanup covers relationships stored on the deleting shard. It retires incident source-owned relationships, including parallel IDs and self loops. Retirement survives graph replay, reconstruction, and snapshot restore. Independently owned fact relationships keep their fact-document lifecycle. An explicit relationship write can revive that exact identity; a meaningful update to the producing document starts a new projection lifecycle. Rewriting an identical document preserves retirement. Explicit relationship deletion also survives replay and restore. Deleting a graph index clears its retirements before an index with the same name can be recreated.

Node traversal returns distinct reached nodes; relationship matches and paths retain relationship IDs. K-shortest paths can differ only by fact ID. Type, weight, and metadata predicates apply to relationships before neighbor admission. Graph degree counts relationships, and existing unweighted PageRank, eigenvector, and HITS use relationship multiplicity; connectivity treats parallel relationships as the same connection. These metrics do not aggregate confidence weights. To compute distinct-neighbor degree or aggregate confidence, materialize a separate pair projection with an application-chosen aggregation rule. Tree types allow one parent relationship, including its ID, per source and type.

For source-owned indexes, outgoing traversal routes directly to the source's shard. Incoming traversal uses an exact, bounded route directory fenced by topology and per-shard read generations; a cold entry uses compact reverse-index probes. When an index maps arbitrary source endpoints, outgoing and bidirectional expansion consult the pinned shard set because the fact can belong to any shard. Work and retained-byte budgets still apply. The owner shard maintains both endpoint adjacency directions locally; range transfer copies owner records and reconstructs endpoint adjacency.

Fact predicates, weighted paths, and counts#

Project traversal-relevant properties into each relation item alongside its endpoints:

{
  "fact": "Alice works at Acme",
  "relations": [{
    "source": "alice", "target": "acme", "confidence": 0.9,
    "group_id": "team-a",
    "valid_at": "2020-01-01T00:00:00Z", "invalid_at": null,
    "created_at": "2020-02-01T00:00:00Z", "expired_at": null
  }]
}

Without an explicit metadata mapping, the relation item becomes edge metadata, preserving JSON nulls and numeric literals through field extraction, projection, repair, and restore. Edge weights are converted separately to double precision. These projected fields are updated with the fact document. Entity or fact hydration does not supply traversal predicates: the edge projection must carry them.

Use edge_filter on traverse, shortest_path, k_shortest_paths, or on each MATCH edge. Every relationship in a variable-length MATCH edge must pass its filter. Filters apply before node deduplication, path ranking, OPTIONAL matching, NOT EXISTS checks, and counting. They work in either direction and across shards.

{
  "graph_queries": {
    "fact_paths": {
      "index": "facts",
      "k_shortest_paths": {
        "from": { "key": "alice" }, "to": { "key": "acme" },
        "k": 2, "objective": "max_weight_product",
        "edge_types": ["RELATES_TO"],
        "edge_filter": {
          "valid_at": "2022-01-01T00:00:00Z",
          "known_at": "2023-01-01T00:00:00Z",
          "properties": [{ "field": "/metadata/group_id", "op": "eq", "value": "team-a" }]
        }
      }
    }
  }
}

valid_at tests the fact's [valid_at, invalid_at) interval; known_at tests its [created_at, expired_at) interval. Missing or null valid-time bounds are open. A knowledge-time test requires a non-null created_at. Invalid timestamp properties fail the filter. These names refer to metadata fields, distinct from the graph's physical creation/update timestamps. RFC3339 timezone offsets and fractions are compared as instants, including dates before 1970.

Property predicates use JSON pointers, including ~0 and ~1 escaping and array indices, and combine with AND. Operators are eq, ne, lt, lte, gt, gte, is_null, and is_not_null. Supply a non-null scalar value for comparisons; omit it for null tests. Missing/null properties fail all comparisons, including ne, matching Cypher's WHERE behavior. Different scalar types do not compare. Numeric metadata literals compare exactly, including large integers, long decimals, and exponent notation. Stored edge weights compare using their shortest round-trip decimal, so a stored double-precision 0.1 matches /weight equal to 0.1; distinct literals remain distinct. Strings compare lexically unless value_type is datetime, which requires RFC3339 strings. At most 64 predicates totaling 64 KiB are accepted, with up to 256 components per JSON pointer. Array indices use canonical unsigned decimal spelling. Metadata numbers retain their precision in graph responses. Filters prepare constants once and scan only the requested metadata paths; temporary decoding uses the graph query memory budget.

Relationship IDs and owner document keys have no separate 64 KiB limit; they consume the existing request, storage, and graph work budgets. HTTP identities are nonempty UTF-8 strings. An explicit owner equal to the logical source is canonicalized to implicit source ownership, so specifying it does not create a second relationship identity.

Embedded C ABI traversal, shortest-path, K-path, and named graph-query requests accept the same edge_filter object as HTTP. Edge and path responses carry edge_id_b64 and owner_document_b64, preserving opaque native identifiers just like source_b64 and target_b64. Path edges also expose metadata_json and traversal_direction (0 outgoing, 1 incoming, 2 both; null for an unoriented self-loop). Traversal paths retain the exact selected fact relationship. Native Zig callers can use findShortestPathWithOptions and findKShortestPathsWithOptions; the original positional methods remain available.

Path objectives are explicit: min_hops, min_weight_sum for nonnegative costs, or max_weight_product for confidence weights in [0,1]. Same-type parallel facts remain separate candidates, with their own weights and metadata. K paths can differ only by relationship ID and owner. Predicates choose the traversable graph first; they are not applied after selecting the cheapest path. Filtered queries use exact edge expansion when an algebraic optimization cannot prove equivalent results.

Algebraic provenance labels are opaque strings. Versioned labels use the prefix antfly:graph-provenance:v1: and a lossless base64url payload. Read relationship IDs, owners, and endpoints from the structured path_edges fields rather than parsing provenance labels.

For a fixed one-hop MATCH, count(*) counts relationship matches, while a distinct target alias counts connected entities. MATCH anchors enumerate stored documents: store entity documents alongside facts when using those entities as anchors. Direct traversal can start from a graph key without a stored entity document. For joins or variable-length patterns, count(*) counts complete matches, not globally unique relationships. For example:

{
  "graph_queries": {
    "counts": {
      "index": "facts",
      "match": {
        "anchor": "a",
        "nodes": { "a": { "filter": { "ids": ["alice"] } }, "b": {} },
        "edges": [{ "from": "a", "to": "b", "types": ["RELATES_TO"],
          "edge_filter": { "valid_at": "2022-01-01T00:00:00Z" } }]
      },
      "return": { "aggregates": {
        "facts": { "count": "*" },
        "entities": { "count": "b", "distinct": true }
      } }
    }
  }
}

Neo4j also allows multiple same-type relationships between two nodes. Its Cypher shortest-path selectors select by hop count and support relationship prefilters; weighted K paths are supplied by GDS Yen's algorithm. Antfly exposes these behaviors through its graph query API rather than accepting Cypher. The two temporal intervals follow Graphiti's fact model.

Storage compatibility#

Implicit relationships retain their existing keys. Explicit identities use a versioned key suffix, and derived journal records containing them use format v8; ordinary batches continue to use v7. New readers accept existing v2–v7 journal records. Upgrade every replica, coordinator, and backup/restore reader before creating explicit relationships or enabling nodes.source. An older reader cannot interpret the new relationship keys, so downgrade requires removing the new projections and rebuilding compatible indexes and replay state. Relationship IDs remain scoped identities rather than globally unique UUIDs. Data-Raft admission rejects document and relationship writes until every applying replica supports protocol version 21, before catalog-dependent graph effects are extracted. HA replay independently requires batch envelope version 20 for these graph apply effects, including document-derived relationships; older standbys reject the envelope before applying any rows. The envelope preserves each control’s original schema and exact Raft receipts.

Portable backups preserve each explicit relationship’s owner, logical endpoints, ID, weight, timestamps, and metadata in a dedicated relationship block. Such archives require AFB reader version 6 or later; older readers reject them before import. Backups containing only implicit relationships retain the legacy edge blocks and reader version 2 requirement. Archives containing durable endpoint retirements require reader version 7. Explicit relationships use derived journal version 8 and data-Raft protocol version 21; existing owner and TTL journals remain readable.

Readiness Is Two Flags, Not One#

Creation starts materialization and replay; it does not mean coverage is complete. Poll GET /db/v1/tables/{table}/indexes/{index}:

readiness.queryable is the serving authority: the index can answer queries.
readiness.complete is the convergence authority: every source is fully covered.
readiness.pending_reasons gives typed aggregate blockers or failure reasons.
readiness.sources lists each artifact stream in configuration order with its state, completeness, and typed blockers, so document-vector lag and chunk-vector lag show up separately.

Every index kind also reports a catalog-assigned incarnation, at the status root and inside readiness. A waiter succeeds only on a runtime observation of that same definition, so dropping and recreating an index under the same name cannot reuse stale readiness or searchable counts.

The document-level coverage object belongs to embedding indexes. Full-text and graph indexes report completion through readiness, including readiness.sources for artifact-backed projections, and replay status. They do not publish embedding producer counters: an absent counter cannot distinguish unfinished extraction from an intentional empty output. Source extraction failures remain visible in source readiness and repair status, including after restart.

Artifact source readiness also waits for verified completion of that source's producer dependencies. A caught-up replay cursor cannot mark unfinished OCR, transcription, chunking, or relation extraction complete before its output is published. Producer work uses durable, revision-fenced summaries; status polling does not scan documents. Each source is checked independently, so relation extraction debt does not block a text stream that has finished. Intentional empty outputs can complete; extraction failures remain unhealthy.

Existing stores establish these summaries through bounded background discovery and verification of retained artifacts. Local producers use the durable paged source-recovery owner and retain the published generation once its receipts and native checkpoint agree. Until discovery finishes, producer-backed sources remain pending. Restart, source edits, and catalog changes retain or reopen the appropriate completion fences. Known terminal failures remain with their repair/retry owner rather than being regenerated on every open.

Internal replay watermarks are diagnostics only.

Changing source membership, projection fields, analyzer semantics, embedding space, or graph mapping changes what the index means and requires a new materialized generation. Execution batching fields tune work without changing artifact identity.

Canonical, Alternative, and Compatibility Forms#

sources is the canonical multi-source API. Three one-source alternatives are supported and not deprecated: full-text artifact_name, embeddings embedding_name, and graph source. Normalized responses expose canonical sources where the released contract requires it.

The released v0.2 embeddings fields remain supported for v0.2 compatibility, with no removal release scheduled. source_artifact_name is descriptive when paired with embedding_name; the matching enrichment is authoritative. Internal graph manifest and edge materialization fields are engine-owned, not alternate public APIs. No graph status aliases were released, and the released Go v0.2 SDK surface (go/pkg/sdk/v0.2.0) creates no additional compatibility obligation.

Use Agent Skills#

Everything above is also encoded in the Antfly skill, so a coding agent can execute this guide for you:

npx skills add antflydb/antfly-skills

Then prompt it with the outcome, for example "Create an artifact-backed embeddings index over my docs table that unions document-level and chunk-level vectors", and use this page to judge the result.

Next Steps#

Document Engine: how the planner composes full-text, vector, filter, and graph inputs over any index
Multimodal Embeddings: a single-source embeddings index over remote media
Tune Hybrid Search: fusion, reranking, and pruning over the results