AntflyAntfly

Artifact-Backed Indexes

How one full-text, vector, or graph index unions the records that enrichments generate, and how to tell when it is ready

An artifact-backed index consumes the records that named enrichments produce, rather than the source documents themselves. Its sources array unions up to 64 streams into one full-text, embeddings, or graph index, and each member keeps the stable identity (artifact, key), so records from different streams never collide even when their keys match. The decision this page is read for: when to search whole documents and their extracted chunks as one index, and how to know when that index is actually serving all of them.

Confirm the deployment supports it first:

curl http://127.0.0.1:8080/db/v1/status

index_capabilities.artifact_sources must be true. Embedded, standalone, and distributed deployments support artifact sources. Serverless rejects them and does not emulate older artifact-index behavior.

Document and Chunk Vectors in One Index#

One document-level embedding captures the whole; the chunks that document extraction produces are more precise. Indexing both as members of one embeddings index closes that gap. Both embedding enrichments must share compatible dimensions and a compatible vector space.

{
  "type": "embeddings",
  "dimension": 768,
  "embedder": { "provider": "antfly", "model": "BAAI/bge-base-en-v1.5" },
  "sources": [
    { "artifact": "document_dense_v1" },
    { "artifact": "document_chunk_dense_v1" }
  ],
  "enrichments": [
    {
      "name": "document_chunks_v1",
      "kind": "chunk",
      "field": "semantic_content",
      "chunk_size": 512
    },
    {
      "name": "document_dense_v1",
      "kind": "embedding",
      "field": "semantic_content",
      "expected_dims": 768,
      "vector_space": "searchaf:v1"
    },
    {
      "name": "document_chunk_dense_v1",
      "kind": "embedding",
      "field": "text",
      "source_artifact_name": "document_chunks_v1",
      "expected_dims": 768,
      "vector_space": "searchaf:v1"
    }
  ]
}

Create it at POST /db/v1/tables/{table}/indexes/document_vectors. vector_space is an application-owned compatibility identifier: every source in a multi-source index must either use the same explicit identifier or use semantically identical producer configurations, and explicit and implicit modes cannot be mixed.

Query document_vectors through the normal semantic search API. Raw member results keep their artifact identity. When the UI needs one source document with its matching chunks nested beneath it, use hierarchy grouping and request the documented top-level field projection.

Full-Text Sources With Different Shapes#

A top-level field is inherited by every source that does not override it. A source-local field lets one index union artifact records with different JSON shapes:

{
  "type": "full_text",
  "field": "text",
  "sources": [
    { "artifact": "document_units_v1", "field": "summary" },
    { "artifact": "document_chunks_v1" }
  ]
}

document_units_v1 indexes summary; document_chunks_v1 inherits text. If neither level names a field, Antfly uses the default text projection. Missing, null, and non-text values add no postings.

To query a named full-text index, set full_text_index together with full_text_search or a scoring text clause in the recursive query AST:

{
  "full_text_index": "document_text",
  "full_text_search": { "query": "retrieval augmented generation" },
  "limit": 10
}

Filter-only and exclusion-only query trees are not scoring text queries and receive HTTP 422 when combined with full_text_index.

Graph Sources and Edge Ownership#

Graph sources are ordered. When two sources materialize the same edge identity, the earlier source wins. Mapping (path, format, nodes, edge, and context) belongs on each source, because streams can use different record shapes. The optional source object is the one-source convenience form and is mutually exclusive with sources.

Artifact-backed edges are owned by the source document key; nodes.target maps the other endpoint. Source ownership is not configurable, so replay, deletion, routing, and range splits share one identity contract. Outgoing traversal routes directly to that source owner. Incoming and bidirectional traversal use an exact, bounded coordinator route directory fenced by the table topology and per-shard read generations. A cold or generation-mismatched entry falls back to compact, bounded reverse-index probes and records the proven source shards; expansion then goes only to those shards. Cross-range paths stay exact without synchronous cross-shard dual writes, and repeated traversals avoid cluster-wide probe fanout.

Readiness Is Two Flags, Not One#

Creation starts materialization and replay; it does not mean coverage is complete. Poll GET /db/v1/tables/{table}/indexes/{index}:

  • readiness.queryable is the serving authority: the index can answer queries.
  • readiness.complete is the convergence authority: every source is fully covered.
  • readiness.pending_reasons gives typed aggregate blockers or failure reasons.
  • readiness.sources lists each artifact stream in configuration order with its state, completeness, and typed blockers, so document-vector lag and chunk-vector lag show up separately.

Every index kind also reports a catalog-assigned incarnation, at the status root and inside readiness. A waiter succeeds only on a runtime observation of that same definition, so dropping and recreating an index under the same name cannot reuse stale readiness or searchable counts.

Internal replay watermarks are diagnostics only.

Changing source membership, projection fields, analyzer semantics, embedding space, or graph mapping changes what the index means and requires a new materialized generation. Execution batching fields tune work without changing artifact identity.

Canonical, Alternative, and Compatibility Forms#

sources is the canonical multi-source API. Three one-source alternatives are supported and not deprecated: full-text artifact_name, embeddings embedding_name, and graph source. Normalized responses expose canonical sources where the released contract requires it.

The released v0.2 embeddings fields remain supported for v0.2 compatibility, with no removal release scheduled. source_artifact_name is descriptive when paired with embedding_name; the matching enrichment is authoritative. Internal graph manifest and edge materialization fields are engine-owned, not alternate public APIs. No graph status aliases were released, and the released Go v0.2 SDK surface (go/pkg/sdk/v0.2.0) creates no additional compatibility obligation.

Use Agent Skills#

Everything above is also encoded in the Antfly skill, so a coding agent can execute this guide for you:

npx skills add antflydb/antfly-skills

Then prompt it with the outcome, for example "Create an artifact-backed embeddings index over my docs table that unions document-level and chunk-level vectors", and use this page to judge the result.

Next Steps#