Table privacy#

Privacy inspects selected string fields for configured entity categories. A policy can report findings (monitor) or redact the documents consumed by search and AI (redact_for_search_and_ai). Originals remain stored. The redact_before_storage mode returns HTTP 422 (privacy_synchronous_ingestion_required) on preview and policy installation, before sampling or provider execution, until synchronous ingestion is available. Policies, previews, and rescans require document storage without an external base source; unsupported storage returns PrivacyRelationalStorageUnsupported before inspection or mutation.

Start in a table's Privacy tab: choose fields and categories, preview a bounded sample, inspect highlighted spans, and adjust sensitivity or individual category thresholds. A protective save requires a fresh preview and an impact acknowledgment. Samples contain original values and require table administrator access. A preview is not a table-wide detection-quality assessment.

API#

All paths below are relative to /db/v1. Privacy operations require table administration permission when access control is enabled.

POST /tables/customers/privacy/preview
Content-Type: application/json

{
  "policy": {
    "mode": "redact_for_search_and_ai",
    "fields": ["body"],
    "category_groups": ["contact_details", "personal_identifiers"],
    "sensitivity": "balanced"
  },
  "sample": {"keys": ["customer-1", "customer-2"]}
}

Preview accepts up to 50 documents and does not install a policy. Save the policy object with PUT /tables/customers/privacy. The response is 202 with an ETag, immutable policy_version, and an automatically scheduled scan_id. Read the policy, resolved producer, and current coverage with GET on that path. Replacing an existing policy requires its exact ETag in If-Match; missing and stale preconditions return 428 and 412 respectively.

Poll GET /tables/customers/privacy/scans/{scan_id}. Jobs progress through queued, running, and processing; processing verifies current receipts and consumer publication before succeeded. Failed jobs expose bounded document IDs and error codes. A semantic policy replacement supersedes earlier jobs. Supersession is committed atomically with replacement and remains terminal if a later policy reuses the same semantic version. Execution-only tuning preserves jobs and their progress.

GET /tables/customers/privacy/findings?field=body&category=email&limit=100

Findings include document, field, category, half-open UTF-8 byte coordinates, policy version, source fingerprint, producer revision, and confidence when the producer supplies it. They omit matched values. Pagination cursors bind the policy and filters; restart pagination after changing either. Treat cursors as opaque strings and pass them unchanged, including for large or escaped document identifiers. Use POST /tables/customers/privacy/findings with the same fields in a JSON body for large cursors or filters that exceed URL limits. The TypeScript SDK and Privacy tab use this body form. Explicit policy_version requests for a superseded policy return 410.

Failure examples are limited to 100 entries and 1 MiB of serialized JSON. Coverage and scan failure counts include documents omitted from those examples.

Start a bounded retry or rescan with POST /tables/customers/privacy/scans, optionally supplying from_key, to_key, and a page limit from 1 to 1000. Source updates also schedule inspection through the ordinary producer pipeline.

Repairing an activation parked before an upgrade#

Upgrading the server or correcting a provider does not erase terminal producer failures. This includes activations affected by the v0.3.0 text/document transport mismatch: their repair ledger can contain InferenceTransportUnavailable and dependent ArtifactDependencyFailed entries even though the provider is now healthy. Protected reads continue to fail closed until current receipts and consumer publication are complete.

After upgrading and verifying the configured extractor endpoint, inspect the policy and its bounded artifact repair ledger:

GET /tables/customers/privacy

POST /tables/customers/repair/issues
Content-Type: application/json

{"target":"artifact","limit":50}

Create a fresh privacy scan to explicitly retry inspection and its dependent consumers under the same installed policy:

POST /tables/customers/privacy/scans
Content-Type: application/json

{"limit":100}

The coordinator advances bounded pages; limit is a per-page bound, not a cap on the total repaired documents. Supply from_key and to_key when only a particular key range needs repair. Poll the returned new scan_id until succeeded; an older failed scan remains historical failure evidence. Verify that coverage reports no pending or protected-blocked documents and that retrieval returns the redacted projection. If the new scan fails, inspect its error codes and repair ledger before retrying again. Do not disable privacy, replace the policy, or delete its artifacts to bypass the serving barrier.

Producers and execution#

The default is the pinned Fastino GLiNER2 PII model at revision 1cb4166094dc58fa8d836429f060d6c95f62b495. It uses Antfly's existing artifact extraction infrastructure, including native admission and cancellation. Long documents use bounded overlapping windows with exact 512-token admission; spans are rebased into the source. Compatible inspections are batched across documents and fields, and compatible windows use Metal/CUDA batches of up to eight. Native CPU execution remains serial.

Override producer using the artifact API's configuration shape:

{
  "type": "extractor",
  "revision": "immutable-deployment-revision",
  "config": {
    "provider": "antfly",
    "model": "my-pinned-extractor",
    "schema_version": 2
  }
}

Generated extractors and generators are supported too. Generators use type: "generator" and an artifact producer config containing generator and output_schema. Evidence must contain valid source spans; invalid evidence fails inspection. Custom producer revisions are mandatory. Credentials must be empty or exact ${secret:key.name} references. Omitting a credential uses the provider’s environment defaults. Literal credentials in nested provider options, headers, endpoint userinfo, query parameters, or fragments are rejected before preview or installation. Schema and entity-label definitions remain user data. Remote producers receive original selected text; the preview and policy response disclose external processing. That disclosure uses execution's route selection, including inference.api_url, explicit producer endpoints, and task-specific linked callback availability. Generated extractors follow their generator's route.

Managed document projections have a source-sized host budget, separate from producer response limits. Large selected values and unrelated document fields therefore do not consume the detector's evidence allowance. Preview still has its own 8 MiB sample and response limits; provider output remains independently bounded. Selected text must also fit the configured producer input budget; for constrained generators this is producer.config.batch.max_input_bytes.

The union of expanded category groups and explicit custom categories is limited to 128 categories. Overrides of an existing group category count once. This limit is checked before preview or policy installation.

execution.batch_items (1–8) and execution.max_attempts (1–5) tune batching and invalid-evidence retries. Changing them preserves semantic policy identity and accepted inspection results. Transport backoff remains owned by the artifact runtime. Dependency priming honors the same durable, source-bound retry clock as ordinary replay, including after restart. A deferred inspection does not prevent independent documents' downstream work from progressing. Changing fields, categories, thresholds, replacement, mode, or producer identity creates a new policy version and schedules backfill.

Protected serving#

Protective policies atomically rewrite default and named full-text/vector consumers to use the sanitized projection. Chunking and embeddings consume sanitized document text. Original authored vectors, unsupported graph or relational consumers, and unsupported extraction chains are rejected during admission. Ordinary index/enrichment mutations cannot replace policy-owned objects while protection is active.

Point reads and hydration verify current source, policy, and storage-owned publication evidence. Search also fences producer completion and physical index checkpoints before candidates, statistics, filtering, and aggregation. Queries can temporarily return unavailable while a protected generation is pending or failed; readiness is conservative across the table's managed consumers. Original values do not become searchable while backfill runs. Original artifact inspection requires separate administrator access.

SQL reads of document tables apply predicates, joins, projections and aggregates to the sanitized document snapshot. SQL mutations and delayed serializable snapshots on protected tables return an unsupported response. Use the ordinary document batch API for writes; it preserves originals and schedules inspection for the new source revision. Pending inspection makes SQL reads retryably unavailable rather than exposing an older source or original values.

Coverage reports current, pending, known failed, documents with findings, category and field counts, and protected readiness. Unavailable or incomplete census results have null counts instead of an optimistic zero. Detection is probabilistic: qualify categories and thresholds against representative text before relying on the policy for your application.

Coverage counts are collected by a durable background census in bounded pages. GET /tables/{table}/privacy reads its latest completed result without scanning the data shards. started_at_millis and completed_at_millis describe the observation interval; these informational counts refresh asynchronously and are not an atomic cross-shard snapshot or permission to serve a document. While refreshing is true, the previous completed counts remain visible. Before the first complete pass, coverage is unavailable. Source and policy receipts still fence every protected read independently of these counts.

Scan history retains at most 16 jobs. Completed, failed, and superseded jobs can be retired; active jobs are preserved. When all 16 are active, another scan request returns HTTP 409 (privacy_scan_limit_exceeded) without losing work.

Backfill discovery queues document references without retaining source bodies. The requested scan limit is a row ceiling; discovery also limits retained identifiers and reference metadata to an 8 MiB page. One oversized identifier can advance alone. Replay captures current sources and admits detector work after discovery; discovery does not rewrite originals or renew TTLs.

Policy saves acknowledge the installed resource version. Background scan and coverage updates do not change its ETag or delay acknowledgement. Equivalent field/category ordering and producer object-key ordering keep the executable producer unchanged; execution tuning still updates runtime controls separately. The policy-bound runtime override survives restart and governs subsequent inspection attempts without invalidating accepted receipts.

Policy replacement preserves the live resolver and curation control state, including proposals created after protection was enabled. Switching to monitor restores the logical consumer declarations while retaining current control state. Re-enabling protection does not resurrect deleted resolver metadata or interpret curation evidence as index or producer configuration. Scan and coverage progress do not invalidate resolver admission or query proofs; authored policy and consumer changes still fence them.

Findings enumerate document IDs and inspect one source and manifest at a time. Pages stop between documents after an 8 MiB inspection budget, or within a document when the finding count or output-byte budget is reached. A document is validated completely before any of its findings are returned, so a single large document can exceed the inspection budget. Follow next_cursor even when a page contains no findings; missing or stale receipts count as pending.

Full-table scan jobs share the background coverage census. Completion requires a fresh pass started after that job entered verification, plus protected serving readiness where applicable. Range scans verify their own bounds in adaptive pages.

Preview accepts at most 50 documents. Samples, including JSON-escaped document identifiers, have an 8 MiB budget; preview responses also have an 8 MiB budget. Exceeding either returns HTTP 422 (privacy_sample_too_large); request fewer or smaller documents. Automatic sampling streams rows and stops before retaining an oversized sample. A rejected preview does not install a policy.

A newly created Raft table can become visible before its data owner is ready. Privacy operations return HTTP 503 with retryable: true and Retry-After: 1 while an owner, required index, or group leader is temporarily unavailable. Retry a preview after that delay. Invalid policies remain HTTP 422, policy CAS failures remain HTTP 412/428, and an unknown mutation outcome remains HTTP 409: inspect policy/scan state before retrying that mutation. Availability handling does not automatically replay policy writes or scan creation.

A successful policy or scan admission returns its prepared 202 response. Reconcile scheduling is a best-effort acceleration of durable work; a failed hint cannot turn an accepted policy into a retryable rejection. A lost or unusable transport response still requires reconciling policy/scan state.

An upstream X-Antfly-Raft-Mutation-Outcome: unknown-v1 marker is preserved through metadata and data-node forwarding, alongside the status, response body, ETag and Retry-After. Treat this marker as an ambiguous write outcome on either 409 or 503: inspect policy/scan state before deciding whether to submit again. The TypeScript SDK exposes these headers on AntflyHTTPError as mutationOutcome ("unknown" when marked) and retryAfterSeconds (a nonnegative numeric delay when supplied). The delay alone never authorizes replaying a mutation; the SDK does not retry policy or scan writes.

For browser clients on an allowed cross-origin site, default CORS permits If-Match and exposes ETag, Retry-After and X-Antfly-Raft-Mutation-Outcome. Explicit cors.allowed_headers and cors.exposed_headers lists replace those defaults; include these headers when customizing the lists so conditional saves and outcome inspection remain usable.

Private range checkpoints have per-table budgets of 256 MiB for staged uploads and 8 GiB for retained payloads, including retired generations awaiting cleanup. Capacity exhaustion returns HTTP 422 (privacy_checkpoint_capacity_exceeded); allow cleanup to finish and use fewer or smaller concurrent range scans before retrying. Metadata protocol 82 is required for new privacy mutations. Older checkpoint headers remain readable and are reclaimed during upgrade.

Table backups retain the policy, its execution controls, managed producers and protected index configuration. Restore creates a fresh full-table scan and coverage census for the destination. Source-table scan history, cursors and coverage counts are not imported; follow the restored resource's new scan_id. Its policy version and ETag remain unchanged. Protected reads still require current source receipts and authoritative publication, independently of census progress. Private artifacts without destination publication are regenerated, so the configured producer may need to run again. Metadata recovery and hot-standby replication retain their own resumable job state; this reset applies to table backup restore.

Privacy operations also support explicit database and namespace scope: /databases/{databaseName}/namespaces/{namespaceName}/tables/{tableName}/privacy and the same /preview, /findings, /scans, and /scans/{scanId} suffixes. All require administration permission on that scoped table. A plain table name always selects default.public; dots and escaped slashes never infer scope.

The TypeScript privacy client accepts either a literal name or an explicit target for each operation. The target uses the shared CatalogTableTarget contract, with database default and namespace public when omitted. For example:

const target = { database: "tenant", namespace: "serving", table: "notes" };
const current = await client.privacy.get(target);
await client.privacy.put(target, policy, { ifMatch: current.etag });

Generated Go, Python, and Rust clients expose the corresponding getNamespaceTablePrivacy, putNamespaceTablePrivacy, previewNamespaceTablePrivacy, startNamespaceTablePrivacyScan, getNamespaceTablePrivacyScan, listNamespaceTablePrivacyFindings, and queryNamespaceTablePrivacyFindings operations (using each language's naming convention).

Index migration while protection is active#

With redact_for_search_and_ai, create a new physical embedding or full-text index through the normal create-index API. Antfly attaches it to the current protected artifact graph in the same catalog commit. Existing index generations and privacy inspection receipts remain usable. Wait for the new index to become complete, then move the serving index alias with expected_index set to the old physical index. After cutover, delete the unreferenced old physical index through the normal delete-index API. Its private chunk/embedding producers retire in the same catalog mutation, so later writes no longer invoke its model. Shared producers and the protected serving guard remain owned by their surviving consumers. Raw external vectors and unsupported graph consumers are rejected. Repeating an identical creation request preserves the physical generation.

Semantic policy changes (including producer revision or replacement style) replace dependent consumer generations and backfill them under the new policy. Identical policy PUTs and execution-only tuning preserve existing generations. An alias that still references a changed consumer blocks the policy replacement with HTTP 409 IndexReferencedByAlias. Remove every alias referencing a changed consumer before changing the policy; rebind only after the protected index is complete. To remove an alias, GET /tables/{tableName}/index-aliases/{aliasName} and DELETE the same path with If-Match set to that response's exact ETag. Missing validators return 428, stale validators return 412, and a repeated deletion returns 404. This CAS also protects against deleting a concurrently recreated alias or one restored from a backup. Physical indexes and retained PUT idempotency receipts remain intact. Protection remains active throughout the transition. Alias mutations require metadata protocol 91 and store artifact-source protocol 4 so older owners cannot misread the new validator state. Legacy alias catalogs continue to admit protocol-3 stores during rolling upgrades until lifecycle validator state is committed.

Missing producer input does not become a successful replay merely because a retry limit elapsed. Its journal window remains pending until input arrives, a durable source deletion or generation proves it retired, or explicit repair resolves the missing state. Workers park instead of polling the same failed window; source or producer commits and explicit repair notifications wake them, including repairs at the same replay sequence. Ordinary query visibility waits do not wake an unchanged missing-input window. Continuing writes also respect a bounded retry backoff; they cannot cause a blocked window to replay at the write rate. A retry requires both a notification and an elapsed deadline, so idle workers do not poll. Reopening retries retained debt. This applies to full-text consumers as well as vector consumers.

Inspect stored vectors as an administrator#

For a dense index protected by redact_for_search_and_ai and backed by text chunk embedding producers, an explicit artifact inspection can use the index name:

GET /db/v1/tables/docs/documents/doc-id/artifacts/semantic-index?detail=raw

This uses the existing table administrator permission for raw artifact inspection. Tenant search and document projections retain their artifact restrictions. Existing asset names take precedence over an index with the same name. Index inspections are explicit lookups and are not added to asset manifest enumeration. Unprotected indexes and document-level, sparse, media, or elided-vector recipes are outside this inspection format.

The response has route_type: "dense_index_inspection", manifest_version: 1, and an empty top-level artifact_id because an index can consume many physical artifacts. Its raw manifest_json is a JSON string with format antfly.dense-index-inspection.v1. It contains the index name, physical coverage generation and recipe fingerprint; the active privacy policy version; the currently sanitized document_json and its SHA-256 fingerprint; and a current publication flag. Each vector carries its real artifact ID, producer name, provider/model identity, producer fingerprint, exact source text and SHA-256 fingerprint, the persisted artifact fingerprint and opaque provider cache identity, and persisted float values. Fingerprints are lowercase SHA-256 hex over the exact represented bytes.

Source, selected chunk heads, recipe, publication guards, and stored vectors are captured while apply is held shared, with the source and vectors read from one storage snapshot. Standalone inspection uses the same source-guarded coverage generations and clean projection checkpoints as protected serving; ordered owners additionally verify per-output causal provenance. The payload identifies this authority and reports the index/checkpoint config hashes, checkpoint status, applied sequence and replay target sequence. It does not substitute asset privacy witnesses for vector authority. current: false and an incomplete_reason do not certify vector equivalence. Pending root publication never yields certified vectors. The operation never calls an embedding provider to reconstruct missing values. It seeks only the selected producers for this document and bounds inspection to 16 producers, 128 vectors, 65,536 float values and 1 MiB of source payloads. Identities are limited to 4 KiB and each producer configuration to 1 MiB. Publication preflight and traversal share a 512-visit, 2 MiB budget, including policy/configuration bytes, readiness nodes, index checks and empty unit heads. The five-second deadline starts at capture entry and is checked throughout. Ordered global readiness verifies one durable catalog authority and each producer counter once, without rebuilding per-producer dependency closures. Standalone vector readiness reads its index-generation coverage once and does not compute source scopes. Full-text scope traversal is linear in the dependency graph and charges the same budget during edge discovery, traversal and hashing; its existing v2 receipt identity is preserved. Fixed payload or traversal limits return HTTP 413 with code inspection_limit_exceeded, retryable: false, and the configured limits; deadline or publication contention remains retryable. No response is truncated. Capture excludes primary and replay publication using the existing snapshot admission gates, then releases those gates before JSON encoding. checkpoint_generation and checkpoint_config_hash can be checked against the independent live index status fields projection_checkpoint_generation and projection_checkpoint_config_fingerprint (the latter is hexadecimal).

An independent oracle should compare these values against a separate invocation of the intended model on independently retrieved sanitized text. Check source revision, policy and index identity as well as the numeric values, and retain tenant permission checks separately.