Cluster Management#

Monitor and manage your Antfly cluster's health and status.

Cluster Health#

The cluster health endpoint provides real-time status of all stores and shards:

  • healthy: All nodes and shards operational
  • degraded: Some shards unavailable but cluster functional
  • unhealthy: Critical failures affecting availability
  • error: Cluster-wide failures

Monitoring#

  • Health check endpoint: GET /healthz
  • Prometheus metrics: GET /metrics on port 4200

Use these endpoints to integrate with monitoring systems and alerting infrastructure.

Common questions about this section
  • How do I check if my cluster is healthy?
  • What do the different health statuses mean?
  • How do I integrate Antfly with Prometheus?

Get cluster status#

GET/status

Returns minimal cluster health and runtime status.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Code Examples#

curl -X GET "/db/v1/status" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

{
  "health": "unknown",
  "message": "string",
  "auth_enabled": true,
  "deployment_mode": "embedded",
  "index_capabilities": {
    "artifact_sources": true,
    "artifact_sources_state": "available"
  },
  "secret_store": {
    "generation": 0,
    "supports_source_generation": true,
    "source_generation": "string",
    "last_reload_failed": true,
    "stale": true,
    "reload_successes": 0,
    "reload_failures": 0
  },
  "runtime_config": {
    "generation": 0,
    "hash": "string",
    "last_reload_failed": true,
    "stale": true,
    "reload_successes": 0,
    "reload_failures": 0
  },
  "storage": {
    "engine": "lite",
    "format": "string",
    "fsync": true,
    "maintenance": {
      "check": true,
      "compact": true,
      "vacuum": true,
      "online": true,
      "asynchronous": true
    }
  }
}

Get cluster topology#

GET/cluster

Returns cluster health plus data-node, range, replica, and group topology for dashboard views.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Code Examples#

curl -X GET "/db/v1/cluster" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

{
  "health": "unknown",
  "message": "string",
  "auth_enabled": true,
  "deployment_mode": "embedded",
  "index_capabilities": {
    "artifact_sources": true,
    "artifact_sources_state": "available"
  },
  "secret_store": {
    "generation": 0,
    "supports_source_generation": true,
    "source_generation": "string",
    "last_reload_failed": true,
    "stale": true,
    "reload_successes": 0,
    "reload_failures": 0
  },
  "runtime_config": {
    "generation": 0,
    "hash": "string",
    "last_reload_failed": true,
    "stale": true,
    "reload_successes": 0,
    "reload_failures": 0
  },
  "storage": {
    "engine": "lite",
    "format": "string",
    "fsync": true,
    "maintenance": {
      "check": true,
      "compact": true,
      "vacuum": true,
      "online": true,
      "asynchronous": true
    }
  },
  "data": {
    "nodes": [
      {
        "data_id": 0,
        "node_id": 0,
        "api_url": "string",
        "raft_url": "string",
        "role": "string",
        "state": "string",
        "health_class": "string",
        "failure_domain": "string",
        "live": true,
        "drain_requested": true,
        "capacity_bytes": 0,
        "available_bytes": 0,
        "lease_pressure": 0,
        "read_load": 0,
        "write_load": 0,
        "active_backfills": 0
      }
    ],
    "ranges": [
      {
        "group_id": 0,
        "range_id": 0,
        "table_id": 0,
        "table_name": "string",
        "start_key": "string",
        "end_key": "string",
        "doc_identity_shard_id": 0,
        "doc_identity_range_id": 0,
        "state": "string",
        "leader_data_id": 0,
        "voter_count": 0,
        "doc_count": 0,
        "disk_bytes": 0,
        "empty": true
      }
    ],
    "replicas": [
      {
        "group_id": 0,
        "data_id": 0,
        "node_id": 0,
        "replica_id": 0,
        "peer_node_ids": [
          0
        ]
      }
    ],
    "groups": [
      {
        "group_id": 0,
        "leader_known": true,
        "leader_data_id": 0,
        "voter_count_known": true,
        "voter_count": 0,
        "healthy_voter_reports": 0,
        "joint_consensus": true,
        "transition_pending": true,
        "replay_required": true,
        "replay_caught_up": true,
        "cutover_ready": true,
        "reads_ready_after_cutover": true,
        "doc_identity_lifecycle": "string",
        "doc_count": 0,
        "disk_bytes": 0,
        "empty": true
      }
    ]
  }
}

List configured external connections#

GET/connections

Enumerates public external connections configured on this node under top-level connections: inference providers, web search providers, external IO endpoints, and CDC replication sources.

The default response is config-derived and avoids slow provider calls. With include=models, each inference provider is queried live for its available models where the provider exposes a listing API. Connections that fail to respond are reported with status "error" instead of failing the whole response. A status of "configured" means the connection exists but was not live-probed in this response.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Parameters#

NameTypeLocationRequiredDescription
typesstringqueryNoComma-separated list of connection kinds to include
(e.g. "inference,external_io,cdc"). Defaults to all
kinds. This filters by the response "kind" field.
includestringqueryNoComma-separated list of expansions. Supported values: models to
live-query inference model listings and status to live-probe
external connections. Live work is opt-in and single-flight per
server.
refreshstringqueryNoSet to "true" to bypass the short server-side cache for requested
live expansions. Live expansion passes are serialized to prevent
concurrent refresh amplification. This does not force a node config
or metadata reload.

Code Examples#

curl -X GET "/db/v1/connections?types=value&include=value&refresh=value" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

{
  "connections": [
    {
      "id": "string",
      "name": "string",
      "display_name": "string",
      "provider": "string",
      "kind": "inference",
      "status": "connected",
      "error": "string",
      "capabilities": [
        "string"
      ],
      "sources": [
        "string"
      ],
      "inference": {
        "provider": "gemini",
        "url": "string",
        "region": "string",
        "project_id": "string",
        "location": "string",
        "names": [
          "string"
        ],
        "configured_model_types": [
          "embedder"
        ],
        "models": {}
      },
      "web_search": {
        "service": "string",
        "max_results": 0,
        "timeout_ms": 0,
        "safe_search": true,
        "language": "string",
        "region": "string",
        "include_content": true,
        "include_highlights": true,
        "endpoint": "string",
        "project_id": "string",
        "location": "string",
        "data_store": "string",
        "serving_config": "string",
        "include_domains": [
          "string"
        ],
        "exclude_domains": [
          "string"
        ],
        "configured": true
      },
      "external_io": {
        "protocol": "s3",
        "endpoint": "string",
        "buckets": [
          "string"
        ],
        "prefix": "string",
        "hosts": [
          "string"
        ]
      },
      "cdc": {
        "provider": "postgres",
        "table_name": "string",
        "source_ordinal": 0,
        "external_table": "string",
        "slot_name": "string",
        "publication_name": "string",
        "phase": "string",
        "lag_records": 0,
        "lag_millis": 0,
        "last_success_at_ms": 0,
        "last_change_applied_at_ms": 0,
        "updated_at_ms": 0
      }
    }
  ]
}

Invoke an Antfly-compatible inference connection#

POST/connections/{connection_id}/inference/{operation}

Invokes an inference operation through the selected connection. Requires inference/* write permission. Generation requests with stream: true return the provider's Server-Sent Events stream; all other responses are returned as buffered JSON.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Parameters#

NameTypeLocationRequiredDescription
connection_idstringpathYes
operationstringpathYesRequires the connection capability models.<operation>.

Request Body#

Example:

{}

Code Examples#

curl -X POST "/db/v1/connections/{connection_id}/inference/{operation}" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{}'

Responses#

{}

List secrets status#

GET/secrets

List all configured secret names and their status (secret-store file, env var, or both). Never returns secret values — only names and configuration status.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Code Examples#

curl -X GET "/db/v1/secrets" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

{
  "secrets": [
    {
      "key": "string",
      "status": "configured_file",
      "env_var": "string",
      "created_at": "2024-01-01T00:00:00Z",
      "updated_at": "2024-01-01T00:00:00Z"
    }
  ]
}

Store a secret#

PUT/secrets/{key}

Store a secret in the configured writable secret-store file. Only available in standalone mode. Returns 503 in multi-node mode.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Request Body#

Example:

{
    "value": "string"
}

Code Examples#

curl -X PUT "/db/v1/secrets/{key}" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
    "value": "string"
}'

Responses#

{
  "key": "string",
  "status": "configured_file",
  "env_var": "string",
  "created_at": "2024-01-01T00:00:00Z",
  "updated_at": "2024-01-01T00:00:00Z"
}

Delete a secret#

DELETE/secrets/{key}

Remove a secret from the configured writable secret-store file. Only available in standalone mode. Returns 503 in multi-node mode.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Code Examples#

curl -X DELETE "/db/v1/secrets/{key}" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

No response body

Backup all tables or selected tables#

POST/backup

Creates a backup of all tables or specified tables. Each table's backup includes:

  • Table metadata (schema, indexes, shard configuration)
  • All shard data (compressed with zstd)

A non-empty backup publishes a cluster-level manifest only after every requested table backup is durable. The manifest is the final commit point and records complete expected/completed table counts. A partial or failed attempt returns per-table diagnostics but does not publish a restorable aggregate manifest. A cluster with no selected tables returns 400 without writing a backup artifact.

Backup IDs are immutable. Reusing an ID that already has a published cluster manifest returns 409 and leaves the existing backup unchanged.

Storage Locations:

  • Local filesystem: file:///path/to/backup
  • Amazon S3: s3://bucket-name/path/to/backup

Backup Structure:

{location}/
├── {backup_id}-cluster-metadata.json   (cluster manifest)
└── generation-scoped table manifests and payloads

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Request Body#

Example:

{
    "backup_id": "cluster-backup-2025-01-15",
    "location": "s3://mybucket/antfly-backups/cluster/2025-01-15",
    "connection": "string",
    "format": "portable",
    "table_names": [
        "users",
        "products"
    ]
}

Code Examples#

curl -X POST "/db/v1/backup" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
    "backup_id": "cluster-backup-2025-01-15",
    "location": "s3://mybucket/antfly-backups/cluster/2025-01-15",
    "connection": "string",
    "format": "portable",
    "table_names": [
        "users",
        "products"
    ]
}'

Responses#

{
  "backup_id": "cluster-backup-2025-01-15",
  "tables": [
    {
      "name": "users",
      "status": "completed",
      "error": "string",
      "code": "backup_outcome_ambiguous",
      "retryable": true,
      "backup_id": "string",
      "artifact_backup_id": "string"
    }
  ],
  "status": "completed"
}

Restore multiple tables from a backup#

POST/restore

Restores tables from a cluster backup. Can restore all tables or a subset.

Restore Modes:

  • fail_if_exists: Abort if any target table already exists (default)
  • skip_if_exists: Skip existing tables and restore the rest
  • overwrite: Stage and validate replacement generations, then atomically publish them over existing tables.

The restore is a durable asynchronous job. The request returns after the job record is persisted and both a durable job store and asynchronous worker are available. Poll the restore job resource for progress. Catalog publication is durably checkpointed per table and is not repeated after restart. If leadership changes before that checkpoint, recovery adopts only an exact, still-active restore intent for the same backup and location; unrelated or ambiguous existing tables fail closed. A job reaches succeeded only after all placement replicas report the restore complete and metadata clears the restore intent. Cancellation is cooperative between table publication boundaries. Remote transfer and staging occur before the per-table write fence. Publication stops new admission and drains current readers and writers before swapping the direct-path generation.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Parameters#

NameTypeLocationRequiredDescription
Idempotency-KeystringheaderNoStable key used to safely retry creation of this restore job. Keys are scoped to the authenticated principal and cluster restore target. Requests without this header create a new job.

Request Body#

Example:

{
    "backup_id": "cluster-backup-2025-01-15",
    "location": "s3://mybucket/antfly-backups/cluster/2025-01-15",
    "connection": "string",
    "table_names": [
        "users",
        "products"
    ],
    "restore_mode": "skip_if_exists"
}

Code Examples#

curl -X POST "/db/v1/restore" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
    "backup_id": "cluster-backup-2025-01-15",
    "location": "s3://mybucket/antfly-backups/cluster/2025-01-15",
    "connection": "string",
    "table_names": [
        "users",
        "products"
    ],
    "restore_mode": "skip_if_exists"
}'

Responses#

{
  "job_id": "string",
  "attempt_id": 0,
  "scope": "table",
  "table_name": "string",
  "backup_id": "string",
  "phase": "queued",
  "cancel_requested": true,
  "durability_pending_table_count": 0,
  "published_table_count": 0,
  "completed_table_count": 0,
  "total_table_count": 0,
  "result": {
    "restore": "triggered",
    "durability": "pending",
    "status": "completed",
    "triggered_table_count": 0,
    "committed_table_count": 0,
    "durability_pending_table_count": 0,
    "skipped_table_count": 0,
    "failed_table_count": 0,
    "failure_details": [
      {
        "table_name": "string",
        "error": "string",
        "table_name_truncated": true
      }
    ],
    "failure_details_truncated": true
  },
  "error": "string",
  "created_at_ms": 0,
  "updated_at_ms": 0,
  "expires_at_ms": 0
}

List durable restore jobs#

GET/restore/jobs

Returns a newest-first, authorization-filtered page of retained restore jobs from the metadata leader.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Parameters#

NameTypeLocationRequiredDescription
limitintegerqueryNo
cursorstringqueryNoOpaque cursor returned by the preceding page.
phasestringqueryNo
scopestringqueryNo

Code Examples#

curl -X GET "/db/v1/restore/jobs?limit=value&cursor=value&phase=value&scope=value" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

{
  "jobs": [
    {
      "job_id": "string",
      "attempt_id": 0,
      "scope": "table",
      "table_name": "string",
      "backup_id": "string",
      "phase": "queued",
      "cancel_requested": true,
      "durability_pending_table_count": 0,
      "published_table_count": 0,
      "completed_table_count": 0,
      "total_table_count": 0,
      "result": {
        "restore": "triggered",
        "durability": "pending",
        "status": "completed",
        "triggered_table_count": 0,
        "committed_table_count": 0,
        "durability_pending_table_count": 0,
        "skipped_table_count": 0,
        "failed_table_count": 0,
        "failure_details": [
          {
            "table_name": "string",
            "error": "string",
            "table_name_truncated": true
          }
        ],
        "failure_details_truncated": true
      },
      "error": "string",
      "created_at_ms": 0,
      "updated_at_ms": 0,
      "expires_at_ms": 0
    }
  ],
  "next_cursor": "string"
}

Get durable restore job status#

GET/restore/jobs/{job_id}

Returns replicated restore-job state. A metadata follower may return the retryable metadata-not-leader 503 until a newly committed job has applied locally; clients should retry instead of treating that response as job absence.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Code Examples#

curl -X GET "/db/v1/restore/jobs/{job_id}" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

{
  "job_id": "string",
  "attempt_id": 0,
  "scope": "table",
  "table_name": "string",
  "backup_id": "string",
  "phase": "queued",
  "cancel_requested": true,
  "durability_pending_table_count": 0,
  "published_table_count": 0,
  "completed_table_count": 0,
  "total_table_count": 0,
  "result": {
    "restore": "triggered",
    "durability": "pending",
    "status": "completed",
    "triggered_table_count": 0,
    "committed_table_count": 0,
    "durability_pending_table_count": 0,
    "skipped_table_count": 0,
    "failed_table_count": 0,
    "failure_details": [
      {
        "table_name": "string",
        "error": "string",
        "table_name_truncated": true
      }
    ],
    "failure_details_truncated": true
  },
  "error": "string",
  "created_at_ms": 0,
  "updated_at_ms": 0,
  "expires_at_ms": 0
}

Request cooperative restore cancellation#

DELETE/restore/jobs/{job_id}

Requests best-effort cancellation. Queued work is cancelled immediately and running work stops at its next safe boundary. If irreversible restore publication completes before cancellation is observed, the job remains succeeded with cancel_requested: true.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Code Examples#

curl -X DELETE "/db/v1/restore/jobs/{job_id}" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

{
  "job_id": "string",
  "attempt_id": 0,
  "scope": "table",
  "table_name": "string",
  "backup_id": "string",
  "phase": "queued",
  "cancel_requested": true,
  "durability_pending_table_count": 0,
  "published_table_count": 0,
  "completed_table_count": 0,
  "total_table_count": 0,
  "result": {
    "restore": "triggered",
    "durability": "pending",
    "status": "completed",
    "triggered_table_count": 0,
    "committed_table_count": 0,
    "durability_pending_table_count": 0,
    "skipped_table_count": 0,
    "failed_table_count": 0,
    "failure_details": [
      {
        "table_name": "string",
        "error": "string",
        "table_name_truncated": true
      }
    ],
    "failure_details_truncated": true
  },
  "error": "string",
  "created_at_ms": 0,
  "updated_at_ms": 0,
  "expires_at_ms": 0
}

List available backups#

GET/backups

Lists one bounded page of cluster-level backups in stable manifest-key order at the specified location. Returns metadata about each backup including the tables included, timestamp, and Antfly version. Pass the returned next_cursor unchanged to retrieve the next page.

Provide your bearer token in the Authorization header when making requests to protected resources.

Example: Authorization: Bearer YOUR_API_KEY

Parameters#

NameTypeLocationRequiredDescription
locationstringqueryYesStorage location to search for backups.
  • Local filesystem: file:///path/to/backup
  • Amazon S3: s3://bucket-name/path/to/backup | | connection | string | query | Yes | Named external_io connection authorized for reading this backup location. | | limit | integer | query | No | Maximum backups returned in one page. | | cursor | string | query | No | Continuation cursor returned by the preceding page. |

Code Examples#

curl -X GET "/db/v1/backups?location=s3://mybucket/antfly-backups/&connection=value&limit=value&cursor=value" \
    -H "Authorization: Bearer YOUR_API_KEY"

Responses#

{
  "backups": [
    {
      "backup_id": "cluster-backup-2025-01-15",
      "timestamp": "2025-01-15T10:30:00Z",
      "tables": [
        "users",
        "products"
      ],
      "location": "s3://mybucket/antfly-backups/cluster/2025-01-15",
      "antfly_version": "v1.0.0",
      "format": "portable"
    }
  ],
  "next_cursor": "string"
}