Decisions#

Standalone decisions use POST /decisions (/ai/v1/decisions on the standalone server). Named question and answer arrays, input, choices, levels and predicate probability follow the OpenAI Decisions conventions. Antfly adds embedding similarity answers, multi-choice selection and text batches. Supported hosted providers also accept multimodal evidence; see the gateway decision contract for provider and method restrictions. Native trained/embedding decisions and the browser surface retain their text restrictions. Choice values are strings and every question needs a unique name. See the decision schema.

/extract retains entities, relations, attributes, structured records and ordinary classification. Standalone GLiDE, Laya, OpenDecider and EmbeddingGemma workflows use /decisions. The unreleased extraction decision output and Boolean and ordinal decision modes have been removed.

For the v0.3.0 development source documented here, standalone public inference requires an explicit opt-in before calling /ai/v1. Save antfly.json:

{"public_inference": {"enabled": true}}
antfly standalone --config antfly.json

There is no enablement CLI flag. Without this setting, public execution returns HTTP 403 even when models are installed. See the public execution policy. This prerequisite applies to database API nodes on this source version; the independent antfly inference run listener and older stable releases have their own policies.

Questions and answers#

{
  "model": "decision-model",
  "input": "Please refund the duplicate charge before tomorrow.",
  "questions": [
    {"name": "route", "type": "choice", "instructions": "Which team?", "choices": [{"value": "billing", "description": "Charges and refunds"}, {"value": "support", "description": "Product troubleshooting"}]},
    {"name": "urgency", "type": "score", "instructions": "How urgent?", "levels": [{"label": "routine"}, {"label": "soon"}, {"label": "immediate"}]},
    {"name": "refund", "type": "predicate", "instructions": "The input requests a refund."}
  ]
}

Trained answers include name, type, decision_method: "typed" and:

TypeFields
choicechoice, probabilities: [{value, probability}], confidence, confidence_method
scorescore, probabilities: [{value, label, probability}], confidence, confidence_method
predicateprobability, optional confidence diagnostics

Scores are expected zero-based ordinal indices and can be fractional. A predicate estimates whether the statement is true. Confidence is a model diagnostic, not the probability that a chosen label is correct. Models with an action head retain act_probability. Applications apply acceptance policy and validate arguments before executing an action.

A single response contains model, an answers array in question order, and aggregate token usage. For batches, replace input with inputs: [{"id": "first", "input": "..."}, {"input": "..."}]. The response uses data: [{input_index, id?, answers}] instead of top-level answers. Questions are shared; IDs and order are preserved. Limits: 128 inputs, 64 questions, 2–64 choices or levels, 1 MiB per input and 16 MiB per request. Model and executor limits can be smaller.

Shared context batches#

Supporting Antfly servers accept the explicit context_version: 1 transport extension. Send immutable context strings once and reference their zero-based indices from individual inputs:

{
  "model": "decision-model",
  "context_version": 1,
  "contexts": ["File: retry.py\nCONTEXT: def retry(): ..."],
  "inputs": [
    {"id": "line-1", "input": "TARGET: retry_count += 1", "context_index": 0},
    {"id": "line-2", "input": "TARGET: return response", "context_index": 0}
  ],
  "questions": [{"name": "relevant", "type": "predicate", "instructions": "Does TARGET implement retry enforcement? Use CONTEXT only to interpret TARGET."}]
}

Each referenced item is evaluated with exactly context + "\n" + input. Unreferenced items keep their original input. Answer IDs, indices, and question semantics are unchanged. Unknown versions, missing context arrays, invalid indices, and references without a version fail validation. Context arrays contain at most 128 nonempty UTF-8 strings; effective inputs remain limited to 1 MiB each and their combined expanded size is limited to 16 MiB before model admission. Model limits may reject smaller requests. Shared transport reduces request bytes; it does not imply shared model computation or lower billed token usage.

antfly-grep --decision-context shared-v1 opts into this format. Its default legacy transport remains compatible with existing endpoints.

Embedding similarity and multi-choice#

EmbeddingGemma 2 advertises embedding_similarity and supports choice and multi_choice. Answers include decision_method: "embedding_similarity", similarity_metric: "cosine" and similarities: [{value, similarity}]. Raw cosine values range from -1 to 1; these answers have no probability or confidence fields. Cosine is fixed because thresholds, margins and calibration depend on its units. Another metric would require a separately qualified contract.

Request-wide embedding_options supplies only geometry defaults: task_type (CLUSTERING or CLASSIFICATION) and dimensions (768, 512, 256 or 128). Each question has its own acceptance embedding_options: min_similarity, min_margin or calibration_id. The embedding endpoint also accepts task_type; decision routing restricts the supported prompt profiles.

calibration_id is a public identifier for a qualified artifact at <model-dir>/calibrations/<id>.json. The runtime verifies model identity, prompt profile, dimensions, renderer, mode and that question's prototypes. Calibration is mutually exclusive with manual thresholds. See the model guide for fitting; thresholds must be evaluated on application data.

For single choice, min_margin is the top-two cosine gap. Ties abstain: choice is null and status is abstained. Multi-choice requires similarity_thresholds (one cosine threshold or a complete value-to-threshold map), unless calibration supplies them. Its min_margin is the nearest absolute distance to any label's threshold. The answer includes effective thresholds and choices: selected means a nonempty accepted set, empty is a valid empty set, and abstained means the boundary margin failed. These states are distinct.

HTTP, CLI and bindings#

curl --fail-with-body http://127.0.0.1:8080/ai/v1/decisions \
  -H 'Content-Type: application/json' --data-binary @decision.json
antfly inference decisions ./models --request decision.json --backend metal

Repeat --request to run several files against one loaded model; the CLI prints one JSON response per line. --repeat N re-runs the files N times, interleaved, and prints each file's median and minimum latency on stderr.

The inference listener serves /decisions directly (default port 8090). Existing binding methods consume the same JSON: C antfly_inference_decide_json, Rust Inference::decide, Go Inference.Decide, Python Inference.decide, TypeScript Inference.decide / decideRaw. HTTP methods remain Python AntflyClient.decide and TypeScript InferenceClient.decide.

response = client.decide(request)
route = next(answer for answer in response.answers if answer.name == "route")
print(route.choice)
const response = await client.decide(request);
const route = response.answers?.find(answer => answer.name === "route");
if (route?.type === "choice") console.log(route.choice);

Reuse handles for cached execution. Bindings preserve error bodies and retry metadata. Free C output buffers on success and failure and close handles; see C API ownership.

SQL providers#

deciders:
  triage:
    provider: antfly
    model: decision-model
    max_rows: 10000
    max_input_tokens: 1000000
    batch_size: 32

SQL references the configured name. Omitting url uses linked inference when available. For HTTP use http://127.0.0.1:8090, or the standalone base including /ai/v1; the provider appends /decisions. OpenAI uses /decisions; Jev's private adapter retains its upstream wire format. Responses normalize to the same public answer arrays.

SELECT ai_decide('Please refund my charge.',
  '[{"name":"refund","type":"predicate","instructions":"The input requests a refund."}]'::jsonb,
  'triage');
SELECT ai_choice('Duplicate charge', 'Which team?',
  '{"billing":"Payments and refunds","support":"Product troubleshooting"}'::jsonb, 'triage');
SELECT ai_score('Respond before tomorrow.', 'How urgent?',
  '["Routine","Soon","Immediate"]'::jsonb, 'triage');
SELECT ai_probability('Please refund my charge.', 'The input requests a refund.', 'triage');

ai_decide accepts named question arrays and returns the full response. Convenience functions return a choice, expected score or true probability. Embedding deciders use decision_method: embedding_similarity; trusted config can supply geometry and acceptance defaults. Serialization moves acceptance onto each question; question policies override those defaults. ai_choice returns SQL NULL on abstention, and ai_decide supports multi-choice. Scores and predicates require trained deciders. A database handle alone does not configure SQL providers.

Validate quality, thresholds and resource limits on your workload before deployment. Preparation and limits are in GLiDE, Laya and EmbeddingGemma 2.

Decider cascades#

A cascade is a named decider that asks a cheap decider first and sends only uncertain inputs to a stronger one. Configure it under deciders with decision_method: cascade; each stage is an inline typed or generated decider.

deciders:
  same-entity:
    decision_method: cascade
    cascade:
      stages:
        - name: fast
          decider: {provider: jev, model: jev-latest}
        - name: strong
          decider:
            decision_method: generated
            probabilities: verdict_only
            generator: {provider: anthropic, model: your-strong-model}
          when:
            question: same
            choice: same
            probability_between: [0.3, 0.9]
      budget: {max_model_calls: 2, deadline_ms: 10000}

Select it like any other decider: "model": "deciders/same-entity" on /ai/v1/decisions, the decider name in SQL, or the inline configuration in a resolver adjudicate policy or decision enrichment.

  • The first stage answers every input. A later stage runs only for inputs whose previous answer falls inside its inclusive probability_between band, and those inputs share one call. question limits the band to one question, and choice measures that value's probability instead of the selected value's.
  • The final answer is the last answering stage's. Usage is summed over the stages that ran for the input. Shared generated-batch usage is apportioned evenly across each call's inputs, with integer remainders assigned in input order, and marked usage.estimated: true. Request totals remain measured. If reported usage exceeds a stage's token limit, every completed call is charged before the stage fails and future windows stop. keep_previous retains those paid costs in the failed-stage trace and final usage; inputs that were not submitted incur no model usage.
  • Only choice and predicate questions are accepted, because bands read probabilities. Embedding-similarity deciders and nested cascades cannot be stages, and a band cannot follow a verdict_only generated stage. This also applies to dynamically returned proxy answers: a verdict's 0/1 assertion cannot drive a confidence band. An unmeasurable result follows its producing stage's failure policy, retaining paid usage and completed siblings. A valid veto stops before any later band's evidence is inspected.
  • Escalation applies to successful answers. It is not failover: a stage error ends the input unless that stage sets on_failure: keep_previous, which keeps the previous answer and records the error. The first stage defaults to fail; explicitly setting on_failure: fail is equivalent to omission, while keep_previous requires a preceding stage. A later stage's operational failure does not discard confident siblings that already completed an earlier stage.
  • budget.max_model_calls (at most 8, at least the stage count) bounds stage invocations; budget.deadline_ms narrows the caller's deadline.
  • Each stage keeps its own rate limits, row and token budgets, and media limits across batches in the same statement. Set row, token and batch limits on the stage deciders. The cascade admits images and message evidence only when every stage does. Question and choice selectors are validated against the request before any stage runs. Native model discovery runs only when a stage is reached, under its deadline and failure policy.

Set require_agreement: true on a later stage to require its vote to match that of the preceding stage before accepting the result:

- name: strong
  decider:
    decision_method: generated
    generator: {provider: openai, model: your-strong-model}
  when: {question: same, choice: same, probability_between: [0.3, 1.0]}
  require_agreement: true

This band sends every positive candidate, including confident positives, for confirmation. A narrower uncertainty band leaves confident earlier answers unchecked. when.choice selects the probability used for escalation; agreement compares selected choice strings. Predicate votes must be strictly on the same side of 0.5; a tie at 0.5 does not agree. when.question scopes the comparison to that question; without it, all questions must agree.

Disagreement stops that input immediately. Later stages cannot override the veto, and other inputs continue independently. Raw final answers and accumulated usage remain available through ai_decide and the Decisions API; SQL ai_choice, ai_score, and ai_probability return NULL. Predicate convenience clients return DecisionAbstained. Resolver adjudication remains REVIEW even with on_failure: mint, because disagreement is an abstention rather than an operational failure.

Guarded stage traces and the aggregate trace contain agreement_status: agreed, disagreed, or unverified. The derived agreed boolean appears for completed comparisons, and sorted disagreed_questions appears only for disagreement. A guarded stage failing with keep_previous retains the earlier answer but reports unverified and abstains. No agreement fields appear when neither a local nor an upstream check ran. A successful check remains recorded if a later unguarded stage runs; guard every later stage whose answer must also be confirmed.

An upstream cascade's disagreed or unverified result also stops an outer cascade, including at its first or an unguarded stage. Stage receipts preserve a bounded upstream_agreement summary with its reported status, disagreement question names, and available cascade identity. They do not recursively embed upstream traces or claim to verify the upstream definition. A local required comparison is unverified when its upstream abstains; the aggregate retains the upstream veto. Resolver policies honor this abstention even when they use a plain typed proxy and on_failure: mint. When an upstream response supplies a full trace, its bounded stage structure, aggregate status, revision metadata, and final answer must be coherent. Metadata-only legacy summaries remain supported; upstream policy definitions are not reconstructed locally. A performed local comparison records sorted agreement_questions on its stage. Consumers recompute that comparison from the preceding and current answers, including strict predicate tie handling. Every successful stage also passes a shared, bounded vote validator before its selected answer can authorize agreement: typed choices must maximize normalized finite distributions, generated choices must match their raw sampled answer, and categorical verdicts must carry consistent 0/1 evidence. Validation does not rebuild producer plans per result. Older full traces without scope remain usable when the answer evidence proves agreement; ambiguous historical scoped agreement abstains as unverified. Reported disagreement must identify actual differing votes. A flat dependency projection cannot coexist with a full trace and hide its generated revisions or definition identity. Projections remain on stage receipts and are not copied onto the public response root.

For provider: antfly, the default typed method selects the Decisions transport. A remote named decider may return typed or generated answers, including a sampled choice that differs from its distribution's argmax. Proxy consumers validate each answer's reported method and probability evidence before honoring agreement. Generated token probabilities and categorical verdicts keep their original meaning; embedding-similarity responses still require explicit configuration. A stage receipt records its configured execution method; each answer retains the method that actually produced it.

Generated choice votes retain the sampled structured answer, which need not be the highest-probability alternative. Resolver thresholds apply to that selected vote; probability mass on another choice cannot accept or reject the pair.

Disabled guards are omitted from forwarded and persisted configuration, so unguarded cascades remain readable at epoch 30. Enabled guards require epoch 31. SDK constructors omit the optional guard unless explicitly configured.

Agreement records matching votes; it does not establish model independence or calibrated confidence. Use suitable independent evidence and evaluate the resulting policy on representative data.

Responses carry a cascade trace per input (top-level for single input, on each batch row otherwise):

{"cascade": {"decider": "same-entity", "definition_sha256": "…", "answered_by": "strong",
  "stages": [
    {"name": "fast", "decision_method": "typed", "model": "jev-latest", "answers": [...], "usage": {...}, "escalated": true},
    {"name": "strong", "decision_method": "generated", "model": "your-strong-model", "answers": [...], "usage": {...}, "escalated": false}]}}

definition_sha256 identifies the cascade definition without credentials. Anthropic stages require probabilities: verdict_only, since they do not provide token logprobs. Such stages can appear only last. Resolver probability thresholds still apply to the probabilistic stages; a terminal verdict-only stage must choose categorically, and its reported 0/1 distribution is not a calibrated confidence estimate. Stage traces also retain a flat upstream_revisions manifest, bounded to 32 entries and eight stage path levels. It records every successful generated model revision and the upstream cascade definitions, including dependencies before a typed final answer. A cascade definition remains a versioned dependency even when every stage is typed or no provider reports a model revision. New projections use upstream_revisions_version: 2 with an explicit upstream_revisions_complete boolean. The node array retains actual model and cascade dependencies; false carries uncertainty without consuming any of its 32 slots. Version-1 projections remain readable, with legacy unverified entries folded into uncertainty. Only a complete empty projection establishes that no versioned dependencies were reported. Older projections may have dropped definition-only subtrees even while retaining other model nodes. Their old hashes (including the preceding synthetic-marker format) are recognized for migration, but cannot establish completeness or authorize identity verdict reuse by themselves. A validated historical veto can still retract MATCH to REVIEW offline at the full 32-dependency limit. Unavailable verification preserves the committed artifact; explicit configuration/source invalidation permits fresh inference. Resolver replay requires an authoritative verifier for composite routes and preserves stored receipts when that verification is unavailable. Stored successful responses are reclassified under the current policy without model inference; an upstream veto retracts a historical accepted verdict to REVIEW and restores the mention's own endpoint. Ordinary low-confidence abstentions still verify model revisions. Veto receipts also refresh when authoritative verification succeeds; an unavailable verifier can retain the recorded veto without another inference call.

Resolution artifacts carry one optional adjudication_execution_hash for their shared policy/source binding, independent of model revisions. This lets changed inputs invalidate old failure receipts before obsolete dependencies need verification. Legacy receipts retain verified migration; a recorded changed mention can also prove invalidation. Ambiguous legacy bindings remain preserved until they can be verified or explicitly invalidated by a new configuration. Revision discovery and inference share one monotonic deadline, narrowed by any cascade deadline. Each lookup receives only the remaining whole milliseconds; discovery exhaustion preserves committed evidence. Executor requests also carry the shared budget; serial and batch completion checks reject late results with Timeout before publishing any identity decisions, including under on_failure: mint. Malformed dependency metadata fails only its own batch row, without discarding healthy sibling results. A validated recorded veto can still retract an older accepted verdict to REVIEW.

Stage traces include model_revision when the provider reports an immutable revision. Resolver adjudication verifies the definition, escalation decisions, stage order, and agreement between the trace and the final answer. Probabilities from different models are not calibrated against each other, so set each band on labelled examples for the models involved and measure how often inputs escalate. The trace has what an evaluation needs: the answering stage, whether each answer escalated, and per-stage usage.