LightOnOCR-3 document OCR

LightOnOCR-3-0.8B and 4B are Apache-2.0 document parsers based on Qwen3.5. Each transcribes page images into Markdown, including HTML tables and LaTeX formulas. Antfly uses the integrated safetensors checkpoint with the native Qwen vision and hybrid-attention decoder paths.

Install#

antfly inference pull lightonai/LightOnOCR-3-0.8B:safetensors@36d8636925327c671bd8e479c517a004f7566bd9

The pull installs the upstream configuration, tokenizer, chat template, image processor, and weights, plus a model_manifest.json declaring generation and reading. Runtime lookup also accepts lightonocr3-0.8b. The checkpoint contains 852,985,920 BF16 parameters, approximately 1.7 GB of weights; working memory is additional. This integration does not add a Qwen3.5 GGUF weight mapping.

The larger model uses the same image-only prompt and /read API:

antfly inference pull lightonai/LightOnOCR-3-4B:safetensors@b71010f095dc3735de7ee6b9969bff8f8f50b39a

Use lightonocr3-4b as the model name. Its 4,539,265,536 BF16 parameters occupy approximately 9.1 GB of weights. It has 32 decoder layers with hidden size 2560, 16 recurrent key heads and 32 value heads, plus a 24-layer vision encoder with hidden size 1024. The 0.8B model has 24 decoder layers with hidden size 1024 and a 12-layer vision encoder with hidden size 768. Both retain 128-channel recurrent heads, four-tap convolution, and partial M-RoPE.

Both Metal models defer paged-attention completion within the active decoder frame, avoiding a GPU wait at every full-attention layer. The rollback is TERMITE_METAL_DISABLE_ACTIVE_PAGED_SLOT_ATTENTION_DEFERRED=1.

On M4 Max, both projection paths use SIMD reductions for single-token projections, including the small recurrent gates, and fuses FFN gate/up projection with SiLU. Its wide decode and attention-head RMS normalizations also use parallel F32 reductions instead of scanning each row in one lane. Eligible multi-token projections use Metal 4 TensorOps on macOS 26.1 or newer. Unsupported shapes, alignment, or shader availability retain the existing implementation. The master rollback is TERMITE_METAL_DISABLE_LIGHTONOCR3_PROJECTIONS=1. Normalization alone can be rolled back with TERMITE_METAL_DISABLE_LIGHTONOCR3_PARALLEL_NORM=1.

F32 activations, accumulation, and recurrent state remain the default. TERMITE_METAL_LIGHTONOCR3_BF16_ACTIVATIONS=1 experiments with BF16 inputs only for eligible multi-token TensorOps projections. It preserves F32 outputs and recurrent state; it does not convert the whole runtime to BF16.

The 4B model automatically sizes its GPU cache and admission envelope from the weight artifact to cover both mapped source weights and prepared device weights. Explicit operator memory limits still take precedence. Working memory is additional to checkpoint size; the local comparison uses 36 GiB of unified memory.

Read a page#

Send a rendered PDF page or other document image to the inference service:

curl http://localhost:8080/ai/v1/read \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "lightonocr3-0.8b",
    "images": [{"url": "data:image/png;base64,..."}],
    "max_tokens": 1024
  }'

Omit prompt or pass an empty string for transcription. This model is trained with an image-only prompt; a generic instruction such as Transcribe this page is rejected by /read. The alternative "prompt": "grounding" returns the model's native labeled bounding-box markers, descriptions, and chart tables in the result's text. These markers use coordinates normalized to 0–1000; they are not converted into pixel-coordinate regions.

On Metal, and on CUDA with the opt-in LightOn batch profile, /read coalesces compatible pages from the same request and from concurrent requests into bounded cohorts. Decoder projections share GPU work; each page keeps independent KV, DeltaNet history, and image positions. Pages retire independently inside the cohort. Vision projection and prefill retain the shared runtime's execution lock. CPU readers retain serial page execution. Response execution counters report pages that actually participated in a multi-page decode step.

/read currently accepts at most 1024 output tokens per page and rejects truncated output. Use /generate with a larger budget for dense pages; its content should contain the image and either no text or the exact grounding keyword, with thinking disabled.

Resolution and validation#

The runtime reads pixel limits and patch geometry from the installed image processor, bounded by a runtime cap of 4096 merged vision tokens per image, and uses the same geometry for admission and execution. Resize page images before submitting them when optimizing latency; the upstream speed evaluation includes a 1540-pixel-long-side setting. Higher-resolution results must be measured separately.

The shared Qwen vision path supplies learned image-position embeddings and two-axis vision rotary positions. Qwen3.5's full-attention layers use three-axis M-RoPE on the first 64 channels of each 256-channel head, preserving the tail; its remaining layers use recurrent linear attention. LightOn has no DeepStack vision taps. On Metal, the qualified 128-channel, four-tap DeltaNet path keeps causal convolution, normalized Q/K, gates, recurrent matrices, and gated RMSNorm on-device. Prefill distributes each value column across a 32-lane SIMD group; single-token decoding uses a separate kernel. Buffers belong to the request, and reset invalidates their history before reuse. The numerical test covers short prefixes, grouped heads, continued decoding, reset, and zero core readbacks against the CPU reference.

Metal batches decoder commands across layers and vision commands within each block. The M4 Max also enables vectorized BF16 weight loads for the qualified 0.8B and 4B text/vision projection shapes; activations and accumulation remain F32. Other shapes retain the existing dispatch. The CPU fallback uses vectorized recurrence and activations. Measure release-build latency on deployment hardware before setting an SLO; this does not qualify CUDA or vLLM throughput.

Run the focused tests from zig/ with the repository's Zig 0.17 toolchain:

zig build inference-test -Dtest-filter=LightOnOCR3

An opt-in smoke test uses a local installed checkpoint and an invoice image:

ANTFLY_TEST_LIGHTONOCR3_MODEL_DIR=/absolute/path/to/model \
  zig build inference-test -Dtest-filter=LightOnOCR3

For a shared-machine 4B benchmark, the opt-in test also accepts ANTFLY_TEST_LIGHTONOCR3_MEMORY_LIMIT_GIB=13 and ANTFLY_TEST_LIGHTONOCR3_SHARED_LIMIT_GIB=24. These set independent process and shared-pool limits using normal operator memory policy; live availability, process usage, and the emergency reserve remain enforced. The shared limit defaults to the process limit when omitted. These test-only options do not change production defaults.

The 0.8B configuration fixture in src/models/testdata/lightonocr3/ comes from the pinned upstream checkpoint, licensed Apache-2.0. The 4B fixtures come from its pinned checkpoint. The invoice fixture is copied from Antfly's existing native Apple OCR tests. Published quality and speed numbers are documented in LightOn's evaluation; they are not Antfly runtime measurements.

CUDA execution#

CUDA supports both original BF16 checkpoints with opt-in recurrent, attention and vision kernels. The experimental DGX Spark presets in scripts/lightonocr3/cuda_ocr_profiles.json require a CUDA build and cuDNN on the dynamic library search path. They preserve operator limits and keep F32 activations, outputs and recurrent state by default. Native attention KV is F16.

python3 zig/pkg/inference/scripts/lightonocr3/cuda_ocr_profile.py lighton08 --print-env
python3 zig/pkg/inference/scripts/lightonocr3/cuda_ocr_profile.py lighton4b -- ./antfly standalone

The experimental compact-KV variants allocate attention storage only for full attention layers. Each recurrent layer retains its own state. Up to 128 physical rows use recurrent and attention packets of at most 64. Client concurrency, admitted live pages and physical GPU rows remain separate limits.

Partial BF16 prefill options retain F32 residuals and recurrent state and do not match all of vLLM's BF16 activation boundaries. They remain experimental. CUDA LightOn throughput is below the installed vLLM reference; the Florence canonical GEMM option remains opt-in while an older rare output mismatch is investigated.

Performance and verification#

Historical shared-machine results are summarized below. They qualify the named builds and three synthetic fixtures, not broad document accuracy or an SLO.

The Metal comparisons are paired. CUDA results are separate capacity screens; LightOn CUDA parity remains outstanding. Metal Florence uses Q4_K weights and an F16 encoder versus an FP16 MLX proxy. Encoded-image, decoded-image and HTTP frontends have different overheads. These are not fully precision-matched runs.

The evidence index contains the versioned archive, hashes, limitations and reproducible validation entry points. Benchmark scripts, fixture definitions and independent kernel checks remain in the source tree; chronological logs and rejected experiments are archived separately.