Benchmarks

What the repository-owned document-routing benchmark currently measures.

The benchmark runs indx against seventeen real documents with pinned source digests, constraints, capability snapshots, policy versions, and hand-written acceptable routes. The committed report covers Japanese securities filings and public-sector documents, a US financial report, drawings, scans, handwriting, encryption, mixed pages, a large file, and a spreadsheet whose chart sheets no installed capability reads.

The advantage in three figures

Three numbers that depend on no label, so nothing below the note has to be true for them to be:

0.05 sseconds per page

$0.00003per page routed, against $0.02 through the vision model

98.8%of pages processed on CPU only

Against that, 15 pages no installed capability reads. The plan hands those to a person, and the economics table prices them at $2 a page: $28.00. That, on its own, is why the corpus-wide comparison comes out behind an all-vision baseline, and it is kept separate here because a page nobody could read is not a page that was cheap to read.

0.05 s — seconds per page

Measured wall time over 17 cases and 1,361 pages. It is one sequential run, not concurrent throughput, and most pages carry a native text layer: a scan is orders of magnitude slower than this average.

$0.00003 — per page routed, against $0.02 through the vision model

One side is the router's economics table and the other is the same table with every page through the vision model. Exactly one page in the run was billed, and the table overpriced that page eight-fold. It is not a bill.

98.8% — of pages processed on CPU only

0 of 1,361 pages used a GPU. This is as much a property of this install's extras as of routing: the run carries the `ocr` extra, which is why scans are read on CPU at all.

The seven axes

The axes the sprint frame says the advantage is expressed on, each with what this benchmark can say about it today. No competitor column appears until a like-for-like number exists.

Axis	State	Where it is
Speed	Measured	Seconds per page above, over every executed page.
Resource requirements	Measured	The CPU-only share above, and zero GPU pages.
Cost	Measured	Routed against all-vision above; both sides priced from the same table.
Parsing accuracy	Measured, labels under review	Text recall and field accuracy, over 14 labelled scopes.
Routing quality	Measured, labels under review	Route accuracy, read acceptability and fallback accuracy, over 28 labelled scopes.
Complex documents and diagrams	Measured, labels under review	One process-chart case and two drawing cases; signature accuracy is 1.0 over a corpus that small.
Few-shot performance	Not yet	Needs a recipe format and an example store; `summary.kr1.few_shot_setup_days` is `null` rather than absent, so the gap is visible in the block.

17real documents

1,361pages executed

0.05seconds per page

96.4%route accuracy

92.9%read acceptability

75%fallback accuracy

98.8%CPU only

1.1%unreadable rate

$28.039routed estimate

$27.22all-VLM estimate

5data types scored

70%entity recall, best lane

95.2%language accuracy

99.9%CPU only, standard workload

What the numbers mean

Route accuracy asks whether the planned capability is one of the acceptable routes labelled for that scope. The current miss is the handwriting-labelled case: cheap preflight sees an image but does not yet distinguish handwriting from printed text.
Read acceptability is separate from exact route accuracy. A different eligible reader may still be acceptable.
Fallback accuracy scores the plan’s declared fallback against the labelled expectation for each scope. It is the weakest number in the report, and it is shown anyway.
Actual cost sums only what capabilities report about themselves. A route nobody billed for reports zero. It is not a bill. One page in the run is billed: the vision-model anchor below.
The executed baseline is the map scan admitted a second time under a quality floor the OCR rung cannot meet, so the plan selects generic-vlm (gpt-5.4-mini) for what the document is. It reads the page in 3.2 s for $0.0024 where the routing table prices it at 3 s and $0.02: the one anchor in the cost comparison that is billed rather than priced, and it says the table overprices this model on this page eight-fold while getting its latency right.
CPU-only rate records pages processed without GPU use. It does not mean every page was read successfully. Beside it, the standard workload is every executed page some capability read: the 15 pages sent to a human carry no device and are reported as their own bucket, and the CPU-only share of the remaining 1,345 pages is 100%.
Text recall and field accuracy are measured against ground truth: fourteen labelled scopes carry the strings and named values the document says, and the run reads 20 of 22 strings and 20 of 20 fields. One miss is a character PP-OCR gets wrong on a map scan; the other is a native page whose font’s ToUnicode map lies, so the reader declares success over the wrong word. Neither is visible to the engine’s own confidence, which is the point. This is the quality side of the cost comparison, over a subset that is small on purpose.
Entity recall per lane is measured against 30 named entities a person reads on six labelled scopes, each installed extractor lane run alone over the same plan: ner-llm (a hosted gpt-5.4-nano) reads 70% for $0.12 over the six cases, patterns 50%, onnx 27%, vibrato 3%. Recall and never precision, because a label names some of a page’s entities and not all of them. Six of the hosted lane’s misses are phone and registration numbers its default label list does not ask for and the pattern lane reads for free. Read entities_missing per lane before generalising any of it; vibrato’s misses are mostly a granularity it cannot express.
Language accuracy is the share of labelled scopes whose detected first language is one the label names, measured only when a detector is installed: 20 of 21 scopes with the lang extra. The miss is the vision model’s transcription of a kanji-only map, which the detector reads as Chinese; none of the scopes is short enough to test the case the deferred declared-language detector waits on.
Unreadable rate stays a reported metric so a page nothing reads is counted, never dropped. The committed run carries the ocr extra, so every scanned page was read; the 1% that is unreadable is the FDIC workbook’s 14 chart sheets, which hold pictures rather than cells.
The spreadsheet case is read rather than refused, which it was until office-extraction shipped. Its 14 chart sheets still are not: the observer reports them with no text layer and the plan sends them to a human up front, which is what the estimate above is carrying.

The KR1 block, and what each number is still missing

summary.kr1 in the committed report carries the five objective numbers as one block, each restating a figure already in summary rather than recomputing it, so the block cannot drift from what it summarizes. Four have a measured value; one has no implementation at all. Every one carries a caveat the report itself does not state:

Routing accuracy compares the planned capability ID against hand-written acceptable_capability_ids. It cannot check the stronger claim — “the lowest-cost candidate meeting the quality threshold” — because actuals.quality is a capability’s self-report, measured against nothing.
CPU-only rate is about this install’s extras rather than about routing: the committed run carries --extra ocr, so every scanned page is read on CPU. It is page-weighted, so a 336-page filing outweighs a 9-page disclosure 37 times over, and it is below 1.0 only because the FDIC workbook’s chart sheets reach manual-review, which carries no device.
TCO reduction is an estimate against an estimate: both sides are priced from indx_router/economics.py, and actual_cost_usd is the $0.0024 one vision page and the hosted extractor’s runs billed. It is emitted with its sign rather than clamped at zero, because the routed side genuinely exceeds the every-page-VLM baseline when chart sheets are planned to a human at $2 and a day per page. A figure floored at zero would hide the one thing this number is for.
Priority data-type coverage counts a type only when this install observes it, some capability reads it, and one admitted case scores it. Eleven types have observed and read for several slices while the figure read 2, which was the rule working rather than a regression.
Few-shot setup for a new similar format has no recipe, no example store and no past-case lookup. It is null rather than absent, because a KR with no implementation should be visible in the block, not missing from it.

Known gaps

Recorded rather than papered over. Each is a measurement this benchmark owes, not a defect in the code it measures.

The run carries --extra vlm and a configured vision model, and one baseline page is executed. Both were owed here and both are in the committed report: the oversize scan descends past the OCR rung and then past the vision rung, each refusing its second page for what it is, and the map scan’s second admission is read by the vision model and billed. The pinned snapshot is now that of an install with a vision model configured, so a run without one drifts every case rather than scoring a different inventory under the same pin.
Widen the labelled subset. case.schema.json carries expected_text and expected_fields since schema 1.3 and expected_entities and expected_languages since 1.4; fourteen scopes carry text and fields, six carry entities, twenty-one carry a language. None of the six unscored media types has a label, no label yet catches a reader that declared success over wrong output on a native page, and the match is exact after a fold, so a one-character error misses the whole string. The Office reader comparison still has nothing to compare against; the entity extraction lanes now do, on 30 entities, which is a number to widen rather than to trust.
The three cases the methodology recorded as gaps are admitted, synthetically. A two-page invoice with a positive signature nomination, a native page whose text layer lies so the label catches the reader, and a scan whose second page is too large to rasterize so the OCR rung refuses it and the ladder descends for a reason the page gave. Each is generated by the repository’s fixture script and marked redistributable; a public document in any of the three shapes would replace it.
Six media types are readable and unscored — .docx, text/plain, text/tab-separated-values, image/tiff, image/png and message/rfc822. For four of them the finding is that the document does not appear to exist publicly rather than that nobody looked: ten Japanese government .docx files all declare exactly one w:sectPr where the page rule needs two, both public text/plain resources carry zero form feeds, and the e-Gov catalogue holds no TSV and no TIFF at all. A 国土地理院 map tile would have covered image/png and was rejected because a tile is re-rendered upstream without notice, so its digest is not a pin. Coverage keys on each case’s own media type, so one image document cannot cover both image/jpeg and image/png.

Run it

just bench::fetch  # download sources and verify every digest
just bench::run    # plan, execute, score, and rewrite benchmarks/report.json

A missing cached source is reported as unavailable and skipped rather than treated as a routing result. A snapshot or policy mismatch is reported as drifted and excluded from scoring.