Chunker

The port that decides what a chunk is.

Protocols:

Chunker.chunk(content: bytes, media_type: str, outputs: tuple[PageOutput, ...]) -> tuple[PageChunks, ...]
ChunkerProvider.chunkers() -> tuple[Chunker, ...]

What

Chunker is the boundary for deciding what a chunk is — the embeddable unit a retriever indexes. It is asked once per encode, after every reader has produced its pages and before anything is embedded, with the same bytes every reader saw: a chunker that knows the format can reopen them for the structure a flat string lost, and one that only needs text reads it off the page outputs.

A chunker claims pages by answering with PageChunks for them, and disclaims a document by answering with nothing. Answers are merged per page, first in chunker order wins, so a chunker is free to be honest about partial knowledge — the PDF chunker has no layout for a scanned page OCR read, and saying nothing about it hands that page to the next chunker in line, indx-chunker-lines, which cuts the lines the OCR reader located into chunks with rectangles.

Four chunker distributions ship with indx through the same entry-point group a third party uses. indx-chunker-pdf splits PDF pages on the text-run layout PDFium reports, merged into visual lines, each chunk carrying a normalized bbox. indx-chunker-pptx cuts a PowerPoint slide into its shapes: one chunk per text-bearing shape or table and one image chunk per raster picture, each at the box the file states, with group transforms applied and placeholder frames inherited from the layout and master. indx-chunker-page is the floor: it claims every readable page of any media type with its whole text as one chunk, which is exactly what build hardcoded before this port existed. An install without it — a stock pip install indx — chunks nothing, the same honest cliff as observing nothing and resolving no URI. indx-chunker-lines cuts the lines a reader located (PageOutput.lines, which generic-ocr fills from the recognizer’s boxes and which never reaches the wire) into one chunk per line at its rectangle, for any media type and without reopening the bytes; a page whose reader stated no lines is disclaimed.

Why

blocks.build minted one chunk per page and marked it as a deferral: split on real structure once something downstream has an opinion about size. The recipe store and the export index are that downstream, and chunk_texts is the only path to a vector, which made chunking the thing that silently decided retrieval quality — measured concretely by the fixture invoice, whose generator draws every glyph as its own text run: boundary knowledge existed in the bytes and had nowhere to be said.

A capability-supplied channel on PageOutput was the other candidate and was rejected for uniformity: a boundary is an opinion about retrieval, not about reading, and a port applied after all reading lets one implementation draw boundaries for pages that different capabilities read.

Contract

The inputs are the loaded source bytes, the detected media type — the same pair PageReader.read and SourceObserver.observe receive — plus every PageOutput the run produced. A chunker distribution therefore depends on indx-interfaces and nothing else.

The output is a tuple of PageChunks, each naming a 1-based page and carrying ordered ChunkPieces. A piece is non-empty text or non-empty image bytes — exactly one, see below — and an optional bbox, normalized the way Block.bbox is: one validate_bbox serves both, so a piece that validates becomes a block that validates. Like PageOutput, it is deliberately not a Block: the executor mints every page:N/chunk:M from piece positions, so ID uniqueness stays with the one thing that can see the whole document.

A piece without a bbox carries a bbox_reason instead, always: no_geometry when the chunker knows no positions, hidden when the source marks the shape hidden, unresolved when the format has positions and this shape’s could not be resolved. The model refuses a piece with neither and a piece with both, so the reason is the chunker’s to state and never a default the executor fills in.

An absent page means “not mine” and the next chunker is asked. Raising means “mine, and broken” — and unlike an observer’s raise, it is a verdict about the chunker rather than the source, so the executor logs it and moves to the next chunker instead of failing an encode the run already paid to read. Chunk text is the chunker’s to write and need not reconstruct the page text: document and page blocks keep the reader’s text either way.

ChunkerProvider is optional, the way SourceObserverProvider is: a provider that reads pages and never draws a boundary never implements it. A chunker names no ID, joins no descriptor, and is not advertised on the capability snapshot at all — what it changes is how execution output is cut, which no outstanding plan recorded and no routing decision consults. Installing one moves nothing a plan is bound to.

Responsibilities and guarantees

Answer with nothing for a document it has no boundary for, so merging can continue; never invent pieces for a page it did not understand.
Never raise for a source it cannot open — the readers already gave those bytes their verdict, and a chunker has no standing to override it. Raising is reserved for the chunker’s own defects, and costs it the document, not the caller the encode.
Order pieces in reading order; the executor’s index and chunk IDs are derived from that order.
A page without text is chunked as a picture or not at all. The executor enforces it — text claimed for an unreadable page is dropped from the answer, an image is kept — because “a chunk of nothing is not addressable content” must not depend on every chunker remembering it.
Keep module scope cheap. Discovery imports every provider module while building a snapshot, so a heavy import belongs inside chunk().

Two attributes sit outside the protocol deliberately, read with a False default, because a member declared on it is a member isinstance and the type checker both demand. builtin = True is what the shipped chunkers set, and the registry orders installed chunkers ahead of them — claiming it can only cost precedence, so nothing verifies it. fallback = True is what indx-chunker-page alone sets, and sorts after everything: an answer for every page must be the last answer asked for, or nothing more specific would ever get one.

Place in the system

DocumentExecutor.encode computes the chunk map once — only when the request asks for chunk granularity, since a chunker may reopen the document — and hands the same map to the embedding path and to blocks.build, so the chunk IDs vectors attach to and the chunk blocks that exist are one enumeration by construction. How many chunks a page yields is therefore installation-dependent, the way resolvable schemes and observable media types already are; one chunk per page is the floor wherever indx-chunker-page is installed.

A chunk that is a picture

A ChunkPiece carries text or image and never both or neither. The image half exists for one case: a page no capability could read. Such a page has no text for any chunker to cut, so it used to leave a page block carrying its reason and nothing retrievable at all — chunk_map dropped every page without text, and a chunk is the only thing document embedding ever vectorizes. indx-chunker-pdf renders it instead, at INDX_CHUNK_DPI (default 120), and an image embedder puts it in the same space as everything else.

Which pages qualify is deliberately narrow. A page that read as an empty string was read — it is blank — and rendering it would spend a rasterization to embed white; only a page a reader reported failed or unreadable is rendered. And a chunker may claim such a page only with an image: a page’s text is a reader’s verdict, reached down the plan’s ladder and past output validation with a capability ID and a trace event attached, so text asserted by a chunker for a page the ladder gave up on would launder an unvalidated read into the result. Pixels claim nothing — they are the page — so they are admitted where invented text is dropped. A page that will not rasterize keeps no chunk and fails no encode.

This is the one part of chunking that moved POLICY_VERSION, and only indirectly: rendering made DOCUMENT_EMBEDDING_MODALITIES gain image, so a space whose only document lane is an image went from an unsatisfied plan to a routable one. Nothing else here is a routing decision.