SourceObserver

Protocols:

• SourceObserver.observe(content: bytes, media_type: str) -> tuple[PageEvidence, ...]
• SourceObserver.sniff(content: bytes) -> str | None (optional)
• SourceObserverProvider.source_observers() -> tuple[SourceObserver, ...]

What

SourceObserver is the boundary for cheap structural evidence about a loaded source, taken before anything expensive runs. It answers one question: what is on each page, in the terms the routing policy reads — is there a text layer, is there an image, is the page genuinely empty.

Optionally it answers a second, earlier one. sniff recognizes a media type from the bytes themselves, which is what lets an installed distribution be believed over a client that mislabelled its upload. One observer therefore answers two questions about one source — what it is, and what it looks like inside — and both halves are the same distribution’s knowledge, which is why there is no separate port for the first.

An observer claims the media types it understands by answering with page evidence, and disclaims every other type by answering with nothing. An empty tuple means “not mine”, and dispatch moves on to the next observer. Bytes that are the right type and cannot be read are a different answer entirely: that is a raised InvalidSourceError, and it ends the search rather than falling through to an observer that would misread the same bytes.

Installing a distribution that declares an observer is what makes a media type plannable, and what makes its bytes recognizable. PDF and image observation reach the router this way too, from indx-observer-pdf, indx-observer-image, indx-observer-office, indx-observer-text, indx-observer-email and indx-observer-dxf, through the same entry-point group a third party uses, and the three magic-byte prefixes indx used to carry moved there with them. There is one dispatch, and no built-in path for an extension path to drift away from.

Why

A capability could always declare any media_types and reach the capability snapshot. Planning still refused the source, because observation was a closed dispatch over PDF, PNG, and JPEG written into the router. An installed distribution could add a reader for a format indx already observed, and could not add a format.

Routing over evidence rather than over file types is what makes the ladder general, and evidence had nowhere to come from but first-party code. This port is where it comes from now.

sniff closes the gate one layer below, and the asymmetry it removes was the argument for it: indx-source recognized three magic-byte prefixes it had written down, so a first-party type was believed over a client that lied about its Content-Type, while a third-party type was only ever recognized from that same lie or from a filename — the weakest signal in the chain. Recognizing a format is knowledge that belongs with whatever parses it, not with the package that counts bytes.

Contract

The inputs are the source bytes and its detected media type — the same pair PageReader.read and SignatureDetector.detect receive. An observer distribution therefore depends on indx-interfaces and nothing else, and the loader stays free to change how the bytes were fetched.

The output is a tuple of PageEvidence, one per page, 1-based and in order. Page evidence carries a TextLayerState, open-string signals (font, image, empty are the ones the first-party observers emit), and optional RegionEvidence.

It is deliberately not a whole preflight context. source_digest is minted by the router from what the loader read, and page_count is derived from the length of the evidence, so an observer cannot forge the identity a plan is bound to or claim a page count nothing produced.

indx-observer-office is what the whole-content rule was written for: .docx, .xlsx and .pptx all begin PK\x03\x04, and only the zip’s member list — which lives in the central directory at the end of the file — separates them. A zip that is no Office package answers None, because the declared type and the filename are still waiting below; a package that is an Office file of the wrong Office type raises, because those are bytes this observer owns and a label the caller got wrong. Every slide, section and worksheet stores characters, so the text layer is USABLE without opening anything but the part that lists the pages – except a chart sheet, a workbook tab that holds a picture rather than cells. Telling one from a worksheet is the sheet part’s root tag and nothing cheaper, so a workbook’s sheets are each opened once and a chart sheet is reported MISSING: the plan sends it past the free rung up front instead of letting the reader discover at execution that there was nothing to read.

sniff receives the whole content and returns a media type, or None for bytes it does not recognize — the same “not mine” rule stated with a different empty value. The whole content rather than a head, because a fixed head is a second length to keep in step with the longest prefix anyone ever adds, and is simply wrong for a zip container, whose member names live at the end. Reading a prefix is the cheap case and stays cheap; anything more inherits the same budget as observe.

SourceObserverProvider is optional, the way EmbeddingSpaceProvider is: a provider that reads pages and observes nothing never implements it. Unlike an embedding space, an observer names no ID and joins no capability descriptor, so it neither collides with another distribution’s nor moves the capability snapshot — installing one changes what can be planned, while the snapshot records what can be run.

Responsibilities and guarantees

• Answer with nothing for a media type the observer does not claim, so dispatch can continue.
• Declare no sniff at all when the format’s bytes do not name it. It is read with a getattr default, so declining is an answer rather than an omission, and two first-party observers decline: indx-observer-text because a .txt, a .csv and a .tsv are the same characters with different separators inside, and indx-observer-email because a message begins with whatever header the sending agent wrote first. The only signatures available there — “decodes as UTF-8”, “the first line looks like Name: value” — would claim JSON, XML, HTML and raw HTTP responses ahead of the declared type that is actually right. The declared type and the filename are then the honest signals, and mimetypes maps every one of those extensions already.
• Answer None from sniff for bytes it does not recognize, and recognize only types it also observes: a type nothing can then look at buys the caller a 422 instead of a 415.
• Raise InvalidSourceError for bytes of a claimed type that cannot be read, which stops the search.
• Stay inside preflight’s budget: cheap, local, deterministic, and free of OCR, models, rendering, and network. Reading structure is the intended cost; reading content is affordable only where structure has run out.
• Number pages from 1, matching ScopeRef.page and Block.index.
• Emit signals from the shared vocabulary in indx_interfaces.preflight, because the routing policy matches those strings. A signal the policy does not read is inert rather than wrong; making a new one meaningful is a first-party policy change with a POLICY_VERSION bump.
• Keep module scope cheap. Discovery imports every provider module while building a snapshot, so a heavy import belongs inside observe().

Three members sit outside the protocol deliberately, because a member declared on it is a member isinstance and the type checker both demand, and none of these is something an observer must have in order to work.

builtin = True is what the first-party observers set, and the registry orders installed observers ahead of them. Claiming it can only cost precedence, never win it, which is why nothing verifies it. Two installed observers claiming one type are resolved by discovery order.

media_types declares what this observer looks at, and is what the capability snapshot reports under resolvable.observable_media_types. It is advertisement and never a gate: dispatch still asks, and still reads an empty answer as “not mine”, because two sources of truth for one question is exactly what a gate would create. It is not what sniff may return, either — an observer that recognizes bytes it does not advertise is not stopped, it is only unlisted.

sniff is what an observer contributes to media-type detection. Two observers recognizing one format are resolved by the same order, first answer wins and quietly, because an observer declares no ID to collide on. A raise from sniff is logged and skipped rather than fatal, which is the opposite of a raise from observe: there it means “mine, and unreadable” and letting the next observer misread the same bytes would be worse than stopping, while here the declared type and the filename are still waiting below. Nothing verifies any of the three: a wrong builtin costs precedence, an unlisted media_types costs visibility, and a broken sniff costs one turn in a chain that continues.

All three are read with an empty default, so an observer written before any of them existed still works — it simply is not advertised, sorts as installed, and contributes nothing to detection.

Place in the system

Router.plan loads a source through an installed SourceLoader — and it is during that load that indx-source asks the same registry’s observers what the bytes are, before any declared Content-Type is trusted. It then refuses a media type no installed capability reads with a 415, and observes the source with those same observers. A type nothing observes is a 422 with code source_unreadable. All of these boundaries are installation-dependent: the same source can be refused on one install and planned on another, and a stock pip install indx recognizes nothing from bytes at all.

Observation is the second place planning reaches an installed implementation. Unlike signature_detection, which is opt-in per request, it runs for every plan — and is held to the same budget, which is why that budget is stated on both ports.

Related reading

• Protocol overview
• SourceLoader
• CapabilityProvider
• CapabilityRegistry
• Planner
• Preflight evidence models
• Capability extension architecture