SourceLoader

The port that fetches bytes behind a URI scheme.

Protocols:

SourceLoader.schemes -> tuple[str, ...]
SourceLoader.fetch(uri: str) -> Iterator[bytes]
SourceLoaderProvider.source_loaders() -> tuple[SourceLoader, ...]

What

SourceLoader is the boundary for getting the bytes behind a URI. A loader declares the schemes it resolves and yields the content of one, in chunks. It does not decide what the bytes are, how large they may be, or what identity they carry – indx-source owns all three, and owns them the same way regardless of who fetched.

Installing a distribution that declares a loader is what teaches an installation a URI scheme. file:, data:, http:, https: and s3: reach indx this way too, from indx-loader-file, indx-loader-http and indx-loader-s3, through the same entry-point group a third party uses. There is one dispatch, and no built-in path for an extension path to drift away from.

The consequence is worth stating plainly: a stock pip install indx resolves no URI at all. That is the same shape observation already had, and it is what “installing a distribution is what makes a scheme resolvable” has to mean if it is true for https: as well as for s3:. An inline source needs no loader – bytes handed over directly were never fetched.

Why

indx_source.uri held a hardcoded SUPPORTED_SCHEMES frozenset, and anything outside it was a 415. An installed distribution could bring a capability, an embedding space and an observer, and could not bring the one thing standing between a caller’s s3:// bucket and every one of them.

The error contract was already built for this: UnsupportedMediaError documents unsupported_source as a distinct code so that “we do not speak s3://” stays separable from “no capability parses DXF”. What was missing was the seam behind it.

Contract

The input is a URI string. The output is an iterator of bytes.

Chunks, not a body. indx-source counts as it consumes and refuses at the first chunk that crosses the advertised max_input_bytes, so an oversized source is rejected part-way rather than after it is buffered whole. That holds only as far as the chunks allow: a loader yielding an entire response as one chunk has already allocated it before anything can object, which makes bounded chunks an obligation of this port rather than a guarantee of it.

The digest is not a loader’s to supply. It is minted from the same pass that counts, by the package that also detects the media type. A plan is bound to that digest, and a loader that supplied its own could name an identity for content it did not return – the same reasoning that keeps source_digest out of an observer’s hands. LoadedSource stays private to indx-source for the same reason.

Declared schemes are load-bearing. This is the one place this port differs from SourceObserver. An observer discovers what it claims by being asked and answering with nothing; a loader is selected by schemes before anything is called, and the same declaration is what a 415 enumerates and what the capability snapshot reports. Lowercase, no trailing colon.

Loaders are not discovered where they are used. indx-source sits below both stages and cannot reach the registry – a second discovery could disagree with the snapshot a plan was decided against. All three load() call sites already hold a snapshot, so they hand the registry’s loaders down with the limit they take from it.

Credentials come from the environment. INDX_* variables, never a descriptor and never the URI’s caller, which is the rule the hosted vision and embedding lanes already follow.

SourceLoaderProvider is optional, the way EmbeddingSpaceProvider and SourceObserverProvider are: a provider that reads pages and fetches nothing never implements it. A provider may equally fetch and read nothing – indx-loader-file, indx-loader-http and indx-loader-s3 all declare descriptors() -> ().

Responsibilities and guarantees

Declare every scheme this loader resolves, and resolve exactly those.
Yield in bounded chunks, so the deployment’s input ceiling stays enforceable.
Raise for a source that cannot be fetched. Anything that is not already an IndxError is wrapped as a 422 source_unreadable, so a bug in an installed distribution is legible rather than a 500.
Refuse a destination this deployment should not reach with 422 source_forbidden, and say in the message which setting would allow it.
Keep module scope cheap. Discovery imports every provider module while building a snapshot, so a client library belongs inside fetch().
Take credentials from INDX_* environment variables.

The registry orders installed loaders ahead of the ones indx ships, using the same optional builtin = True attribute the observers use. It is not part of the protocol and is read with a False default, so a third party never declares it – claiming it can only cost precedence, never win it. Two installed loaders claiming one scheme are resolved by discovery order: a loader declares no ID, so there is nothing to collide on the way duplicate capability IDs do, and that stays first-wins rather than fatal.

What a loader is trusted with

An installed loader is code an operator chose to install, and it makes outbound requests on the request path. indx bounds its bytes and mints its digest; it does not police where it connects, and cannot.

What indx does guard is its own fetching. indx-loader-http refuses loopback, private, link-local, reserved and multicast destinations unless INDX_LOADER_HTTP_ALLOW_PRIVATE_HOSTS says otherwise, re-checks every redirect hop, and refuses a redirect into a scheme it does not speak. indx-loader-file reads any path by default and only the configured roots once INDX_LOADER_FILE_ROOTS is set. indx-loader-s3 fetches from any bucket its credentials reach until INDX_LOADER_S3_BUCKETS names the ones it may, and refuses the rest. In every case the log line carries what was actually resolved and names the variable that would permit it, while the response carries neither – telling an untrusted caller which internal address a hostname resolves to is the answer they were probing for.

Neither closes DNS rebinding: the name is resolved to be checked and resolved again to connect. The ceiling is marked in check_destination, and the upgrade path is connecting to the vetted address with an explicit Host header.

Place in the system

Router.plan and DocumentExecutor.encode both load before they do anything else, so a scheme nothing resolves is the first refusal a source can meet – ahead of the 415 for a media type no capability reads, and ahead of the 422 for one no observer can look at. All three now depend on what is installed.

The snapshot advertises the resolvable set on CapabilitySnapshot.resolvable, excluded from its content hash for the same reason limits is: gaining a scheme widens what can be planned next without changing what any outstanding plan selected, and hashing it would reject every plan in flight with a 409 for an unrelated reason. POLICY_VERSION does not move either – nothing here is a routing decision.