Source loader

Add a loader that makes a new URI scheme resolvable at all.

A URI scheme no loader resolves is a 415 before anything else can happen. A SourceLoader declares the schemes it resolves and yields the bytes behind one, in chunks. Installing a distribution that declares one is what teaches an installation a scheme. file:, data:, http:, https: and s3: reach indx this way too, from indx-loader-file, indx-loader-http and indx-loader-s3, through the same entry-point group your distribution uses. A stock pip install indx resolves no URI at all. Package and install it as the overview describes.

from collections.abc import Iterator
from typing import Any

from indx_interfaces import (
    CapabilityDescriptor,
    CapabilityId,
    InvalidSourceError,
    SourceLoader,
)

_CHUNK_BYTES = 64 * 1024


class GcsLoader:
    schemes = ("gs",)

    def fetch(self, uri: str) -> Iterator[bytes]:
        # The client library is imported here, not at module scope:
        # discovery imports every provider module while building a snapshot.
        import google.cloud.storage as storage

        bucket, _, key = uri.removeprefix("gs://").partition("/")
        if not bucket or not key:
            raise InvalidSourceError(
                f"{uri} does not name a bucket and a key",
                code="source_unreadable",
                param="source.uri",
            )
        blob = storage.Client().bucket(bucket).blob(key)
        with blob.open("rb") as body:
            yield from iter(lambda: body.read(_CHUNK_BYTES), b"")


class Provider:
    def descriptors(self) -> tuple[CapabilityDescriptor, ...]:
        # This distribution fetches bytes and reads nothing.
        return ()

    def create(self, capability_id: CapabilityId) -> Any:
        raise ValueError(f"this provider declares no capabilities, so not {capability_id}")

    def source_loaders(self) -> tuple[SourceLoader, ...]:
        return (GcsLoader(),)

Unlike an observer’s media_types, schemes is load-bearing: it selects the loader before anything is called, it is what a 415 enumerates, and it is what the snapshot advertises under resolvable—lowercase, no trailing colon. Installed loaders are ordered ahead of the ones indx ships, so you can take over a scheme indx already resolves.

A loader only fetches. The digest a plan is bound to, the media type, and the input ceiling stay with indx-source, which counts as it consumes—so yield bounded chunks rather than one whole body, or an oversized source is allocated before anything can refuse it. resolvable is excluded from the snapshot’s content hash: installing a loader widens what can be planned next without invalidating any plan in flight.

Checklist

Declare every scheme the loader resolves, and resolve exactly those.
Yield in bounded chunks, so the deployment’s input ceiling stays enforceable.
Raise for a source that cannot be fetched; refuse a destination this deployment should not reach with code source_forbidden, and say in the message which setting would allow it.
Take credentials from INDX_* environment variables—never from a descriptor, never from the URI’s caller.

Source loader

Checklist

Reference

Developer