Embedding space

An embedding provider implements EmbeddingSpaceProvider: it advertises every encoder as an EMBEDDER capability and declares the complete spaces those encoders form. create() returns the VectorEncoder for one of them; encode_one() stands for your code. Package and install it as the overview describes.

from collections.abc import Sequence
from typing import Any

from indx_interfaces import (
    CapabilityDescriptor,
    CapabilityId,
    CapabilityKind,
    Device,
    DistanceMetric,
    EmbedderConfig,
    EmbedderId,
    EmbedderModality,
    EmbedderRole,
    EmbeddingSpace,
    EmbeddingSpaceId,
    VectorNormalization,
    embedder_fingerprint,
)

CAPABILITY_ID = CapabilityId("acme-text-embedder")
EMBEDDER_ID = EmbedderId("acme-text-embedder")
SPACE_ID = EmbeddingSpaceId("acme-text")
DIMENSION = 384
PREPROCESSING: dict[str, Any] = {"lowercase": True}

class Encoder:
    def encode(
        self, inputs: Sequence[str | bytes]
    ) -> tuple[tuple[float, ...], ...]:
        return tuple(tuple(encode_one(value)) for value in inputs)

class Provider:
    def descriptors(self) -> tuple[CapabilityDescriptor, ...]:
        return (
            CapabilityDescriptor(
                id=CAPABILITY_ID,
                version="1",
                kind=CapabilityKind.EMBEDDER,
                devices=(Device.CPU,),
                available=True,
            ),
        )

    def embedding_spaces(self) -> tuple[EmbeddingSpace, ...]:
        revision = "model-revision-1"
        config = EmbedderConfig(
            id=EMBEDDER_ID,
            provider="acme",
            model="acme-text-model",
            revision=revision,
            roles=(EmbedderRole.DOCUMENT, EmbedderRole.QUERY),
            modalities=(EmbedderModality.TEXT,),
            preprocessing=PREPROCESSING,
            fingerprint=embedder_fingerprint(
                "acme", "acme-text-model", revision, PREPROCESSING
            ),
        )
        return (
            EmbeddingSpace(
                id=SPACE_ID,
                version="1",
                dimension=DIMENSION,
                metric=DistanceMetric.COSINE,
                normalization=VectorNormalization.L2,
                embedders=(config,),
            ),
        )

    def create(self, capability_id: CapabilityId) -> Any:
        if capability_id != CAPABILITY_ID:
            raise ValueError(f"unknown capability: {capability_id}")
        return Encoder()

Return exactly one vector per input and exactly dimension values per vector. A single provider owns a whole space: place encoders together only when you have tested that their vectors are directly comparable.

DOCUMENT_EMBEDDING_MODALITIES in indx-interfaces states what a run hands a document embedder, in preference order: (text, image). Text for any chunk that carries text, and the rendered page for a chunk standing in for a page nothing could read as text — which indx-chunker-pdf produces. A document-role embedder declaring a modality outside that list is never called, and a space declaring no document lane it covers comes back unsatisfied when a plan names it. Declaring image for the document role is supported and is what the shipped clip-vit-b32 space does; it is called only for pages that were rendered, because text wins wherever a page has both. Query lanes are unconstrained by any of this.

Checklist

• Test encoder count, dimension, fingerprint changes, and document/query compatibility.

Reference

• Embedding spaces
• Worked extension example