Documentation

Everything you need to install, use, and trust SimLens — from a zero-setup first call to trained, certified, named concept bundles. New to the ideas? Read How SimLens works first.

Overview

SimLens is faithful, vector-only similarity & ranking attribution. Given two embeddings and a metric, it decomposes the similarity score into additive, completeness-checked contributions at three zoom levels — raw dimensions (exact), learned SAE features, and named concepts.

• Vector-only: it reads embeddings, never your model or your database — so it works with any embedder, any modality, any store.
• Faithful: for dot & cosine the contributions provably sum to the score; every result carries a completeness residual quantifying any gap.
• Fast: the per-query hot path is a compact Rust core (simlens-core) exposed through PyO3 with a zero-copy numpy FFI.
• Auditable: bundles are content-hashed and signable; every explanation is traceable to the exact artifacts that produced it.

Installation

pip install simlens

# optional extras (any combination):
pip install "simlens[qdrant,openai,train]"

Build from source

python -m venv .venv && . .venv/bin/activate
pip install maturin numpy pytest
maturin develop --release      # builds the Rust core into your environment
cargo test -p simlens-core     # Rust unit tests
pytest -q                      # Python tests
python examples/quickstart.py  # full end-to-end demo

Requires Python ≥ 3.8. Prebuilt abi3 wheels ship on PyPI for Linux and macOS (universal2).

Quickstart

The exact Level-1 explanation needs nothing but two vectors and a metric:

import simlens

ex = simlens.Explainer(metric="cosine")
attr = ex.explain(query_vec, candidate_vec)
print(attr.as_sentence())
→ "Matched mainly on 'financial-regulation' (61%), '2024-filings' (22%)."

To get named, human-readable explanations, point autofit at your store — it samples vectors, trains the dictionary, and names features from your payload fields (plus an optional LLM), all in one line:

bundle = simlens.autofit(store)                       # zero manual labeling
# or: simlens.autofit(store, namer=simlens.naming.from_provider("openai"))

Or walk the full pipeline by hand — the bundled demo does exactly this on a synthetic space with known concepts:

from simlens import train

# 1. train a sparse autoencoder + auto-label features
sae = train.SAE(dim=dim, expansion=4, l1=1e-3).fit(X, epochs=60)
bundle = train.build_bundle("synthetic-v1", "cosine", sae, X, labelers=labelers)

# 2. register named concepts (CAVs) from example vectors
bundle.add_concept("finance", positives, negatives, aspect="topic")

# 3. save / load / verify (audit)
bundle.save("demo.simlens")
ex = simlens.Explainer(simlens.Bundle.load("demo.simlens"))

# 4. explain at any level
ex.explain(q, c, level="feature", top_k=5)
ex.explain(q, c, level="concept")
ex.explain_margin(q, better, worse, level="concept")

Capability reference

Explanation levels

Pass level= to ex.explain(...). All levels share one additive contract: contributions (each tagged shared / query-only / candidate-only) plus a completeness residual.

Centered “why” (anisotropy correction)

Real embedding spaces are anisotropic: all vectors share a large common component, so a raw decomposition is dominated by the shared-mean baseline (“everything matches on ‘being English text’”). Centering removes that baseline and surfaces the discriminative concepts — the reasons this pair matched and others didn't.

# per-call:
ex.explain(q, c, center=True)

# or fit a correction (mean / ABTT / whitening) from a sample:
ctr = simlens.fit_centering(X, method="abtt")
simlens.apply_centering(ctr, vecs)

Centering is the default in the integrations — on real embeddings it's what makes the surfaced topic actually discriminative.

Learned metrics — Integrated Gradients

Dot and cosine decompose exactly, but rerankers and cross-encoders don't. For any learned or non-linear scorer, LearnedMetricExplainer uses Integrated Gradients, which comes with a completeness axiom — attributions sum to the difference between the score and a baseline, and the residual is still reported.

lex = simlens.LearnedMetricExplainer(scorer)   # scorer: (q, c) → score
attr = lex.explain(q, c)

Multi-vector (late interaction)

For ColBERT-style late-interaction models, where a query and a document are each a set of vectors, MultiVectorExplainer attributes the score over token-level vector pairs:

attr = simlens.MultiVectorExplainer().explain(Q, C)

Bundles

A bundle packages the learned dictionary (SAE), the feature names, and the named concepts into a small directory you train once and reuse. It is stamped with a content hash; every explanation it produces carries that hash, so any rationale is reproducible and traceable to the exact artifacts that made it.

bundle.save("mybundle.simlens")
b = simlens.Bundle.load("mybundle.simlens")
b.verify()          # hash (and signature) check
ex = simlens.Explainer(b)

You can also bring your own dictionary: SAEs trained elsewhere drop in via import_safetensors_sae (safetensors / SAELens format) instead of retraining.

autofit — store to bundle in one line

simlens.autofit(store) samples vectors from a store adapter, trains the dictionary, and names features from your payload fields — optionally with an LLM naming provider:

bundle = simlens.autofit(store)
bundle = simlens.autofit(store, namer=simlens.naming.from_provider("openai"))

SAE training

simlens.train ships real sparse-autoencoder trainers with trustworthy defaults:

• Architectures: TopK (default), BatchTopK, and JumpReLU — with AuxK dead-feature revival.
• Backends: a zero-dependency numpy backend, plus an optional torch/GPU backend (extra: train) for research scale.
• Train == inference: the training-time sparsity gate is reproduced exactly at inference, so the residual you certify is the residual you serve.

from simlens import train
sae = train.SAE(dim=768, expansion=8).fit(X, epochs=60)
bundle = train.build_bundle("prod-v1", "cosine", sae, X, labelers=labelers)

Feature naming

Ranking contributions is exact math; naming them is the hard part — so SimLens measures its names instead of asserting them. Names carry a balanced-accuracy score (the name used as a classifier over held-out activations), and low-scoring names are dropped rather than shown as confident nonsense. Unnamed-but-important features are shown, not hidden.

• Labelers: supply label functions / label arrays over your corpus and build_bundle auto-names features against them.
• LLM providers: simlens.naming.from_provider("openai") or "gemini" name features from top-activating payload examples.
• Reliability audit: simlens.eval.reliability(bundle, X, labelers) calibrates and audits naming confidence.

Faithfulness certification

bundle.certify(vectors) computes a signed quality scorecard baked into the bundle and covered by its content hash:

• FVU — fraction of variance unexplained by the SAE,
• L0 — average active features per vector,
• dead % — features that never fire,
• deletion / insertion AUC — do the top attributions actually move the score?
• detection accuracies — how well the names classify.

simlens info <bundle> prints the scorecard, so regressions between bundle versions are visible at a glance.

Integrations (system extensions)

Thin, customizable wrappers that adapt SimLens to a system type — you bring the vectors, they bring the explanation. Centered “why” is the default. Each ships a runnable notebook (examples/notebook_*.py) validated on a real embedder — see Validation.

from simlens.integrations.rag    import RagExplainer           # why retrieved / why ranked
from simlens.integrations.recsys import RecsysExplainer        # "because you liked …", steer
from simlens.integrations.kg     import KnowledgeGraphExplainer # explain / propose / type edges
from simlens.integrations.audit  import AuditLog               # signed, hashed decision records

Store adapters

SimLens doesn't replace your vector database — it explains its output. Adapters pull candidate vectors (and payloads, for naming) from common stores behind one interface:

Swapping Memory for Qdrant or Pgvector leaves integration code unchanged.

CLI & serving

The simlens command inspects, verifies, and evaluates bundles from the shell; a small HTTP sidecar serves explanations to any language or service.

python -m simlens.serve --bundle b.simlens

Benchmarks

Generated by python benchmarks/bench.py --md. Times are per-call medians on the host CPU; treat them as relative, not absolute. explain dim is the exact Level-1 decomposition (no bundle); explain feature is Level-2 over a TopK SAE (expansion 8, k=32); encode paths use the zero-copy numpy FFI.

Run cargo bench -p simlens-core for the Rust-kernel micro-benchmarks.

Real-system validation

Each integration ships a runnable notebook that runs on a real embedder (fastembed / bge-small-en-v1.5) when available and falls back to an offline hashing embedder so it runs anywhere — including CI, where tests/test_notebooks.py executes all four end-to-end as a guard. Every notebook reports a qualitative explanation and one quantitative faithfulness signal:

What the results say

• Faithful attribution transfers to real embeddings. Level-1 (dim) and margin decompositions are exact (residual 0); Level-2 residual is just SAE reconstruction error and is small on similar pairs. Deletion curves beat random on real vectors.
• The centered “why” earns its keep. On anisotropic real embeddings the raw feature decomposition is dominated by the shared-mean baseline; the centered view surfaces the discriminative topic instead.
• The concept level stays honestly partial. Concept attribution reports a nonzero completeness residual by construction — the ranking is right, the decomposition is a subspace projection, and the warning says so. That's a finding, not a failure.

Project layout

Status & roadmap

v0.2 — trustworthy defaults, measured not asserted. Everything in v0.1 plus:

• Real SAE trainers — TopK (default), BatchTopK, JumpReLU with AuxK dead-feature revival; numpy and torch/GPU backends; train == inference sparsity; safetensors/SAELens import.
• Detection-scored naming — names carry a measured balanced accuracy; low-scoring names are dropped.
• Centered “why” — anisotropy correction (mean / ABTT / whitening); the default in the integrations.
• Integrated Gradients — explain any learned/non-linear scorer with a completeness axiom.
• Faithfulness certification — a signed quality scorecard baked into the bundle and covered by its content hash.
• Zero-copy numpy FFI, criterion + Python benchmarks, CI + abi3 wheels + PyPI publishing, property-based & adversarial tests, and per-extension real-system validation.

Frontier (post-v0.2): a native Rust simlens-serve, cross-modal and hierarchical concepts.

Limitations

• Only Level 1 is exact. Levels 2–3 always report a residual and warn when it grows — heed the warning.
• Naming is genuinely hard. Labels carry measured confidence; unnamed-but-important features are surfaced, not hidden.
• No domain priors out of the box. On an unfamiliar space, supply label functions and example sets from your own domain to make the names good.

SimLens is released under the Apache License 2.0.