why-this-chunk 发布：为 RAG 检索结果提供归因解释与最小修复建议

Retrieval explainer for RAG — shows why a chunk ranked where it did, the lexical-vs-dense split, and the one config change that would have surfaced the right answer.

Point it at a query and an existing retriever/index; it produces per-result explanations: sentence-level (or token-level) attribution of the similarity score and a lexical (BM25) vs dense contribution split for hybrid search. When a known-correct chunk is missing, it classifies why and finds the smallest config change that fixes that one query.

Why

RAG debugging today is mostly guesswork. This tells you which layer is actually at fault — index, chunking, ranking, or the embedding model — and the smallest change that fixes a specific failing query.

Features

Score attribution — per-result sentence/token attribution of the similarity score via deterministic occlusion (model-agnostic, no model internals needed).
Lexical vs dense split — for hybrid retrievers, decompose the score into BM25 and dense contributions and report which dominated.
Failure taxonomy — classify a failed (query, expected_chunk) into missing_from_index / lost_to_chunking / out_ranked / embedding_blind_spot, with explicit evidence and an unambiguous, documented decision order.
Counterfactual minimal fix: search the smallest config change (top-K, chunk size, hybrid alpha, reranker on/off) that surfaces the right chunk, with a documented per-axis cost. One axis at a time by default; --max-axes 2 adds a second pass over pairs for the query that needs two.
Offline by default — a deterministic hashing embedder (FakeEmbedder) makes the whole pipeline reproducible with zero downloads; real local embeddings and rerankers are opt-in extras.
CLI + tiny web inspector — a rich terminal UI (plus Markdown/JSON output) and an optional read-only FastAPI view.

How it works

Given a query and (optionally) the chunk you expected to win, it attributes the score by re-scoring the chunk with each sentence occluded, then — if the chunk failed — classifies the cause and searches a bounded set of single-axis config changes for the cheapest one that pulls the right chunk into the top-K. Cause (why it ranks where it does) and fixability (the cheapest change that surfaces it) are reported as separate fields, so a cause label never reads as "unfixable".

Installation

Requires Python 3.11+.

pip install why-this-chunk            # core: numpy, rank_bm25, typer, rich, pydantic

# optional extras
pip install "why-this-chunk[st]"      # real local embeddings + cross-encoder rerank (sentence-transformers)
pip install "why-this-chunk[faiss]"   # FAISS dense backend (faiss-cpu)
pip install "why-this-chunk[web]"     # read-only FastAPI inspector
pip install "why-this-chunk[qdrant]"  # explain an existing Qdrant collection
pip install "why-this-chunk[pg]"      # explain an existing PostgreSQL + pgvector table

From a checkout, for development:

pip install -e ".[dev]"

Quickstart

CLI

The corpus is a JSON-Lines file with one {"id": ..., "text": ...} object per line (optional metadata, source_document_id, span). A sample lives in examples/corpus.jsonl.

# Explain why each top-K result ranked where it did (sentence attribution + split).
why-this-chunk explain "What is the capital of France?" --corpus examples/corpus.jsonl --k 3

# Diagnose why an expected chunk failed, and print the minimal fix.
why-this-chunk diagnose "NumPy fast numerical arrays for Python data science" \
    --expect python --corpus examples/corpus.jsonl --k 1

# Just the fix suggestion(s).
why-this-chunk fix "NumPy fast numerical arrays for Python data science" \
    --expect python --corpus examples/corpus.jsonl --k 1 --all

# Diagnose a whole file of (query, expected chunk) pairs and aggregate the results.
why-this-chunk batch --queries examples/queries.jsonl --corpus examples/corpus.jsonl --k 1

# Use real local embeddings and cross-encoder reranking (requires the [st] extra).
why-this-chunk diagnose "NumPy fast numerical arrays for Python data science" \
    --expect python --corpus examples/corpus.jsonl --k 1 --embedder st --rerank

batch reads a queries JSON-Lines file with one object per line (a sample lives in examples/queries.jsonl). Each row needs query and exactly one of expect (chunk id, unique metadata value, or unique text substring), expect_text, or expect_meta. It runs the same diagnose + fix path over each row and prints an aggregate: the count per failure class, the most common suggested fix axis, and a per-query table. Pass --out PATH to also write that report as a CI artifact (--format md writes Markdown; --format json or the default rich view writes JSON). The file is created (parents included) before --fail-on trips, so a failing gate still leaves the artifact behind.

diagnose / fix resolve --expect the same way: exact id first, then a unique metadata value (or source_document_id), then a unique case-insensitive substring of the chunk text. Ambiguous locators exit with a usage error listing the candidate ids; a real id that is simply absent still classifies as missing_from_index.

Every command takes --format {rich,md,json} (default rich) to choose the output shape; --json is kept as a deprecated alias for --format json. Pick the retriever with --mode {bm25,dense,hybrid} (default hybrid). All CLI commands run offline by default using the deterministic FakeEmbedder; pass --embedder st for the real SentenceTransformerEmbedder and --rerank (with an optional --rerank-model override) to rerank the candidate pool with a cross-encoder — both require the [st] extra and fail with a clear error if it isn't installed. Pass --faiss to use the FAISS dense backend (dense and hybrid modes); that requires the [faiss] extra and likewise fails clearly if it isn't installed. The active backend (faiss or numpy) is reported in the command output.

Library

from why_this_chunk import (
    Corpus,
    FakeEmbedder,
    BM25Retriever,
    DenseRetriever,
    HybridRetriever,
    RetrievalConfig,
    explain_chunk,
    diagnose,
    search_fixes,
)

corpus = Corpus.from_jsonl("examples/corpus.jsonl")
embedder = FakeEmbedder(seed=0)  # deterministic, offline
retriever = HybridRetriever(
    DenseRetriever(corpus, embedder),
    BM25Retriever(corpus),
    alpha=0.5,
)

# Explain the top result.
top = retriever.search("What is the capital of France?", k=1)[0]
explanation = explain_chunk(retriever, "What is the capital of France?", top)
print(explanation.sentences[0].sentence, explanation.sentences[0].share)
print(explanation.split.dominant)  # "dense" or "lexical"

# Diagnose a failure and find the cheapest fix.
config = RetrievalConfig(top_k=1, alpha=0.5)
query = "NumPy fast numerical arrays for Python data science"
result = diagnose(retriever, query, "python", config)
print(result.failure_class)  # e.g. FailureClass.OUT_RANKED
print(search_fixes(retriever, query, "python", config).best)  # e.g. raise top_k to 2

To make the chunk_size axis and lost_to_chunking check evaluable, build the corpus with provenance from raw documents:

from why_this_chunk import Corpus, SourceDocument, FixedSizeChunker

corpus = Corpus.from_sources(
    [SourceDocument(id="doc1", text="...full document text...")],
    FixedSizeChunker(overlap=0),
    chunk_size=512,
)

Example demo

A runnable, zero-download demo of explain and diagnose lives in examples/demo.py:

python examples/demo.py

Explaining an index you already have (optional)

The built-in retrievers own their corpus in-process. To explain a store that already holds your chunks and embeddings, wrap it instead of re-ingesting it. For a Qdrant collection:

from qdrant_client import QdrantClient
from why_this_chunk import explain_chunk
from why_this_chunk.retrievers.adapters.qdrant import QdrantRetriever

retriever = QdrantRetriever(
    QdrantClient(url="http://localhost:6333"),
    "my_chunks",
    embedder,  # the same embedder that produced the stored vectors
    text_field="text",  # payload key holding the chunk body
    id_field="chunk_id",  # payload key holding your chunk id (else the point id)
)

top = retriever.search("why did this rank here?", k=1)[0]
print(explain_chunk(retriever, "why did this rank here?", top).sentences[0].share)

For a PostgreSQL table with a pgvector column — same adapter boundary, same scores, driven by one ORDER BY <distance> LIMIT k so your existing index does the work:

from why_this_chunk.retrievers.adapters.pgvector import PgVectorRetriever

retriever = PgVectorRetriever(
    "postgresql://localhost/mydb",
    "chunks",  # or "schema.chunks"
    embedder,
    id_column="chunk_id",
    text_column="body",
    embedding_column="embedding",
    metadata_column="meta",  # optional jsonb column, carried onto the chunk
    metric="cosine",  # must match the operator class of your index
)

The table must hold each chunk's text as well as its vector: attribution occludes sentences of the chunk body. Table and column names are validated as SQL identifiers and quoted, never interpolated raw; the query vector and k are bound parameters. The connection is opened lazily and reused — pass connection= to reuse one you already own, and it stays yours to close.

What you get and what you don't, stated up front rather than faked:


Sentence-level occlusion attribution	✅ — `score_text` re-embeds locally, no store round-trip
`explain` / `diagnose` / the `top_k` and `alpha` reasoning	✅
Lexical-vs-dense split	❌ — a vector store has no lexical modality; `split` is `None`
`chunk_size` counterfactual axis	❌ — re-chunking means rewriting the store, so it is reported unevaluable, never silently skipped

Scores are true cosine similarities on the same scale as the built-in dense retriever, which the Embedder contract's L2-normalized vectors make exact for Qdrant's Cosine, Dot and Euclid collections and for pgvector's cosine, ip and l2 operators. A Manhattan collection has no such identity and is rejected at construction rather than mis-scored.

Web inspector (optional)

pip install "why-this-chunk[web]"
why-this-chunk serve --corpus examples/corpus.jsonl
# open http://127.0.0.1:8000

Two-axis fixes

fix searches one axis at a time. When nothing single-axis works, the answer is often not "unfixable" but "two things are wrong at once": a chunk that is both split badly and out-ranked needs a smaller chunk_size to put the answer in one piece, and only then does a higher alpha pull it into the top-K. Neither alone moves it, and reporting that query as unfixable sends someone off to re-embed a corpus when two knobs would have done it.

why-this-chunk fix "..." --expect chunk-42 --corpus corpus.jsonl --max-axes 2

The second pass runs only when the first found nothing, because it costs a reindex per combination: single-axis is the sum of the per-axis candidate counts, pairs is the product. --max-combinations (default 64) bounds it, and hitting that bound is reported as capped rather than as "no fix". Those two answers are not the same and do not print the same.

A one-axis fix always outranks a two-axis one, even when the two-axis one is cheaper on the cost scale. A reviewer applying it has to reason about one thing instead of two, and this tool exists to be read by a person.

A pair is unevaluable the moment either half is, so an axis missing its capability or its corpus provenance keeps every pair it appears in out of the search rather than producing a fix nobody can apply.

Every output distinguishes a pair from a single change: the terminal labels it 2 axes and indents both changes under it, Markdown gives it an axes column, and the JSON adds best_plan (one shape whether the answer is one change or two), pair_fixes and capped.

Bring your own retriever

A third-party retriever only needs to implement search(query, k) -> list[ScoredChunk]. Richer behaviour is advertised through two boolean capabilities — supports_components (the dense/lexical split) and supports_reindex (returning a new retriever under a different RetrievalConfig). Any feature or counterfactual axis that depends on a missing capability or on corpus provenance is reported unevaluable rather than silently skipped.

Determinism

The default FakeEmbedder produces stable vectors from token hashes, so retrieval, attribution, and the counterfactual search are fully reproducible with no network access. All tie-breaks use stable id ordering. The real SentenceTransformerEmbedder and cross-encoder reranker are opt-in via the [st] extra.

Development

pip install -e ".[dev]"
ruff check .
ruff format --check .
mypy src
pytest -q --cov=why_this_chunk

Status & roadmap

Beta. The core explainer, taxonomy, counterfactual search, CLI, and a minimal web inspector are implemented and tested offline.

Score attribution + lexical/dense split for a hybrid retriever
Per-query failure taxonomy classifier
Counterfactual minimal-config-fix search (single axis, and pairs via --max-axes 2)
CLI (explain / diagnose / fix / batch) with rich / Markdown / JSON output
Optional local embeddings, cross-encoder rerank, FAISS backend, read-only web inspector
Qdrant adapter over an existing collection
pgvector adapter over an existing table