Read the results
Finding codes
The ten things a diagnosis can say, what each one means, and what to try next.
Reading a finding#
Every finding has the same parts:
{
"code": "TOPK_CUT",
"stage": "top_k",
"primary": true,
"chunk_ids": ["refund-policy#2"],
"message": "Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.",
"evidence": { "chunk_id": "refund-policy#2", "effective_rank": 2, "score": 0.498, "top_k": 1, "eligible_count": 12 },
"next_step": "Raise top_k, or add a reranker so a larger candidate pool can be narrowed by relevance."
}evidence holds the values from the trace that make the finding true. Nothing in a finding comes from outside the trace.
Evidence was lost#
These name the stage where a labeled chunk dropped out, in pipeline order.
| Code | Stage | What happened |
|---|---|---|
NOT_RETRIEVABLE |
Index | No chunk matching the label appeared in the unfiltered search at all. |
FILTER_REMOVED |
Filter | The chunk exists, but your metadata filter excluded it. |
THRESHOLD_REMOVED |
Threshold | The chunk scored below your similarity threshold. |
TOPK_CUT |
Top K | The chunk passed the filter and threshold but ranked below top_k. |
UNEXPLAINED_EXCLUSION |
Top K | The chunk should have been returned and wasn't. |
RERANK_DROPPED |
Rerank | The chunk was retrieved, then the reranker cut it. |
CONTEXT_DROPPED |
Context | The chunk was selected but left out of the prompt. |
NOT_RETRIEVABLE#
The index can't produce this chunk for this query. Check that the source document is indexed, and that chunking didn't split the phrase your label looks for. If you are sure it is indexed, it may rank below the shadow query's reach: raise shadow_k.
FILTER_REMOVED#
The evidence names the filter key, the value you asked for, and the value the chunk has. Either the filter is too strict or the chunk's metadata is wrong. Mistagged tenants and stale status fields are the usual causes.
THRESHOLD_REMOVED#
The evidence gives the chunk's score and your threshold. Lower the threshold, or improve the query or the embedding so the chunk scores higher.
TOPK_CUT#
The evidence gives the chunk's rank among the chunks that were eligible. Raise top_k, or retrieve more and add a reranker to narrow them down.
UNEXPLAINED_EXCLUSION#
The chunk passes your filter and threshold and ranks within top_k in the shadow query, yet the real search didn't return it. Usually this is approximate nearest-neighbour search returning slightly different results on each call, or a filter your store applies that the trace doesn't know about. Run the query again and compare.
RERANK_DROPPED#
The evidence gives the chunk's rank before and after reranking, and the rule that cut it (top_n or the score threshold). Raise top_n, or look at why the reranker scores this chunk low.
CONTEXT_DROPPED#
The evidence gives the reason: token_budget, max_chunks, dedupe, or your own. For a budget, raise it, shorten your chunks, or put the most relevant chunks first.
Evidence arrived#
| Code | Stage | What happened |
|---|---|---|
EVIDENCE_IN_CONTEXT |
Generate | The labeled chunk reached the prompt. |
Retrieval did its job for this chunk. If the answer is still wrong, look at the prompt and the model, not at retrieval. On a correct answer, this is simply what a healthy trace looks like.
The trace can't say#
These are not failures of your pipeline. They say what the trace is missing.
| Code | What it means | What to do |
|---|---|---|
NO_LABEL |
The trace has no label, so there is nothing to look for. | Add a label to the query. |
SHADOW_MISSING |
The trace has no shadow query, so the filter, threshold and top_k stages can't be observed. | Turn on diagnose mode. |
Which finding is primary#
A trace can have several findings, one per labeled chunk. Exactly one is primary: the loss at the earliest stage. Fix that one first. Stages after it never saw the chunk, so changing them would not have helped.