Skip to content

Read the results

Finding codes

The ten things a diagnosis can say, what each one means, and what to try next.

Reading a finding#

Every finding has the same parts:

JSON
{
  "code": "TOPK_CUT",
  "stage": "top_k",
  "primary": true,
  "chunk_ids": ["refund-policy#2"],
  "message": "Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.",
  "evidence": { "chunk_id": "refund-policy#2", "effective_rank": 2, "score": 0.498, "top_k": 1, "eligible_count": 12 },
  "next_step": "Raise top_k, or add a reranker so a larger candidate pool can be narrowed by relevance."
}

evidence holds the values from the trace that make the finding true. Nothing in a finding comes from outside the trace.

Evidence was lost#

These name the stage where a labeled chunk dropped out, in pipeline order.

Code Stage What happened
NOT_RETRIEVABLE Index No chunk matching the label appeared in the unfiltered search at all.
FILTER_REMOVED Filter The chunk exists, but your metadata filter excluded it.
THRESHOLD_REMOVED Threshold The chunk scored below your similarity threshold.
TOPK_CUT Top K The chunk passed the filter and threshold but ranked below top_k.
UNEXPLAINED_EXCLUSION Top K The chunk should have been returned and wasn't.
RERANK_DROPPED Rerank The chunk was retrieved, then the reranker cut it.
CONTEXT_DROPPED Context The chunk was selected but left out of the prompt.

NOT_RETRIEVABLE#

The index can't produce this chunk for this query. Check that the source document is indexed, and that chunking didn't split the phrase your label looks for. If you are sure it is indexed, it may rank below the shadow query's reach: raise shadow_k.

FILTER_REMOVED#

The evidence names the filter key, the value you asked for, and the value the chunk has. Either the filter is too strict or the chunk's metadata is wrong. Mistagged tenants and stale status fields are the usual causes.

THRESHOLD_REMOVED#

The evidence gives the chunk's score and your threshold. Lower the threshold, or improve the query or the embedding so the chunk scores higher.

TOPK_CUT#

The evidence gives the chunk's rank among the chunks that were eligible. Raise top_k, or retrieve more and add a reranker to narrow them down.

UNEXPLAINED_EXCLUSION#

The chunk passes your filter and threshold and ranks within top_k in the shadow query, yet the real search didn't return it. Usually this is approximate nearest-neighbour search returning slightly different results on each call, or a filter your store applies that the trace doesn't know about. Run the query again and compare.

RERANK_DROPPED#

The evidence gives the chunk's rank before and after reranking, and the rule that cut it (top_n or the score threshold). Raise top_n, or look at why the reranker scores this chunk low.

CONTEXT_DROPPED#

The evidence gives the reason: token_budget, max_chunks, dedupe, or your own. For a budget, raise it, shorten your chunks, or put the most relevant chunks first.

Evidence arrived#

Code Stage What happened
EVIDENCE_IN_CONTEXT Generate The labeled chunk reached the prompt.

Retrieval did its job for this chunk. If the answer is still wrong, look at the prompt and the model, not at retrieval. On a correct answer, this is simply what a healthy trace looks like.

The trace can't say#

These are not failures of your pipeline. They say what the trace is missing.

Code What it means What to do
NO_LABEL The trace has no label, so there is nothing to look for. Add a label to the query.
SHADOW_MISSING The trace has no shadow query, so the filter, threshold and top_k stages can't be observed. Turn on diagnose mode.

Which finding is primary#

A trace can have several findings, one per labeled chunk. Exactly one is primary: the loss at the earliest stage. Fix that one first. Stages after it never saw the chunk, so changing them would not have helped.