Skip to content

Read the results

How a diagnosis works

A fixed set of rules reads one trace and names the first stage that lost the evidence. No model is involved.

Rules, not a model#

A diagnosis is a pure function of the trace. The same trace always gives the same findings, and every finding carries the field values that prove it. Nothing is inferred, scored by a model, or looked up elsewhere.

The same code runs in three places: in your process (run.diagnose()), in the CLI, and on the server. A verdict on your laptop is the verdict in production.

The stages#

A chunk has to survive every stage to reach the model:

  1. Index. Is the chunk in the index, and does it come back for this query at all?
  2. Filter. Does it pass your metadata filter?
  3. Threshold. Does it score above your similarity threshold?
  4. Top K. Does it rank within top_k?
  5. Rerank. Does the reranker keep it?
  6. Context. Does it fit in the prompt?
  7. Generate. The model reads the context and answers.

The steps#

  1. Find the labeled chunks. The chunks your label matches: by keyword, by document, or by id.
  2. No label, no diagnosis. The engine reports NO_LABEL and stops. Without expected evidence it can describe a trace but not diagnose it.
  3. No shadow query, limited sight. The engine reports SHADOW_MISSING. It can still see reranking and context assembly, but not what the filter, threshold and top_k removed.
  4. Walk each labeled chunk through the stages. The first stage that lost the chunk becomes a finding, with the values that prove it.
  5. Mark one finding primary. The loss at the earliest stage.

How a removal is proven#

The shadow query returns chunks with no filter and no threshold. For a labeled chunk that is there but missing from your real results, the engine re-applies your settings in order:

  • It re-runs your filter on the chunk's metadata. If it fails, that is FILTER_REMOVED, and the evidence names the key and both values.
  • It compares the chunk's score to your threshold. Below it is THRESHOLD_REMOVED.
  • It ranks the chunk among everything that passed both. A rank beyond top_k is TOPK_CUT.
  • If the chunk passes all three and still wasn't returned, that is UNEXPLAINED_EXCLUSION. The engine says so rather than guessing.

One primary finding#

Several chunks can match a label, and each can be lost at a different stage. The primary finding is the earliest loss, because later stages never saw that chunk. Fixing them first would change nothing.

Engine version#

The rule set has a version number, shown in the dashboard as engine v1 and stored with every diagnosis. It goes up when a change to the rules could give a different finding for the same trace, so you can always tell which rules produced a result.

Limits today#

  • One retrieval per trace. A pipeline that searches several times for one question records one of them.
  • Filters are equality, $in, $ne and numeric ranges, joined by AND.
  • A new engine version applies to new traces. Existing diagnoses are not re-run.