Read the results
How a diagnosis works
A fixed set of rules reads one trace and names the first stage that lost the evidence. No model is involved.
Rules, not a model#
A diagnosis is a pure function of the trace. The same trace always gives the same findings, and every finding carries the field values that prove it. Nothing is inferred, scored by a model, or looked up elsewhere.
The same code runs in three places: in your process (run.diagnose()), in the CLI, and on the server. A verdict on your laptop is the verdict in production.
The stages#
A chunk has to survive every stage to reach the model:
- Index. Is the chunk in the index, and does it come back for this query at all?
- Filter. Does it pass your metadata filter?
- Threshold. Does it score above your similarity threshold?
- Top K. Does it rank within
top_k? - Rerank. Does the reranker keep it?
- Context. Does it fit in the prompt?
- Generate. The model reads the context and answers.
The steps#
- Find the labeled chunks. The chunks your label matches: by keyword, by document, or by id.
- No label, no diagnosis. The engine reports
NO_LABELand stops. Without expected evidence it can describe a trace but not diagnose it. - No shadow query, limited sight. The engine reports
SHADOW_MISSING. It can still see reranking and context assembly, but not what the filter, threshold andtop_kremoved. - Walk each labeled chunk through the stages. The first stage that lost the chunk becomes a finding, with the values that prove it.
- Mark one finding primary. The loss at the earliest stage.
How a removal is proven#
The shadow query returns chunks with no filter and no threshold. For a labeled chunk that is there but missing from your real results, the engine re-applies your settings in order:
- It re-runs your filter on the chunk's metadata. If it fails, that is
FILTER_REMOVED, and the evidence names the key and both values. - It compares the chunk's score to your threshold. Below it is
THRESHOLD_REMOVED. - It ranks the chunk among everything that passed both. A rank beyond
top_kisTOPK_CUT. - If the chunk passes all three and still wasn't returned, that is
UNEXPLAINED_EXCLUSION. The engine says so rather than guessing.
One primary finding#
Several chunks can match a label, and each can be lost at a different stage. The primary finding is the earliest loss, because later stages never saw that chunk. Fixing them first would change nothing.
Engine version#
The rule set has a version number, shown in the dashboard as engine v1 and stored with every diagnosis. It goes up when a change to the rules could give a different finding for the same trace, so you can always tell which rules produced a result.
Limits today#
- One retrieval per trace. A pipeline that searches several times for one question records one of them.
- Filters are equality,
$in,$neand numeric ranges, joined by AND. - A new engine version applies to new traces. Existing diagnoses are not re-run.