Skip to content

Instrument your app

Diagnose mode

One extra, unfiltered search per query. It is how a trace can show what your filter, threshold and top_k removed.

The problem it solves#

A vector store returns what survived. It does not tell you what it left out. If the chunk you needed is missing from the results, the results alone can't say whether the filter removed it, the threshold removed it, or it simply ranked eleventh when you asked for ten.

What it does#

With diagnose: true, the SDK runs your search twice:

  1. Your real search, with your filter, threshold and top_k.
  2. A shadow query: the same question with no filter, no threshold, and a larger k.
JavaScript
const hits = await run.retrieve(store, {
  top_k: 5,
  similarity_threshold: 0.75,
  metadata_filter: { tenant: "acme" },
  diagnose: true,
});

Your app only sees the real results. The trace carries both, and the rule engine compares them. A chunk that is in the shadow results but not the real ones was removed by something, and the engine works out what by re-applying your filter, threshold and top_k in order.

What it costs#

One extra vector search. No model calls, and no extra latency in the answer if you run the trace after responding.

The shadow query asks for top_k × 10 results, and at least 50. To change it:

JavaScript
diagnose: { shadow_k: 200 }

A larger shadow_k sees further down the ranking. Raise it if you get NOT_RETRIEVABLE for chunks you know are indexed.

When to turn it on#

  • On a query you are debugging: always.
  • On a sample of production traffic: for a fraction of the traces.
  • On every request: usually not worth it.

Without it#

A trace without a shadow query still records what was returned, reranked and put in the prompt. Findings from those stages (RERANK_DROPPED, CONTEXT_DROPPED, EVIDENCE_IN_CONTEXT) work as usual.

What it can't show is anything before that. If the labeled chunk was not returned, the finding is SHADOW_MISSING, and the filter, threshold and top_k stages are drawn dashed in the dashboard: not observable.