Reference
CLI
Diagnose a trace file in your terminal. No account and no network.
Run it#
The chunktrace command comes with the SDK. It reads one trace file and prints the diagnosis.
npx chunktrace diagnose traces/demo-04-topk.jsonTrace demo-04-topk
Query: "How many days do I have to request a refund?"
Label: {"type":"keyword","values":["30 days"]}
Timeline
retrieve index=support-docs top_k=1 filter={"tenant":"acme"} -> 1 candidates (shadow: 13 unfiltered, k=50)
context included 1, excluded 0, used 17
generate answer_correct=false
Shadow diff: filter removed 1, threshold removed 0, top_k cut 11
Findings
PRIMARY [TOPK_CUT] Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.
next: Raise top_k, or add a reranker so a larger candidate pool can be narrowed by relevance.Get a trace file#
Write traces to disk with the file sink:
import { createProfiler, fileSink } from "@chunktrace/sdk";
const profiler = createProfiler({ sink: fileSink("./traces") });Each finished run becomes ./traces/<trace_id>.json.
As JSON#
Add --json for the full report, to pipe into other tools:
npx chunktrace diagnose traces/demo-04-topk.json --json{
"trace_id": "demo-04-topk",
"findings": [
{
"code": "TOPK_CUT",
"stage": "top_k",
"primary": true,
"chunk_ids": ["refund-policy#2"],
"message": "Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.",
"evidence": { "…": "…" },
"next_step": "Raise top_k, or add a reranker so a larger candidate pool can be narrowed by relevance."
}
],
"summary": {
"labeled_chunk_ids": ["refund-policy#2"],
"shadow_present": true,
"context_included_count": 1,
"context_excluded_count": 0,
"context_tokens_used": 17,
"filter_removed_count": 1,
"threshold_removed_count": 0,
"topk_cut_count": 11
}
}In a test#
The CLI is a thin wrapper. In code, call the same function:
import { diagnose } from "@chunktrace/sdk";
import { readFileSync } from "node:fs";
const trace = JSON.parse(readFileSync("traces/demo-04-topk.json", "utf8"));
const primary = diagnose(trace).findings.find((f) => f.primary);
assert.equal(primary.code, "EVIDENCE_IN_CONTEXT");A saved trace plus this assertion is a regression test for your retrieval settings.