Skip to content

Reference

CLI

Diagnose a trace file in your terminal. No account and no network.

Run it#

The chunktrace command comes with the SDK. It reads one trace file and prints the diagnosis.

Terminal
npx chunktrace diagnose traces/demo-04-topk.json
Output
Trace demo-04-topk
Query: "How many days do I have to request a refund?"
Label: {"type":"keyword","values":["30 days"]}

Timeline
  retrieve  index=support-docs top_k=1 filter={"tenant":"acme"} -> 1 candidates (shadow: 13 unfiltered, k=50)
  context   included 1, excluded 0, used 17
  generate  answer_correct=false

Shadow diff: filter removed 1, threshold removed 0, top_k cut 11

Findings
  PRIMARY [TOPK_CUT] Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.
           next: Raise top_k, or add a reranker so a larger candidate pool can be narrowed by relevance.

Get a trace file#

Write traces to disk with the file sink:

JavaScript
import { createProfiler, fileSink } from "@chunktrace/sdk";

const profiler = createProfiler({ sink: fileSink("./traces") });

Each finished run becomes ./traces/<trace_id>.json.

As JSON#

Add --json for the full report, to pipe into other tools:

Terminal
npx chunktrace diagnose traces/demo-04-topk.json --json
JSON
{
  "trace_id": "demo-04-topk",
  "findings": [
    {
      "code": "TOPK_CUT",
      "stage": "top_k",
      "primary": true,
      "chunk_ids": ["refund-policy#2"],
      "message": "Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.",
      "evidence": { "…": "…" },
      "next_step": "Raise top_k, or add a reranker so a larger candidate pool can be narrowed by relevance."
    }
  ],
  "summary": {
    "labeled_chunk_ids": ["refund-policy#2"],
    "shadow_present": true,
    "context_included_count": 1,
    "context_excluded_count": 0,
    "context_tokens_used": 17,
    "filter_removed_count": 1,
    "threshold_removed_count": 0,
    "topk_cut_count": 11
  }
}

In a test#

The CLI is a thin wrapper. In code, call the same function:

JavaScript
import { diagnose } from "@chunktrace/sdk";
import { readFileSync } from "node:fs";

const trace = JSON.parse(readFileSync("traces/demo-04-topk.json", "utf8"));
const primary = diagnose(trace).findings.find((f) => f.primary);
assert.equal(primary.code, "EVIDENCE_IN_CONTEXT");

A saved trace plus this assertion is a regression test for your retrieval settings.