Skip to content
New Docs, and a five-minute quickstart

Find the stage that lost the answer.

ChunkTrace replays one query trace through index, filter, threshold, top_k, rerank and context, and names the first stage that dropped the chunk the answer needed. Deterministic rules, no model in the analysis path, every finding proven from the trace.

No LLM calls in analysis TypeScript SDK and CLI Hosted dashboard and API

chunktrace.com/traces/tr_first_trace
The ChunkTrace dashboard showing one trace. The question was answered wrong, and the pipeline marks the Top K stage: twelve chunks went in, one came out, and the chunk with the answer was cut.
Same question, broken at

Findings

Every finding carries its proof.

A finding is not a guess. It names the chunk, the stage, and the exact field values from the trace that make it true: the rank, the score, the top_k, the tokens left in the budget. Then it says what to change next.

  • Exactly one finding is primary: the earliest-stage loss. Later stages never saw the chunk, so fixing them wouldn't help.
  • NO_LABEL and SHADOW_MISSING say what the trace can't prove instead of guessing.
  • UNEXPLAINED_EXCLUSION catches the chunk that should have been returned and wasn't: usually ANN nondeterminism or a filter the SDK didn't capture.
All ten finding codes
A finding in the dashboard: Primary, TOPK_CUT, stage top_k, chunk refund-policy#2. The message says the chunk ranked 2 among eligible results and top_k=1 cut it. Below are the next step and the evidence as JSON.

SDK

Five calls around the retrieval you already have.

createProfiler() wraps your vector store and reranker. retrieve, rerank, buildContext and finish record the trace; diagnose() returns the report, or a sink ships it to the API.

  • Pass diagnose: true on a failing query, or a sample of traffic, and the SDK runs one extra unfiltered search so removals become visible. One vector search, no inference.
  • Chunk text never leaves unless you set includeText: true. Label matching runs client-side and ships only matches_label.
  • Only the metadata keys your filter references are sent, unless metadataKeys says otherwise.
Trace a query, step by step
support-bot.ts@chunktrace/sdk
import { createProfiler, httpSink, pgvectorStore } from "@chunktrace/sdk";

const profiler = createProfiler({
  app: { name: "support-bot" },
  sink: httpSink({ endpoint: "https://api.chunktrace.com", apiKey: process.env.CHUNKTRACE_KEY! }),
});

const run = profiler.trace({
  query: userQuestion,
  label: { type: "keyword", values: ["30 days"] },  // what the answer should cite
});

let hits = await run.retrieve(store, {
  top_k: 5,
  similarity_threshold: 0.75,
  metadata_filter: { tenant: "acme" },
  diagnose: true,  // one extra unfiltered query so removals become visible
});
hits = await run.rerank(myReranker, hits, { top_n: 3 });
const { text } = run.buildContext(hits, { token_budget: 2000 });
const answer = await llm(text, userQuestion);
await run.finish({ answer, answer_correct: false });

console.log(run.diagnose().findings.find(f => f.primary)?.message);
// "Chunk refund-policy#2 ranked 6 among eligible results; top_k=5 cut it."

Rule engine

Deterministic, versioned, and the same everywhere.

diagnose() is a pure function in @chunktrace/core. The CLI, the SDK and the API worker call the same code, so a verdict is identical on a laptop and in production. ENGINE_VERSION is stored with every diagnosis, so you can tell which rule set produced it.

Ten finding codes

  • NOT_RETRIEVABLE
  • FILTER_REMOVED
  • THRESHOLD_REMOVED
  • TOPK_CUT
  • UNEXPLAINED_EXCLUSION
  • RERANK_DROPPED
  • CONTEXT_DROPPED
  • EVIDENCE_IN_CONTEXT
  • NO_LABEL
  • SHADOW_MISSING
How a diagnosis works
Terminalchunktrace diagnose
$ npx chunktrace diagnose traces/demo-04-topk.json
Trace demo-04-topk
Query: "How many days do I have to request a refund?"
Label: {"type":"keyword","values":["30 days"]}

Timeline
  retrieve  index=support-docs top_k=1 filter={"tenant":"acme"} -> 1 candidates (shadow: 13 unfiltered, k=50)
  context   included 1, excluded 0, used 17
  generate  answer_correct=false

Shadow diff: filter removed 1, threshold removed 0, top_k cut 11

Findings
  PRIMARY [TOPK_CUT] Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.
           next: Raise top_k, or add a reranker so a larger candidate pool can be narrowed by relevance.

HTTP API

One endpoint in, your keys.

Post traces one at a time or in batches of a hundred. Each is checked against the trace schema, stored, and diagnosed moments later. The SDK's httpSink does this for you; anything that can send JSON can do it too.

  • Keys belong to a project and carry a scope. A full-access key sends traces from your servers; a read-only key can only fetch them.
  • A retry of the same trace is a 200 duplicate, never a second diagnosis.
  • List, filter and count traces over the same API the dashboard uses.
HTTP API reference
Terminalapi.chunktrace.com
$ curl -X POST https://api.chunktrace.com/v0/traces \
    -H "authorization: Bearer $CHUNKTRACE_KEY" \
    -H "content-type: application/json" \
    --data @traces/demo-04-topk.json
202 {"trace_id":"demo-04-topk","status":"accepted"}
$ curl https://api.chunktrace.com/v0/traces/demo-04-topk \
    -H "authorization: Bearer $CHUNKTRACE_KEY"
{"trace_id":"demo-04-topk","status":"diagnosed",
 "primary_code":"TOPK_CUT","engine_version":1,
 "trace":{…},"diagnosis":{"findings":[…],"summary":{…}}}

How a diagnosis works

Five steps, in order, no model.

The rule engine never reads anything but the trace. Validation is strict on shape and lenient on content: a trace that validates is always diagnosable.

  1. 01
    Compute the labeled set. Chunk ids that match the developer's label: a keyword, a document, or explicit ids.
  2. 02
    No label, no diagnosis. Emit NO_LABEL and stop. Without expected evidence the trace can be described, not diagnosed.
  3. 03
    No shadow query, limited sight. Emit SHADOW_MISSING; filter, threshold and top_k effects can't be observed.
  4. 04
    Find the first stage that lost each labeled chunk. Each finding carries the exact field values that prove it in evidence.
  5. 05
    Mark exactly one finding primary. The earliest-stage loss. Later stages never saw the chunk, so fixing them wouldn't help.

Also in the box

The rest of it.

Dashboard
Headline numbers, traces per day split by answer correctness, and the primary findings ranked by how often they occur. Filters for time range, app, environment, answer and query text live in the URL, so a link reproduces the view.
pgvector and memory adapters
pgvectorStore against any table with an embedding column; memoryStore with a hashed bag-of-words embedder for tests and the sample app.
Sinks
fileSink, httpSink and multiSink. The file sink needs no server at all; the CLI reads what it writes.
Filter re-evaluation
The engine re-runs your metadata filter over the shadow results: equality, $in, $ne, numeric ranges, AND.
Listing and stats
GET /v0/traces pages newest first, with filters on code, app, env, answer, time range and query text. GET /v0/stats counts by status, code and answer.
Accounts and keys
Sign in with an email and a password. Each project has its own keys, full-access or read-only, created and revoked in Settings. A key's secret is shown once.
Offline CLI
chunktrace diagnose trace.json prints the same report in a terminal. No account and no network.
Strict validation
A zod schema for Trace Schema v0: unknown keys stripped, scores and ranks must be numbers, timestamps must be ISO-8601.

Not yet: one retrieval per trace (multi-query needs a retrievals[] array), and no re-diagnose job when ENGINE_VERSION moves. Both are on the list.

Get started

Send your first trace in five minutes.

Create an account, make a key, and run one deliberately broken query. The SDK ships an in-memory store, so there is no vector database to set up.

# 1. Create an account, then Settings → Create key (full access)
$ export CHUNKTRACE_KEY=ct_…
# 2. Install the SDK from GitHub Packages (two lines of .npmrc, in the quickstart)
$ npm install @chunktrace/sdk
# 3. Run the quickstart script: one query, with top_k set too low
$ node first-trace.mjs
Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.