New
Docs, and a five-minute quickstart
Find the stage that lost the answer.
ChunkTrace replays one query trace through index, filter, threshold, top_k, rerank and context, and names the first stage that dropped the chunk the answer needed. Deterministic rules, no model in the analysis path, every finding proven from the trace.
Findings
Every finding carries its proof.
A finding is not a guess. It names the chunk, the stage, and the exact field values from the trace that make it true: the rank, the score, the top_k, the tokens left in the budget. Then it says what to change next.
- Exactly one finding is primary: the earliest-stage loss. Later stages never saw the chunk, so fixing them wouldn't help.
NO_LABELandSHADOW_MISSINGsay what the trace can't prove instead of guessing.UNEXPLAINED_EXCLUSIONcatches the chunk that should have been returned and wasn't: usually ANN nondeterminism or a filter the SDK didn't capture.
SDK
Five calls around the retrieval you already have.
createProfiler() wraps your vector store and reranker. retrieve,
rerank, buildContext and finish record the trace;
diagnose() returns the report, or a sink ships it to the API.
- Pass
diagnose: trueon a failing query, or a sample of traffic, and the SDK runs one extra unfiltered search so removals become visible. One vector search, no inference. - Chunk text never leaves unless you set
includeText: true. Label matching runs client-side and ships onlymatches_label. - Only the metadata keys your filter references are sent, unless
metadataKeyssays otherwise.
import { createProfiler, httpSink, pgvectorStore } from "@chunktrace/sdk";
const profiler = createProfiler({
app: { name: "support-bot" },
sink: httpSink({ endpoint: "https://api.chunktrace.com", apiKey: process.env.CHUNKTRACE_KEY! }),
});
const run = profiler.trace({
query: userQuestion,
label: { type: "keyword", values: ["30 days"] }, // what the answer should cite
});
let hits = await run.retrieve(store, {
top_k: 5,
similarity_threshold: 0.75,
metadata_filter: { tenant: "acme" },
diagnose: true, // one extra unfiltered query so removals become visible
});
hits = await run.rerank(myReranker, hits, { top_n: 3 });
const { text } = run.buildContext(hits, { token_budget: 2000 });
const answer = await llm(text, userQuestion);
await run.finish({ answer, answer_correct: false });
console.log(run.diagnose().findings.find(f => f.primary)?.message);
// "Chunk refund-policy#2 ranked 6 among eligible results; top_k=5 cut it."
Rule engine
Deterministic, versioned, and the same everywhere.
diagnose() is a pure function in @chunktrace/core. The CLI, the SDK and
the API worker call the same code, so a verdict is identical on a laptop and in production.
ENGINE_VERSION is stored with every diagnosis, so you can tell which rule set produced it.
Ten finding codes
- NOT_RETRIEVABLE
- FILTER_REMOVED
- THRESHOLD_REMOVED
- TOPK_CUT
- UNEXPLAINED_EXCLUSION
- RERANK_DROPPED
- CONTEXT_DROPPED
- EVIDENCE_IN_CONTEXT
- NO_LABEL
- SHADOW_MISSING
$ npx chunktrace diagnose traces/demo-04-topk.json
Trace demo-04-topk
Query: "How many days do I have to request a refund?"
Label: {"type":"keyword","values":["30 days"]}
Timeline
retrieve index=support-docs top_k=1 filter={"tenant":"acme"} -> 1 candidates (shadow: 13 unfiltered, k=50)
context included 1, excluded 0, used 17
generate answer_correct=false
Shadow diff: filter removed 1, threshold removed 0, top_k cut 11
Findings
PRIMARY [TOPK_CUT] Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.
next: Raise top_k, or add a reranker so a larger candidate pool can be narrowed by relevance.
HTTP API
One endpoint in, your keys.
Post traces one at a time or in batches of a hundred. Each is checked against the trace schema, stored,
and diagnosed moments later. The SDK's httpSink does this for you; anything that can send
JSON can do it too.
- Keys belong to a project and carry a scope. A full-access key sends traces from your servers; a read-only key can only fetch them.
- A retry of the same trace is a
200 duplicate, never a second diagnosis. - List, filter and count traces over the same API the dashboard uses.
$ curl -X POST https://api.chunktrace.com/v0/traces \
-H "authorization: Bearer $CHUNKTRACE_KEY" \
-H "content-type: application/json" \
--data @traces/demo-04-topk.json
202 {"trace_id":"demo-04-topk","status":"accepted"}
$ curl https://api.chunktrace.com/v0/traces/demo-04-topk \
-H "authorization: Bearer $CHUNKTRACE_KEY"
{"trace_id":"demo-04-topk","status":"diagnosed",
"primary_code":"TOPK_CUT","engine_version":1,
"trace":{…},"diagnosis":{"findings":[…],"summary":{…}}}
How a diagnosis works
Five steps, in order, no model.
The rule engine never reads anything but the trace. Validation is strict on shape and lenient on content: a trace that validates is always diagnosable.
- 01Compute the labeled set. Chunk ids that match the developer's label: a keyword, a document, or explicit ids.
- 02No label, no diagnosis. Emit
NO_LABELand stop. Without expected evidence the trace can be described, not diagnosed. - 03No shadow query, limited sight. Emit
SHADOW_MISSING; filter, threshold and top_k effects can't be observed. - 04Find the first stage that lost each labeled chunk. Each finding carries the exact field values that prove it in
evidence. - 05Mark exactly one finding primary. The earliest-stage loss. Later stages never saw the chunk, so fixing them wouldn't help.
Also in the box
The rest of it.
- Dashboard
- Headline numbers, traces per day split by answer correctness, and the primary findings ranked by how often they occur. Filters for time range, app, environment, answer and query text live in the URL, so a link reproduces the view.
- pgvector and memory adapters
pgvectorStoreagainst any table with an embedding column;memoryStorewith a hashed bag-of-words embedder for tests and the sample app.- Sinks
fileSink,httpSinkandmultiSink. The file sink needs no server at all; the CLI reads what it writes.- Filter re-evaluation
- The engine re-runs your metadata filter over the shadow results: equality,
$in,$ne, numeric ranges, AND. - Listing and stats
GET /v0/tracespages newest first, with filters on code, app, env, answer, time range and query text.GET /v0/statscounts by status, code and answer.- Accounts and keys
- Sign in with an email and a password. Each project has its own keys, full-access or read-only, created and revoked in Settings. A key's secret is shown once.
- Offline CLI
chunktrace diagnose trace.jsonprints the same report in a terminal. No account and no network.- Strict validation
- A zod schema for Trace Schema v0: unknown keys stripped, scores and ranks must be numbers, timestamps must be ISO-8601.
Not yet: one retrieval per trace (multi-query needs a retrievals[] array), and no re-diagnose job when ENGINE_VERSION moves. Both are on the list.
Get started
Send your first trace in five minutes.
Create an account, make a key, and run one deliberately broken query. The SDK ships an in-memory store, so there is no vector database to set up.
# 1. Create an account, then Settings → Create key (full access)
$ export CHUNKTRACE_KEY=ct_…
# 2. Install the SDK from GitHub Packages (two lines of .npmrc, in the quickstart)
$ npm install @chunktrace/sdk
# 3. Run the quickstart script: one query, with top_k set too low
$ node first-trace.mjs
Chunk refund-policy#2 ranked 2 among eligible results; top_k=1 cut it.