Skip to content

Instrument your app

Trace a query

Five calls around the retrieval you already have. Each one records what that stage kept and what it dropped.

The shape of it#

One query is one trace. You start a run, pass your retrieval steps through it, and finish it. The run does the work you were already doing and writes down what happened.

support-bot.mjs
import { createProfiler, httpSink, pgvectorStore } from "@chunktrace/sdk";

const profiler = createProfiler({
  app: { name: "support-bot", env: "prod" },
  sink: httpSink({ endpoint: "https://api.chunktrace.com", apiKey: process.env.CHUNKTRACE_KEY }),
});

const store = pgvectorStore({ name: "docs", db: pgPool, table: "chunks", embed });

const run = profiler.trace({
  query: userQuestion,
  label: { type: "keyword", values: ["30 days"] }, // what a right answer has to cite
});

let hits = await run.retrieve(store, {
  top_k: 5,
  similarity_threshold: 0.75,
  metadata_filter: { tenant: "acme" },
  diagnose: true, // one extra search, so removals become visible
});
hits = await run.rerank(myReranker, hits, { top_n: 3 });
const { text } = run.buildContext(hits, { token_budget: 2000 });
const answer = await llm(text, userQuestion);
await run.finish({ answer, answer_correct: false });

createProfiler(options)#

Create one profiler when your app starts and reuse it.

Option Default What it does
app none { name, env, version }. Shown on every trace. The dashboard filters by name and env.
sink in memory Where finished traces go. See Where traces go.
includeText false Send chunk text in traces. See What leaves your app.
metadataKeys filter keys only Which metadata keys to send: a list of names, or "all".
countTokens characters ÷ 4 A function from text to a token count, used for the context budget. Pass your tokenizer for exact numbers.

profiler.trace({ query, label, trace_id })#

Starts a run for one query.

  • query is the user's question.
  • label says what a right answer should cite. Without it the trace is recorded but can't be diagnosed. See Labels.
  • trace_id is optional. Pass your own request id to find the trace later; otherwise one is generated. Read it from run.trace_id.

run.retrieve(store, options)#

Searches the store and returns the hits, in rank order. Call it once per run.

Option What it does
top_k How many chunks to return. Required.
similarity_threshold Drop chunks that score below this.
metadata_filter Keep only chunks whose metadata matches. See filters.
diagnose true runs a second, unfiltered search so the trace can show what the filter, threshold and top_k removed. See Diagnose mode.

store is a vector store adapter: pgvector, the in-memory store, or your own.

run.rerank(reranker, hits, options)#

Optional. Scores the hits with your reranker and returns the ones that survive, best first.

Option What it does
top_n Keep this many after reranking.
score_threshold Drop hits the reranker scores below this.

run.buildContext(hits, options)#

Assembles the prompt context and records which chunks made it in. Returns { hits, text }: the chunks that were included, and their text joined together.

Option Default What it does
token_budget no limit Skip a chunk that would push the context over this many tokens.
max_chunks no limit Include at most this many chunks.
separator blank line What goes between chunks in text.

A chunk that appears twice is included once. Every skipped chunk is recorded with its reason: token_budget, max_chunks or dedupe.

If your own code drops a chunk after this, say so, or the trace will claim it reached the prompt:

JavaScript
run.exclude("refund-policy#2", "failed the relevance gate");

run.finish({ model, answer, answer_correct })#

Ends the run and sends the trace to the sink. All three fields are optional.

  • answer_correct is your judgement of the answer: true, false, or leave it out if you don't know. The dashboard charts it and filters on it.
  • finish waits for the sink. With httpSink that is one HTTP request, so in a request handler you may want to respond to the user first and finish after.

run.diagnose()#

Runs the rule engine on the finished trace, in your process, and returns the report. Use it in tests, scripts and logs:

JavaScript
const report = run.diagnose();
const primary = report.findings.find((f) => f.primary);
console.log(primary.code, primary.message, primary.next_step);

It is the same code the server runs, so the result is the same in both places.

Which queries to trace#

You don't have to trace everything. Two patterns work well:

  • Failing queries. When a user reports a wrong answer, replay the query with a label and diagnose: true.
  • A sample of traffic. Trace a few percent of requests to watch for regressions. Leave diagnose off for most of them; it costs one extra vector search.