---
title: "Retrieval"
type: reference
tags: [signpost, docs, reference]
proposed_by: tok_bc09b340db294c709763789154d8113d
model: kmem-docs-publisher
extracted_at: "2026-10-08T22:28:29.988Z"
---
# Retrieval

Retrieval is the product. `recall` returning the right three paragraphs inside a token budget is the
thing kmem is judged on, and it is the one part of the system that cannot be fixed later by
polishing an interface over it.

> ## Revised 2026-08-14. Two things below are no longer true and one of them was the headline.
>
> **1. `precision@5 ≥ 0.80` is no longer the phase 2 gate.** It was retired on 2026-08-14 on a
> validity proof rather than on a shortfall: a deliberately degraded BM25-only configuration that
> fails to find the answer at all on two of 111 questions scored **0.8180**, while the full real
> pipeline that finds it on every one scored **0.7748**. A gate the worse product passes and the
> better one fails is measuring the wrong thing. The gate is now `hit@5` = 1.000 **and**
> `precision@1` ≥ 0.90, with `precision@5` ≥ 0.60 promoted from advisory prose to a hard floor. This
> revision box is the record of that retirement. **Every
> `precision@5`-against-0.80 sentence in this document is history, and the tuning it describes is
> still exactly right** — 0.3159 → 0.7720 was a real defect being fixed, not a number being chased.
>
> **2. `hit@5` is not 1.0000 on the real pipeline over the whole set. It is 0.9936.** The claim
> below — "1.0000 on every split and on both retrievers" — was true of every population that had
> been measured and was never true of the one that had not. The whole 196-question set was run
> against the deployment for the first time on 2026-08-14: **156 of 157**. The miss is
> `pp-single-points-of-failure`, *"Which parts of the system depend on exactly one person?"*,
> held-out, `hard`, five relevant notes spread across `people` and `state`. The deployment ranks the
> best relevant claim **7th**; the reference retriever ranks it **2nd**. `hit@10` is 1.000, so it is
> retrieved and mis-ranked. **It is the one thing standing between this project and a green phase 2
> automated gate**, and the threshold is not moving to 0.99 to accommodate it.
>
> **The current numbers, both retrievers, whole set, 2026-08-14:**
>
> | | reference | deployment `eaa6ccee` |
> | --- | --- | --- |
> | `hit@5` | 1.0000 | **0.9936** |
> | `precision@1` | 0.8981 | **0.9427** |
> | `precision@5` | 0.7720 | 0.7503 |
> | MRR / nDCG@10 | 0.9375 / 0.8150 | **0.9663 / 0.8654** |
> | answer text in full | 0.8981 | **0.9363** |
> | `used_tokens` median / p95 | 1,868.5 / 2,378 | 2,409.5 / 2,702 |
> | gate | **PASS** 15/15 | **FAIL** 9/11 |
>
> The real pipeline beats the reference on every metric except `hit@5` and `precision@5` — which is
> the shape D10 predicted, because `precision@5` charges for the pack diversity the real pipeline
> provides.

**Retrieval fails its gate. The final figure is `precision@5` 0.7720 on the whole 196-question set
against a required 0.80** (reference retriever), and **0.7748 on the dev split through the real
`bge-m3` + Vectorize + `bge-reranker-base` pipeline**, from 0.3159 before tuning. Both are short of
0.80. **`hit@5` is 1.0000 on every split and on both retrievers** — a relevant note is in the top five
for 157 of 157 scored questions on the whole set, and the first relevant claim sits at rank 1 on 141
of them (106 of 111 on the real pipeline). Read those two sentences together before reading anything
else: the system finds the answer every single time, and the gate metric still says no.

**Answer first, for a reader who wants one line.** *When you ask this knowledge base something it
knows, does it tell you?* **Yes — 157 times out of 157.** What fails is `precision@5`, and
[round 5](#2026-08-13--round-5-nothing-shipped-precision5-is-a-duplication-metric-now-proved-from-both-sides)
and [round 8](#2026-08-13--round-8-final-numbers-and-the-verdict) prove, from both sides and now on real
model output, that on this eval `precision@5` is a monotone *decreasing* function of how many distinct
notes the pack contains. Five chunks of the one correct note score 1.00; the correct note at rank 1
plus four genuinely useful neighbours scores 0.20. The metric and the reader disagree, and it is the
metric that is wrong here.

The three splits, reference retriever, all 2026-08-13: **whole set 0.7720 (157 scored) · dev 0.7856
(111 scored) · held-out 0.7391 (46 scored)**. The held-out figure has now been measured **twice** —
first in [round 6](#2026-08-13--round-6-the-held-out-measurement-and-the-honest-verdict), then again in
round 8 — and it reproduced to four decimals, 0.7391 both times. That second run is a *second spend*
and is labelled as one everywhere it appears. The dev-minus-held-out gap is **0.0465**, whose bootstrap
interval is **[-0.0360, 0.1322]** — it crosses zero, so 46 questions cannot separate mild overfitting
from noise, and it survives standardisation on question difficulty, source file and relevant-set size,
so it is not a question-mix artefact either. Every ranking metric that measures *whether the right note
is found* is equal or better on held-out: `hit@5` **1.0000** on all three splits, `precision@1` 0.9130
against dev's 0.8919, MRR 0.9475 against 0.9333, block-level `precision@5` 0.5818 against 0.5569. Only
the `precision@k` family is lower. Two facts frame everything below:

1. **The candidate set was never the problem, and is not now.** The pre-tuning diagnosis: a relevant
   claim was already ranked first *entering* stage 7 on 129 of 155 queries and first *leaving* it on
   40, because stage 7 multiplied a relevance signal whose range within one result list is **1.373x**
   by a freshness signal whose range is **499.8x**. That is fixed. On the whole set today the funnel
   loses nothing at any stage — `0/0/0/0/0/0` — and a relevant note is in the final top five on
   **157 of 157** scored questions. On the real pipeline, over the dev split, the same funnel is
   `0/0/0/0/0/0` and BM25 recall@50 and dense recall@50 are each **1.0000**.
2. **The real pipeline has now run, and it is not the deployed Worker.** Stages 2, 3 and 6 — BM25 over
   FTS5, Vectorize plus `bge-m3`, and the `bge-reranker-base` cross-encoder — now exist and have been
   measured end to end against **real Cloudflare services**: real `@cf/baai/bge-m3` query embeddings,
   real queries against the `signpost-eval` Vectorize index holding 459 real vectors, real
   `@cf/baai/bge-reranker-base` scores, over real FTS5 in a real local D1. That is
   `--retriever=hybrid`, and [round 8](#2026-08-13--round-8-final-numbers-and-the-verdict) is its record.
   **What is still not proven is the transport**: every remote call went over the Cloudflare REST API
   rather than through a Worker's `AI` and `VECTORS` bindings. And
   **the deployed instance still cannot search at all** — its Vectorize index holds
   **0 vectors**, `/health` reports `embedding_coverage: 0`, and deploying the index-building half needs
   a `wrangler` credential nobody in this environment has. Nothing in this document is a measurement of
   the live instance.

   > **Superseded 2026-08-14, noted 2026-08-18.** Both halves of that paragraph ended. The
   > transport was proven when the whole 196-question set ran against the deployment through the
   > Worker's own bindings, and the deployment searches: the table at the top of this document is
   > that run — `hit@5` 0.9936, `precision@1` 0.9427, `/health` `status: ok`, every chunk
   > embedded. The paragraph above stays as the dated record of 2026-08-13; do not read it as the
   > current state.

Written for a contributor who has never seen the code and has to improve the number. Nothing here
assumes you have read anything else.

**Sections 4 and 5 below were measured on 2026-08-10** against the code at commit `6224a2de` on
branch `build/waves-a1-a2`, on macOS 26.5.2 / Apple Silicon / Node v24.15.0. This document is not
committed. Every figure is next to the command that produces it. Where a number could not be
verified, it says so. Appendix D is the table of figures that earlier drafts of this document,
`eval/README.md` and several task briefs got wrong, with what they should be.

> **Sections 4 and 5 are the pre-tuning baseline and are deliberately left as measured.** They are
> the diagnosis this wave acted on, and rewriting them would destroy the record of why. What has
> changed since is in [section 0](#0-tuning-log), which is the current state of the number.

- [0. Tuning log](#0-tuning-log)
- [1. The pipeline](#1-the-pipeline)
- [2. Configuration](#2-configuration)
- [3. The eval](#3-the-eval)
- [4. Current results](#4-current-results)
- [5. Why it fails](#5-why-it-fails)
- [6. The fix plan](#6-the-fix-plan)
- [7. The decision reserved for a human](#7-the-decision-reserved-for-a-human)
- [8. Tuning guide: symptom to knob](#8-tuning-guide-symptom-to-knob)
- [Appendix A: the reference implementations](#appendix-a-the-reference-implementations)
- [Appendix B: the tokeniser is quadratic in one unbroken run](#appendix-b-the-tokeniser-is-quadratic-in-one-unbroken-run)
- [Appendix C: honest limitations](#appendix-c-honest-limitations)
- [Appendix D: documents that disagree with the code](#appendix-d-documents-that-disagree-with-the-code)
- [Appendix E: where the code is](#appendix-e-where-the-code-is)

---

## 0. Tuning log

One entry per change to the ranking, newest first. Each records what changed, the number before and
after **on the same command**, and what was learned — including from changes that were reverted,
because a failed hypothesis that is not written down gets retried.

Rounds 1 to 5 are measured on `--split=dev` (`eval/split.json`, 139 questions, 111 scored). A
`--split=dev` run can never pass `--gate`; the gate is the whole 196-question set. **The held-out
split was spent in round 6, after the tuning finished, and spent a second time in round 8.** It must
not be run again: every configuration change made after reading it is informed by it, and the next
held-out number would be a training number wearing a held-out label. `eval/runner.mjs` refuses
`--split=heldout` without `--final` for exactly that reason. Round 8's re-run is labelled a second
spend wherever its number appears, and [round 8](#2026-08-13--round-8-final-numbers-and-the-verdict)
states what it cost.

### 2026-08-13 — round 8: final numbers, and the verdict

**Outcome: nothing shipped in `packages/core` or `apps/worker`. This round re-ran every measurement,
brought the documentation up to the code, and answered the question the gate does not.** One source
change: seven `console.log` calls in `eval/hybrid/probe-corpus.mts` became `process.stdout.write`, the
convention every other file in `eval/` already follows, which took `pnpm lint` from 7 errors back to 0.

#### The final table

Reference retriever, `node eval/runner.mjs [--split=…]`, `DEFAULT_RETRIEVAL_CONFIG` unmodified, corpus
anchor `2026-08-09`, budget 4,000, no `--config` override:

| Metric | **Whole set** | **Held-out (2nd spend)** | **Dev** | Target |
| --- | --- | --- | --- | --- |
| questions / scored | 196 / 157 | 57 / 46 | 139 / 111 | — |
| **precision@5** | **0.7720** | **0.7391** | **0.7856** | **≥ 0.80** |
| precision@1 / @3 / @10 | 0.898 / 0.847 / 0.554 | 0.913 / 0.819 / 0.515 | 0.892 / 0.859 / 0.570 | — |
| **hit@5 / hit@10** | **1.000 / 1.000** | **1.000 / 1.000** | **1.000 / 1.000** | — |
| MRR / nDCG@10 | 0.937 / 0.815 | 0.947 / 0.829 | 0.933 / 0.809 | — |
| block-level precision@5 | 0.564 (73 q) | 0.582 (22 q) | 0.557 (51 q) | — |
| answer text returned in full | 0.898 | 0.891 | 0.901 | — |
| empty-recall / **false-confidence** | 0.941 / **0.059** | 1.000 / **0.000** | 0.917 / **0.083** | 0 |
| **used_tokens** median / p95 / max | **1,868.5** / 2,378 / 2,594 | 1,880 / 2,381 / 2,501 | 1,860 / 2,358 / 2,594 | **median < 2,500** |
| budget violations | **0** of 196 | 0 of 57 | 0 of 139 | 0 |

Real pipeline, `npx tsx eval/runner.mjs --retriever=hybrid --split=dev` — real `@cf/baai/bge-m3`, real
Vectorize `signpost-eval`, real `@cf/baai/bge-reranker-base`, real FTS5 over real D1:

| Metric | value |
| --- | --- |
| **precision@5** | **0.7748** |
| precision@1 / @3 / @10 | **0.955** / 0.874 / 0.572 |
| **hit@5 / hit@10** | **1.000 / 1.000** |
| MRR / nDCG@10 | **0.976** / 0.862 |
| block-level precision@5 | 0.596 |
| answer text returned in full | **0.955** |
| empty-recall / false-confidence | 0.917 / 0.083 |
| **used_tokens** median / p95 / max | **2,420** / 2,812 / 2,954 |
| budget violations | **0** of 139, worst 73.9% of budget |

**The reference-versus-real delta, stated plainly.** The real pipeline is **0.0108 below** the
reference on the gate metric and **above it on every other ranking measure**: `precision@1` +0.063
(0.892 → 0.955), MRR +0.043 (0.933 → 0.976), `nDCG@10` +0.053 (0.809 → 0.862), answer-text-returned-in-
full +0.054 (0.901 → 0.955), block-level `precision@5` +0.039 (0.557 → 0.596). Decomposed in round 6's
companion measurement: the real embedder is worth **+0.0450** `precision@5` (0.7856 → 0.8306 with no
cross-encoder) and the real cross-encoder is worth **-0.0558** (0.8306 → 0.7748) while simultaneously
buying +0.0451 `precision@1`. **That is the whole finding in two numbers.** The cross-encoder does its
job — it puts one correct note at rank 1 on 106 of 111 dev questions — and `precision@5` charges it for
spending the other four slots on different notes.

#### A third measurement, and why its higher number is worth less

`node eval/runner.mjs --retriever=worker --base-url=http://127.0.0.1:8798 --sign-in-local
--wrangler-config=.wrangler/eval/wrangler.eval.json --split=dev` now runs. It drives
`POST /api/recall` over HTTP against a real `wrangler dev` instance holding the real corpus — 100 notes,
459 chunks, `/health` `status: ok`, `embedding_coverage: 1`. It scores **`precision@5` 0.8180**, which
is the only number in this document above 0.80, **and it must not be quoted as a pass.** `env.AI` and
`env.VECTORS` are `not supported` in a local session without a credential, so the worker log carries
265 `recall_degraded` events (`stage=dense`, `stage=rerank`) and the run is **real FTS5 BM25 and
nothing else**. Its `hit@5` is **0.982**, the only sub-1.000 `hit@5` measured anywhere, and its
`false-confidence` of 0.000 is an artefact: every pack carries a degradation gap, so no pack can be
falsely confident. It is a useful proof that the HTTP boundary, the auth chain, real D1, real FTS5 and
the shipped packing all work end to end. It is not a retrieval score.

The three, side by side on the dev split:

| | reference | real BM25 only (worker, degraded) | full real pipeline |
| --- | --- | --- | --- |
| precision@5 | 0.7856 | **0.8180** | 0.7748 |
| precision@1 | 0.892 | 0.883 | **0.955** |
| hit@5 | **1.000** | 0.982 | **1.000** |
| MRR | 0.933 | 0.924 | **0.976** |
| answer text in full | 0.901 | 0.910 | **0.955** |
| false-confidence | 0.083 | 0.000 (artefact) | 0.083 |
| median used_tokens | 1,860 | 2,393 | 2,420 |

Read the middle column against the right one and the metric's problem is on the page: the configuration
that **cannot find the answer on two questions** scores 0.043 *better* on `precision@5` than the one
that finds it on all of them and ranks it first 7% more often.

#### NF1 latency, measured independently, and it fails

n=60 sequential production-shaped calls against real Cloudflare services, in two batches of 20 and 40
on 2026-08-13, plus 139 in-process observations from the hybrid run:

| leg | n | p50 | p90 | p95 | max |
| --- | --- | --- | --- | --- | --- |
| `@cf/baai/bge-m3`, one query | 60 | 120 | 177 | 200 | 515 |
| Vectorize query, `topK` 100, real filter | 60 | 355 | 366 | 369 | 386 |
| `@cf/baai/bge-reranker-base`, 80 contexts / 16,480 chars | 60 | 522–1,016 | 672–7,104 | 739–7,564 | 13,199 |
| **the three in sequence, pooled** | **60** | **1,088** | **6,274** | **7,588** | **13,663** |
| in-process (real FTS5, fusion, graph, 7–9, packing) | 139 | 23.9 | 28.9 | 31.4 | 63.8 |
| **end to end** | — | **≈ 1,112 ms** | — | **≈ 7,619 ms** | — |

**Against NF1's `recall` p95 under 800 ms: over budget by 1.4× at the median and 9.5× at p95.** It
fails at the *median*, before the tail is even considered.

Two things this measurement adds beyond round 6's. First, **Vectorize is not the problem and neither is
the embedder**: 355 ms and 120 ms at p50, and Vectorize is astonishingly tight (339–386 ms across all
60 calls). Second, **the cross-encoder's tail is not stable between sessions and that is the headline
risk**. Three independent samples of the same call with the same payload shape: 1 of 15 over 3 s, 1 of
20 over 3 s, then **15 of 40 over 3 s** with p95 7,564 ms. Round 6 measured 3 of 30 over 3 s and p95
7,554 ms — which agrees with the third sample almost exactly. So the range is real, it is
service-side, and a p95 quoted from a lucky sample is worthless. Budget for the bad session.

The transport caveat cuts the right way, as it did in round 6: the embed call shares the same REST
transport and costs 120 ms, so Vectorize's 355 ms and the reranker's 522 ms are service time, not
transport. **A binding-based Worker would still miss 800 ms at p95 by an order of magnitude.** Meeting
NF1 needs a design decision — run the two legs concurrently instead of sequentially, rerank fewer
contexts, skip rerank below a candidate-count threshold, or move the budget — not a parameter sweep.
See [section 8.7](#87-everything-is-worse-after-a-change-and-you-cannot-tell-why) for how not to
approach it.

#### NF1 measured per stage on the deployment, 2026-08-14 — and it is not the cross-encoder alone

The paragraph above says the cross-encoder is "the whole of it". Measured on the deployment, through
the bindings rather than REST, that is wrong by about half. `/api/recall` now reports its own
breakdown as a `Server-Timing` header and a `recall_timing` log line, so this is the endpoint
measuring itself rather than a harness measuring something adjacent.

n=40 sequential production-shaped calls, warm, dev-split questions, against the deployed Worker
before any change (version `89982957`):

| stage | port | p50 | p95 | share of p50 |
| --- | --- | --- | --- | --- |
| 1 intent | pure | 0 | 0 | — |
| 2 lexical (BM25) | D1 | 14 | 22 | hidden |
| — corpus_count | D1 | 14 | 22 | hidden |
| — index_health | D1 | 20 | 26 | hidden |
| — embed (`bge-m3`, + one D1 count) | Workers AI | 69 | 127 | 5.0% |
| 3 dense | Vectorize | 260 | 269 | 18.8% |
| 4 fuse | pure | 0 | 0 | — |
| — materialise | D1 | 65 | 88 | 4.7% |
| 5 graph_edges | D1 | 16 | 20 | 1.2% |
| 5 graph_chunks | D1 | 137 | 179 | 9.9% |
| 6 rerank (`bge-reranker-base`, 80 contexts) | Workers AI | 789 | 1,273 | 57.0% |
| 7–9 adjust, dedup, pack | pure | 0 | 0 | — |
| **total (pipeline)** | | **1,385** | **2,081** | |
| **total (client, incl. network)** | | **1,893** | **2,648** | |

Four things this says that the REST measurement could not.

**Steps 2 and 3 already run concurrently** — `pipeline.ts` has put them on one `Promise.all` since it
was written, so "run the two legs concurrently" was never available to do. The concurrent block costs
`max(14, 69 + 260, 14, 20)` = 329 ms: BM25 at 14 ms is free, and the block is exactly the dense leg.

**Bindings are much faster than REST, and the ranking of the costs changed with them.** Vectorize
260 ms against 355 ms over REST, the embedder 69 ms against 120 ms. The cross-encoder is 57% of the
median rather than all of it, and **the five D1 round trips after the concurrent block — materialise,
graph_edges, graph_chunks — are 218 ms, more than the embedder and Vectorize together.**

**The pure stages really do read 0, and they really are ~0.** `Date.now()` does not advance during
synchronous execution in a Worker, so steps 1, 4, 7, 8 and 9 report zero by construction. The tail's
`cpuTime` is the independent check and agrees: 79 ms at p50 for the whole request, 1,093 ms on a cold
start. Cold start is a real cost and it is one request in many, not a per-request one.

**Network is 508 ms at p50 from this workstation**, of which ~125 ms is Worker time outside the
pipeline (session lookup, rate limit, body) and ~383 ms is the wire. NF1 does not say which of the
two totals it is measured at; both miss.

#### What was changed, and what it bought — `rerankCandidateLimit`

Three candidates were costed against the measurement before one was written.

**Rejected: splitting the cross-encoder call into concurrent sub-batches.** The scores are pointwise,
so k concurrent calls return byte-identical scores, and the fixed cost is ~411 ms of a call that is
otherwise linear in the text sent (fitted over 137 production calls: `duration_ms ≈ 411.2 + 0.0226 ×
input_tokens`). It looks free. It is not: NF1 is a **p95**, and the wall time of k concurrent calls is
the *maximum* of k draws from a distribution whose tail is service-side and heavy — 8.8% of production
calls exceed 2× the median. Resampling the measured residual, 200,000 trials: k=2 improves p50 by
106 ms and moves p95 from 3,194 ms to **5,849 ms**; k=4 does not improve p50 at all. Batching trades
the number NF1 is written against for the one it is not.

**Rejected: caching query embeddings.** 69 ms at p50, and only on a repeat question.

**Shipped: `RetrievalConfig.rerankCandidateLimit`, default 30.** `recall` handed the cross-encoder
`fusedLimit + graphExpandLimit` = 80 contexts in order to choose the `rerankLimit` = 12 that survive
step 7. Step 5 leaves that list sorted by `compareScoredCandidates`, so cutting it keeps the
highest-scoring candidates rather than an arbitrary prefix — which is the difference between this and
the step-6 cut that was removed on 2026-08-13. It is applied **only when the model runs**: below
`rerankMinCorpus` there is no call to make cheaper, and truncating a fusion-ordered list that nothing
will rescore is that failure exactly. An abstaining reranker returns the **uncut** list, so the
invariant that an abstention and a skip produce the same bytes still holds — an existing test in
`packages/core/test/retrieval/pipeline.test.ts` caught the first version, which did not.

30 was chosen by sweep, not assumed. Reference retriever, dev split, `--reranker=lexical`, which
isolates the cut because whether the answer survives the top N is decided by fusion and graph order
rather than by the reranker:

| `rerankCandidateLimit` | `precision@5` | `hit@5` | MRR |
| --- | --- | --- | --- |
| ∞ (80) | 0.7387 | 0.9820 | 0.903 |
| 40 | 0.7414 | 0.9910 | 0.900 |
| **30** | **0.7441** | **0.9910** | 0.915 |
| 20 | 0.7315 | 0.9730 | 0.912 |

Measured on the deployment afterwards (version `eaa6ccee`), same instance, same corpus, only the knob
changed. Quality, dev split over `/api/recall`:

| | before | after |
| --- | --- | --- |
| `precision@5` | 0.7730 | 0.7712 |
| `precision@1` | 0.946 | 0.946 |
| `hit@5` | 1.0000 | 1.0000 |
| MRR | 0.970 | 0.970 |
| nDCG@10 | 0.860 | 0.865 |
| answer text in full | 0.955 | 0.946 |
| `used_tokens` median / violations | 2,409 / 0 | 2,410 / 0 |

Latency, n=100 sequential after the change, against n=40 before:

| stage | before p50 / p95 | after p50 / p95 |
| --- | --- | --- |
| rerank | 789 / 1,273 | **454 / 1,091** |
| total (pipeline) | 1,385 / 2,081 | **983 / 1,649** |
| total (client) | 1,893 / 2,648 | **1,180 / 1,867** |

And over the larger like-for-like sample — the 139-question dev split at concurrency 4, read from the
Worker's own `wallTime` — p50 1,510 → 1,154, p90 2,337 → 1,923, p95 3,804 → 2,489, max 16,587 →
7,727. Cost fell with it: **5.502 → 1.759 neurons per `recall`**, a 68% cut in the only thing this
product spends money on, and the dev split went from 739.9 to 309.2 neurons.

**NF1 still fails, and the arithmetic now says it cannot be met with a cross-encoder in the request
path.** 4 of 100 requests came in under 800 ms measured at the pipeline, 0 measured at the client.
Take the cross-encoder to zero and the floor is still `329 (dense leg) + 218 (D1 chain) + ~33 (CPU)` =
**580 ms at p50**, before a single model call and before the network — and one Workers AI call costs
411 ms of fixed cost whatever is in it, which is measured, not assumed. The remaining options are
structural (collapse the three sequential D1 round trips into one; drop graph expansion from the
request path) or the budget moves. That is a product decision and it is a person's to take, not an
agent's.

#### What round 8 verified, with the exit code

| command | result |
| --- | --- |
| `pnpm lint` | **exit 0**, 0 errors, 8 warnings (all in `eval/`; 7 pre-existing, 1 from `eval/hybrid/mcp-bridge.mjs`) |
| `pnpm typecheck` | **exit 0** |
| `pnpm test` | **exit 0** — 168 files, **6,760 tests**, 0 failures, 314.18 s |
| `pnpm licence-check` | **exit 0** — one dev-only LGPL exception, unchanged |
| `node eval/validate-questions.mjs` | **exit 0** — 196 questions, 0 errors, 13 warnings |
| `node eval/runner.mjs --gate` | **exit 1** — `precision@5` 0.7720 against 0.80. 14 of 15 gate checks pass |

Nothing was lowered, no test was weakened, no threshold moved, and the eval was not changed to flatter
a number. The one gate check that fails is `precision-at-5`.

#### The cost of the second held-out spend, stated so it is not repeated

Held-out `precision@5` reproduced at **0.7391**, four decimals identical to round 6. That is the useful
part: it proves round 7's two changes (the gap rules and `rerankLimit` 20 → 12) moved nothing on the
held-out ranking, which is what round 7 claimed from dev evidence and one spot-checked question id.
`false-confidence` on the held-out split went **0.200 → 0.000** and `empty-recall` **0.800 → 1.000**,
so round 7's open-question rule is confirmed out of sample. **The cost is that the held-out split is
now informed twice and can no longer be treated as clean.** Nothing was tuned after reading it, and
nothing should be. If a future round needs an honest out-of-sample number, it needs new questions
written against the corpus by someone who has not read the failures.

### 2026-08-13 — round 7: the two gates that were defects, not metric artefacts

**Outcome: both shipped. `false-confidence` 0.2353 → 0.0588 on the whole negative set and 0.250 →
0.083 on dev; median `used_tokens` 3,148 → 1,860 on the reference retriever and 3,836 → 2,420 on the
real pipeline. `precision@5` is unchanged to four decimals on both — 0.7856 reference, 0.7748 real.**
This round touches no ranking code. It changes `gaps.ts` and one constant in `types.ts`.

Command, before and after: `node eval/runner.mjs --split=dev`, and the same run with
`--retriever=hybrid` for the real cross-encoder.

| | reference before | reference after | real pipeline before | real pipeline after |
| --- | --- | --- | --- | --- |
| `precision@5` | 0.7856 | **0.7856** | 0.7748 | **0.7748** |
| `precision@1` | 0.8919 | 0.8919 | 0.9550 | 0.9550 |
| `hit@5` | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| MRR | 0.9333 | 0.9333 | 0.9760 | 0.9760 |
| **false-confidence** | 0.250 | **0.083** | 0.167 | **0.083** |
| empty-recall | 0.750 | **0.917** | 0.833 | **0.917** |
| **median `used_tokens`** | 3,148 | **1,860** | 3,836 | **2,420** |
| `nDCG@10` | 0.8379 | 0.8094 | 0.877 | 0.862 |
| answer text in full | 0.937 | 0.901 | 0.982 | 0.955 |

#### Both quantities section 6.3 and the task brief proposed were measured, and neither carries the signal

This is the part worth keeping. The dead `low_confidence` floor ([5.3](#53-the-low_confidence-relevance-floor-is-unreachable))
had two candidate replacements on the table. Both were measured on the dev split before anything was
written, and **both are wrong in the same way — they rank the retrieval, and the retrieval is fine.**

| candidate quantity | on the 3 dev false-confidence questions | verdict |
| --- | --- | --- |
| share of query terms the best claim contains ([6.3](#63-replace-the-low_confidence-floor-with-a-quantity-that-carries-information)) | **1.000, 1.000, 1.000** — the maximum | anti-correlated. A threshold that fires here fires on 61 of 111 positives |
| the cross-encoder's calibrated score | **0.0080, 0.686, 0.995** | 0.995 is the *highest* score in the whole negative set, and 24 of 111 positives score below 0.686 |

Nine further features were computed over all 139 dev questions and every one is flat: distinct notes
in the pack, top-to-second score ratio, top-to-median ratio, `noteSupport`, `linkConsensus`, `rrf`,
BM25 rank of the top claim, dense rank of it, and the fraction of the pack covering the whole query.
Positive p10/p50/p90 and negative p10/p50/p90 overlap on all nine.

**Why no score can work.** These three retrievals are *correct*. "What happens to a merchant that
fails KYC twice?" retrieves `architecture/merchant-onboarding`, which is exactly the right note, and
the cross-encoder scores it 0.995 because the note contains the question **verbatim** as a
`[question]` observation. A relevance model is being asked whether the passage is about the query. It
is. It is about the query and contains no answer, and those two are the same passage on every axis a
ranker has. The `neverJoinedGaps` rule gets closest and still cannot see it: on all three questions
the query's rarest term co-occurs with its least-joined partner in **exactly one claim of twenty**,
and the rule demands zero. Relaxing it to "at most one" fires on **59 of 99** positives.

#### What shipped instead: two rules that read the retrieved content

**1. `recordedQuestionGaps` — the knowledge base has written this question down as an open one.**
`- [question] …` is one of kmem's own observation categories (`blocks/observation.ts`,
`SUGGESTED_OBSERVATION_CATEGORIES`), and `observation_category` is a column in the PRD §3.4 schema.
A note carrying one has explicitly recorded that nobody knows. The rule parses each retrieved line
with the real `parseObservation` and fires when a `question` observation *is* this question.

Overlap is required in **both directions**, scored by the smaller. One direction alone fires on two
questions the corpus answers in full: "what does the vault hold" against a recorded "Do we need our
own token vault if the schemes already hold them?", and `conv-001` "how long is an idempotency key
good for" against a recorded question about idempotency records. Both reduce to one or two content
terms that the recorded question happens to contain. Two-directional, the rule fires on **0 of 111
positives, 0 of 12 adversarial, 1 of 12 negatives** — and every floor from 0.5 to 0.8 gives that same
verdict, so 0.75 is chosen inside a flat region rather than at an edge.

One implementation note that would have made this rule dead in production. `parseObservation` parses
a *source* line and needs the `- ` bullet. A chunk has none: `Block.text` for an observation block is
the line **after** its list prefix is split off — verified against the real parser, which returns
`"[question] What happens …"`. The rule puts the canonical bullet back rather than restating the
grammar, so there is still exactly one definition of what an observation is.

**2. `unmetDurationGaps` — the query asks how long and nothing retrieved says how long.** Round 5
already identified the shape: "in two of the three the words carrying the ask (`how long`, `how many`)
are question frames `DEFAULT_GAP_STOPWORDS` removes before the detector runs, correctly, because a gap
naming 'long' reads as a bug". So the frame is read rather than thrown away, and checked against the
claims that mention the query's **rarest** term rather than against the whole pack — a corpus that
states retention precisely for settlement files, logs and Kafka topics has durations everywhere, and
"the pack contains a duration" is true of it whatever was asked.

Dev split: 13 questions carry a duration frame, 11 of them scored. The rule fires on **both negatives
and none of the 11 positives.** That is round 5's own validation population, which named it as the
thing such a rule "would have to test … a new heuristic family needing its own validation".

`how many` was measured and **deliberately left out**. Its answers are bare numerals; the pack for
`neg-payout-failure-volume` already contains `40,000 events`, `312 sellers` and `90 minutes`, so the
rule is silent on the one question it would exist for, and any relaxation that fires there also fires
on `conv-016`, `pp-two-approvals-on-money-code` and `st-webhook-retry-schedule`.

`never` was in the open-ended-duration list for one revision and was removed. It is a duration in "the
token is never deleted" and an ordinary adverb in "cardholder names never appear in a log", and a
claim of the second kind silenced the rule. A word that only sometimes means a duration makes the gap
sentence sometimes false.

#### What it fixes, and the one it does not

| negative question | split | before | after |
| --- | --- | --- | --- |
| `neg-kyc-second-failure` | dev | no gap | `architecture/merchant-onboarding records this as an open question.` |
| `neg-warehouse-pii-retention-paraphrase` | dev | no gap | `No claim about ClickHouse says how long.` |
| `neg-commercial-card-false-positive-rate` | **held out** | no gap | `state/open-questions records this as an open question.` |
| `neg-payout-failure-volume` | dev | no gap | **still no gap** |

The third row is the only out-of-sample evidence this round has and it is worth stating plainly: the
open-question rule was designed against one dev question and, unprompted, fixes the held-out one as
well. Only the five negative questions were run by id — no `precision@k` was computed on the held-out
split and round 6's spend stands.

**`neg-payout-failure-volume` is not fixable by a lexical rule over the retrieved set, and this is the
honest limit of the round.** "How many payouts failed last month?" demands a count of failed payouts;
the corpus holds no operational payout volumes at all but holds a typical batch size and a seller
count from the double-payment incident, both of which sit beside the word `payout`. Every test that
fires on it also fires on questions the corpus answers. Whole-set `false-confidence` is therefore
**1 of 17 = 0.0588**, not 0.

#### Gap precision, the number that decides whether this was worth doing

Zero of 111 scored questions gained a gap. Diffing every question's `knowledgeGaps` against the
baseline, **exactly three changed and all three are negatives**; no `refs` list and no `used_tokens`
figure moved, which is the mechanical reason `precision@5` is identical rather than merely close.

#### `rerankLimit` 20 → 12, verified independently over a 7× sweep

Re-measured from scratch rather than taken from an earlier round. On the dev split
`precision@5` 0.7856, `precision@1` 0.8919, MRR 0.9333, `hit@5` 1.0000 and block-level 0.5569 are
**identical at 20, 16, 14, 12, 10 and 8**, while median `used_tokens` runs 3,148 / 2,538 / 2,172 /
1,860 / 1,584 / 1,291. On the real pipeline, 20 and 12 give identical `precision@5` 0.7748,
`precision@1` 0.9550, MRR 0.9760 and `hit@5` 1.0000 at median 3,836 and 2,420. The knob moves cost and
nothing else, because step 7 computes every list-relative factor over the whole candidate list and
cuts by the composed score afterwards.

12 is the largest value meeting PRODUCT.md §9's published sub-2,500 median **on both paths** — 8 and
10 are cheaper and buy nothing the product has asked for. The cost is depth: `nDCG@10` 0.8379 → 0.8094
and answer-text-in-full 0.937 → 0.901 on the reference, 0.877 → 0.862 and 0.982 → 0.955 on the real
pipeline.

> **This contradicts PRD §4.1 step 6, which reads "rerank (bge-reranker) → top 20", and it closes a
> question the project owner had explicitly reserved** ("The shipped value
> is still 20; this is still yours to take"). It was taken here because this round was briefed to
> verify and ship it. **The specification owes an amendment for that line**, and the divergence is
> asserted rather than hidden: `pipeline.test.ts` pins `rerankLimit: 12` with a comment naming the text
> it now disagrees with. Reverting is one constant. The reason it was a product decision still stands
> and is now the live question — a caller can raise `budget_tokens` and cannot raise `rerankLimit`, so
> if twenty claims is a real use case this has to become a request parameter.

#### What did not move

`precision@5` 0.7856 dev / 0.7748 real, `hit@5` 1.0000, and the funnel's `0/0/0/0/0/0`. The gate is
still red at 0.80 and nothing in this round was aimed at it. All four safety probes pass, `used_tokens
<= budget_tokens` on 139 of 139 with the worst pack at 64.8% of budget, "no pack has empty claims and
empty gaps" on 139 of 139, and the corpus-gap traps still hold. `pnpm test:core` 4,010 pass,
`pnpm test:worker` 2,750 pass, `pnpm typecheck` clean, `biome` clean over `packages/core`.

### 2026-08-13 — round 6: the held-out measurement, and the honest verdict

**Outcome: no change to the ranking, and the tuning is closed.** `packages/core`, `apps/worker` and
`eval/` are untouched by this round — the only files it changes are this one, `README.md` and the
regenerated `eval/results/`. What it produced is the number the project gets to report and the
evidence for reading it.

#### The three splits, one command each

All three at the shipped `DEFAULT_RETRIEVAL_CONFIG`, the corpus anchor `2026-08-09`, budget 4,000,
`--reranker=none`, no `--config` override:

```sh
node eval/runner.mjs --split=heldout --final     # spent once, 2026-08-13
node eval/runner.mjs --split=dev
node eval/runner.mjs                             # the whole 196-question set; --gate exits 1
```

| | **held-out** | **dev** | **whole set** |
| --- | --- | --- | --- |
| questions / scored | 57 / **46** | 139 / **111** | 196 / **157** |
| **`precision@5`** | **0.7391** | **0.7856** | **0.7720** |
| 95% bootstrap CI | [0.6696, 0.8087] | [0.7387, 0.8306] | [0.7325, 0.8102] |
| `precision@1` | **0.9130** | 0.8919 | 0.8981 |
| `precision@3` | 0.8188 | 0.8589 | 0.8471 |
| `precision@10` | 0.5239 | 0.5757 | 0.5605 |
| **`hit@5` / `hit@10`** | **1.0000 / 1.0000** | **1.0000 / 1.0000** | **1.0000 / 1.0000** |
| MRR | **0.9475** | 0.9333 | 0.9375 |
| nDCG@10 | **0.8451** | 0.8379 | 0.8400 |
| `recall@5` | **0.8486** | 0.8266 | 0.8330 |
| distinct-note `precision@5` | 0.3783 | 0.4144 | 0.4038 |
| block-level `precision@5` | **0.5818** (22 q) | 0.5569 (51 q) | 0.5644 (73 q) |
| answer text in full | 0.913 | 0.937 | 0.930 |
| oracle `precision@5` ceiling | 0.9913 | 0.9964 | 0.9949 |
| `used_tokens` median / p95 / max | 3,155 / 3,820 / 3,865 | 3,148 / 3,825 / 3,917 | 3,151.5 / 3,820 / 3,917 |
| budget violations | **0** | **0** | **0** |
| empty-recall / false-confidence | 0.800 / 0.200 | 0.750 / 0.250 | 0.765 / 0.235 |

The CI is the percentile bootstrap over per-question `precision@5`, 20,000 resamples, seed 20260813.
**All three intervals contain 0.80.** That is not an argument that the gate is met; it is the
statement that this eval set, at these sizes, cannot resolve the difference between 0.74 and 0.80.

#### The overfitting gap: 0.0465, real in sign, not resolvable in size

Five rounds of tuning against dev means some gap is expected. It is **0.0465** (dev 0.7856 minus
held-out 0.7391). Three readings, in the order that matters:

1. **It is not statistically distinguishable from zero.** The two-sample bootstrap on the difference
   (20,000 resamples, seed 777) gives **[-0.0360, 0.1322]** with `P(gap <= 0) = 0.139`. At 46 held-out
   questions a gap of this size is one standard error.
2. **It is not a question-mix artefact.** Standardising each split onto the other's composition moves
   it almost not at all: on difficulty **0.0460**, on source file **0.0444**, on the size of the
   question's relevant set **0.0435**, against a raw 0.0465. The split is stratified over
   (file, difficulty, intent) and the stratification did its job.
3. **It is concentrated in twelve questions.** Decomposed by difficulty and weighted by the held-out
   mix: `medium` contributes **-0.0000** (0.8000 on both splits, n=58 and n=23), `easy` **+0.0064**
   (0.7538 / 0.7273), and `hard` **+0.0396** — 0.7852 on dev against 0.6333 on 12 held-out questions.
   **85% of the gap is twelve questions.** By file, `people` (0.8714 / 0.6667, n=14 / n=6) and `state`
   (0.8400 / 0.7500, n=15 / n=4) are the two other small cells that move; `runbooks` (0.8516 / 0.8286,
   n=31 / n=14) and `architecture` (0.6500 / 0.6857) barely do, and `decisions` is *better* held out.

**The verdict on overfitting: mild, real in sign, and not the reason the gate is missed.** The
decisive evidence is which metrics moved. Every reading that measures *whether the right note was
found* is equal or better on the held-out split — `hit@5` 1.0000 on both, `precision@1` 0.9130 against
0.8919, MRR 0.9475 against 0.9333, nDCG@10 0.8451 against 0.8379, `recall@5` 0.8486 against 0.8266,
block-level `precision@5` 0.5818 against 0.5569. If five rounds had fitted the *ranking* to 111
questions, `precision@1` and MRR would be the first casualties and they are the largest gains. What is
worse held out is `precision@3`, `@5`, `@10` and the distinct-note reading — and mean distinct notes in
the top five is 2.4565 held out against 2.5766 on dev. Round 5 proved `precision@5` here is a monotone
decreasing function of pack diversity in both directions. **The held-out packs are more diverse, and
this metric charges for that.** That is not the whole 0.0465, but it is the direction of most of it.

#### Slot accounting and the honest ceiling, on all three splits

The same instrument as round 4, so the three are comparable. A *forced* miss is a top-five slot the
pack could not fill with a relevant chunk; an *ordering error* is one it could and did not.

| | held-out (230 slots) | dev (555 slots) | whole set (785 slots) |
| --- | --- | --- | --- |
| relevant slots | 170 | 436 | 606 |
| forced | 22 | 46 | 68 |
| ordering errors | **38** | **73** | **111** |
| relevant slots that repeat a note already counted | 99 | 245 | 344 |
| duplicate *irrelevant* slots | 18 | 24 | 42 |

Oracles over the **shipped pack** — the best reordering of what the packer already returned, under a
constraint. `at most n per note` caps duplication; `promotion allowed` lets a note not currently in the
top five take one slot and caps every note already there at the slots it already holds:

| oracle over the shipped pack | held-out | dev | whole set |
| --- | --- | --- | --- |
| control (no reordering) | 0.7391 | 0.7856 | 0.7720 |
| at most 1 chunk per note | 0.4043 | 0.4486 | 0.4357 |
| at most 2 chunks per note | 0.7304 | 0.7622 | 0.7529 |
| **promotion allowed, no note exceeds the slots it holds** | **0.8130** | **0.8523** | **0.8408** |
| at most 3 chunks per note | 0.8652 | 0.8739 | 0.8713 |
| unconstrained | 0.9043 | 0.9171 | 0.9134 |

The dev column reproduces round 4's numbers to four decimals, which is the check that the oracle is the
same instrument. **The gate is reachable on every split without any extra duplication** — 0.8130,
0.8523, 0.8408, all above 0.80 — by finding the promotions six rounds of feature work did not find.
That is the same conclusion round 4 reached on dev, now confirmed on questions no tuning has seen.

#### What a user actually experiences, which is the reading the headline number hides

`precision@5` is not the question a user asks. The question is: *when I ask this knowledge base
something it knows, does it tell me?*

- **A relevant note is in the top five on 157 of 157 scored questions**, on every split. `hit@5` is
  exactly 1.0000 held out, on dev and on the whole set. There is no question in the committed eval set
  where the answer is missing from the pack.
- **It is ranked first on 89.8%** of the whole set, 91.3% held out. The rank of the *first* relevant
  claim, over all 157 scored questions: **1 on 141, 2 on 8, 3 on 4, 4 on 1, 5 on 3, and never worse
  than 5.** Mean 1.197, median 1, MRR 0.9375.
- **The funnel loses nothing.** `0 never retrieved, 0 at fusion, 0 at rerank, 0 at stage 7, 0 at stage
  8, 0 at stage 9` on all three splits. Stage 7 moves the best relevant claim on 35 of 157 questions:
  27 up, 8 down, mean -0.62 ranks.
- **The answer text comes back in full on 93.0%** of questions; every claim is a real block with a real
  permalink.
- **`used_tokens` never exceeds `budget_tokens`**, worst case 97.925% of a 4,000 budget over all 196.

What the 0.7720 is measuring, then, is that of the five claims returned, on average 3.86 come from a
note judged relevant and 1.14 do not. The other 1.14 are not noise from a broken retriever — the pack
holds a relevant chunk lower down for 111 of those slots.

**The one number that is a genuine user-facing defect** is `falseConfidence` **0.235**: 4 of the 17
negative questions return a full pack of claims and state no gap. That is the pipeline answering
confidently about something the corpus does not contain, which is the failure PRODUCT.md 9 cares most
about, and it is the same 4 questions on the whole set as the 3 on dev plus one. Round 5 measured the
fix section 6.3 prescribes and proved it cannot fire; see that entry before touching it.

> **Superseded by round 7, 2026-08-13.** Three of the four now state a gap and `falseConfidence` is
> **0.0588** — 1 of 17. The survivor is `neg-payout-failure-volume`, and the round 7 entry argues it
> is not reachable by any lexical rule over the retrieved set. The figures in this round's tables
> below are left as they were measured.

#### Verification run beside the measurement, all offline

The PRD phase 2 automated gate had 15 checks at the time of this run. **14 pass; `precision-at-5`
is the only failure**, on `node eval/runner.mjs --gate` (exit 1). Two of the 15 no longer exist:
since ADR 0027 the superseded and soft-deleted probes run once each, not twice, so the same gate
would have 13 checks today. Named individually because the task asked for them:

| gate | result | how |
| --- | --- | --- |
| `used_tokens <= budget_tokens` on 100% of queries | **pass** | 196 queries, worst 97.925% of budget (`neg-merchant-fee-schedule`, 3,917 of 4,000), 0 violations |
| …including adversarial inputs | **pass** | 22 adversarial questions, 0 violations, worst 90.5%; queries from 0 to 13,999 characters |
| empty knowledge base → `claims: []` and non-empty `gaps` | **pass** | 3 probes: a full question, the empty string, one bare word |
| never both empty | **pass** | 196 packs, 0 silent |
| superseded content never appears | **pass** | 2 probes, `includeUnverified` false and true. One probe since ADR 0027 removed the flag |
| soft-deleted content never appears | **pass** | 2 probes, both settings. One probe since ADR 0027 |
| a 400-day-old `state` note below a 5-day-old one on equal relevance | **pass** | `npx vitest run --project core -t "PRD phase 7 gate"` — 4 tests |
| determinism: the same query twice gives an identical pack | **pass** | 33 tests across 15 files, plus two independent 196-question runner processes whose packs are byte-identical over 1,454,110 bytes of serialised claims, gaps, conflicts and token counts |
| the shipped configuration was used | **pass** | `DEFAULT_RETRIEVAL_CONFIG`, unmodified |
| every committed question was scored | **pass** | all 196 |
| the corpus anchor and default budget were used | **pass** | now = 2026-08-09, budget = 4,000 |
| gap premises still hold | **pass** | 10 questions pinned to `eval/corpus/gaps.json` |
| adversarial contracts | **pass** | 22 inputs met their own `expect` block |
| **`precision@5` >= 0.80** | **FAIL** | **0.7720 over 157 scored** |

Suite, same day, same tree: `pnpm test` **168 files / 6,754 tests / exit 0**; `pnpm typecheck` exit 0;
`pnpm lint` exit 0 with 7 warnings, all in `eval/` (6 complexity, 1 for the size of the committed
`eval/results/results.json`), none in `packages/` or `apps/`; `pnpm licence-check` passed;
`node eval/validate-questions.mjs` 0 errors, 13 warnings. The four named non-negotiables re-run by
name on core: `-t "phase 7"` 7, `-t "superseded"` 9, `-t "budget"` 39, `-t "gaps"` 132 — the same
counts as rounds 1 to 3. `npx vitest run --project worker apps/worker/test/services/agents-md.test.ts`
54 passed.

#### What this does not measure, and it is most of the product

**No Cloudflare credential was available for this round, and none of the three numbers above is the
pipeline the PRD specifies.** `--retriever=reference` runs entirely in process over
`eval/corpus`: stage 3 is a signed-hashing embedder rather than `@cf/baai/bge-m3`, stage 6 is
`--reranker=none` so no cross-encoder executes at all, and stage 2 is an in-memory BM25 rather than
FTS5 over D1. The hashing embedder is measurably *worse than having no dense retriever*
(section 5). Nobody has ever measured what a real cross-encoder separates on this corpus, and section 7
names that as the single number the whole plan is most sensitive to.

What is written and unrun for want of a credential is, in short: provisioning `signpost-eval` and its
seven metadata indexes; confirming
`signpost-chunks` carries all seven, which nothing inside the Worker can answer because the Vectorize
binding cannot list metadata indexes; the 459-chunk embed through `bge-m3`; `/health` coverage reaching
1.0; D1-versus-Vectorize parity; the self-neighbour probe; `embed-to-dense.remote.test.ts`; the
`--retriever=worker` measurement against an indexed instance; and the live deploy.

#### What round 7 should do, and what it must not

**Do not open another tuning round against this split.** Rounds 1 to 5 closed the lexical feature space,
the relation graph, the floor surface, duplication in both directions, the list-size axis and three
further consensus spaces. Round 6 adds that the split cannot resolve the remaining 0.028 in any case.

The work that is left is [6.4](#64-land-stage-6-and-recalibrate-everything-that-reads-a-relevance-score-in-the-same-change):
land stages 3 and 6 against real Vectorize and real Workers AI, re-run `--retriever=worker`, and read
the number that comes out. **Everything stage 7 does is calibrated against a relevance base whose range
within one list is 1.373x.** A cross-encoder changes that range, and every floor in
`DEFAULT_RETRIEVAL_CONFIG` — `freshnessFloor`, `supportFloor`, `coverageFloor`, `noteCoverageFloor`,
`linkConsensusFloor` — is a bound expressed relative to it. They will all need re-sweeping in the same
change, and the sweeps in rounds 1 to 5 are the method, not the answers.

Two guard rails for whoever does it. The oracle rows in section 6 predict **0.943 to 0.955** with a
perfect cross-encoder and **0.749** with the rejected clamp; neither predicts `bge-reranker-base`. And
D2's own stop condition still stands: **if stage 6 lands and the number does not move past 0.590, the
design is the problem rather than the missing halves.**

### 2026-08-13 — round 5: nothing shipped. `precision@5` is a duplication metric, now proved from both sides

**Outcome: no change to the ranking. `precision@5` stays 0.7856.** Round 4 concluded that the last
0.0144 is reachable only by duplicating notes, from a set of changes that all happened to *increase*
duplication. That is a one-sided argument, and a one-sided argument about a metric is exactly the kind
this file has had to retract before. This round closed it from the other side, measured four hypotheses
nobody had tried, and found that the fix `6.3` prescribes for the one remaining live defect **cannot
fire on any of the questions it was written for**. Nothing here beat the control, so nothing ships.

#### The headline: the metric is monotone in duplication, in both directions

Round 4 measured `noteLead` — cluster a note's chunks — and got +0.0180 by adding 23 duplicated slots
and 0 promotions. The mirror image had never been run. A **within-note rank discount** is the same
knob turned the other way: the *k*-th chunk of a note in score order is multiplied by a shrinking
factor, so a note cannot hold consecutive slots cheaply. Four shapes, twenty-five settings, over the
shipped packs:

| discount | setting → `precision@5` |
| --- | --- |
| linear, `× (1 − dk)` | 0.02 → **0.7802** · 0.05 → 0.7622 · 0.08 → 0.7405 · 0.1 → 0.7297 · 0.15 → 0.7153 · 0.2 → 0.6685 · 0.3 → 0.6216 |
| geometric, `× r^k` | 0.98 → **0.7802** · 0.95 → 0.7640 · 0.9 → 0.7351 · 0.85 → 0.7207 · 0.8 → 0.6883 · 0.7 → 0.6378 |
| harmonic, `× 1/(1+dk)` | 0.05 → 0.7640 · 0.1 → 0.7387 · 0.2 → 0.7189 · 0.5 → 0.6396 |
| first *N* free, then linear | free2: 0.7748 / 0.7604 / 0.7369 / 0.6865 · free3: **0.7802** / 0.7730 / 0.7658 / 0.7532 |

**Every cell is below the control, at every strength, and the loss is monotone in the strength.** The
gentlest setting measurable — a 2% discount on the second chunk of a note — already costs 0.0054. And
the diagnostic is the exact inverse of `noteLead`'s: at `d = 0.3` the discount puts **28 relevant notes
into the top five and removes none** (`noteLead` at floor 0.5 removes nine and adds none), raises mean
distinct notes 2.577 → 3.757, and scores **0.6216**. A hard per-note cap says the same thing without
the tuning: caps of 1 / 2 / 3 / 4 score **0.4198 / 0.6432 / 0.7441 / 0.7784**, the cap of 1 buying 39
promotions at the price of 242 lost slots.

So `precision@5` on this eval is a monotone decreasing function of pack diversity over the whole
measurable range. Round 4's conclusion was right and is now two-sided: **there is no ordering of the
shipped pack that raises the published number without making the answer more repetitive.**

**And `distinct-note precision@5` cannot referee this, as a matter of arithmetic rather than of luck.**
It is exactly 0.4144 in all twenty-five rows above and in all seven of round 4's `noteLead` rows. The
reason is structural: `eval/runner.mjs` builds it from `distinct(retrieved)`, which keeps each note's
*first* occurrence, and any re-weighting of the form `score × f(within-note index)` preserves the order
of first occurrences whatever `f` is. Round 4 found this empirically for one family and read it as a
metric being blind; it is the whole family, in both directions, and it is a theorem. **Do not cite that
reading as corroboration for anything that reorders a note's own chunks.** `hit@5` and block-level
`precision@5` do register these changes and are the two to publish beside any future attempt.

#### The four hypotheses this round measured for the first time

1. **The list-relative factors are computed over the wrong list, and fixing that loses.** Since round 2
   the cut to `rerankLimit` happens at the *end* of stage 7, so `noteSupport`, `noteCoverage` and
   `linkConsensus` are aggregated over all **80** candidates while only **20** survive — a consensus
   statistic describing a list that is then thrown away. Recomputing all three over the survivors and
   re-sorting (a second pass, no new port call) scores **0.7838** against 0.7856, with `hit@5` 1.0000 →
   0.9910 and block-level 0.5569 → 0.5490. Worse on all three readings. The idea is sound and the corpus
   disagrees; the aggregates are apparently better estimated from the wider list than from the narrower
   one, which is the opposite of the pseudo-relevance-feedback intuition that motivated it.
2. **Three more consensus spaces, none as good as the edges.** Round 3 took the relation graph because
   the lexical space was closed. The three remaining author-written structures were screened the same
   way — pairwise over the 359 (irrelevant claim in the top five, relevant claim below it) pairs, then
   swept as a bounded multiplier. **Tag consensus** (other candidate notes sharing a tag) separates
   63.8% of the pairs it separates at all and tops out at **0.7892**; **note-type consensus** 57.3% and
   **0.7874**; **vocabulary consensus** (share of a chunk's terms appearing in ≥ 3 of the list's notes)
   59.7% and **0.7892**. Against `linkConsensus`'s 76.8% when it was chosen, and every one of the three
   reaches its maximum with the duplication signature (`dup` gains outnumbering promotions 3:1 or worse)
   and a *falling* block-level reading. Structural features — heading depth, chunk position within its
   note, chunk length, block count, note size — all screen between 44% and 61% and every one loses.
3. **The candidate-list size is now completely irrelevant to precision.** `rerankLimit`
   12 / 16 / 20 / 26 / 30 / 40 / 60 / 80 all score **0.7856**, with block-level 0.5569 and `hit@5`
   1.0000 unchanged at every value, at median `used_tokens` of 1,860 / 2,538 / **3,148** / 3,826 /
   3,848 / 3,846 / 3,840 / 3,833. Round 3 established this over 12–24; over a 7× range it is not a
   plateau, it is the statement that **the top five is decided entirely by the composed score and not
   at all by how many candidates it ranks**. The `rerankLimit` cut is therefore now a pure cost
   decision with no precision term at all: 20 → 12 saves 1,288 median tokens for nothing.
4. **Stage 8 does not exist on this corpus.** `semanticDedup` collapses nothing on **0 of 111** dev
   questions — `adjusted` is 20 and `kept` is 20 every time — so the "tighten dedup" lever has no
   surface to act on and the pack's repetitiveness is not dedup's failure to fire. It is not obviously
   a bug: complete-link clustering at 0.92 over a *hashing* embedder is expected to be conservative, and
   `dedup.ts` argues at length that over-merging costs answers where fragmenting costs tokens. It does
   mean **no number in this file has ever exercised stage 8**, which is worth knowing before anyone
   tunes `dedupCosine` against a real embedding model.

#### The fix `6.3` prescribes for `low_confidence` cannot fire, and here is the measurement

This is the correction most worth having. [6.3](#63-replace-the-low_confidence-floor-with-a-quantity-that-carries-information)
proposes replacing the dead relevance floor with "the fraction of the query's non-stopword terms that
the best claim actually contains", emitting the gap below a threshold, and names
`falseConfidenceRate` 0.2353 → 0 as the expected effect. **That quantity is exactly 1.000 on all three
of the dev split's false-confidence questions.** The best claim already contains every word they use:

| negative question | best-claim coverage | gap emitted today |
| --- | --- | --- |
| How long do we keep cardholder names in ClickHouse? | **1.000** | none |
| How many payouts failed last month? | **1.000** | none |
| What happens to a merchant that fails KYC twice? | **1.000** | none |
| the other nine dev negatives | 0.400 – 0.833 | one or more, already |

No threshold separates them. At 1.0 the rule fires on 9 of 12 negatives — every one of which already
emits a gap by another rule — and on **61 of 111 positive questions**, which is the "cries wolf"
failure `gaps.ts` opens by warning against. At 0.6 it is 5 negatives and 15 positives. The proposed fix
buys nothing and costs the detector's credibility.

The reason is visible once the detector is traced per question. Every negative that *does* emit a gap
has a query term with document frequency **0** in the pack, and `absentTermGaps` fires on it
(`neg-failover-duration` is the single exception and is caught by `neverJoinedGaps`). The three
false-confidence questions have **every** term present *and* every pair of terms co-occurring inside
some single claim, so neither rule can reach them. What they actually ask for — a retention period, a
count, the consequence of a second failure — is a *quantity or a condition the corpus never states*,
and in two of the three the words carrying that ask ("how long", "how many") are question frames that
`DEFAULT_GAP_STOPWORDS` removes before the detector runs, correctly, because a gap naming "long" reads
as a bug.

**So `falseConfidence` 0.25 is not a mis-tuned threshold and not a missing lexical rule.** Section 6.3
should be rewritten before anyone implements it. The only lead with any evidence behind it: two of the
three are quantity-shaped questions (`how long`, `how many`), a shape 14 of 111 positives also have, so
a rule would have to test "asks for a quantity **and** no claim states one in a matching unit" — a new
heuristic family needing its own validation against those 14, which is more than a tuning round.

#### What this leaves

The pack ceiling is unchanged at **0.9171** and round 4's honest ceiling at **0.8523**; the funnel still
loses nothing (`0/0/0/0/0/0`) and the five-relevant-chunk ceiling reads 0.9640 entering stage 7, 0.9351
after the cut, 0.9351 after dedup, 0.9171 in the pack. 555 top-five slots hold 436 relevant, 46 forced
and 73 ordering errors, of which **245 of the 436 are a second-or-later chunk of a note already
counted** and only 24 duplicate slots are held by irrelevant notes — which is the whole reason every
diversity change trades at 10:1 against itself. Rounds 1 to 5 have now closed the lexical space (round
1), the graph (round 3), the floor surface (round 4), duplication in both directions (round 5), the
list-size axis (round 5) and three further consensus spaces (round 5). Read with round 4's confidence
interval — **[0.7351, 0.8234]**, which contains 0.80 — the honest position is that this split has been
tuned as far as it can distinguish, and the next real gain is stages 3 and 6 (a semantic embedder and a
cross-encoder), not a tenth multiplier. See [6.4](#64-land-stage-6-and-recalibrate-everything-that-reads-a-relevance-score-in-the-same-change).

### 2026-08-13 — round 4: nothing shipped. The last 0.0144 is note duplication, and that is measured

**Outcome: no change to the ranking. `precision@5` stays 0.7856.** Six hypotheses were implemented or
swept and every one was reverted. This entry exists because the most useful thing round 4 produced is
the reason the remaining gap is not worth closing the way it is reachable, and because round 3 handed
round 4 a specific recommendation that turns out to be the wrong one to take.

#### The headline: `noteLead` clears the gate and makes the answer worse

Round 3 measured `noteLead` — every chunk inherits a share of its own note's best composed score,
`floor + (1 - floor) × (this note's best ÷ the list's best)` — at **0.8018** (floor 0.6) and **0.8036**
(floor 0.5), and recommended it for this round. It was implemented in `decay.ts` behind a
`noteLeadFloor` knob whose `1` is bit-identical to the control, swept on the real runner, and
**reproduces exactly**:

| `noteLeadFloor` | p@5 | p@1 | hit@5 | MRR | block-level | median tokens |
| --- | --- | --- | --- | --- | --- | --- |
| **1 (control)** | **0.7856** | 0.8919 | **1.0000** | **0.9333** | 0.5569 | 3,148 |
| 0.8 | 0.7946 | 0.8919 | 1.0000 | 0.9302 | 0.5647 | 3,121 |
| 0.7 | 0.7946 | 0.8919 | 0.9910 | 0.9285 | 0.5647 | 3,121 |
| 0.6 | **0.8018** | 0.8919 | 0.9910 | 0.9285 | **0.5725** | 3,109 |
| 0.5 | **0.8036** | 0.8919 | 0.9910 | 0.9270 | 0.5686 | 3,104 |
| 0.4 | **0.8054** | 0.8919 | 0.9910 | 0.9270 | 0.5686 | 3,096 |
| 0.3 | 0.8036 | 0.8919 | 0.9820 | 0.9270 | 0.5686 | 3,095 |

Then the gain was decomposed against the control, claim by claim, in the real pipeline
(`--output=` on both runs). At floor 0.5:

| | |
| --- | --- |
| relevant top-five slots gained by **duplicating a note already in the top five** | **23** |
| relevant top-five slots gained by **promoting a note that was not there** | **0** |
| relevant top-five slots lost | 13 |
| distinct relevant notes **entering** the top five | **0** |
| distinct relevant notes **leaving** the top five | **9** |
| mean distinct notes in the top five | 2.577 -> **2.396** |

**It scores +0.0180 by removing nine relevant notes from the caller's top five and adding none.**
Every floor in the table has the same shape; at 0.8 it is 12 duplicated slots and 0 promotions.
`distinct-note precision@5` is *exactly* unchanged at 0.4144 at every floor, and round 3 read that as
reassurance. It is not: `eval/runner.mjs` builds that metric from `distinct(retrieved)`, which keeps
**the first occurrence of each note and drops the rest**, so clustering a note's chunks cannot move it
by construction. The metric is blind to this change rather than exonerating it. `hit@5` falling
1.0000 -> 0.9910 is the one published number that does register it, and it registers it as a loss.

This is the shape round 1 rejected twice. It is not shipped, and it should not be shipped in round 5.

#### Why no honest change was found, with the ceiling that says one exists

Slot accounting over the 555 top-five slots on the dev split: **436 relevant, 46 forced** (the pack
does not hold five relevant chunks) and **73 ordering errors**. Splitting the 73 by what would fix
them: **37 need a relevant note promoted that is not in the top five, 36 need a second chunk of a note
that already is.** So honest headroom exists, and four oracles bound it — each the best reordering of
the *shipped pack* under a constraint:

| Oracle over the shipped pack | precision@5 |
| --- | --- |
| control (no reordering) | 0.7856 |
| at most one chunk per note | 0.4486 |
| at most two chunks per note | 0.7622 |
| **promotion allowed, no note exceeds the slots it already holds** | **0.8523** |
| at most three chunks per note | 0.8739 |
| unconstrained | 0.9171 |

**The gate is reachable without any extra duplication — 0.8523 against 0.80.** No feature tried this
round finds those 37 promotions. Across every variant measured, `precision@5` tracked the number of
duplicated slots and was uncorrelated with the number of promotions: `noteLead` 23 duplications / 0
promotions scores 0.8036; `max(termCoverage, noteCoverage)` 35 / 5 scores 0.7874; `min` of the same two
3 / 4 scores 0.7604. That relationship is the finding.

#### The five other hypotheses, all reverted

1. **Query-term selection is not the problem, though it looked like it.** `termCoverage` and
   `noteCoverage` are keyed on `gaps.ts`'s `queryTerms`, which drops any word under
   `gapMinTermLength` 4 that is not a proper noun. On `arch-016` — *"when do we send a buyer to 3ds"* —
   the terms are `send` and `buyer`; **`3ds`, the subject of the question, is dropped**, and the two
   coverage factors then rank `runbooks/merchant-data-export` (covers 0.68 / 0.80) above
   `decisions/sca-delegation` (0.35 / 0.60) despite the two having the *same* RRF to three digits
   (0.03175 against 0.03178) and `sca-delegation` being BM25 rank 1. 32 of 111 scored questions
   (28.8%) drop a short non-stopword word this way. **Fixing it is worth nothing**: `gapMinTermLength`
   3 and 2 both score `precision@5` **0.7856**, unchanged to four decimals, with block-level falling
   0.5569 -> 0.5529 and `hit@5` 1.0000 -> 0.9910 against a distinct-note reading that rises 0.4144 ->
   0.4198. The dropped words carry *less* information than the kept ones on average — mean corpus IDF
   3.182 against 3.504 — so admitting them adds as much noise as signal. Raising the floor to 5 is
   worse (0.7802). **The diagnosis was right and the fix is worthless**, which is worth writing down
   because the diagnosis is very convincing to read. Note also that this is the one change measured
   this round that moves the *distinct-note* reading up while leaving the gate metric alone, which is
   the opposite trade to everything else here and may be the honest thread for round 5 to pull.
2. **IDF-weighted coverage is monotonically worse, now confirmed on the larger list.** Weighting each
   term's contribution by `log((L + 1) / (df + 1)) ** strength` over the `L` candidates stage 7 holds,
   `strength` 0 being bit-identical to the unweighted share: 0.25 / 0.5 / 0.75 / 1 / 1.5 / 2 / 3 score
   **0.7874 / 0.7820 / 0.7766 / 0.7712 / 0.7676 / 0.7604 / 0.7261**, with block-level falling at every
   step. This was measured once before, on 2026-08-12, when stage 7 saw a 20-candidate list; the cut
   moved on 2026-08-13 and it now sees 80, which is the only reason it was worth re-running. It is the
   same answer. **The unweighted share stands; stop re-deriving this.**
3. **Intent classification is not worth repairing.** Stage 1 leaves `typeWeight` at exactly 1 on
   **71 of 111** scored questions and fires no cue at all on 68. But an *oracle* that knows the
   majority note type of each question's relevant set and boosts it tops out at **0.8090** (at ×2;
   ×1.25 gives 0.7982, ×1.5 0.8054, ×3 0.7982). A perfect classifier buys 0.023, and only 14 of 111
   questions have relevant notes of a single type. Not a lever.
4. **The two coverage factors double-count, and correcting it loses.** `noteCoverage` is the union
   over a note's candidates *including this chunk*, so multiplying it by `termCoverage` counts one
   piece of evidence twice; the statistically correct treatment of two correlated readings is their
   geometric mean. Measured **offline**, by re-ordering the shipped packs rather than by re-running
   the pipeline: geometric mean **0.7712**, `min` 0.7604, `max` 0.7874, and clamping the product at
   0.25 / 0.3 / 0.35 / 0.4 / 0.5 gives 0.7820 / 0.7820 / 0.7820 / 0.7802 / 0.7568. That harness
   reproduces the real runner exactly where the two overlap — it returns 0.7856 for the control and
   0.8036 for `noteLead` at floor 0.5, both to four decimals — but it cannot change *which* candidates
   reach the pack, so treat these five as a screen rather than as a result. The double-count is real
   and every correction for it scores below the control.
5. **The joint floor sweep round 3 asked for has now been run, and the surface is flat.** The full
   5 × 5 grid of `coverageFloor` × `noteCoverageFloor` spans **0.7748 to 0.7964** — 12 slots out of
   555 — with the maximum at `cov` 0.55 with `ncov` 0.3 or 0.5. Round 3's two single-knob leads
   reproduce (`noteCoverageFloor` 0.5 → 0.7910, `coverageFloor` 0.55 → 0.7892) and stack to 0.7964,
   **still short of the gate**. Neither cell at that maximum is corroborated: `ncov` 0.3 holds `hit@5`
   at 1.0000 but drops block-level to 0.5490, `ncov` 0.5 holds block-level at 0.5569 but drops `hit@5`
   to 0.9910, and **both drop distinct-note precision** (0.4090 and 0.4072 against the control's
   0.4144). A 0.0108 move that costs a reading on every alternative metric, on a surface whose whole
   range is 12 slots, is a fit to 111 questions rather than a finding.

#### The number that reframes rounds 1 to 5

`precision@5` is 0.7856 over 111 questions with a per-question standard deviation of 0.2497, so the
**standard error of the mean is 0.0237** and the 95% bootstrap confidence interval (20,000 resamples)
is **[0.7351, 0.8234]**. **The gate at 0.80 is inside it.** The dev split cannot tell 0.7856 from 0.80;
the gap being chased is 0.6 standard errors.

Paired comparisons are a different matter and remain worth making: the paired bootstrap on
`noteLead` gives +0.0180 with a 95% interval of [0.0036, 0.0288], significant because it changes only
12 of 111 questions and the pairing removes the between-question variance. **So a tuning delta is
measurable on this split and an absolute level is not.** Round 5 should read every headline figure in
this log — 0.5640, 0.6685, 0.7027, 0.7171, 0.7568, 0.7676, 0.7748, 0.7856 — as a paired improvement
over its own control, which is how each was measured, and should not treat "0.7856 against 0.80" as a
gap that 111 questions can resolve. The honest statement of where retrieval stands is that the dev
split puts `precision@5` somewhere around 0.79 ± 0.02.

### 2026-08-13 — score whether the rest of the evidence points here. precision@5 0.7748 -> 0.7856

**The change.** One new multiplier in stage 7, `linkConsensus`: how many *other notes in this
candidate list* link to a candidate's note, capped at `linkConsensusCap` **6** and mapped onto
`[linkConsensusFloor, 1]` with the floor at **0.8**.

```text
consensus(note) = min(other candidate notes linking to it, 6) ÷ 6
linkConsensus   = 0.8 + 0.2 × consensus(note)
score           = base × typeWeight × graphPenalty × noteSupport × termCoverage × noteCoverage
                       × linkConsensus × freshness × verification × authorKind
```

At `linkConsensusFloor: 1` the factor is 1 everywhere and the pipeline is **bit-identical** to the
previous entry's: `node eval/runner.mjs --split=dev --config='{"linkConsensusFloor":1}'` reproduces
0.7748 and a median `used_tokens` of 3,147 exactly, so that is the control below rather than a quoted
number, and the whole delta is this one factor.

**It is the first factor in stage 7 that is not made of words**, which is why it was the thing to
try. The previous two entries added a fifth and a sixth reading of how the query's tokens line up
against a chunk's, and round 1 closed that space with a measurement rather than an opinion: a
logistic model over fourteen lexical features, *fitted on the dev split and scored on the same
split*, ranks at 0.7423 — below the shipped composition. An overfit ceiling under the shipped number
is the end of a feature space. This is a different space: the edges the author wrote in
`## Relations`.

**The edges cost nothing.** Stage 5 already fetches them and already builds `GraphExpansion.related`
out of them so the packer can fill `Claim.related`. Until now the ranker never read that map. The
pipeline hands it to stage 7 as `ScoreAdjustInput.links`; there is no new port call, no new query and
no new latency.

**The statistic that chose it.** Over the 111 dev questions and the 20 candidates each list holds,
counting the *other* candidate notes that link to each candidate note:

| in-list inbound citations | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| candidate notes | 428 | 274 | 163 | 81 | 31 | 12 | 4 | 3 |
| cumulative | 43.0% | 70.5% | 86.8% | 95.0% | 98.1% | 99.3% | 99.7% | 100% |

A relevant note is cited by a mean **2.045** other candidate notes; an irrelevant one by **0.757** —
a ratio of **2.701x**, against `noteSupport`'s 1.323x and the relevance base's 1.106x. The separation
lives at zero: **53.5%** of irrelevant candidate notes have no in-list citation at all against
**10.9%** of relevant ones. Read against the errors this round had to fix rather than marginally —
every pair of (an irrelevant claim in the top five, a relevant claim below it), 308 such pairs — it
ranks the relevant one higher in **76.8%** of the pairs it separates at all (152 wins, 46 losses, 110
ties). That is the best of the sixteen quantities measured; the next is the undirected reading at
74.9%, and `termCoverage` scores 2.1%, meaning the shipped ranker has already extracted everything it
can from that one and what remains is where it is wrong.

**It can only demote.** The map tops out at 1, so a note forty others link to is not *boosted* — it
merely escapes a discount an unlinked note takes. That is the whole safety argument against the hub
failure `expand.ts` documents at length ("an architecture-overview page that forty notes link to
accumulates forty small contributions and outranks the one note that actually answered"). Nothing
here accumulates, nothing here exceeds 1, and the total range is `1 / 0.8` = **1.25x**, the narrowest
of any factor in stage 7 except freshness. `seedCeiling` caps a graph neighbour's reading at its
seed's for the same reason it caps the other four, plus one specific to this factor: the edge that
put a neighbour in the list is itself an edge from a candidate note, so an uncapped reading would
score a neighbour with the very link that fetched it.

**It is consensus, not popularity, and that was measured rather than asserted.** Substituting the
corpus-wide inbound link count for the in-list one — same map, same cap, same floor, the popularity
reading of the same graph — scores **0.7802** at (6, 0.8) and **0.7748**, exactly the control, at
(4, 0.7). Reading the same edges backwards as outbound degree scores **0.7712**, *below* the control.
The signal is agreement among the retrieved set.

**The sweep**, same selection rule as the two entries below: the interior maximum of precision@5,
then the largest floor that reaches it, so the factor swings the ordering least.

| cap \ floor | 0.6 | 0.7 | 0.75 | 0.8 | 0.85 | 0.9 |
| --- | --- | --- | --- | --- | --- | --- |
| 2 | 0.7820 | 0.7784 | — | 0.7820 | — | 0.7766 |
| 3 | 0.7820 | 0.7820 | — | 0.7784 | — | 0.7766 |
| 4 | 0.7838 | 0.7838 | — | 0.7802 | — | 0.7766 |
| 5 | — | — | — | 0.7838 | — | — |
| **6** | 0.7820 | 0.7856 | 0.7856 | **0.7856** | 0.7802 | 0.7784 |
| 7 | — | — | — | 0.7856 | — | — |
| 8 | — | — | 0.7856 | 0.7838 | — | — |
| 10 | — | — | — | 0.7820 | — | — |

Every cell beats the control's 0.7748. The maximum 0.7856 is reached at (6, 0.75), (6, 0.8), (7, 0.8)
and (8, 0.75); 0.8 is the largest floor on it and 6 the smallest cap reaching it there, so the shipped
point is the least-swinging corner of the maximum. The undirected reading — counting an edge in
either direction — tops out at 0.7838 (cap 6, floor 0.7 or 0.8) and is not what ships.

**Before and after, `node eval/runner.mjs --split=dev`.** Both columns measured on 2026-08-13 from the
same binary; the "before" column is the `linkConsensusFloor: 1` control described above.

| Metric | Before | After |
| --- | --- | --- |
| precision@5 | 0.7748 | **0.7856** |
| precision@1 | 0.8919 | 0.8919 |
| precision@3 | 0.8529 | 0.8589 |
| precision@10 | 0.5658 | 0.5757 |
| distinct-note precision@5 | 0.4108 | 0.4144 |
| recall@5 | 0.8183 | 0.8266 |
| hit@5 / hit@10 | 0.9910 / 1.0000 | **1.0000** / 1.0000 |
| MRR | 0.9281 | 0.9333 |
| nDCG@10 | 0.8267 | 0.8379 |
| block-level precision@5 (51 questions) | 0.5412 | **0.5569** |
| answer text returned in full | 0.937 | 0.937 |
| `used_tokens` median / p95 / max / violations | 3,147 / 3,820 / 3,926 / 0 | 3,148 / 3,825 / 3,917 / 0 |
| empty-recall / false-confidence / silent | 0.750 / 0.250 / 0.000 | 0.750 / 0.250 / 0.000 |
| oracle ceiling | 0.9964 | 0.9964 |

**Every metric moves up or stays flat. None regresses.** That includes both of the readings that
caught the two changes round 1 rejected as metric gaming — block-level precision@5, which a factor
merely promoting siblings of a claim already in the pack could not move, rises 0.0157, *more than the
headline*; and distinct-note precision@5, which note-first ordering pushed down, rises.

**`hit@5` reaches 1.0000 and the pipeline now loses no answer at any stage.** The funnel reads `0
never retrieved, 0 at fusion, 0 at rerank, 0 at stage 7, 0 at stage 8, 0 at stage 9`, against one lost
at stage 7 before. `eval/report.mjs`'s failure section prints "Nothing relevant was missing from the
top five on any scored question" for the first time, which is worth knowing because it means **the
table of per-claim score components no longer renders on a passing run** — force it with
`--config='{"freshnessFloor":0}'` if you need to read the decomposition.

**Per question file:** architecture 0.5875 -> **0.6500** (+0.0625), people 0.8429 -> 0.8714 (+0.0286),
conventions 0.6947 -> 0.6947, decisions 0.7750 -> 0.7750, runbooks 0.8516 -> 0.8516, state 0.8533 ->
0.8400 (-0.0133). **The worst file gains the most and by a wide margin**, which is the opposite of
the previous entry's trade. One file regresses, `state`, by one question moving one slot. By
difficulty every band improves: easy 0.7462 -> 0.7538, medium 0.7897 -> 0.8000, hard 0.7704 -> 0.7852.
Ten questions better and four worse: `arch-001`, `arch-012`, `arch-018`, `arch-019`, `arch-020`,
`conv-022`, `pp-who-owns-risk-and-fraud`, `pp-two-approvals-on-money-code`,
`rb-saved-cards-during-outage`, `rb-webhook-replay-window` against `conv-006`,
`rb-cross-region-failover-authority`, `rb-rollback-with-migration`, `st-deploy-freeze-window`.
`conv-022` — "who gets woken up if money moves wrong", the only question in the split scoring **zero**
— finally moves off it.

**Where the remaining loss is now.** The pack's own ceiling on precision@5 moves 0.9135 -> 0.9171 and
questions holding fewer than five relevant chunks in the pack go 22 -> 23, so this change is almost
entirely *ordering* rather than headroom: of the 119 non-relevant claims across all the top-fives, 46
are forced (the pack does not hold five relevant chunks) and **73 are ordering errors** across 51
questions, against 48 and 77 across 52. The gate at 0.80 remains inside what reordering the existing
pack can reach, by 0.1315.

**Tests.** `packages/core/test/retrieval/decay.test.ts` gains a `the link-consensus multiplier`
block, 14 cases. They were proved to bite by mutating the implementation six ways and running them,
then reverting each:

| Mutation | Cases that failed |
| --- | --- |
| count chunks rather than distinct notes | 3, including `counts a note once however many chunks of it are in the list` |
| let a note cite itself | 2 |
| drop the cap on the count | 2, including `never exceeds 1, so it discounts isolation and never rewards popularity` |
| drop the seed ceiling | 1, `caps a graph neighbour at its seed's consensus` |
| return the floor rather than the multiplier off when there are no in-list edges | **16** |
| guard the cap with the floor's guard (`>= 0` rather than finite and `>= 1`) | 1 |

The sixteen-failure mutation is the one worth explaining. `linkConsensusOverList` returns "off" when
*no candidate links to another candidate*, which is `coverageOverList`'s rule for a query that named
nothing and is right here for the same reason: a list in which nothing points anywhere carries no
evidence about consensus, so every candidate belongs at the top of the range rather than uniformly at
the bottom of it. The two readings are indistinguishable in the *ordering* — a factor every candidate
takes equally cannot move a rank — and very distinguishable in the *score*, which is a published
number that should not shrink by 20% on a corpus with no relations in it. Without that rule every
score this repository has ever recorded changes and sixteen existing assertions in that file break,
which is exactly what the mutation demonstrates.

#### What this round measured and did not ship

**Filtering relation-only chunks earlier does not pay, and the reason is worth recording.**
`relation-chunks.ts` says the chunker should not index a pure `## Relations` list at all and that
applying the predicate at stage 9 is the interim. Measured: **12.9% of the stage-7 list is
relation-only** (286 of 2,220), they take **40 of the 555 top-five slots**, and the packer drops
every one of them. Moving the filter *before* stage 7, so those slots go to real candidates and the
list-relative factors are computed without them, scores **0.7694** — below the control — because
removing a note's relation chunk lowers that note's `noteSupport`. Moving it *after* the multipliers
and *before* the cut, which changes only which candidates survive, scores **0.7748**: exactly the
control, to four decimals, and 450 tokens more expensive at the median (3,200 -> 3,650), because the
packer was already dropping them and promoting the next candidate by score — which is precisely what
an earlier filter does. Raising `rerankLimit` to 24, 30 or 40 on top of it changes nothing. **The
stage-9 filter costs no precision and the follow-up in `relation-chunks.ts` is a cost and clarity
change, not a ranking one.** A useful side fact fell out of measuring it: 38 of the 40 top-five
candidates the packer drops are relevant *by note* and **0 of 40 are relevant by block**, which is
what a note-level metric looks like when it is counting a link list as evidence for its note.

**Every one of the five existing floors was re-swept on this round's control, and four are still at
an optimum — but `noteCoverageFloor` has moved.** Coordinate sweep, one knob at a time, against the
0.7748 control: `coverageFloor` 0.2 / 0.25 / **0.35** / 0.45 / 0.55 / 0.65 scores 0.7658 / 0.7694 /
**0.7748** / 0.7712 / 0.7766 / 0.7784; `supportFloor` 0.3 / 0.4 / **0.5** / 0.6 / 0.7 scores 0.7748 /
0.7712 / **0.7748** / 0.7640 / 0.7568; `noteCoverageFloor` 0.4 / 0.5 / **0.6** / 0.7 / 0.8 scores
**0.7838** / 0.7802 / **0.7748** / 0.7694 / 0.7640; `freshnessFloor` 0.85 / **0.9** / 0.93 / 0.95 /
0.97 scores 0.7730 / **0.7748** / 0.7712 / 0.7712 / 0.7694; `typeWeightStrength` 0.2 / **0.35** / 0.5
/ 0.65 scores 0.7676 / **0.7748** / 0.7658 / 0.7622. Re-run *on top of* the shipped change,
`noteCoverageFloor` 0.3 / 0.4 / **0.5** / 0.6 scores 0.7910 / 0.7892 / **0.7910** / 0.7856, and 0.5 is
the one that also keeps `hit@5` at 1.0000 and block-level at 0.5569. `coverageFloor` 0.55 scores
0.7892. **Neither is taken here, because this round changes one thing**, and both are the strongest
single-knob moves available to round 4. A true joint sweep of all six still has never been run.

**Graph consensus and note-lead compound, and together they clear the gate on this split.** The other
candidate this round measured is `noteLead`: every chunk inherits a share of its own note's best
chunk score, `floor + (1 - floor) × (best score of this note ÷ best score in the list)`, which is the
cluster hypothesis as a bounded multiplier. Alone at floor 0.6 it scores **0.7892** with block-level
0.5569 and `hit@5` unchanged; on top of the shipped change it scores **0.8018** at floor 0.6 and
**0.8036** at 0.5, with block-level **0.5725** — the highest reading of that metric anything has
produced — `hit@5` 0.9910, MRR 0.9285, and a *cheaper* median pack. It is deliberately not shipped
this round, and it is the weaker of the two claims for a reason that should be written down rather
than discovered again: `noteLead` leaves distinct-note precision@5 **exactly** unchanged at 0.4108 at
every floor from 0.9 to 0.3, which means it does not reorder *notes* at all — it only pulls a note's
chunks together, and note-level precision@5 rewards that mechanically. Its rank-based variant is
worse still: at floor 0.4 it scores 0.7982 with `hit@5` collapsing to 0.9640, which is round 1's
note-first-ordering signature to three decimal places. Whoever takes it in round 4 should take the
score-share form, at 0.5 or 0.6, and should publish distinct-note precision beside the headline.

### 2026-08-13 — cut the list with the score, not with fusion's order. precision@5 0.7676 -> 0.7748

**The change.** One line moved. The pipeline keeps `rerankLimit` candidates and throws the rest away;
that cut used to happen at the end of stage 6, in **fusion order**. It now happens at the end of
stage 7, by the **composed score**. Nothing else about the pipeline changes: the cross-encoder already
scored the whole list before anything was discarded, so there is no extra model call, and stages 8
and 9 are handed the same twenty candidates as before.

```text
before   fuse 60 → expand +20 → rerank → take first 20 of 80 → score → dedup → pack
after    fuse 60 → expand +20 → rerank → score all 80 → take best 20 → dedup → pack
```

The previous entry named this as the pipeline's largest single loss and left it on the table.
It is the loss because fusion order is the **weakest ranking function the pipeline owns** — 0.5946
`precision@5` on its own, against the composed score's 0.7676 — and it was deciding what stages 7, 8
and 9 would ever see. The list-relative factors (`noteSupport`, `termCoverage`, `noteCoverage`) are
now computed over everything fusion and graph expansion produced rather than over an arbitrary
prefix of it, which is the second half of what this buys.

**Where it is.** `decay.ts`'s `adjustScores` sorts and then slices, because stage 7 is the first
point at which a score worth cutting by exists, and because every instrument — the eval funnel,
`apps/worker/src/retrieval/trace.ts` — reads a step's output and so keeps observing the cut for free.
`pipeline.ts` re-applies the same limit to whatever the step returned, for the same reason
`applyRetrievalFilter` runs a second time over materialised candidates: the steps are injected, and
the size of the list stages 8 and 9 work on must be a property of the pipeline rather than of the
caller's `AdjustScores`.

**This is a deviation from PRD 4.1**, whose diagram draws the cut at step 6 (`rerank → top 20`). It is
recorded as one in the module header of `pipeline.ts` and it needs writing into the PRD; the gate
metric it exists to serve is in the same document.

**Before and after, `node eval/runner.mjs --split=dev`.** Both columns re-measured on 2026-08-13 by
reverting the change, running, and restoring — not quoted from the entry below.

| Metric | Before | After |
| --- | --- | --- |
| precision@5 | 0.7676 | **0.7748** |
| precision@1 | 0.8919 | 0.8919 |
| precision@3 | 0.8619 | 0.8529 |
| precision@10 | 0.5072 | 0.5658 |
| distinct-note precision@5 | 0.3874 | **0.4108** |
| recall@5 | 0.7755 | 0.8183 |
| hit@5 / hit@10 | 0.9820 / 0.9820 | 0.9910 / 1.0000 |
| MRR | 0.9282 | 0.9281 |
| nDCG@10 | 0.8102 | 0.8267 |
| block-level precision@5 (51 questions) | 0.5098 | **0.5412** |
| block-endorsed hits / note-projected hits | 167 / 231 | 208 / 300 |
| answer text returned in full | 0.865 | 0.937 |
| `used_tokens` median / p95 / max / violations | 3,270 / 3,790 / 3,937 / 0 | 3,147 / 3,820 / 3,926 / 0 |
| empty-recall / false-confidence / silent | 0.750 / 0.250 / 0.000 | 0.750 / 0.250 / 0.000 |

**It is not the gaming shape.** The two changes the previous entry rejected — note-first ordering and
loosening dedup — both raised note-level `precision@5` while `hit@5`, block-level precision and
distinct-note precision fell. Here every one of those three rises: block-level +0.0314 (larger than
the headline), distinct-note +0.0234, `hit@5` +0.0090, and the median pack got 123 tokens *cheaper*.
`endorsementRate` falls 0.7229 → 0.6933 because note-projected hits grew faster (231 → 300) than
block-endorsed ones (167 → 208), which are both up in absolute terms.

**Three question files regress**, which is why the previous round's selection rule left this on the
table: architecture 0.5250 -> **0.5875**, conventions 0.6737 -> 0.6947, decisions 0.7500 -> 0.7750,
people 0.8571 -> 0.8429, runbooks 0.8645 -> 0.8516, state 0.8800 -> 0.8533. The gain is concentrated
in the worst file and the losses in the three best, which is the trade a cut by score makes: it stops
protecting a question whose fused prefix happened to be right. Fifteen questions improve and ten get
worse; `arch-007` and `dec-022` go from **zero** relevant claims in the top five to one, and `conv-022`
goes the other way, 0.4 -> 0.0. By difficulty: easy 0.7538 -> 0.7462, medium 0.7621 -> 0.7897, hard
0.7926 -> 0.7704.

**The ceiling it raises is the point.** Reading the funnel as `min(relevant chunks, 5) / 5`, the
pack's own ceiling on `precision@5` goes 0.8505 -> **0.9135**, and 39 -> **22** questions have fewer
than five relevant chunks in the pack. The gate is 0.80. Before this change a perfect ranker of the
pack could reach 0.8505; it can now reach 0.9135, and the remaining 0.1387 is ordering.

**So the shape of the remaining loss changed, and it is worth being explicit about it.** Of the 125
non-relevant claims across all the top-fives, **48 are forced** — the pack does not hold five
relevant chunks — and **77 are ordering errors**, against 83 forced and 46 ordering before. Fifty-two
questions now have some ordering loss, against 31. That is the trade: the pipeline stopped losing
answers it could not get back and started mis-ranking answers it now has. Every remaining point is
reachable by ranking better, which is where the next rounds have to look.

**`rerankLimit` stops being a precision lever, which settles half of a pending decision.** With the
cut moved, every value from 12 to 24 produces the *same* `precision@5` 0.7748, `precision@1` 0.8919,
`hit@5` 0.9910 and block-level 0.5412. Only cost moves:

| `rerankLimit` | median `used_tokens` | p95 | nDCG@10 | answer text in full |
| --- | --- | --- | --- | --- |
| 12 | **1,860** | 2,391 | 0.8030 | 0.901 |
| 16 | 2,552 | 3,084 | 0.8260 | 0.919 |
| **20 (shipped, unchanged)** | 3,147 | 3,820 | 0.8267 | 0.937 |
| 24 | 3,738 | 3,923 | 0.8310 | 0.937 |

"Cut `rerankLimit` 20 → 12" is open as a
**product** decision, on the grounds that it is the only change that makes the published sub-2,500
median true and that a caller can raise `budget_tokens` but cannot raise `rerankLimit`. The precision
half of that argument is now measured and the answer is that it costs nothing. **The decision is still
not this document's to take** and the shipped value is unchanged at 20. What is new is that whoever
takes it is trading 1,287 median tokens against nDCG@10 0.8267 -> 0.8030 and answer coverage 0.937 ->
0.901, and against nothing else.

#### What this round measured and did not ship

**A per-note slot cap in the packer is monotonically harmful, and not as an artefact.** Capping the
claims one note may contribute to the top five at 1 / 2 / 3 / 4 / uncapped scores `precision@5`
0.4144 / 0.6270 / 0.7351 / 0.7676 / **0.7748**, and block-level precision moves the same way — 0.2824
/ 0.4353 / 0.5216 / 0.5412 / 0.5412 — so the cap removes passages that genuinely answer the question
rather than merely deflating a note-level count. The pack is not repetitive either: the top five
holds two distinct notes on 52 of 111 questions and three or more on 53, and only six questions are
answered by a single note five times over.

**Stage 8 collapses nothing on this corpus, at any threshold anyone would ship.** Instrumented over
the 111 dev questions: 2,220 candidates in, 2,220 out, **zero groups** at `dedupCosine` 0.6, 0.7,
0.8, 0.88, 0.92 and 0.96; six collapse at 0.5 and 46 at 0.3, and even at 0.3 not one metric moves.
Dedup is therefore neither the cause of the remaining loss nor a lever on it, and — more
importantly — **the eval does not exercise stage 8 at all**. The reference `CandidateStore` supplies
no `embedding`, so `dedupCandidates` runs its lexical fallback rather than the cosine path the stage
is named for. Nothing in any tuning entry, this one included, is evidence about how dedup behaves on
real `bge-m3` vectors. The previous entry's "dedup is doing its job at 0.92" is not supported: the
difference it measured between the pack and the pre-dedup twenty was the packer's budget truncation.

**Graph consensus is the strongest untried signal, and is deliberately not shipped this round.** How
many *other notes in the same candidate list* link to a candidate's note separates a relevant
candidate from an irrelevant one by **3.08x** — against `noteSupport`'s 1.32x and the relevance
base's 1.11x — over exactly the edges `expandGraph` already fetches, so it needs no new port call.
Re-ranking the pack by `score × map(min(citedBy, 4) / 4, 0.8)` scores **0.7820** with `precision@1`
0.8919, `hit@5` 1.0000, block-level 0.5490 and distinct-note 0.4162; the undirected reading at
`cap 4, floor 0.7` scores 0.7856. Both are offline re-rankings of the pack and must be re-measured
end to end before they are believed. It is left for the next round because this round changes one
thing, and because it is a new quantity in stage 7 rather than a repair of a defect — which is the
weaker of the two claims on a round.

**Four of the five floors are at a local optimum; `coverageFloor` is the exception and is left open.**
A coordinate sweep around the shipped point — one knob at a time, *not* a joint sweep, which nobody
has run: `freshnessFloor` 0.85 / **0.9** / 0.93 / 0.95 scores 0.7658 / **0.7676** / 0.7676 / 0.7676;
`supportFloor` 0.4 / **0.5** / 0.6 / 0.7 scores 0.7658 / **0.7676** / 0.7622 / 0.7568;
`noteCoverageFloor` 0.5 / **0.6** / 0.7 scores 0.7658 / **0.7676** / 0.7640; `typeWeightStrength` 0.2
/ **0.35** / 0.5 scores 0.7658 / **0.7676** / 0.7676. But `coverageFloor` 0.25 / **0.35** / 0.45 /
0.55 scores 0.7640 / **0.7676** / 0.7658 / **0.7712** — 0.55 is 0.0036 *above* the shipped 0.35, on a
four-point grid, against the finer sweep that chose 0.35 two entries below. It is not taken here
because this round changes one thing, and a 0.0036 move on 111 questions is one question moving one
slot. Somebody should sweep it properly on the new baseline. All figures measured against the 0.7676
control, before this round's change.

### 2026-08-13 — score how much of the question the note answers. precision@5 0.7568 -> 0.7676

**The change.** One new multiplier in stage 7, `noteCoverage`: the share of the terms the query named
that appear in **any** of a note's candidates in this list, mapped onto `[noteCoverageFloor, 1]` with
`noteCoverageFloor` **0.6**.

```text
coverage(note) = query terms appearing in any candidate of the note ÷ query terms
noteCoverage   = noteCoverageFloor + (1 - noteCoverageFloor) × coverage(note)
score          = base × typeWeight × graphPenalty × noteSupport × termCoverage × noteCoverage
                      × freshness × verification × authorKind
```

At `noteCoverageFloor: 1` the factor is 1 everywhere and the pipeline is bit-identical to the previous
entry's. It is computed in the **same pass** as `termCoverage` — a note's coverage is the union of its
candidates' covered-term sets, so the second reading costs one `Set` per note and no second scan of any
text, and the two readings cannot disagree about whether a note contains a word.

**Why it is a different quantity from the two factors it sits between.** `termCoverage` asks whether
*this passage* answers the question; `noteSupport` asks how *much* of the note matched. Neither can say
that a passage covering two of four terms sits in a note that covers the other two somewhere else. A
large note can put half its chunks in the list while covering one term the query named, and a two-chunk
note can cover everything across both. Measured: the exponent on every existing factor is already at its
optimum at 1 (see the dead ends below), so this had to be a new quantity or nothing.

**The floor sweep**, same selection rule as the entry below — the interior maximum of precision@5,
provided no question file regresses and provided the composition beats both of its parts.

| `noteCoverageFloor` | p@5 | p@1 | hit@5 | MRR | block-level p@5 | better/worse | files regressing |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 (off, the control) | 0.7568 | 0.8919 | 0.9820 | 0.9290 | 0.5020 | — | — |
| 0.9 | 0.7586 | 0.8919 | 0.9820 | 0.9297 | 0.5059 | 2 / 1 | conventions |
| 0.8 | 0.7604 | 0.9009 | 0.9820 | 0.9342 | 0.5059 | 4 / 2 | none |
| 0.7 | 0.7640 | 0.9009 | 0.9820 | 0.9327 | 0.5098 | 7 / 3 | none |
| **0.6 (shipped)** | **0.7676** | 0.8919 | 0.9820 | 0.9282 | 0.5098 | 9 / 3 | none |
| 0.5 | 0.7658 | 0.8919 | 0.9820 | 0.9270 | 0.5059 | 9 / 4 | conventions |
| 0.4 | 0.7676 | 0.8919 | 0.9820 | 0.9270 | 0.5059 | 9 / 4 | conventions |
| 0.35 | 0.7658 | 0.9009 | 0.9820 | 0.9315 | 0.5020 | 9 / 5 | conventions |
| 0.25 | 0.7676 | 0.9009 | 0.9820 | 0.9308 | 0.5059 | 10 / 5 | none |
| 0.2 | 0.7676 | 0.9009 | 0.9820 | 0.9308 | 0.5059 | 10 / 5 | none |
| 0 | 0.7676 | 0.9009 | 0.9820 | 0.9308 | 0.5059 | 10 / 5 | none |

The maximum is a plateau from 0.6 down to 0, not a spike. **0.6 is the largest floor that reaches it**,
so it is the one that lets this factor swing the ordering least: a median **1.5333x** within one list,
against 4.2000x at 0.2 and the relevance base's 1.3782x. Its cost is MRR 0.9290 -> 0.9282 and nDCG
0.8106 -> 0.8102, one question moving one rank; 0.7 gives that back and costs 0.0036 of precision@5.
The two coverage factors compound, so the shipped total coverage swing is `1 / (0.35 × 0.6)` =
**4.762x** — that is the number to argue with, and `decay.test.ts` pins it.

**It is over the range rule by the same margin `supportFloor` is, and passes the test that rule is
replaced by.** Note coverage alone ranks at **0.7099**, the relevance base alone at 0.5946, the two
together with every other multiplier off at **0.7189**, and the shipped composition at 0.7676. It
combines; it does not dominate. Its discriminative ratio — mean over a pack's relevant claims divided by
the mean over its irrelevant ones — is **1.279x**, between `noteSupport`'s 1.323x and the base's 1.106x.

**Before and after, `--split=dev`, `node eval/runner.mjs --split=dev`.**

| Metric | Before | After |
| --- | --- | --- |
| precision@5 | 0.7568 | **0.7676** |
| precision@1 | 0.8919 | 0.8919 |
| precision@3 | 0.8408 | 0.8618 |
| precision@10 | 0.5045 | 0.5072 |
| hit@5 | 0.9820 | 0.9820 |
| MRR | 0.9290 | 0.9282 |
| nDCG@10 | 0.8106 | 0.8104 |
| block-level precision@5 (51 questions) | 0.5020 | 0.5098 |
| distinct-note precision@5 | 0.3874 | 0.3874 |
| `used_tokens` median / p95 / violations | 3,270 / 3,790 / 0 | 3,270 / 3,790 / 0 |
| empty-recall / false-confidence / silent | 0.750 / 0.250 / 0.000 | 0.750 / 0.250 / 0.000 |

Per file: architecture 0.5000 -> 0.5125, conventions 0.6737 -> 0.6737, decisions 0.7250 -> 0.7500,
people 0.8429 -> 0.8571, runbooks 0.8581 -> 0.8710, state 0.8800 -> 0.8800. None regresses. The three
questions made worse are `arch-012`, `conv-022` and `rb-3ds-failure-triage`, all 0.4 -> 0.2 or 0.8 -> 0.6.

**It is not a clumping artifact.** Block-level precision@5 moves 0.5020 -> 0.5098 — the reading that
judges the specific paragraph, which a factor merely promoting siblings of a claim already in the pack
could not move — while distinct-note precision@5 is unchanged at 0.3874.

**The four non-negotiables, re-run by name.** (1) `-t "phase 7"`, 7 passed. (2) `-t "superseded"`,
9 passed. (3) `-t "budget"`, 39 passed, 0 violations over 139 questions. (4) `-t "gaps"`, 132 passed,
never-silent 0.000. `pnpm test:core` 53 files / 3,983 tests, `pnpm test:worker` 115 files / 2,750 tests,
`pnpm typecheck` and `pnpm lint` (6 known `eval/` warnings) all clean.

**The failure report's table multiplies out again.** `eval/report.mjs` gained a `cov` column when
`termCoverage` landed and did not; it now carries `cov` and `note cov`, so the twelve columns are the
identity in `types.ts` and can be read as one.

#### The dead ends this round measured, so no later round repeats them

Everything below was measured on `--split=dev` against the 0.7568 control. **Nothing in this list beat
it by more than 0.0036 except where marked, and the two that did are rejected for the stated reason.**

**Re-weighting the five existing factors is exhausted.** Ranking the pack by
`base^a × typeWeight × noteSupport^b × termCoverage^c × freshness`, every exponent's optimum is already
1: base 0 / 0.25 / 0.5 / 0.75 / **1** / 1.5 / 2 / 3 scores 0.7423 / 0.7459 / 0.7532 / 0.7532 / **0.7568**
/ 0.7423 / 0.7333 / 0.7189; coverage 0 / 0.5 / **1** / 1.5 / 2 / 3 scores 0.7171 / 0.7387 / **0.7568** /
0.7514 / 0.7387 / 0.7351; support 0 / 0.5 / **1** / 1.5 / 2 / 3 scores 0.6883 / 0.7297 / **0.7568** /
0.7550 / 0.7568 / 0.7441.

**The features these five are computed from contain no more information.** A logistic model over
fourteen features — the five factors, both retrieval ranks, chunk-level and structural coverage, term
occurrences, coverage density, chunk length, note size, heading depth — **fitted on the dev split itself
and scored on the same split** ranks at **0.7423**, below the shipped composition. An overfit ceiling
under the shipped number is the end of the lexical feature space, not a tuning failure.

**Every alternative composition loses.** Reciprocal-rank fusion over the factors' own within-list
rankings (the trick stage 4 uses, applied to stage 7, which equalises their dynamic ranges automatically)
scores 0.7099 / 0.7081 / 0.7099 / 0.7045 / 0.7045 / 0.6991 at k = 1 / 3 / 5 / 10 / 20 / 60; adding
freshness to it, 0.6685 to 0.6901; weighted variants at k = 10, 0.6505 to 0.7315. Replacing the raw
shares with within-list percentiles scores 0.7423 / 0.7495 / 0.7495 at floors 0.5 / 0.35 / 0.2.

**Field-weighted coverage loses, and the marginal statistic that suggested it is a between-query
artifact.** Query terms present in the heading path, tags or permalink separate a relevant candidate
from an irrelevant one by **5.42x** marginally, against 2.55x for the chunk body — but weighting them
scores 0.7532 / 0.7514 / 0.7459 / 0.7495 / 0.7477 / 0.7459 / 0.7171 at body weights 0.9 / 0.75 / 0.6 /
0.5 / 0.35 / 0.25 / 0, and a separate structural-coverage multiplier tops out at 0.7586. `Candidate.text`
carries the heading-path prefix, so those terms are already in the body bag.

**Corpus-IDF-weighted coverage loses monotonically**, which corroborates the previous round's
list-IDF result with the better-founded quantity: 0.7514 / 0.7495 / 0.7477 / 0.7441 / 0.7297 at IDF
powers 0.5 / 1 / 1.5 / 2 / 3 (floor 0.35). The unweighted share remains the best form.

**Term occurrences, density, proximity, earliness and chunk length all lose.** Occurrence-capped
coverage 0.7333 to 0.7387; log-occurrence 0.7351 to 0.7387; earliest-match position 0.7477 to 0.7532;
span-based proximity 0.7387 to 0.7514; shorter-is-better 0.7477 to 0.7568; longer-is-better 0.7405 to
0.7477; coverage density 0.7477 to 0.7586.

**A bigger candidate pool does not help, at either end.** `rerankLimit` 12 / 20 / 24 / 30 / 40 / 50 / 60
scores 0.7027 / **0.7568** / 0.7676 / 0.7658 / 0.7550 / 0.7477 / 0.7604, and everything above 20 costs
550 tokens of median `used_tokens` (3,270 -> 3,825). Raising the retrievers instead is worse:
`lexicalLimit`/`denseLimit` 100 with `fusedLimit` 120 scores 0.7440, 150/200 scores 0.7410.

**Graph expansion cannot be made to pay.** It contributes nothing today — `graphPenalty` is 1.00 on
every claim in every pack — and opening it up is monotonically worse: `graphPenalty` 0.6 (shipped) /
0.8 / 0.9 / 0.95 / 1.0 scores 0.7568 / 0.7550 / 0.7500 / 0.7440 / 0.7390, with block-level precision
falling the same way. `graphExpandLimit` 40 and `graphChunksPerNote` 6 change nothing and 0 changes
nothing.

**A perfect intent classifier is worth nothing on this metric.** Boosting the note type the question
*declares* — an oracle stage 1 cannot beat — scores 0.7568 / 0.7532 / 0.7297 / 0.6991 / 0.6739 at
boosts of 1.25 / 1.5 / 2 / 3 / 5. Only **63%** of relevant candidates carry the declared intent's note
type (435 of 691), so any type boost demotes the other 37%. It lifts precision@1 (0.8919 -> 0.9099) and
MRR, and precision@5 is flat or worse at every strength. Stage 1 is not where the gate is.

**Two changes that do raise precision@5 and are rejected.** Ordering the pack by note first —
notes ranked by their best claim, then that note's claims together — scores **0.7784**, and it is metric
gaming: `hit@5` falls 0.9820 -> 0.9640, MRR 0.9290 -> 0.9166 and block-level precision@5 0.5020 ->
0.4941. Loosening `dedupCosine` has the same signature: ranking the full pre-dedup 20 rather than the
pack scores **0.7820** while block-level precision falls 0.5020 -> 0.4902. Both inflate the note-level
reading by returning more near-duplicates. Dedup is doing its job; leave it at 0.92.

#### Where the ceiling actually goes, measured for the first time

Every funnel number published before this round asks "is *a* relevant note present". `precision@5`
needs **five relevant chunks**, and nobody had measured the fifth. Reading the same funnel as
`min(relevant chunks, 5) / 5`:

| Stage | ceiling on precision@5 | questions with fewer than five |
| --- | --- | --- |
| relevant chunks in the corpus | 0.9964 | 2 of 111 |
| BM25 top 50 | 0.9495 | 12 |
| fused top 60 | 0.9550 | 11 |
| **the 20 stage 7 scores** | **0.8775** | **31** |
| the pack, after dedup and budget | 0.8505 | 39 |

**The 60-to-20 cut is the single largest loss in the pipeline — 0.0775 of ceiling — and it is performed
by fusion order, the weakest ranking function the pipeline owns** (0.5946 alone, against the composed
score's 0.7568). Note that it destroys *headroom* rather than *result*: moving the truncation to after
stage 7, so the pipeline keeps the best 20 of all 80 expanded candidates by its final score, raises the
pack ceiling 0.8505 -> **0.9063** and every secondary metric (`hit@5` 0.9820 -> 0.9910, MRR 0.9290 ->
0.9351, nDCG 0.8106 -> 0.8308, block-level 0.5020 -> **0.5333**, median tokens 3,338 -> 3,228) — and
moves precision@5 only 0.7568 -> 0.7604, with 15 questions better, 13 worse and three question files
regressing. **The cross-encoder already scores every candidate before the cut** (`runRerank` calls the
port on the full list and truncates afterwards), so this costs no extra model calls. It is on the table
for a later round; it fails this round's selection rule.

After the cut, 73 of 111 questions have **zero** ordering loss and the remaining 0.0937 is concentrated
in 17. `architecture` carries the most (mean 0.2125 per question, against `runbooks`' 0.0516). Read
`arch-018` before designing anything for them: the query is "how does a merchant verify a webhook came
from us", the pack holds six relevant claims, and `runbooks/webhook-replay` takes ranks 2, 3 and 4 with
identical `noteSupport` 0.833 and `termCoverage` 0.675 to the relevant
`architecture/webhook-delivery-service` at ranks 7, 9 and 10. **No lexical statistic separates them**,
which is the same conclusion [5.5](#55-how-much-of-the-03159-is-the-two-missing-stages) reached from the
other end.

### 2026-08-12 — score how much of the question a passage answers. precision@5 0.7171 -> 0.7568

**The change.** One new multiplier in stage 7, `termCoverage`: the share of the terms the query
*named* that appear in the candidate, mapped onto `[coverageFloor, 1]` with `coverageFloor` **0.35**.

```text
coverage(c)  = query terms appearing in c ÷ query terms
termCoverage = coverageFloor + (1 - coverageFloor) × coverage(c)
score        = base × typeWeight × graphPenalty × noteSupport × termCoverage
                    × freshness × verification × authorKind
```

At `coverageFloor: 1` the factor is 1 everywhere and the pipeline is **bit-identical** to the previous
entry's — the control below scores 0.7171 exactly, so the whole delta is this one factor. Nothing else
moved: no retriever, no fusion, no rerank, no limit, no half-life, no other weight, no other floor.

**It is not a new quantity; the pipeline already computed it and threw it away.** Stage 9 tells the
caller *"No claims found about verify"* and stage 7 never asked which claims did. The terms are
`gaps.ts`'s own `queryTerms(query.text, config)`, matched with `gaps.ts`'s own `containsSequence` over
`gaps.ts`'s own `normalisedTokens`, against the same four fields `buildClaimIndex` reads — chunk text,
heading path, tags and permalink. Those three helpers moved to a new `retrieval/terms.ts` so that
there is literally one of each: **a claim credited here for covering a word cannot have that word
reported as a gap in the same pack.** `gaps.ts` already imports `ageInDays` from `decay.ts`, so
without the extraction the dependency would have been a cycle.

**Why this and not something else.** The funnel entering this round said the answers were in the top
twenty and in the wrong order — `hit@5` 0.9640, a top-20 ceiling of 0.8775, and 0.7171 actual. Three
measurements decided it.

First, **every factor stage 7 had was a fact about a note, not about the question.** Type, size, age,
verification and authorship cannot tell a passage that answers the question from a passage sitting in
the same note as one that does. Reading `arch-018` ("how does a merchant verify a webhook came from
us") makes it concrete: every base in the list of 20 lay between 2.4e-2 and 3.3e-2, `noteSupport`
between 0.800 and 0.875, `freshness` between 0.908 and 0.981, and `verification` was 0.85 on all
twenty. `runbooks/webhook-replay` took ranks 1, 3 and 4 with the answer at rank 2.

Second, **the metric is clumping-sensitive and the obvious counter-move is the wrong one.** Damping
the *j*-th chunk of one note — the standard site-collapsing remedy — is monotonically worse at every
setting: geometric 0.95/0.9/0.85/0.8 scores 0.6811 / 0.6378 / 0.5946 / 0.5477, a hard cap of
4/3/2/1 per note scores 0.6919 / 0.6414 / 0.5586 / 0.3622, harmonic 0.5/1/2 scores 0.4649 / 0.3694 /
0.3586. All against 0.7171. This retires the whole diversity family, and it also says that the note
one relevant chunk belongs to usually *does* hold the next one — which is why coverage, a per-chunk
property, is the shape of thing that can help without gaming the count.

Third, **fusion is not where the money is, and that was checked rather than assumed.** BM25's own
ordering beats the fused ordering at every depth to 50 (at depth 5, precision 0.6703 against 0.6000;
at 20, 0.3329 against 0.3113), because the reference "dense" retriever is a hashing embedder — a
cruder copy of the lexical signal rather than a complement. Replacing RRF with score fusion was
measured end to end: min-max 0.6450, max-normalised 0.7063, sum-normalised **0.7279**, half RRF and
half max-normalised 0.7153. Sum-normalised beats the control on all nine metrics and is worth
+0.0108, a quarter of what coverage is worth, and every one of them is tuned against a stand-in
embedder. Not taken, and recorded so the next round does not re-measure it.

**Four weightings of the terms were measured. The unweighted share is the best one.**

| Coverage shape | floor 0.5 | floor 0.3 | floor 0.2 |
| --- | --- | --- | --- |
| **unweighted share (shipped)** | 0.7387 | **0.7514** | 0.7532 |
| inverse document frequency over the list | 0.7495 | 0.7477 | 0.7333 |
| IDF squared | 0.7459 | 0.7225 | 0.7135 |
| a proper noun or quoted phrase counts double | 0.7423 | 0.7477 | 0.7477 |

**The floor sweep**, with the selection rule fixed first: the interior maximum of precision@5,
provided no question file regresses and provided the composition still beats both of its parts.

| `coverageFloor` | p@5 | p@1 | hit@5 | recall@5 | MRR | median range in one list |
| --- | --- | --- | --- | --- | --- | --- |
| 1 (off, the control) | 0.7171 | 0.8108 | 0.9640 | 0.7440 | 0.8719 | 1.000x |
| 0.9 | 0.7207 | 0.8468 | 0.9730 | 0.7590 | 0.8962 | — |
| 0.8 | 0.7351 | 0.8559 | 0.9730 | 0.7650 | 0.9007 | — |
| 0.7 | 0.7369 | 0.8829 | 0.9730 | 0.7658 | 0.9181 | 1.321x |
| 0.6 | 0.7405 | 0.8919 | 0.9730 | 0.7748 | 0.9242 | — |
| 0.5 | 0.7387 | 0.8829 | 0.9730 | 0.7703 | 0.9220 | 1.750x |
| 0.4 | 0.7495 | 0.8919 | 0.9820 | 0.7778 | 0.9275 | 2.125x |
| **0.35 (shipped)** | **0.7568** | 0.8919 | 0.9820 | 0.7755 | 0.9290 | **2.393x** |
| 0.3 | 0.7514 | 0.8919 | 0.9820 | 0.7755 | 0.9297 | 2.750x |
| 0.25 | 0.7532 | 0.8919 | 0.9820 | 0.7755 | 0.9282 | 3.250x |
| 0.2 | 0.7532 | 0.9009 | 0.9820 | 0.7755 | 0.9327 | 4.000x |
| 0.1 | 0.7459 | 0.9009 | 0.9820 | 0.7755 | 0.9330 | — |
| 0 | 0.7405 | 0.9009 | 0.9820 | 0.7800 | 0.9315 | infinite |

0.35 sits inside a 0.4-to-0.2 plateau spanning 0.0073 rather than on a spike, and it makes the fewest
questions worse of anything on that plateau (7, against 9 at 0.2).

**This is the first multiplier allowed to be wider than the relevance base, and that needs an
argument.** The three entries below were bought with the rule *no multiplier may swing the ordering
harder than the base it multiplies*, and at 0.35 this one spans a median 2.393x against the base's
1.378x. The rule is a guard against a factor that knows nothing about the question overriding the one
that does — the 500x freshness decay that turned stage 7 into a recency sort. Coverage is a second
measurement of relevance, so the guard has to be applied directly rather than through its proxy: **does
the composition beat both of its parts, or converge on one of them?**

| Ranking | precision@5 |
| --- | --- |
| the relevance base alone | 0.5946 |
| coverage alone | 0.6414 |
| coverage × base, every other multiplier off | 0.6865 |
| **the shipped composition** | **0.7568** |

It combines. It does not dominate. And it is the most *discriminative* quantity in the stage: mean
over a pack's relevant claims divided by the mean over its irrelevant ones is **1.487x**, against
1.323x for `noteSupport`, 1.106x for the undamped `typeWeight` and **1.097x for the relevance base it
multiplies**. Damping it to respect the range rule would down-weight the better signal in favour of
the worse one; the honest cost of doing so is stated anyway — floor 0.7, the widest value under the
base's range, scores 0.7369.

**Before and after, `--split=dev`, 139 questions, 111 scored.**

| Metric | Before | After |
| --- | --- | --- |
| precision@5 | 0.7171 | **0.7568** |
| precision@1 | 0.8108 | 0.8919 |
| precision@3 | 0.7928 | 0.8408 |
| precision@10 | 0.4928 | 0.5045 |
| hit@5 | 0.9640 | 0.9820 |
| hit@10 | 0.9730 | 0.9820 |
| recall@5 | 0.7440 | 0.7755 |
| MRR | 0.8719 | 0.9290 |
| nDCG@10 | 0.7630 | 0.8106 |
| block-level precision@5 (51 questions) | 0.4824 | 0.5020 |
| distinct-note precision@5 | 0.3712 | 0.3874 |
| answer text returned in full | 0.8649 | 0.8649 |
| oracle precision@5 | 0.9964 | 0.9964 |
| `used_tokens` median / p95 / violations | 3,270 / 3,790 / 0 | 3,270 / 3,790 / 0 |
| empty-recall / false-confidence / silent | 0.750 / 0.250 / 0.000 | 0.750 / 0.250 / 0.000 |

**It is not a clumping artifact, and that is measured rather than asserted.** The two readings that
would *not* move if a factor were winning claim-level precision@5 by promoting siblings both move:
block-level precision@5 0.4824 -> 0.5020, which judges the specific paragraph, and distinct-note
precision@5 0.3712 -> 0.3874, which collapses several chunks of one note to one. `recall@5`
(note-level) and `hit@5` move the same way, and `hit@5` is the one a diversity-gaming change cannot
move at all.

**Stage 7 has stopped losing answers.** It now retains **100%** of what rerank hands it into the top
five (was 98.2%), and it moves the best relevant claim on 22 of 109 questions — **18 up, 4 down, mean
-0.50 ranks**, against 13 up / 12 down / -0.17 before and 11 up / 16 down / +0.05 two rounds ago.

**Per slice, none of which regresses.**

| By file | Before | After |   | By difficulty | Before | After |
| --- | --- | --- | --- | --- | --- | --- |
| architecture (16) | 0.4875 | 0.5000 |   | easy (26) | 0.7231 | 0.7538 |
| conventions (19) | 0.6737 | 0.6737 |   | medium (58) | 0.7138 | 0.7448 |
| decisions (16) | 0.6625 | 0.7250 |   | hard (27) | 0.7185 | **0.7852** |
| people (14) | 0.7714 | 0.8429 |   |  |  |  |
| runbooks (31) | 0.8258 | 0.8581 |   |  |  |  |
| state (15) | 0.8000 | 0.8800 |   |  |  |  |

**`hard` gains the most, which is the evidence that matters against the overfitting worry.** A signal
that merely rewarded verbatim echo would show up as a gain concentrated in `easy`. By classified
intent: note 0.7400 -> 0.8400, decision 0.6625 -> 0.7500, state 0.7286 -> 0.7857, reference 0.7240 ->
0.7520, convention 0.7238 -> 0.7143 — the one slice that falls, by one question in twenty-one.

**What it costs, by name.** 29 questions improve, 75 are unchanged, 7 get worse: `arch-004` 0.6->0.2,
`arch-016` 0.8->0.4, `conv-022` 0.8->0.4, `dec-016` 1.0->0.8,
`pp-two-approvals-on-money-code` 0.8->0.6, `rb-3ds-failure-triage` 1.0->0.8,
`st-postgres-17-upgrade` 0.8->0.6. All seven are the same trade: a passage that answers the question
in the corpus's vocabulary losing to one that answers it in the asker's. The floor is what bounds
that — at 0.35 a passage covering nothing keeps 35% of its score and still wins any relevance gap
above 2.86x — and `0`, which deletes such a passage from contention outright, scores worse (0.7405).

**The four non-negotiables, re-run by name.** (1) A 400-day-old `state` note below a 5-day-old one on
equal relevance: `npx vitest run --project core -t "phase 7"`, 7 passed. (2) Superseded and
soft-deleted never appear: `-t "superseded"`, 9 passed, plus all four eval safety probes.
(3) `used_tokens <= budget_tokens`: `-t "budget"`, 39 passed, 0 violations over 139 questions.
(4) `gaps` populated whenever `claims` is empty: `-t "gaps"`, 132 passed, never-silent 0.000. Every
count identical to the three rounds before.

**The worker caveat, unchanged and for the unchanged reason.** `--retriever=worker --split=dev` scores
**0.0991**, bit-identical to rounds 1 to 4, because `ports.ts:451` is
`rerank: async (_query, contexts) => contexts.map(() => 0)` and stage 7's base is
`components.rerank ?? components.rrf`. A new multiplier applied to zero is still zero. Worker latency
on this run: median 313.69 ms, p95 534.48 ms, max 648.95 ms over 139 requests.

**What did not move, and why that is deliberate.** `false-confidence` 0.250 and `empty-recall` 0.750,
unchanged for the sixth round — the dead `low_confidence` rule ([5.3](#53-the-low_confidence-relevance-floor-is-unreachable)),
left alone so this round changes one thing. The top-20 ceiling (0.8775) and the pack ceiling (0.8505)
are unchanged to four places, because stage 7 cannot change which twenty candidates it is handed: the
entire gain is order within the pack, and **0.0937 of reordering headroom remains**.

**Where the rest of the gate is, measured this round.** The fused top 60 supports precision@5
**0.9550** (mean 8.67 relevant chunks, only 11 of 111 lists holding fewer than five). The top 20 that
survives the cut supports 0.8775 (mean 6.23, 31 of 111 short). **The 60-to-20 truncation costs 0.0775
of ceiling, and it is done by RRF order because the reference reranker preserves rank.** That number
is what [5.5](#55-how-much-of-the-03159-is-the-two-missing-stages)'s 0.94-0.96 estimate for a real
cross-encoder is: a perfect reranker over the fused 60 *is* the 0.9550 above. Ordering alone can no
longer reach 0.80 comfortably — it needs 62% of the headroom left, from a base that separates relevant
from irrelevant by 1.097x.

### 2026-08-12 — divide note support by the note, not by the list. precision@5 0.7027 -> 0.7171

**The change.** One quantity, in one function. `noteSupport` was the number of candidates a note
contributed to step 7's list, divided by the best-supported note's count. It is now the share of the
note's **own** chunks that reached the list:

```text
before:  support(note) = count(note) / max over the list of count
after:   support(note) = count(note) / Candidate.noteChunkCount        (clamped to 1)
noteSupport = supportFloor + (1 - supportFloor) × support(note)        (unchanged)
```

`supportFloor` stays at **0.5**, deliberately un-retuned, so this entry changes exactly one thing and
the previous entry's pipeline is the control. Nothing else moved: no retriever, no fusion, no rerank,
no limit, no half-life, no other weight, no other floor.

It needs a fact the pipeline did not carry, so `Candidate` gains an optional `noteChunkCount` and both
`CandidateStore` implementations supply it — `ReferenceCandidateStore` by counting its own corpus,
`apps/worker/src/retrieval/candidates.ts` by a fourth phase, a `GROUP BY note_id` over the same
`chunks`-to-live-`notes` join the other three use. **When any candidate in a list arrives without it
the whole list falls back to `count / max count`**, never one note on one scale and its neighbour on
another: that is the mistake `seedCeiling` exists to prevent one stage earlier.

**Why this and not something else.** The funnel entering this round said the answers were in the pack
and in the wrong order — `hit@5` 0.9369 against `precision@5` 0.7027, `precision@1` 0.7928, and a pack
oracle of 0.8505. Two measurements decided it.

First, **the count form is the largest remaining blamed factor**. Over the 111 dev questions, taking
each irrelevant claim in a pack's top five against the best relevant claim below it and naming the
factor that most favours the wrong one: `noteSupport` 65 of 130, relevance base 42, `typeWeight` 18,
`freshness` 5. Second, **switching each factor off in turn** says support is also the most valuable
thing in the product: relevance alone scores 0.5946, `supportFloor: 1` scores 0.5640, `freshnessFloor:
1` scores 0.7099 and `typeWeightStrength: 0` scores 0.7027. Support is worth **+0.1387** and is
simultaneously the factor getting the top five wrong most often. That is the signature of a good
quantity measured badly, not of a factor that should be turned down.

What it measured badly is named in the previous entry's own closing paragraph: *"a count cannot tell a
one-chunk note that matched entirely from a six-chunk note that matched once."*

**Four candidate denominators were measured. Only the note's own size works.**

| Denominator | precision@5 | Needs |
| --- | --- | --- |
| the list's best-supported note (the control) | 0.7027 | nothing |
| **the note's chunk count in the corpus** | **0.7171** | a new `Candidate` field and a D1 `GROUP BY` |
| the note's count in the pre-rerank candidate list | 0.5315 | nothing |
| the note's count in the fused top 60 | 0.5315 | nothing |

The two cheap ones are not near misses, they are **0.17 worse than the control**, and the reason is
worth writing down: a count over a list that is about to be truncated measures *retention through the
cut*, so a note with one chunk retrieved and one kept scores a perfect 1.0. Reshaping the count without
changing its denominator does not work either — measured at floor 0.5 / 0.35 / 0.25, capping it at 2
scores 0.6739 / 0.6739 / 0.6757, at 3 0.6901 / 0.7009 / 0.7027, at 4 0.6955 / 0.7081 / 0.7081, its
square root 0.6937 / 0.7009 / 0.7027 and its logarithm 0.6991 / 0.7045 / 0.7045. **Every list-local
rearrangement is at or below the count form. The note's size is information the list does not
contain, and there is no substitute for going and getting it.**

**It buys the gain by being a better signal, not a louder one.** Measured on the same runs, at the same
floor:

| | count / max count | count / note size |
| --- | --- | --- |
| median range within one list | 1.600x | **1.548x** |
| mean over a pack's relevant claims ÷ over its irrelevant ones | 1.287x | **1.323x** |

**Narrower and more discriminative at once**, which is the only kind of change to this pipeline that is
free. The three entries below all bought their gain by *removing* authority from a factor; this one is
the first that adds information instead. For scale, the relevance base spans a median 1.378x in the
same lists and separates relevant from irrelevant by 1.095x.

**The floor was deliberately not re-tuned, and that costs 0.0072.** The sweep at the new quantity,
`node eval/runner.mjs --split=dev --config='{"supportFloor":F}'`:

| floor | precision@5 | precision@1 | hit@5 | MRR | support's median range in one list |
| --- | --- | --- | --- | --- | --- |
| 1 (multiplier off) | 0.5640 | 0.7297 | 0.9189 | 0.8074 | 1.000x |
| 0.8 | 0.6559 | 0.8018 | 0.9640 | 0.8604 | 1.190x |
| 0.65 | 0.7009 | 0.7928 | 0.9730 | 0.8612 | 1.319x |
| **0.5 (shipped, unchanged)** | **0.7171** | **0.8108** | **0.9640** | **0.8719** | **1.548x** |
| 0.45 | 0.7189 | 0.8108 | 0.9640 | 0.8764 | 1.645x |
| 0.35 | 0.7243 | 0.8108 | 0.9640 | 0.8706 | 1.898x |
| 0.25 | 0.7207 | 0.8108 | 0.9459 | 0.8672 | 2.275x |
| 0 (un-floored) | 0.7171 | 0.8108 | 0.9189 | 0.8608 | 5.000x |

The curve now has an interior maximum, at 0.35, and 0.35 is not what ships. Two reasons, in order.
**A round that changes two things teaches nothing**, and the floor is the control this entry is
measured against. And the rule the three entries below were bought with — *a multiplier may not swing
the ordering harder than the relevance signal it multiplies* — puts 0.35's 1.898x at 38% over the
base's 1.378x. The shipped 0.5 is already 12% over and is inherited, not chosen here. Re-deriving this
floor under that rule is a round of its own, and on this evidence it would move it **up**, not down.

**Measured effect**, `node eval/runner.mjs --split=dev`, reference backend, before -> after:

| metric | before | after |
| --- | --- | --- |
| precision@5 | 0.7027 | **0.7171** |
| precision@1 | 0.7928 | 0.8108 |
| precision@3 | 0.7658 | 0.7928 |
| precision@10 | 0.4874 | 0.4928 |
| hit@5 | 0.9369 | **0.9640** |
| hit@10 | 0.9730 | 0.9730 |
| MRR | 0.8467 | 0.8719 |
| nDCG@10 | 0.7479 | 0.7630 |
| recall@5 | 0.7282 | 0.7440 |
| block-level precision@5 | 0.4314 | **0.4824** |
| distinct-note precision@5 | 0.3640 | 0.3712 |
| answer text returned in full | 0.8649 | 0.8649 |
| stage 7 retention (reranked -> adjust@5) | 95.4% (104/109) | **98.2% (107/109)** |
| answers lost at stage 7 | 5 | **2** |
| a relevant note in the final top five | 93.7% (104/111) | 96.4% (107/111) |
| stage 7's displacement of the best relevant claim | 0.05 ranks over 27 of 109, 16 down / 11 up | **−0.17 ranks over 25 of 109, 12 down / 13 up** |
| used_tokens median / p95 / max / violations | 3,271 / 3,788 / 3,933 / 0 | 3,270 / 3,790 / 3,935 / 0 |
| oracle ceiling | 0.9964 | 0.9964 |

**Two rows deserve to be read out loud.** `hit@5` moves for the first time in three rounds — the last
two changes reordered the top five, this one puts an answer into a top five that did not have one, on
three more questions. And stage 7's mean displacement is **negative**: for the first time in this log
the score adjustment moves the best relevant claim *up* more often than down. Stage 7 has stopped
being damage to be bounded.

**Block-level precision moves further than the headline** (0.4314 -> 0.4824, +11.8% relative against
+2.0%), and distinct-note precision moves with it (0.3640 -> 0.3712). Those are the two readings that
would *not* move if this were a clumping artifact — the block reading judges the specific paragraph and
the distinct reading collapses several chunks of one note to one — so this is not the claim-level
metric being gamed by promoting siblings.

**What it costs, by name.** Question by question against the control: **14 improve, 91 are unchanged,
6 get worse.** The six are `arch-011` 1.0 -> 0.8, `arch-019` 0.6 -> 0.4, `conv-016` 1.0 -> 0.8,
`dec-013` 1.0 -> 0.8, `dec-023` 0.6 -> 0.4 and `rb-refund-missing-ledger-postings` 0.8 -> 0.6. They
are the mirror image of the ten the previous quantity cost: a large relevant note that showed three of
its eleven chunks used to be the best-supported note in its list and is now scored at 0.27 of itself.
**Anyone raising `supportFloor` is buying those six back at about 0.008 precision@5 each.**

**One slice regresses.** By question file: architecture 0.4625 -> 0.4875, conventions 0.6211 ->
0.6737, **decisions 0.6875 -> 0.6625**, people 0.7286 -> 0.7714, runbooks 0.8258 -> 0.8258, state
0.8000 -> 0.8000. `decisions` is two of the six above (`dec-013`, `dec-023`) over 16 questions, and it
is stated rather than averaged away. By difficulty: easy 0.7000 -> 0.7231, medium 0.6966 -> 0.7138,
hard 0.7185 -> 0.7185. By answer type: `reference` 0.7319 -> 0.7429, `convention` 0.7100 -> 0.7400,
`note` 0.7158 -> 0.7684, `state` 0.6595 -> 0.6811, `decision` 0.6865 -> 0.6811. By classified intent:
`note` 0.7042 -> 0.7211, `reference` 0.7333 -> 0.7600, `convention` 0.7200 -> 0.7333, `state` 0.9000
-> 0.9000, `decision` 0.5500 -> 0.5250.

**No test was weakened, and one test changed outcome in the direction this change exists to produce.**
`pipeline.test.ts`'s "keeps a live sibling of the excluded content" pinned the previous entry's cost in
miniature: a planted **one-chunk** note carrying the query verbatim, demoted out of rank 1 because it
contributed one candidate to a list whose best-supported note contributed four. It is rank 1 again —
one of one chunk is a fully supported note — and the assertion now says so with the new arithmetic
beside it. Its two load-bearing assertions (the superseded twin is absent, the live sibling is present)
are untouched. `decay.test.ts`'s existing `describe("the note-support multiplier")` is renamed to *on
the fallback scale* and keeps every assertion it had, because its helper builds candidates with no
`noteChunkCount` and it now pins the fallback branch exactly. A new
`describe("note support as a share of the note")` adds 8 tests for the shipped branch: the arithmetic
written out with the score identity recomputed bit-identically, a one-chunk note taking the full
multiplier, the 1.714x bound asserted in both directions, a fast-check property that support is
monotone in the *share*, the per-list fallback (one missing count takes the whole list back to the old
scale), rejection of a zero / negative / fractional / `NaN` / infinite chunk count, the clamp when a
store reports fewer chunks than the list holds, and the graph-seed cap on the new scale.

**The four non-negotiables, re-run by name.** `npx vitest run --project core -t "phase 7"` 7 passed,
`-t "superseded"` 9 passed, `-t "budget"` 39 passed, `-t "gaps"` 132 passed — the same four counts as
the three entries below. All seven of the eval's own safety probes pass, including the two that plant a
superseded chunk whose text is the query verbatim and the one that has the candidate store refuse a
tombstoned chunk. 13 of the 15 gate checks pass; the two that fail are `precision@5` and
`whole-eval-set`, the latter because `--split=dev` is not the whole set.

**The worker backend still cannot measure any of this**, for the fourth round running.
`node eval/runner.mjs --retriever=worker --split=dev --base-url=http://127.0.0.1:8798 --sign-in-local`
scores **0.09909909909909909**, bit-identical to all three entries below. Verified again on this run:
all **2,594** returned claims have `score` exactly 0, because `ports.ts:451` is
`rerank: async (_query, contexts) => contexts.map(() => 0)` and `runRerank` sets
`score = rerank × graphPenalty`. Worker latency this run: median 303.54 ms, p95 534.23 ms, max
613.10 ms over 139 requests; the same 5 adversarial questions are refused HTTP 400 at the zod boundary.
**The `noteChunkCount` half of this change is therefore committed and unmeasured on the worker path** —
`candidates.ts` supplies it and `notIndexedPorts` means nothing reads it yet. Its own suite covers it:
the count is the note's and not the request's, it excludes tombstoned notes, and a graph neighbour
loaded under `graphChunksPerNote: 2` still reports its note's real size of 5.

**What did not move.** `false-confidence` 0.250 and `empty-recall` 0.750, unchanged for the fifth run:
[5.3](#53-the-low_confidence-relevance-floor-is-unreachable)'s dead `low_confidence` rule, still
deliberately untouched. The oracle ceiling is unchanged at 0.9964, as it must be — this changes an
ordering, not a candidate set. And the **pack oracle is unchanged at 0.8505**: re-ranking each shipped
pack perfectly scores the same before and after, because pack membership is bit-identical. The whole
delta is order within the pack, and the remaining reordering headroom is now 0.1333 rather than 0.1477.

**Gates.** `pnpm lint` 0 (416 files, the same 6 pre-existing warnings, none in a file this change
touched), `pnpm typecheck` 0, `pnpm test:core` 0 (53 files, 3,952 tests, +8),
`pnpm test:worker` 0 (114 files, 2,735 tests, +3).

**The next binding constraint, from this round's evidence.** Not stage 7 any more. It moves the best
relevant claim up on net and retains 98.2% of what the reranker hands it; **two** questions still lose
their answer there, against two more lost at the `rerankLimit` truncation before it, and those four
are the whole of `hit@5`'s remaining 3.6%. The two numbers that bound everything
else: **the 20 candidates entering stage 7 support a ceiling of 0.8775** and **the pack that leaves it
supports 0.8505**, against a corpus ceiling of 0.9964. So 0.80 is still reachable by ordering alone,
but it now needs 66% of the reordering headroom that is left, from a relevance base spanning 1.378x.
Four things this round measured and rejected are recorded so they are not retried, all on the shipped
tree:

- **`--reranker=lexical`**, the "obvious next diagnostic" the previous entry named and the last stale
  control from section 5's era, scores **0.5838** against 0.7171. It is worse, and the mechanism is
  worth keeping: a term-coverage score *replaces* the relevance base rather than adjusting it, and it
  is coarse enough to tie most of the list. The same quantity applied as a bounded seventh multiplier
  instead of as a replacement was swept in a [section 8.8](#88-how-to-run-an-experiment-without-touching-the-repository)
  harness against the *previous* entry's pipeline and is worth at most +0.011 there, with `hit@5`
  falling — so it is not the next lever either.
  **This retires the stale-control question the previous entry left open.**
- **`rerankLimit`** at 12 / 16 / 24 / 30 / 40 scores 0.6739 / 0.6901 / 0.7243 / 0.7189 / 0.7063
  against 20's 0.7171. 24 is +0.0072 and is the same size as the floor question above; it also costs
  `precision@1` and 20% more tokens per pack, and it is a second knob.
- **`dedupCosine`** at 0.85 and at 0.999 both score **exactly 0.7171**. Step 8 is inert on this
  corpus and the dedup threshold is not a lever — worth knowing before anyone spends a round on
  "the dedup threshold collapsing genuinely distinct claims".
- **`denseLimit: 0`** scores **0.7369**, so the hashing embedder now costs 0.0198. That is a fact
  about the stand-in and not an argument against `bge-m3`; see [section 3](#3-the-eval).

### 2026-08-12 — bound the intent type weight. precision@5 0.6685 -> 0.7027

**The change.** One new knob, `RetrievalConfig.typeWeightStrength`, default **0.35**. Step 7 pulls
stage 1's per-type weight towards the neutral value of 1 before multiplying it in:

```text
typeWeight = 1 + typeWeightStrength × (intent.typeWeights[noteType] - 1)
```

At `typeWeightStrength: 1` the factor is exactly what step 1 emitted, the pipeline is bit-identical
to the previous entry's, and that is the control this entry is measured against. Nothing else moved:
no retriever, no fusion, no rerank, no limit, no half-life, no floor, no other weight. The map is
strictly increasing in the raw weight, so **the classifier still decides which type a query prefers**
— only how much that preference is worth changes.

**Why this and not something else.** It is [5.1](#51-stage-7-destroys-most-of-the-ordering-the-earlier-stages-produced)'s
defect in a third costume, and the table that found it twice finds it again. Measured over the 111
dev questions, taking max/min of each factor within each question's own result list, and the mean of
each factor over the pack's relevant claims against its irrelevant ones:

| Factor | Range within one list | mean over relevant / over irrelevant claims |
| --- | --- | --- |
| relevance base (`rerank ?? rrf`) | median 1.3737x | 0.0292 / 0.0264 = 1.106x |
| **`typeWeight`, undamped** | **median 2.2000x**, max 5.7600x | **1.0861 / 1.0278 = 1.057x** |
| `noteSupport` at the shipped floor | median 1.6000x | 0.9139 / 0.7102 = 1.287x |
| `freshness` at the shipped floor | median 1.0796x | 0.9181 / 0.9172 = 1.001x |

**The type weight swings the ordering 1.6x harder than the relevance signal it multiplies while
separating a relevant claim from an irrelevant one by a twentieth as much.** It is the widest factor
left in the product and the least informative one. It also fires on only **40 of the 111** questions
— `classifyIntent` returns flat weights on the other 71, [5.7](#57-what-the-eval-cannot-see-and-therefore-what-section-5-does-not-cover)'s
third coverage gap — so on a third of the question set a signal worth 1.057x was given more authority
over the top five than relevance itself.

**The sweep, and the rule that chose 0.35.** `node eval/runner.mjs --split=dev
--config='{"typeWeightStrength":S}'`, reference backend. The last column is the median range of the
damped factor within one list, over the 40 lists where the classifier fires at all — flat lists
carry no range at any strength and would make the median 1.000x everywhere:

| strength | precision@5 | precision@1 | hit@5 | MRR | nDCG@10 | type weight's range in one list |
| --- | --- | --- | --- | --- | --- | --- |
| 1 (PRD 4.1 verbatim, control) | 0.6685 | 0.7297 | 0.9369 | 0.8031 | 0.7169 | 2.2000x |
| 0.75 | 0.6811 | 0.7568 | 0.9369 | 0.8264 | 0.7295 | 1.8491x |
| 0.5 | 0.6937 | 0.7838 | 0.9369 | 0.8418 | 0.7433 | 1.5357x |
| 0.4 | 0.7027 | 0.7928 | 0.9369 | 0.8467 | 0.7470 | 1.4196x |
| **0.35 (shipped)** | **0.7027** | **0.7928** | **0.9369** | **0.8467** | **0.7479** | **1.3633x** |
| 0.3 | 0.7063 | 0.7928 | 0.9369 | 0.8467 | 0.7487 | 1.3082x |
| 0.25 | 0.7099 | 0.7928 | 0.9459 | 0.8470 | 0.7490 | 1.2542x |
| 0.2 | 0.7081 | 0.7928 | 0.9459 | 0.8472 | 0.7500 | 1.2013x |
| 0.1 | 0.7063 | 0.8018 | 0.9459 | 0.8530 | 0.7499 | 1.0987x |
| 0 (multiplier off) | 0.7027 | 0.7928 | 0.9459 | 0.8455 | 0.7477 | 1.0000x |

**Every strength below 1 beats the control on every headline metric**, so the eval cannot pick
between them and something else has to. The rule was fixed before the precision column was read, and
it is the arithmetic the two entries below already acted on twice: *a multiplier may not swing the
ordering harder than the relevance signal it multiplies, or it replaces that signal rather than
adjusting it.* The relevance base spans a median 1.3737x. The largest strength at or under that is
**0.35** (1.3633x); 0.4 is already over (1.4196x).

That rule costs **0.0072 precision@5** against the 0.7099 available at 0.25, and it is stated rather
than buried. Two honest caveats on it:

- **It is stricter than the rule that chose `supportFloor`.** Note support spans 1.6000x, above the
  base's 1.3737x, and this rule applied to it would demand a floor of about 0.64. Support has 5x the
  discriminative power of the type weight (1.287x against 1.057x), so the two are not comparable on
  range alone — but "authority in proportion to evidence" is the honest version of the principle and
  neither round has stated it that way. Re-deriving `supportFloor` under it is a round of its own.
- **The rule uses medians.** At strength 0.35 the *widest* damped list still spans 2.0292x, against
  the base's widest 1.4675x. A per-list bound would be a different and more invasive change.

**Measured effect**, `node eval/runner.mjs --split=dev`, reference backend, before -> after:

| metric | before | after |
| --- | --- | --- |
| precision@5 | 0.6685 | **0.7027** |
| precision@1 | 0.7297 | 0.7928 |
| precision@3 | 0.7087 | 0.7658 |
| precision@10 | 0.4811 | 0.4874 |
| hit@5 | 0.9369 | 0.9369 |
| hit@10 | 0.9640 | 0.9730 |
| MRR | 0.8031 | 0.8467 |
| nDCG@10 | 0.7169 | 0.7479 |
| recall@5 | 0.7042 | 0.7282 |
| block-level precision@5 | 0.4078 | 0.4314 |
| distinct-note precision@5 | 0.3532 | 0.3640 |
| answer text returned in full | 0.8649 | 0.8649 |
| stage 7 retention (reranked -> adjust@5) | 95.4% (104/109) | 95.4% (104/109) |
| answers lost at stage 7 | 5 | 5 |
| stage 7's mean displacement of the best relevant claim | 0.26 ranks over 32 of 109 | **0.05 ranks over 27 of 109** |
| used_tokens median / p95 / violations | 3,271 / 3,790 / 0 | 3,271 / 3,788 / 0 |
| oracle ceiling | 0.9964 | 0.9964 |

**Nothing regressed, and that is not a figure of speech.** Question by question against the control:
**13 improve, 98 are unchanged, 0 get worse.** This is the first change in this log with no cost
column, and the reason is arithmetic rather than luck — damping a weight of 1 towards 1 is 1, so the
71 questions `classifyIntent` reads as neutral are bit-identical before and after. Every one of the
13 that moved is a question where the classifier fired:

| classified intent | n | before | after |
| --- | --- | --- | --- |
| `note` (flat weights) | 71 | 0.7042 | 0.7042 |
| `reference` | 15 | 0.6533 | 0.7333 |
| `convention` | 15 | 0.6000 | 0.7200 |
| `decision` | 8 | 0.4500 | 0.5500 |
| `state` | 2 | 0.9000 | 0.9000 |

By answer type: `reference` 0.7011 -> 0.7319, `convention` 0.6750 -> 0.7100, `decision`
0.6541 -> 0.6865, `note` 0.6421 -> 0.7158, `state` 0.6324 -> 0.6595. By file: architecture
0.4500 -> 0.4625, conventions 0.6000 -> 0.6211, decisions 0.6500 -> 0.6875, people 0.6857 -> 0.7286,
runbooks 0.7613 -> 0.8258, state 0.8000 -> 0.8000. By difficulty: easy 0.6462 -> 0.7000, medium
0.6621 -> 0.6966, hard 0.7037 -> 0.7185.

**This retires a claim section 5 makes.** [5.7](#57-what-the-eval-cannot-see-and-therefore-what-section-5-does-not-cover)
says "removing the type weights moves `precision@5` by 0.008", and [5.1](#51-stage-7-destroys-most-of-the-ordering-the-earlier-stages-produced)'s
composition table has "type weights off, everything else on" at 0.308 against 0.3159 shipped. Both
were measured through the un-floored freshness multiplier, which was swamping everything including
this. With freshness bounded the same factor is worth **0.0342**, four times what section 5 reports.
**A factor measured through a broken one has not been measured.** The other conclusion from the same
era that is now stale for the same reason is `--reranker=lexical`; re-running it is the obvious next
diagnostic.

**No test was weakened.** Two assertions in `decay.test.ts` changed outcome, and both were assertions
about PRD 4.1 rather than about the shipped composition: "applies the intent's per-type weights" and
"caps nothing on a candidate a retriever returned directly" both pinned a raw weight of 2. Each now
asserts the PRD's number verbatim under a new `FULL_TYPE_WEIGHT` config (the shipped one with the
strength back at 1) **and** the shipped number beside it — the pattern the two entries below already
use, so the old behaviour is one config value away and this is a configuration rather than a
regression. A new `describe("the type-weight strength")` adds 8 tests: the default and the rule that
derived it, the damping arithmetic written out (1.35 and 0.825) with the score identity recomputed
bit-identically, the 1.6364x bound asserted in both directions, a fast-check property that the
classifier's preference is never inverted, identity at 1 and off at 0, a neutral classifier
unaffected at every strength, rejection of NaN / negative / >1 / ±Infinity falling back to PRD 4.1,
and the graph-seed cap computed on the damped scale.

That property test found something worth recording. **Strict monotonicity is exact-arithmetic and
only nearly true in doubles**: at strength 0.01 the map sends 7.9999999999999725 and
7.999999999999973 — one ulp apart — to the same double. A tie cannot reorder anything, and
`normaliseWeights` divides products of table constants and cannot emit two weights one ulp apart, so
the test asserts non-inversion unconditionally and strictness at any separation a classifier could
reach.

**Graph neighbours are capped on the damped scale**, for the reason `seedCeiling` exists and
`decayOverHalfLife` already carries: a ceiling computed on a different scale from the value it caps
is not a ceiling. A `decision` neighbour of a `state` seed is capped at 0.825, not at 0.5.

**Gap detection was deliberately not touched.** `gaps.ts`'s `impliedType` still tests the raw
`intent.typeWeights` against `gapImpliedTypeWeight: 1.25`, because "did the classifier ask for a note
type" is a question about stage 1, not about what stage 7 chose to pay for it. Damping there as well
would have retuned gap detection inside a ranking change, and the unchanged `false-confidence` and
`empty-recall` rates below are the evidence that it did not.

**The worker backend still cannot measure any of this**, for the third round running.
`node eval/runner.mjs --retriever=worker --split=dev --base-url=http://127.0.0.1:8798
--sign-in-local` scores **0.09909909909909909**, bit-identical to both entries below. Verified again
on this run: all **2,594** returned claims have `score` exactly 0, because `ports.ts:451` is
`rerank: async (_query, contexts) => contexts.map(() => 0)` and `runRerank` sets
`score = rerank × graphPenalty`. **A multiplier applied to zero is zero**, so no stage-7 change of
any kind can appear on the worker path until `indexedPorts` lands. Worker latency this run: median
310.57 ms, p95 515.96 ms, max 611.39 ms over 139 requests; the same 5 adversarial questions are
refused HTTP 400 at the zod boundary.

**What did not move.** `false-confidence` 0.250 and `empty-recall` 0.750, unchanged for the fourth
run: [5.3](#53-the-low_confidence-relevance-floor-is-unreachable)'s dead `low_confidence` rule,
deliberately untouched so this round stays attributable. `hit@5` is also flat at 0.9369 — this change
reorders the top five, it does not put an answer in one that did not have it. The oracle ceiling is
unchanged at 0.9964, as it must be.

**Gates.** `pnpm lint` 0 (416 files, the same 6 pre-existing warnings, none in a file this change
touched), `pnpm typecheck` 0, `pnpm test:core` 0 (53 files, 3,944 tests, +8),
`pnpm test:worker` 0 (114 files, 2,732 tests). The four non-negotiables re-run by name:
`-t "phase 7"` 7 passed, `-t "superseded"` 9 passed, `-t "budget"` 39 passed, `-t "gaps"` 132
passed, and the eval's own seven safety probes pass.

> **The test runner is flaky on this machine and it is not the tests.** Two of the four suite runs
> above failed the first time with `[vitest-pool-runner]: Timeout waiting for worker to respond` —
> five test *files* that never started, counted as five failed tests — and took 1,008 s against a
> normal 22 s. `--maxWorkers=4` makes it reproducible. Check the failure text before believing a red
> suite.

**The next binding constraint, from this round's evidence.** The relevance base, again and more so.
Stage 7 now moves the best relevant claim by a mean of 0.05 ranks, so there is very little ordering
damage left to undo: `precision@1` is 0.7928 and `MRR` 0.8467, but `precision@5` is 0.7027, which
says the right note takes rank 1 and the next four slots go elsewhere. Re-ranking the *shipped pack*
perfectly scores **0.8505** — measured by sorting each pack's own claims by whether they came from a
relevant note — so 0.80 is reachable by ordering alone, but only just, and only with a signal the
pipeline does not have. 39 of the 111 packs contain fewer than five relevant claims at all, which is
what puts that ceiling below the 0.9964 oracle. `rerankLimit` is not the lever: 12 scores 0.6631,
20 (shipped) 0.6685, 30 0.6667, 40 0.6414, 60 0.6126, all measured this round on the control.

### 2026-08-12 — score how much of a note matched. precision@5 0.5640 -> 0.6685

**The change.** One new factor in step 7's product, `components.noteSupport`, and one new knob,
`RetrievalConfig.supportFloor`, default **0.5**. `adjustScores` counts, in one pass over the list it
is handed, how many candidates each note contributed, and maps that count onto `[supportFloor, 1]`:

```text
support(note) = candidates in this list whose permalink is that note's
noteSupport   = supportFloor + (1 - supportFloor) × support(note) / max support
```

At `supportFloor: 1` the factor is exactly 1 everywhere and the pipeline is bit-identical to the
previous entry's, which is the control this entry is measured against. Nothing else moved: no
retriever, no fusion, no rerank, no limit, no half-life, no other weight.

**Why this and not something else.** After the freshness floor, the funnel said the answers were
already in the pack — `hit@5` 0.9189 against `precision@5` 0.5640, with an oracle ceiling of 0.9964.
A relevant note was in the top five for 91.9% of questions while only 2.82 of the five slots held
one, so the remaining loss was **which of the other four slots get filled**, not retrieval. Reading
the packs for what distinguishes a relevant note from an irrelevant one gave a quantity the pipeline
was already computing and throwing away:

> Over the 111 dev questions with the multiplier off, a note the question names contributes **2.57**
> claims to the pack; a note it does not contributes **1.22**. That is 2.11x, where the relevance
> base spans 1.374x within the same list. A chunk score says how well one passage matched. It never
> says how many of a note's passages matched, and on this corpus the second is the wider signal.

**The sweep.** `node eval/runner.mjs --split=dev --config='{"supportFloor":F}'`, reference backend.
The last column is the widest support ratio inside one pack, median over the 111 questions,
against a relevance base that spans 1.374x in the same lists:

| floor | precision@5 | precision@1 | hit@5 | MRR | support's range in one list |
| --- | --- | --- | --- | --- | --- |
| 1 (multiplier off) | 0.5640 | 0.7207 | 0.9189 | 0.7998 | 1.000x |
| 0.65 | 0.6649 | 0.6937 | 0.9459 | 0.7923 | 1.356x |
| 0.55 | 0.6667 | 0.7207 | 0.9459 | 0.8038 | 1.509x |
| **0.5 (shipped)** | **0.6685** | **0.7297** | **0.9369** | **0.8031** | **1.600x** |
| 0.35 | 0.6865 | 0.7658 | 0.9369 | 0.8275 | 1.951x |
| 0.25 | 0.6919 | 0.7838 | 0.9279 | 0.8357 | 2.286x |
| 0 (un-floored) | 0.7063 | 0.7748 | 0.9279 | 0.8290 | 4.000x |

**`precision@5` rises all the way down, and 0 is not the answer.** The selection rule was fixed
before the column was read: *the largest floor — the least authority handed to a multiplier the PRD
does not specify — that leaves every headline metric strictly above the multiplier-off control and
leaves the PRD phase 7 gate tests passing as written.* That is 0.5.

- Going lower breaks the eval's unanimity in one direction and the test suite in the other. At 0.35
  `decay.test.ts`'s convention-at-the-same-age gate fails; at 0.25 both phase 7 gate tests fail. The
  cause is one sentence: a one-chunk note carrying the query verbatim is demoted below a four-chunk
  note that merely mentions it.
- Going higher costs `precision@1`, which ties the control at 0.55 and is below it from 0.6 up.
- And at 0 this multiplier spans **4.000x** against a base that spans 1.374x, which is the arithmetic
  the previous entry spent itself removing from freshness. A signal wider than the thing it
  multiplies does not adjust a ranking, it replaces one.

0.5 costs **0.0378 precision@5** against the un-floored value — about four questions — and that is
the price of the three bullets above, stated rather than buried.

**Measured effect**, `node eval/runner.mjs --split=dev`, reference backend, before -> after:

| metric | before | after |
| --- | --- | --- |
| precision@5 | 0.5640 | **0.6685** |
| precision@1 | 0.7207 | 0.7297 |
| precision@3 | 0.6246 | 0.7087 |
| hit@5 | 0.9189 | 0.9369 |
| hit@10 | 0.9550 | 0.9640 |
| MRR | 0.7998 | 0.8031 |
| nDCG@10 | 0.6934 | 0.7169 |
| recall@5 | 0.6502 | 0.7042 |
| block-level precision@5 | 0.3569 | 0.4078 |
| distinct-note precision@5 | 0.3243 | 0.3532 |
| answer text returned in full | 0.8649 | 0.8649 |
| stage 7 retention (reranked -> adjust@5) | 93.6% (102/109) | **95.4% (104/109)** |
| answers lost at stage 7 | 7 | 5 |
| used_tokens median / violations | 3,273 / 0 | 3,271 / 0 |
| oracle ceiling | 0.9964 | 0.9964 |

The **distinct-note** row is the one that says this is not a metric artifact. The obvious objection
to a note-level multiplier is that it wins `precision@5` by clumping several chunks of one note into
the top five, which the claim-level metric rewards; the distinct reading collapses those to one and
it moves the same way, 0.3243 -> 0.3532. So does block-level precision, 0.3569 -> 0.4078, which
judges the specific paragraph rather than the note.

**No slice regressed.** By answer type: `reference` 0.6066 -> 0.7011, `convention` 0.5650 -> 0.6750,
`decision` 0.5405 -> 0.6541, `note` 0.5789 -> 0.6421, `state` 0.5351 -> 0.6324. By file:
architecture 0.3625 -> 0.4500, conventions 0.5263 -> 0.6000, decisions 0.4625 -> 0.6500, people
0.5857 -> 0.6857, runbooks 0.6645 -> 0.7613, state 0.7067 -> 0.8000. By difficulty: easy
0.5385 -> 0.6462, medium 0.5552 -> 0.6621, hard 0.6074 -> 0.7037.

**What this costs, stated plainly, because it is a real cost and not a rounding one.** Support is a
count over one list, so a relevant note that contributes a single candidate is demoted by it.
Question by question against the control: **52 of 111 improve, 49 are unchanged, 10 get worse**, and
3 of the 10 lose the answer from the top five altogether. They are all the same shape — `arch-012`
("what does the vault hold") wants `architecture/vault-service`, which holds one candidate against
`runbooks/token-vault-key-rotation`'s three; `dec-019` wants `conventions/testing`, one against
`conventions/on-call`'s two. The corpus has a one-chunk note (`state/deploy-freeze`) and its mean is
4.59, so this is not hypothetical. **Anyone lowering `supportFloor` is buying `precision@5` with
exactly those questions, and should read them first.**

`pipeline.test.ts`'s "keeps a live sibling of the excluded content" is the same case in miniature and
now says so: the planted one-chunk note carrying the marker verbatim is 1.08x more relevant than the
four-chunk note that beats it, which is not enough to pay a 1.6x support penalty. The test keeps its
load-bearing assertion (the superseded twin is absent, the live sibling is present), pins the new
rank with that arithmetic, and asserts that under `supportFloor: 1` the old rank-1 ordering is back —
so it is a configuration and not a regression. No assertion was deleted or loosened.

**The four sites that assert the score identity** (§6.1's table) were all updated in this change, as
that section requires: `decay.test.ts`'s `recomputeScore`, `expand.test.ts`'s `identity`, the
exhaustive `toEqual` on a neighbour's components — which now names `noteSupport: 1`, because step 5
does not know how much of a note matched — and `eval/report.mjs`'s failure table, which gains a
`support` column between `graph` and `fresh` so the ten columns still multiply out to the score.

**Graph neighbours are capped at their seed's support**, for the reason `seedCeiling` exists. A
neighbour whose note happens to hold more of the list than its seed's note would otherwise turn PRD
4.1's `score × 0.6` into a bonus — the 203.92x defect in a second costume — so the seed's own support
is a ceiling on the neighbour's, found through the seed's `chunkId`.

**The worker backend still cannot measure any of this.** `node eval/runner.mjs --retriever=worker
--split=dev --base-url=http://127.0.0.1:8798 --sign-in-local` scores **0.0991**, bit-identical to the
previous entry (0.09909909909909909 both times) for the reason recorded there: `ports.ts:451`'s
`rerank: async (_query, contexts) => contexts.map(() => 0)` makes every score exactly 0, verified
again on this run over all **2,594** returned claims. A multiplier applied to zero is zero. Worker
latency this run: median 307.68 ms, p95 544.30 ms, max 609.98 ms over 139 requests.

**What did not move.** `false-confidence` 0.250 and `empty-recall` 0.750, unchanged for the third
run: that is §5.3's dead `low_confidence` rule, deliberately untouched so this round stays
attributable. The oracle ceiling is unchanged at 0.9964, as it must be — nothing about the candidate
set changed.

**Gates.** `pnpm lint` 0 (416 files, the same 6 pre-existing warnings), `pnpm typecheck` 0,
`pnpm test:core` 0 (53 files, 3,936 tests, +8), `pnpm test:worker` 0 (114 files, 2,732 tests). The
four non-negotiables re-run by name: `-t "phase 7"` 7 passed, `-t "superseded"` 9 passed,
`-t "budget"` 39 passed, `-t "gaps"` 132 passed, and the eval's own seven safety probes pass.

**The obvious next move on this evidence, and why it is not a floor sweep.** Support is a count, and
a count cannot tell a one-chunk note that matched entirely from a six-chunk note that matched once.
The quantity that can is the *fraction* of a note's chunks present, which needs a per-note chunk
count that `Candidate` does not carry and `CandidateStore` would have to supply. That change removes
the cost above rather than bounding it, and it is worth more than any value of this knob.

---

### 2026-08-12 — bound the freshness multiplier. precision@5 0.3279 -> 0.5640

**The change.** One new knob, `RetrievalConfig.freshnessFloor`, default **0.9**, applied in
`packages/core/src/retrieval/decay.ts`:

```text
freshness = freshnessFloor + (1 - freshnessFloor) × exp(-ageDays / halfLife)
```

At 0 it is PRD 4.2's formula unmodified, so the knob is a superset of the old behaviour and the old
behaviour is still one config value away. Nothing else moved: no retriever, no fusion, no rerank, no
limit, no other weight.

**Why this and not something else.** Section 5.1 said stage 7 multiplies a relevance base spanning a
median 1.373x by a decay spanning a median 500x. The dev-split funnel put a number on the damage — of
the 109 questions whose answer survived into the reranked top 20, only 70 still had it in the top
five after stage 7 — and reading the 39 losses individually gave the decisive fact:

> In **32 of the 39** questions lost at stage 7, the demoted relevant claim had a **higher** relevance
> base than the claim that displaced it. Median relevance ratio 1.115x in the relevant claim's
> favour; median freshness ratio 0.141x against it. All 39 lost on freshness, none on relevance.

A signal that is wrong 32 times out of 39 when it overrules the ranking is not mis-weighted, it is
out of range. Bounding its range is the minimum change that addresses that; the alternatives in the
brief (weighted sum over normalised components, moving decay after rerank) all also change what the
other four multipliers mean, and none of them could have been attributed afterwards.

**The sweep.** `node eval/runner.mjs --split=dev --config='{"freshnessFloor":F}'`, reference backend:

| floor | precision@5 | precision@1 | hit@5 | MRR | answers lost at stage 7 |
| --- | --- | --- | --- | --- | --- |
| 0 (PRD 4.2) | 0.3279 | 0.279 | 0.631 | 0.453 | 39 |
| 0.3 | 0.4072 | 0.324 | 0.766 | 0.515 | 24 |
| 0.5 | 0.4955 | 0.441 | 0.865 | 0.612 | 13 |
| 0.7 | 0.5297 | 0.559 | 0.919 | 0.705 | 6 |
| 0.8 | 0.5568 | 0.640 | 0.937 | 0.753 | 5 |
| **0.9 (shipped)** | **0.5640** | **0.721** | **0.919** | **0.800** | **7** |
| 0.95 | 0.5640 | 0.730 | 0.919 | 0.806 | 7 |
| 0.98 | 0.5694 | 0.721 | 0.919 | 0.799 | 7 |
| 1 (decay off) | 0.5712 | 0.721 | 0.919 | 0.800 | 7 |

0.9 is the **smallest floor on the plateau**, chosen by that rule rather than by picking the largest
number in the column: it captures 0.5640 / 0.5712 = **98.7%** of everything the change has to give,
and among the values that do, it is the one that leaves freshness the most authority. Two claims can
now differ in freshness by at most 1/0.9 = **1.111x**, so staleness still decides a near-tie and can
no longer overturn a claim that is 12% more relevant. The gain from 0.9 to 0.98 is 0.0054 — under one
question — bought by reducing freshness to a 2% effect, which is not a trade worth making.

**Measured effect**, `node eval/runner.mjs --split=dev`, reference backend, before -> after:

| metric | before | after |
| --- | --- | --- |
| precision@5 | 0.3279 | **0.5640** |
| precision@1 | 0.2793 | 0.7207 |
| precision@3 | 0.3453 | 0.6246 |
| hit@5 | 0.6306 | 0.9189 |
| MRR | 0.4530 | 0.8000 |
| nDCG@10 | 0.4663 | 0.6931 |
| block-level precision@5 | 0.2157 | 0.3571 |
| answer text returned in full | 0.8559 | 0.8649 |
| stage 7 retention (reranked -> adjust@5) | 64.2% (70/109) | **93.6% (102/109)** |
| answers lost at stage 7 | 39 | 7 |
| answers lost at stage 9 (packing) | 1 | 0 |
| stage 7 moved the best relevant claim | 86 of 109, mean 3.04 ranks | 22 of 109, mean 0.34 ranks |
| used_tokens median / violations | 3,282 / 0 | 3,273 / 0 |
| oracle ceiling | 0.9964 | 0.9964 |

**No slice regressed.** By answer type: `reference` 0.3209 -> 0.6066, `state` 0.2703 -> 0.5351,
`decision` 0.3027 -> 0.5405, `note` 0.3263 -> 0.5789, `convention` 0.5250 -> 0.5650. By file,
`conventions` is flat at 0.5263 and every other file rises; `state` questions rise most
(0.3600 -> 0.7067). That `state` — the recency-sensitive type — is the biggest winner refutes the
obvious objection that this trades recency for relevance, and it refutes a per-type floor before
anyone builds one: if a low floor helped anywhere it would help there, and it does not.

**What did not move.** `false-confidence` is unchanged at 0.250, `empty-recall` at 0.750: those are
section 5.3's dead `low_confidence` rule, which is a separate defect and is untouched. The oracle
ceiling is unchanged, as it must be — nothing about the candidate set changed.

**What this cost, stated plainly.** The un-floored decay could overturn *any* relevance gap. It can
now overturn at most 1.111x. One test asserted the old behaviour as a product claim — a 400-day-old
`state` note losing to a fresh one even when it was 5x more relevant, verified and human-written —
and with the floor the old note wins that comparison. That test now pins the new contract in both
directions and says why (`decay.test.ts`, "no longer beats every other signal at once"). PRD phase
7's actual gate is equal relevance, and it still holds exactly: the ordering is unchanged and the
margin is now 0.9 / 0.96997 = 0.928 instead of 4e-13.

**The status glyph keeps the un-floored decay.** The design draws freshness as bar opacity "on a
continuous scale from 1.0 (fresh) to 0.35 (at half-life)", which a number bounded to `[0.9, 1]`
cannot express. `/api/waymark` and the note summaries therefore read
`FRESHNESS_DISPLAY_CONFIG` — the shipped config with the floor removed — so the floor stays what it
is, a bound on what age may buy **in a ranking**, and not a claim about how stale anything is. One
formula, one function, two configs, and the difference is greppable by that constant's name.

**The worker backend cannot measure this yet, and its number did not move.**
`node eval/runner.mjs --retriever=worker --split=dev --base-url=http://127.0.0.1:8798
--sign-in-local` scores **0.0991**, bit-identical to the same command before the change. That is not
a disagreement with the reference number, it is `ports.ts:451`'s degraded reranker:
`rerank: async (_query, contexts) => contexts.map(() => 0)` makes `score = rerank × graphPenalty`
exactly 0 for every candidate, so stage 7 multiplies zero and **no** stage-7 change of any kind can
show up. Verified on this run: all **2,595** returned claims have score exactly 0. Until that
fallback returns 1 or the fused RRF score, the worker path measures the defect and not the ranking.

**Gates.** `pnpm lint` 0 (416 files, the same 6 pre-existing warnings), `pnpm typecheck` 0,
`pnpm test:core` 0 (53 files, 3,928 tests, +6), `pnpm test:worker` 0 (114 files, 2,732 tests).

---

## 1. The pipeline

A question goes in. A `ContextPack` comes out: ranked claims with provenance, the disagreements
between them, and an explicit list of what the knowledge base does **not** know.

```
query
 ├─ 1. intent classify        → type weights, recency bias (pure fn, no LLM)
 ├─ 2. BM25 over chunks_fts   → top 50
 ├─ 3. dense over Vectorize   → top 50   (filter: not superseded, not deleted)
 ├─ 4. RRF fuse               → top 60          score = Σ 1/(k + rank), k=60
 ├─ 5. graph expand 1-hop     → +20 via relations, score × 0.6
 ├─ 6. rerank (bge-reranker)  → top 20   [skip if corpus < 200 chunks]
 ├─ 7. score adjust           → × freshness_decay(type, age)
 │                             × verification_weight
 │                             × author_kind_weight
 ├─ 8. semantic dedup         → collapse near-duplicates (cosine > 0.92)
 └─ 9. pack to budget         → ContextPack
```

That diagram is PRD 4.1 verbatim. The code follows it stage for stage;
`packages/core/src/retrieval/pipeline.ts` is the file, and its numbered comments match the numbers
above.

### Which stages exist, and which do not

Six stages are arithmetic over plain data and live in `packages/core`, which has **zero Cloudflare
imports** and no clock: time enters as an explicit `now` parameter on every call. Three stages need
bindings — D1's FTS5, Vectorize, Workers AI — and are reached through narrow interfaces called
*ports*, declared in `packages/core/src/retrieval/ports.ts`. Seven interfaces, no Cloudflare type
named anywhere.

| Stage | Kind | Where the algorithm lives | Deployed status |
| --- | --- | --- | --- |
| 1 intent | pure | `packages/core/src/retrieval/intent.ts` | works |
| **2 BM25** | **port `LexicalRetriever`** | D1 `chunks_fts` | **NOT IMPLEMENTED** |
| **3 dense** | **ports `DenseRetriever` + `Embedder`** | Vectorize, `@cf/baai/bge-m3` | **NOT IMPLEMENTED** |
| 4 fuse | pure | `fuse.ts` | works |
| 5 graph expand | half: scoring pure, walk is port `GraphExpander` | `expand.ts` | **port NOT IMPLEMENTED** |
| **6 rerank** | **port `Reranker`** | `@cf/baai/bge-reranker-base` | **NOT IMPLEMENTED** |
| 7 score adjust | pure | `decay.ts` | works |
| 8 dedup | pure | `dedup.ts` | works |
| 9 pack | pure | `pack.ts` | works |
| conflicts | pure | `conflict.ts` | works |
| gaps | pure | `gaps.ts` | works |

**"NOT IMPLEMENTED" is literal, and it is deliberate for this wave.**
`apps/worker/src/retrieval/ports.ts` wires `recall` to `notIndexedPorts`, in which six of the seven
ports answer every call with an empty list. Only `IndexHealth` is real, because it is two counts over
`chunks` and needs no model. From the file:

```ts
export function notIndexedPorts(ctx: ProjectContext): RetrievalPorts {
	const empty: RankedList = { retriever: "lexical", ids: [] };
	const lexical: LexicalRetriever = { search: async () => empty };
	const dense: DenseRetriever = { search: async () => ({ retriever: "dense", ids: [] }) };
	// …
```

The consequence on the live instance is that `recall` returns `claims: []` with a populated `gaps`.
That is correct behaviour for a knowledge base with no index — PRD 4.3 makes `gaps` load-bearing
precisely so that "the index does not exist yet" can be said in words rather than as a 501 — but it
means **the deployed instance cannot be searched and has never been measured**. Verified over the
network on 2026-08-10: `GET /health` on the deployment returns HTTP 200 with
`"status":"degraded"`, `embedding_coverage: 0`, `chunks_total: 1`, `chunks_embedded: 0`,
`lag_seconds: 2494`. `POST /api/recall` returns 401, because `SIGNPOST_AUTH_SECRET` is unset and no
session can be minted.

The seam for the next wave is named in that file's own header: a new
**`apps/worker/src/retrieval/indexed.ts`** exporting `indexedPorts(ctx)`, with `LexicalRetriever`
from `chunks_fts MATCH`, `DenseRetriever` from `ctx.vectors.query`, `Embedder` from
`ctx.ai.run("@cf/baai/bge-m3")` and `Reranker` from `ctx.ai.run("@cf/baai/bge-reranker-base")`.
`IndexHealth` imports the existing `liveIndexHealth` unchanged. The whole change in `ports.ts` is two
function bodies.

The upside of the port design is that the entire algorithm runs, is measured and is debugged in plain
Node in seconds, with no account and no network — see
[appendix A](#appendix-a-the-reference-implementations).

### Entry point

```ts
import { createRecall, DEFAULT_STEPS, DEFAULT_RETRIEVAL_CONFIG } from "@signpost/core/retrieval";

const recall = createRecall(DEFAULT_STEPS);
const pack = await recall(
  {
    text: "what did we decide about auth",
    budgetTokens: 4000,
    now: Date.now(),          // the only clock read in the whole pipeline, at its boundary
  },
  ports,                      // the seven Cloudflare-bound implementations
  DEFAULT_RETRIEVAL_CONFIG,   // every tunable, in one object
);
```

`recall` takes a fourth argument, `steps`, holding the eight pure functions.
`createRecall(DEFAULT_STEPS)` binds them and gives back the three-argument signature above. They are
injected rather than imported so that each pure stage is written against a compile-checked contract
(`RetrievalSteps` in `retrieval/types.ts`) and a signature drift fails to compile in the module that
caused it.

`DEFAULT_STEPS` lives in `packages/core/src/retrieval/steps.ts`. It is the only place that says which
implementation of each stage ships: a second packer or a second gap detector would have to appear in
that object to be reachable, and two entries cannot occupy one key. Injecting a replacement step is
also how every experiment in [section 5](#5-why-it-fails) was run without editing the repository.

### 1. Intent classification — `intent.ts`

**What it does.** Reads the query text and produces a per-note-type multiplier and a recency bias.
"Why did we choose Postgres" wants `decision`; "how do we name branches" wants `convention`; "where
are we on the ledger migration" wants `state`.

**Why it exists.** BM25 and a vector model both answer "which text is about this". Neither answers
"which *kind* of note does this question want", and the corpus contains five kinds whose words
overlap heavily.

**How.** A table of 14 rules (`INTENT_RULES`), each a set of cues and a set of type boosts. No model
call, so it costs nothing and cannot vary between runs. Matching is over a token array, never a regex
against raw text.

| Signal family | Rules | Boost |
| --- | --- | --- |
| `decision.*` | `why`, `what-was-decided`, `who-decided` | `decision` × 2.4 / 2.4 / 2 |
| `convention.*` | `how-do-we`, `permission`, `rule` | `convention` × 2.2 / 2.2 / 2 |
| `state.*` | `where-are-we`, `status`, `is-done` | `state` × 2.6 / 2.6 / 2.2, recency +0.7 / +0.7 / +0.6 |
| `reference.*` | `what-is`, `how-it-works`, `where-is` | `reference` × 1.8 / 2 / 1.8 |
| `recency.*` | `current`, `historical` | recency +0.5 / −0.5 |

**The rule that matters most.** The five type weights are normalised so their **mean is exactly 1**.
Intent redistributes attention; it never inflates it. Without that, a query firing three rules would
produce uniformly larger scores than one firing none, and every absolute threshold downstream would
mean a different thing per query.

**How much it is worth today: almost nothing.** The declared `intent` matches `classifyIntent` on
**46 of 157** scored questions (0.293) — measured from `results.json`, and `node
eval/validate-questions.mjs --intent` reports each disagreement by name. Removing the type weights
entirely moves `precision@5` from 0.3159 to 0.308. Stage 1 is close to a no-op on this question set.
Decide what it is *for* before tuning it.

### 2. BM25 over `chunks_fts` — port, NOT IMPLEMENTED

**What it does.** Full-text search over the contentless FTS5 index, returning the top 50 chunk ids,
best first.

**Why it exists.** Lexical search is the half of retrieval that never hallucinates. If the question
contains the word `idempotency` and a note contains the word `idempotency`, no embedding model is
needed to connect them, and no embedding model should be trusted to be the only thing that does.

**Notes for whoever writes it.** `chunks_fts` has three columns (`text`, `title`, `tags`) and is
contentless, so it is kept in sync by explicit inserts in `CommitService`, never by triggers. FTS5's
implicit operator is `AND`, which over a natural-language question matches nothing; the query has to
be tokenised and joined with `OR`. FTS5's `bm25()` returns a **negative** number so that SQL's
ascending sort puts the best row first — flip the sign or read the order backwards, and be deliberate
about which.

`ReferenceLexicalRetriever` already implements the real algorithm with FTS5's own parameters, so the
eval measures a faithful stand-in for this stage. It is the one Cloudflare-bound stage whose numbers
should transfer.

### 3. Dense search over Vectorize — port, NOT IMPLEMENTED

**What it does.** Embeds the query with `@cf/baai/bge-m3` (1024 dimensions, L2-normalised) and
returns the 50 nearest chunks.

**Why it exists.** It is the half that handles paraphrase. "How often do we pay out" and "payout
cadence" share no content word.

**Invariants (PRD 2.5).** Assert `vector.length === 1024` and an L2 norm within `[0.99, 1.01]` on
every embedding before it is written. A comparable product shipped un-normalised vectors and silently
degraded to keyword-only search; the failure is invisible without the assertion. Store `model_id` and
`embedding_version` with every vector, and treat a model change as a full reindex, never a silent
mix.

This is the only stage that takes the `RetrievalFilter`, because Vectorize applies metadata filters
inside the index where it is far cheaper than fetching 50 vectors and discarding 40. The same filter
is applied again as a pure predicate over materialised candidates. That belt-and-braces is
deliberate: it is what stops superseded content reaching a pack through the lexical door, which takes
no filter at all.

**The stand-in is worse than nothing, and this is not obvious.** `ReferenceEmbedder` is a signed
hashing trick over stemmed tokens — a bag-of-words model that cannot match a synonym. Fusing it into
BM25 **costs** `precision@5`. Under a relevance-only ranking, measured:

| Retrievers fused | `precision@5` | `precision@1` |
| --- | --- | --- |
| BM25 only (`denseLimit: 0`) | **0.6446** | 0.8471 |
| BM25 + hashing dense (shipped) | 0.5898 | 0.8153 |
| hashing dense only (`lexicalLimit: 0`) | 0.4573 | 0.6879 |

Two consequences nobody has written down. First, the eval's number is depressed by its own
scaffolding as well as by the missing semantics, so "a floor, not a prediction" is true for a second
reason. Second, **fusion parameters tuned against the reference stack are tuned against a retriever
that is actively wrong**, and any `rrfWeights` work done now will have to be redone. This is not an
argument against Vectorize: `bge-m3` is a different object, and paraphrase recall is the entire point
of stage 3.

### 4. Reciprocal rank fusion — `fuse.ts`

**What it does.** Merges the two ranked lists into one: `rrf = Σ over the lists containing this id of
1 / (k + rank)`, with `k = 60` and `rank` 1-based. Keeps the top 60.

**Why it exists.** BM25 scores and cosine similarities are not commensurable, and normalising them
against each other requires a corpus-shaped constant someone would have to re-tune whenever the
corpus changed. Ranks are commensurable: "third best according to BM25" and "third best according to
the vectors" mean the same thing. A document both retrievers like beats a document only one of them
likes, and it falls out of the arithmetic rather than being configured.

**Absence is scored too, and this is the correction that mattered most.** A retriever asked for 50
results that returned 50 has not said *nothing* about the 51st — it has said "worse than fiftieth,
according to me". Classic RRF throws that away, and with `k = 60` over two lists of 50 the
consequence is severe: the best a single list can give is `1/61 = 0.016393` and the worst two lists
can give is `2/110 = 0.018182`, so **every dual-retriever hit outranked every single-retriever hit,
unconditionally**. Measured across all 196 questions, a single-list entry was ranked above a dual-list
entry **zero times** in 11,292 fused entries; the first single-list result sat at fused position 27
on average, past `rerankLimit`; and on "when is the christmas freeze" the note saying "No production
deploys between 18 December and 4 January" was **BM25 rank 1 of 438** and the pack replied *"No claims
found about christmas freeze."* `rrfAbsentRankOffset` fixes it; the arithmetic is under
[fusion knobs](#fusion-stage-4).

**Ordering.** `rrf` descending, then `chunkId` ascending. The tiebreak is not a detail: floating-point
scores collide constantly, and an unspecified tiebreak makes results flap between runs and the eval
set unreproducible. Contributions are summed in a canonical ascending order so that permuting the
input lists cannot change the last bit of a sum.

**Weights.** `fuse.ts` also exports `weightedRrfFuse`, where each list carries a multiplier. It is not
wired in and should stay that way unless there is evidence: unequal weights reintroduce the one
corpus-shaped calibration constant RRF exists to avoid — and today they would be fitted to a hashing
embedder.

### 5. Graph expansion — `expand.ts` + port

**What it does.** Walks one hop out from the notes that matched, along the relations their authors
wrote, and admits up to 20 neighbouring chunks at a discount.

**Why it exists.** The answer to "how do we roll back a migration" is often not in the note that
matches the words; it is in the note that one links to. A knowledge base with a graph in it that
retrieval does not use is a knowledge base with a decorative graph.

**Four rules, each answering a specific failure.**

1. **A neighbour inherits, it does not accumulate.** When two matched notes both link to the same
   third note, that note's base score is the **maximum** of the two, never the sum. Summing is the
   classic graph-expansion failure: an architecture-overview page that forty notes link to
   accumulates forty small contributions and outranks the note that answered the question. Under the
   maximum, being popular buys nothing.
2. **The verb decides how much survives.** `depends_on` lands on something you probably need;
   `supersedes` lands on the content default recall exists to keep out.

   | Verb | Weight | Reasoning |
   | --- | --- | --- |
   | `depends_on` | 1.0 | what this note relies on is usually needed to act on it |
   | `implements` | 0.9 | the concrete form of an abstract answer |
   | `part_of` | 0.9 | the containing whole |
   | `blocks` | 0.7 | related work, one step further away |
   | `relates_to` | 0.6 | the author said "see also" and no more |
   | anything else | 0.6 | an author's invented verb is a deliberate link, not a mistake |
   | `supersedes` | 0 | never followed |

3. **The discount is a ceiling, and it was not one.** `graphPenalty = relationWeight × 0.6 ^ hops`,
   stored in `ScoreComponents.graphPenalty` with `graphHops`. Stage 5 hands the neighbour the seed's
   *fused* relevance and leaves the other multipliers at 1, because it has neither `now` nor the
   intent classification. Stage 7 then computed them from the **neighbour's own** metadata, and the
   discount inverted: a fresh `reference` neighbour one link from a stale `state` seed kept the seed's
   relevance and swapped in its own, much slower, decay. Worst case measured, `dec-009`: the neighbour
   came out **203.92 times** its parent's score, and 15 of the 49 graph-expanded candidates that
   reached a pack were above the seed they borrowed from.

   The repair is `ScoredCandidate.graphSeed`: stage 5 records which seed a neighbour borrowed from,
   and `decay.ts` caps each of the neighbour's four trust multipliers at that seed's. `min` rather
   than substitution, so a neighbour that is *staler* than its seed keeps its own worse number — the
   discount is a ceiling on what borrowing can buy, not a licence to inherit someone else's freshness.
4. **`related` reports what the author wrote, not what retrieval followed.** Every one-hop outbound
   edge appears in `Claim.related`, `supersedes` and unresolved forward references included. Weights
   govern retrieval; `related` is provenance. Unresolved forward references (invariant I3) are dropped
   from *expansion* — there are no chunks to retrieve — but never from `related`.

**Graph expansion is inert on the current eval, and that is a gap in the eval, not a defect.** Across
196 queries, 27 of the 3,820 candidates entering stage 7 are graph-expanded and they survive to the
packer in 2 of 196 packs. Verified: `node eval/runner.mjs --config='{"graphExpandLimit":0}'` produces
`precision@5` 0.3159, `hit@5` 0.643, MRR 0.438, nDCG 0.459 — identical to the shipped run on all four
ranking metrics. Not *everything* is identical: median `used_tokens` moves 3,284.5 → 3,283, so the
two packs are not byte-for-byte the same. The `seedCeiling` mechanism above, which cost two critical
defects to get right, is not exercised by any scored number in this report.

### 6. Reranking — port, NOT IMPLEMENTED

**What it does.** A cross-encoder scores each candidate against the query directly, rather than
comparing two independently-computed vectors. **It no longer cuts** — see the note below.

**Why it exists.** Bi-encoders compress a chunk into one vector before the query is known. A
cross-encoder reads both together and is markedly better at "does this passage actually answer this
question", which is the exact discrimination the top of the list needs. **It is the only change that
can carry the gate** — see [section 5.5](#55-how-much-of-the-03159-is-the-two-missing-stages).

**Skip the model, never the cut — but the cut moved on 2026-08-13.** Below `rerankMinCorpus` (200
chunks) the model call is skipped and the list is still truncated to `rerankLimit`; the shape of the
pipeline must not change with the size of the corpus, or a small knowledge base runs stages 8 and 9
over eighty candidates while a large one runs them over twenty, and the two get tuned against each
other. What changed is *where*: the cut is made at the end of stage 7, by the composed score, rather
than here in fusion order. The eval forces the skip branch — `eval/retriever.mjs` sets
`rerankMinCorpus: Infinity` for `--reranker=none` — so **the list entering stage 7 in every
measurement below the 2026-08-13 entry is the RRF order truncated to 20, and in every measurement
after it is the whole RRF order, up to 80**. See [section 0](#0-tuning-log).

**Score semantics.** The rerank score *replaces* the relevance base: afterwards
`score = rerank × graphPenalty`, and every other factor is still `1`, so the score identity holds.
Two downstream consumers assume that base is an RRF-scale quantity, and both break when a real
cross-encoder lands — the gap floor ([5.3](#53-the-low_confidence-relevance-floor-is-unreachable))
and any fixed-scale normalisation ([6.1 risks](#61-replace-the-multiplicative-composition-in-stage-7)).

### 7. Score adjustment — `decay.ts`

**What it does.** Multiplies in eight things: the intent type weight, note support, term coverage,
note coverage, link consensus, freshness decay, the verification weight and the author-kind weight.
Then re-sorts, and — since 2026-08-13 — cuts to `rerankLimit`. **This was the stage that was
broken**; the analysis is [section 5](#5-why-it-fails) and the six changes that answered it are in
[section 0](#0-tuning-log).

**The intent type weight** is damped towards 1 before it is applied:
`1 + typeWeightStrength × (intent.typeWeights[noteType] - 1)`. Undamped it was the widest factor in
the product (median 2.200x within one list) and the least informative (1.057x between a relevant
claim and an irrelevant one). Damping preserves the classifier's ordering exactly and bounds only
what a note type may buy. Added 2026-08-12.

**Note support** is the only factor that is a property of the *list* rather than of one candidate:
`supportFloor + (1 - supportFloor) × (this note's candidates in the list / the note's own chunk
count)`. It says how much of a note matched, which a chunk score never says. Added 2026-08-12; its
denominator moved from the best-supported note's count to the note's own size later the same day,
which is worth 0.7027 -> 0.7171 and needs `Candidate.noteChunkCount`. A list in which any candidate
arrives without that count falls back — as a whole list, never note by note — to the count form.

**Term coverage** is the only factor in this stage that is about the *question*:
`coverageFloor + (1 - coverageFloor) × (the query's named terms appearing in this candidate / the
query's named terms)`. The terms, the token identity and the contiguous-phrase rule are stage 9's, from
`retrieval/terms.ts`, so a claim ranked for covering a word cannot have that word reported as a gap in
the same pack. It is the most discriminative quantity in the stage (1.487x between a relevant claim and
an irrelevant one, against 1.097x for the relevance base) and the only one permitted to span more than
the base. Added 2026-08-12, worth 0.7171 -> 0.7568.

**Note coverage** is the same measurement one level up — the share of the query's terms appearing in
*any* of a note's candidates in this list, over `[noteCoverageFloor, 1]`. It is a different fact
rather than the same one rescaled: a passage naming two of four terms sits either in a note that
covers the other two somewhere else or in one that never mentions them, and no per-chunk quantity
separates those. Computed in the same pass as term coverage, as a `Set` union per note, so the two
cannot disagree about whether a note contains a word. Added 2026-08-13, worth 0.7568 -> 0.7676.

**Link consensus** is the only factor in this stage — in the whole pipeline after fusion — that is
not made of words: `linkConsensusFloor + (1 - linkConsensusFloor) × (min(other notes in this list
that link to this one, linkConsensusCap) / linkConsensusCap)`. It asks whether the rest of the
retrieved evidence points here, over the edges stage 5 already fetched for `Claim.related`, so it
needs no port call. It is the widest separator in the stage — 2.701x between a relevant note and an
irrelevant one, against term coverage's 1.487x — and it tops out at 1, so it discounts isolation
rather than rewarding popularity, which is what keeps it clear of the hub failure `expand.ts`
documents. Added 2026-08-13, worth 0.7748 -> 0.7856 and `hit@5` to 1.0000.

**Freshness decay** is `exp(-ageDays / halfLife)`, with the half-life chosen per note type:

| Type | Half-life | At 30 days | At 180 days | At 400 days | Rationale |
| --- | --- | --- | --- | --- | --- |
| `convention` | 365 d | 0.92 | 0.61 | 0.33 | rules change slowly |
| `decision` | 180 d | 0.85 | 0.37 | 0.11 | durable but revisitable |
| `reference` | 180 d | 0.85 | 0.37 | 0.11 | — |
| `state` | 14 d | 0.12 | 2.6e-6 | 3.9e-13 | "auth is 70% done" rots fast |
| `note` (default) | 90 d | 0.72 | 0.14 | 0.012 | — |

Two details worth carrying:

- **Age is measured from `provenance.createdAt`, not from the note's `created_at`.** Block ids are
  content-derived, so a rewritten paragraph is a new block with a new provenance row: age is the age
  of the words, not the age of the file.
- **`exp(-age/T)` is an e-folding time, not a half-life.** At `age = T` the multiplier is
  `1/e ≈ 0.368`; it halves at `0.693 × T`. The formula and the numbers are the PRD's; only the label
  is loose. The table above gives the real values.

**Verification and author kind** are lookup tables:

| Verification state | Weight |   | Author kind | Weight |
| --- | --- | --- | --- | --- |
| `verified` | 1.0 |   | `human` | 1.0 |
| `unverified` | 0.85 |   | `agent` | 0.9 |
| `stale` | 0.5 |   |  |  |
| `superseded` | 0.2 |   |  |  |

The resulting trust order is human-verified > agent-verified > human-unverified > agent-unverified.
`superseded`'s weight of 0.2 looks redundant given that superseded content is excluded structurally —
it is there for the explicit-request path, where a caller has asked for superseded content on purpose
and still wants it ranked below everything current. Do not simplify it away.

**A person's own words are human-verified from the moment the person writes them.** Since
2026-09-16 the editor, the API, the CLI and an upload store a person's new block as `verified` by
that person. `CommitOptions.verifies` in `services/commit.ts` carries the rule. So a person's own
text gets 1.0 × 1.0. Migration
`0023_human_verified_backfill.sql` applied the same rule to older rows. Human-unverified is now a
smaller set:

- a block that a history restore, a session undo or a tombstone restore put back,
- a block of a deleted doc that a save brought back unchanged (`authoredProvenance`),
- a block from a repository pull (`git-sync` is not an account, so it never verifies),
- an older row that the migration left alone, for example a row near an accepted proposal.

These rows keep 0.85. An accepted agent paragraph gets 0.95 × 0.9 = 0.855, which is level with
them and below a person's own text. The comment on `verificationWeights` in
`packages/core/src/retrieval/types.ts` gives the calibration.

**Both trust multipliers are constant across the entire eval.** Every chunk in `eval/corpus/` has
`provenance.state = "unverified"` and `authorKind = "human"`, so `verification` is 0.85 and
`authorKind` is 1.00 on every candidate in every pack. `decay.ts`'s whole trust half — the
verified-agent-beats-unverified-human inequality PRODUCT.md bet 4 rests on — is **unmeasured end to
end**. It is covered by `decay.test.ts` unit tests and by nothing else. For the same reason the
in-process eval cannot see the human-verified rule above: `eval/retriever.mjs` builds every row as
human and unverified, which is what a repository pull stores. `scripts/ingest-eval-corpus.mjs`
writes the same text to a deployment through `POST /api/notes`, which stores it at 1.0. Measured
2026-09-16 with a local edit that stored every eval row as human-verified: every ranking metric was
the same as the baseline (hit@5 1.000, precision@1 0.917, precision@5 0.622, MRR 0.950). Only the
median `used_tokens` moved, from 2,306 to 2,285.5, because each claim then says `verified` and not
`unverified`. A constant multiplier cannot change an order. A corpus that mixes the two rungs is the
eval that could see the rule, and none exists yet.

**Recency bias** from stage 1 has no component of its own. It shortens the effective half-life:

```
effectiveHalfLife = halfLives[noteType] / (1 + recencyBias × recencyBiasStrength)
```

so its entire effect is visible inside `components.freshness` and no factor is missing from the
decomposition. Freshness is clamped to a floor of `2.2e-308` rather than allowed to reach zero: `exp`
hits a hard zero at about 10,432 days for a `state` note, and `Claim.freshness` is documented as lying
in `(0, 1]`.

### 8. Semantic dedup — `dedup.ts`

**What it does.** Collapses candidates whose cosine similarity is at or above `0.92` into one
survivor, keeping a record of what was collapsed.

**Why it exists.** A note that restates its own body in an observation line, and two notes that copied
the same paragraph, otherwise spend the token budget three times on one claim.

**Complete link, not single link.** A candidate joins a cluster only if it clears the threshold against
the representative **and** every member already in it. Single-link clustering collapses `A~B~C` where
`A` and `C` are genuinely different into one group and silently deletes an answer — which is precisely
the failure `conflicts` exists to prevent, committed one stage earlier.

**Election order** decides who survives: score, then verified over unverified, then human over agent,
then fresher, then chunk id. `kept` is re-sorted into presentation order afterwards.

**Similarity** is cosine when both sides carry a usable equal-width embedding, and otherwise falls back
to normalised-text equality and then a 3-word-shingle Jaccard. A half-reindexed corpus is a fallback,
never a throw: `recall` still has to answer.

**Collapsed members are kept, not discarded.** Two chunks that say almost the same thing in almost the
same words are exactly where "24h" and "1h" hide, so the conflict detector searches them.

Dedup is why a claim's rank in the pack is one to six places better than its rank leaving stage 7.
When reading a worked example, be clear which of the two you are quoting.

### 9. Packing — `pack.ts`

**What it does.** Turns surviving candidates into `Claim`s, admits them in rank order while they fit,
and emits the `ContextPack`.

**The algorithm.** Drop any candidate that is nothing but a note's `## Relations` list. Measure the
conflicts, the gaps and the pack shell first, trimming the echoed query if echoing it whole would
breach the budget on its own. Fill greedily in score order, **skipping** a claim that does not fit
rather than stopping at it, so one enormous chunk does not starve everything below it. Then make one
attempt to truncate the highest-scoring claim that missed, at a sentence or line boundary, else a word
boundary, else drop it whole. If anything was dropped or cut, add the `budget_truncated` gap. Finally,
re-measure the assembled pack and drop claims from the bottom until it fits.

**A link list is not a claim.** PRD 3.3 gives every note a `## Relations` section, the chunker has no
notion of one, and so **99 of the seeded corpus's 459 chunks (21.6%) are pure `verb [[target]]`
lists**. They rank well for the worst possible reason: a four-line link list has a tiny `|D|` against
the corpus average, and BM25's length normalisation rewards exactly that. Before they were filtered
they took **228 of 3,765 pack claims (6.1%) across 88 of 196 packs, 50 top-five slots, 23 top-three
slots, rank one three times, and 5.2% of the token spend**. Nothing is lost by dropping them: the pack
already carries a note's edges in `Claim.related`, addressed and machine-readable, at a fraction of the
tokens.

**Where that filter goes matters, and the obvious placement is worse.** Three positions were measured
over the whole eval set (before `rrfAbsentRankOffset` landed, so the absolute numbers are the earlier
ones; the ordering is what the table is for):

| Where | precision@5 | hit@5 | MRR | median tokens |
| --- | --- | --- | --- | --- |
| nowhere | 0.312 | 0.611 | 0.413 | 3,543 |
| **stage 9 only (shipped)** | **0.313** | **0.643** | **0.430** | **3,391** |
| before stage 5, on the seeds | 0.297 | 0.599 | 0.418 | 3,628 |
| between stages 5 and 6 | 0.297 | 0.599 | 0.418 | 3,628 |

Filtering earlier is better engineering — a real cross-encoder should not spend inference on link
lists — and it measures *worse*, identically in both earlier positions. The mechanism is not graph
perturbation: the two earlier placements produce byte-identical results, so the difference is entirely
that three more candidates reach the pack. **Stage 7 re-sorts after the `rerankLimit` cut**, so every
candidate admitted at rank 21 to 23 is another ticket in the freshness lottery of
[section 5](#5-why-it-fails), and they win often enough to displace correct answers. Revisit this
placement the day stage 7 is fixed.

**The wire conventions.** Refs go through `blockRef`/`noteRef` from `ref.ts`, so a permalink stored as
`/backend/auth.md` addresses the same note the `read` tool resolves; an unaddressable block id falls
back to the note ref rather than emitting an address nobody can open. `created_at` is rendered to the
second (`2026-03-14T09:22:00Z`), which is PRD 4.3's format. `score`, `freshness` and
`embedding_coverage` are rounded to three **significant digits** — significant digits rather than
decimal places, so a thoroughly decayed `3.02e-13` survives as itself instead of being published as a
`0` that contradicts `Claim.freshness`'s documented `(0, 1]` range.

### The ContextPack and its two hard contracts

```jsonc
{
  "query": "what did we decide about auth",
  "budget_tokens": 4000,
  "used_tokens": 2847,
  "claims": [{
    "ref": "kmem://backend/auth-decision#blk_7f3a",
    "kind": "decision",
    "text": "Session cookies over JWT — chosen for revocation simplicity.",
    "provenance": { "author_kind": "human", "author": "ankur",
                    "created_at": "2026-03-14T09:22:00Z", "state": "verified" },
    "freshness": 0.91, "score": 0.94,
    "related": ["kmem://backend/session-store"]
  }],
  "conflicts": [{ "topic": "token expiry", "refs": ["kmem://a#b1","kmem://a#b2"],
                  "note": "24h (Mar) vs 1h (Jun). Neither superseded." }],
  "gaps": ["No claims found about refresh-token rotation."],
  "index_health": { "embedding_coverage": 1.0, "lag_seconds": 2 }
}
```

**Contract 1 — `used_tokens <= budget_tokens`, always.** No exceptions, no off-by-one, for every
budget including `0`. To be provable it is defined mechanically:

```
claimCost(claim) = countTokens(JSON.stringify(claim))
used_tokens      = sum of claimCost over the admitted claims
```

`JSON.stringify` rather than the prose alone because the caller pays for the whole serialised entry.
`countTokens` is `estimateTokens` behind the guard in `retrieval/tokens.ts` — a real `cl100k_base` BPE
count from `js-tiktoken`, the same counter the chunker and the `token_count` column use. **Swapping
one without the other breaks the proof.** See [appendix B](#appendix-b-the-tokeniser-is-quadratic-in-one-unbroken-run)
for why the guard exists.

**Conflicts and gaps are not charged.** They are the pack's safety apparatus, not retrieved content.
Charging them would put the two contracts in direct conflict at small budgets — a pack with no claims
must still carry a gap, and at `budgetTokens: 0` there is no room for one. Their cost is reported
separately as `PackerResult.reportTokens`, and the packer additionally keeps the **whole serialised
pack** inside the budget whenever it is at least `reportReserveTokens` (512), by trimming the echoed
query if that is what it takes.

Measured 2026-08-10 over the 100-note corpus at the default 4,000-token budget, 196 questions: median
`used_tokens` **3,284.5**, p50 3,284, p95 3,765, p99 3,905, max 3,911, **0 violations**, worst pack
97.8% of budget.

**Contract 2 — empty `claims` implies non-empty `gaps`.** When retrieval finds nothing the agent must
be told so in words. Handed an empty array it invents an answer, and the knowledge base becomes a
hallucination amplifier — the failure this product exists to prevent. The packer is responsible for it
and `recall` re-checks it before returning, because the failure is silent: nothing throws, no test goes
red, the agent is simply handed nothing and told nothing.

The gap detector (`gaps.ts`) does better than the fallback sentence wherever it can:

| Reason | When | Emissions in 196 packs |
| --- | --- | --- |
| `no_results` | nothing came back, or a query term appears in no claim | 75 (5 whole-query, 70 per-term) |
| `low_confidence` — never-joined | both halves of the question are present and nothing joins them | 50 |
| `low_confidence` — stale | the freshest note of the demanded type is heavily decayed | 27 |
| ~~`low_confidence` — relevance floor~~ | ~~nothing cleared the relevance floor~~ | **0 — unreachable, and deleted on 2026-08-13. See [5.3](#53-the-low_confidence-relevance-floor-is-unreachable)** |
| `low_confidence` — recorded open question | a retrieved `[question]` observation *is* this question | replaced it, 2026-08-13. 2 in 17 negatives, 0 in 111 positives |
| `low_confidence` — unmet duration | the query asks `how long` and no claim about its subject says | replaced it, 2026-08-13. 2 of 13 dev duration questions, both negative |
| `filtered_out` | the caller's own `types`, `since` or verification filters removed the matches | 0 |
| `budget_truncated` | claims were dropped to fit the budget | 11 |
| `index_incomplete` | part of the corpus is not embedded, so dense search could not see all of it | 0 |
| implied-type | the query demands a note kind the pack has none of | 3 |

Counts measured by classifying the gap text of all 196 packs in `results.json`. **Anyone grepping
`results.json` for `low_confidence` will find 77 hits and conclude the floor rule is alive. It is
not** — those 77 are the never-joined and stale rules, which share the reason code.

### The score, decomposed

Every multiplier that produced a score is recorded. This identity holds for every candidate the
pipeline emits, at every intermediate stage, because each factor is `1` until the stage that owns it
has run:

```
base  = components.rerank ?? components.rrf
score = base
      × components.typeWeight      (stage 1, damped and applied in 7)
      × components.graphPenalty    (stage 5, relation weight × 0.6 ^ hops)
      × components.noteSupport     (stage 7, a property of the list)
      × components.termCoverage    (stage 7, how much of the query this passage covers)
      × components.noteCoverage    (stage 7, how much of it this passage's note covers)
      × components.linkConsensus   (stage 7, how many other notes in the list link to it)
      × components.freshness       (stage 7)
      × components.verification    (stage 7)
      × components.authorKind      (stage 7)
```

`lexicalRank`, `denseRank`, `graphHops` and `rerank` are diagnostics rather than factors. **A scorer
whose output cannot be decomposed cannot be tuned**, and a number that appears in `score` but in no
component is a number nobody can explain six months later.

A worked example. A `state` note, written by an agent 40 days ago, nobody has checked, found at rank 3
by BM25 and rank 7 by the vectors, on a query that fired `state.status`:

```
rrf          = 1/(60+3) + 1/(60+7)          = 0.01587 + 0.01493 = 0.03080
typeWeight   = 2.6 / mean(2.6,1,1,1,1)      = 2.6 / 1.32        = 1.970
graphPenalty = 1                            (a direct hit, not a neighbour)
freshness    = exp(-40 / (14 / (1 + 0.7)))  = exp(-4.857)       = 0.00777
verification = 0.85                         (unverified)
authorKind   = 0.9                          (agent)
score        = 0.03080 × 1.970 × 1 × 0.00777 × 0.85 × 0.9       = 0.000361
```

Reading that back: the intent classifier roughly doubled it and freshness cut it by a factor of 129.
That ratio is the whole of [section 5](#5-why-it-fails) in one line.

**That example is the pre-tuning arithmetic and is kept because it is the diagnosis.** On the shipped
configuration the same candidate takes `typeWeight` 1.340 rather than 1.970 (`typeWeightStrength`
0.35), `freshness` 0.9008 rather than 0.00777 (`freshnessFloor` 0.9), and three further factors:
`noteSupport` in `[0.5, 1]` depending on how much of its note reached the list, `termCoverage` in
`[0.35, 1]` depending on how much of the question it answers, `noteCoverage` in `[0.6, 1]`
depending on how much of the question its note answers, and `linkConsensus` in `[0.8, 1]` depending
on how many other notes in the same list link to it — a score near
`0.0284 × noteSupport × termCoverage × noteCoverage × linkConsensus` rather than 0.000361.
[Section 0](#0-tuning-log) is where those six numbers came from.

**Note that any fix to stage 7 breaks this identity, and the identity is asserted in `types.ts` and in
tests.** Redefining the decomposition is part of the change, not a follow-up.

### Conflicts

A conflict needs all three of: a shared subject, a named clash, and no recorded resolution. Clashes
come in three kinds — `quantity` (units normalised, so 4h against 24h and 24h against 7d both fire),
`choice` (the name in front of the copula: Adyen against Stripe) and `polarity` ("never stores" against
"held encrypted").

Granularity is **statements, not chunks**: chunk-against-chunk washes the signal out, because a payout
convention mentions both "once per day" and "48 hours" in the same paragraph. The note copy follows PRD
4.3 exactly — `24h (Mar) vs 1h (Jun). Neither superseded.` — with the year added only when the two
differ. `refs` is **not** narrowed to the members that survived packing; narrowing turns a two-sided
warning into a one-sided one, which is worse than not warning at all.

---

## 2. Configuration

Almost every tunable lives in one object, `RetrievalConfig` (`packages/core/src/retrieval/types.ts`),
and `DEFAULT_RETRIEVAL_CONFIG` is the shipped set. The rule is that **no number appears in
`src/retrieval/` that is not in that object**, so a tuning session is a diff against one file. Six
modules do not meet it yet and are listed at the end; that is known debt, not design.

Two things to know before turning anything.

- `node eval/runner.mjs --config='{"…":…}'` changes a value for one run without committing it, and
  the run **cannot pass the gate** — the `no-config-overrides` check fails whatever it scores. That is
  the intended tuning loop.
- **Changing a default is a decision, not a tuning step.** See
  [section 7](#7-the-decision-reserved-for-a-human).

### Fusion (stage 4)

| Knob | Default | What it does and why |
| --- | --- | --- |
| `rrfK` | `60` | the `k` in `1/(k + rank)`, from the original Cormack RRF paper. Raising it flattens the difference between rank 1 and rank 20; lowering it makes the very top of each list dominate. **Read `rrfAbsentRankOffset` before touching this**: 60 was tuned on TREC runs a thousand documents deep and means something different over lists of 50. Lowering it is [rejected as a fix](#66-rejected-lower-rrfk). |
| `rrfAbsentRankOffset` | `1` | how far past the end of a list a document **absent** from it is assumed to sit. `absentRank = max(ids.length, the limit it was asked for) + this`. At `1` a rank-1 hit from one retriever scores `1/61 + 1/111 = 0.025399`, between "rank 1 of both" (0.032787) and "rank 50 of both" (0.018182). Set it to `Infinity` for classic RRF, where absence contributes nothing — correct when the lists are long enough for `k` to do the job alone, whose condition is `listLength > k + 2`. |
| `lexicalLimit` | `50` | BM25 candidates requested. Also the notional list length `rrfAbsentRankOffset` measures absence against. |
| `denseLimit` | `50` | Vectorize candidates requested. Set to `0` to measure BM25 alone. |
| `fusedLimit` | `60` | candidates kept after fusion |

**Why `rrfAbsentRankOffset` exists rather than a lowered `rrfK`.** With `rrfK = 60` and both limits at
50 the reachable score ranges do not overlap, so fusion is a de facto AND. Lowering `k` below 48 also
breaks the overlap, and it is a point solution: the condition is `listLength > k + 2`, so a `k` that
works at a limit of 50 has to be revisited whenever the limit moves, and it distorts the whole curve to
repair one relationship. The absent-rank correction repairs that relationship and leaves every other
comparison alone, at any `k` and any limit.

Measured when it landed, whole eval set: precision@1 0.229 → 0.261, precision@5 0.312 → 0.316, hit@5
0.611 → 0.643, MRR 0.413 → 0.438, answer-text coverage 0.764 → 0.847. Chunks at rank 5 or better of
exactly one retriever that were cut before stage 7 fell from **177 to 1**; queries where no
single-retriever result reached stage 7 at all fell from **173 of 196 to 42**.

### Graph expansion (stage 5)

| Knob | Default | What it does and why |
| --- | --- | --- |
| `graphPenalty` | `0.6` | what a one-hop neighbour's inherited score is multiplied by. Related is worth something and is not worth as much as a direct hit. |
| `graphHops` | `1` | how far to walk. Two hops in a densely linked corpus reaches most of it. |
| `graphExpandLimit` | `20` | neighbours admitted in total. Setting it to `0` changes no *ranking* metric today; median `used_tokens` moves 3,284.5 → 3,283. |
| `graphChunksPerNote` | `3` | chunks loaded per neighbour note; bounds the work one over-linked note can cause |

### Reranking (stage 6)

| Knob | Default | What it does and why |
| --- | --- | --- |
| `rerankLimit` | `20` | candidates kept, whether or not the model ran — **applied at the end of stage 7 since 2026-08-13**, not here; the name is historical and kept on purpose (see `types.ts`). **The direct lever on `used_tokens`** — the pack is always full at about 173 tokens per claim, so the median is linear in this number. It stopped being a precision lever when the cut moved, and has stayed that way through the change after it: 12 / 16 / 20 / 24 all score `precision@5` 0.7856, `hit@5` 1.0000 and block-level 0.5569 on the dev split, at medians of 1,860 / 2,538 / 3,148 / 3,724 tokens. That is the whole of [6.2](#62-cut-reranklimit-from-20-to-12), now free. |
| `rerankMinCorpus` | `200` | corpus size below which the model call is skipped. There is not enough to discriminate between and it is latency spent for nothing. The eval sets this to `Infinity`, which is how stage 6 is switched off without changing the pipeline's shape. |

### Score adjustment (stage 7)

| Knob | Default | What it does and why |
| --- | --- | --- |
| `halfLives` | `{convention: 365, decision: 180, reference: 180, state: 14, note: 90}` | days, per note type (PRD 4.2). The most opinionated numbers in the system: the reason a `state` note from last year loses to one from last week at equal relevance. It used to say "however well it matches", and `freshnessFloor` is the measurement that took that clause away — a year-old `state` note now loses a near-tie and wins anything wider than 1.111x. |
| `freshnessFloor` | `0.9` | the smallest value freshness may take, so the multiplier is `floor + (1 - floor) × exp(-age / halfLife)` and lives in `[0.9, 1]` instead of `(0, 1]`. Bounds how much relevance staleness may overturn to `1/floor` = 1.111x. `0` is PRD 4.2 unmodified. **Added 2026-08-12; it is worth 0.3279 -> 0.5640 on the dev split** — see [section 0](#0-tuning-log) for the sweep and the reasoning. |
| `supportFloor` | `0.5` | the smallest value the note-support multiplier may take, so it is `floor + (1 - floor) × (this note's candidates in the list / the note's own chunk count)` and lives in `[0.5, 1]`. Says how much of a *note* matched, which no chunk score says. Bounds what that evidence may buy to a measured median 1.548x. `1` switches it off, which is the control. **Added 2026-08-12 (worth 0.5640 -> 0.6685); its denominator moved to the note's own size the same day (worth 0.7027 -> 0.7171) and the floor was deliberately left where it was.** The curve now has an interior maximum at 0.35 worth a further 0.0072, not taken because 0.35 makes this factor span 1.898x against a relevance base of 1.378x — see [section 0](#0-tuning-log) for the sweep and the six questions it makes worse. |
| `coverageFloor` | `0.35` | the smallest value the term-coverage multiplier may take, so it is `floor + (1 - floor) × (the query's named terms present in this candidate / the query's named terms)` and lives in `[0.35, 1]`. Says how much of the *question* a passage answers — the only factor in stage 7 that is about the query rather than about the note — reusing stage 9's terms, token identity and contiguous-phrase rule from `retrieval/terms.ts`. `1` switches it off, which is the control. **Added 2026-08-12; it is worth 0.7171 -> 0.7568 on the dev split with 29 questions better, 7 worse and no question file regressing.** It is the only multiplier allowed to span more than the relevance base (a median 2.393x against 1.378x), because it is a second measurement of relevance rather than a fact about the note; the test applied instead of the range rule is that the composition beats both of its parts (0.7568 against 0.6414 for coverage alone and 0.5946 for the base alone) — see [section 0](#0-tuning-log) for the sweep, the four term weightings that lost to the unweighted share, and the seven questions it makes worse. |
| `noteCoverageFloor` | `0.6` | the smallest value the note-coverage multiplier may take, so it is `floor + (1 - floor) × (the query's named terms present in any of this note's candidates / the query's named terms)` and lives in `[0.6, 1]`. Says how much of the *question* a whole note answers, which neither `coverageFloor` (this passage) nor `supportFloor` (how much of the note matched) can: a passage covering two of four terms sits either in a note that covers the other two somewhere else or in one that never mentions them. Computed in the same pass as `termCoverage` — a note's coverage is the union of its candidates' — so the two cannot disagree about whether a note contains a word. `1` switches it off, which is the control. **Added 2026-08-13; it is worth 0.7568 -> 0.7676 on the dev split with 9 questions better, 3 worse and no question file regressing.** 0.6 is the largest floor reaching a plateau that runs down to 0, chosen so the factor swings the ordering least (median 1.5333x against 4.2000x at 0.2); the two coverage factors compound, so the shipped total coverage swing is 1 / (0.35 × 0.6) = 4.762x — see [section 0](#0-tuning-log). |
| `linkConsensusFloor` | `0.8` | the smallest value the link-consensus multiplier may take, so it is `floor + (1 - floor) × (min(other notes in this list that link to this one, linkConsensusCap) / linkConsensusCap)` and lives in `[0.8, 1]`. **The only factor in stage 7 that is not made of words** — it asks whether the rest of the evidence the retrievers returned points at this note, over the edges stage 5 already fetched for `Claim.related`, so it costs no port call. A relevant note is cited by a mean 2.045 other candidate notes against an irrelevant note's 0.757 (2.701x, the widest separation of any quantity measured); 53.5% of irrelevant candidate notes have no in-list citation at all against 10.9% of relevant ones. It tops out at 1, so it discounts isolation and never rewards popularity, which is what keeps it clear of the hub failure `expand.ts` documents. `1` switches it off, which is the control. **Added 2026-08-13; it is worth 0.7748 -> 0.7856 on the dev split with 10 questions better and 4 worse**, `hit@5` to 1.0000, block-level precision@5 0.5412 -> 0.5569 and every other metric flat or up — see [section 0](#0-tuning-log) for the cap-by-floor grid and for the two controls (corpus-wide inbound count 0.7802, outbound degree 0.7712) that show it is consensus rather than popularity. |
| `linkConsensusCap` | `6` | the in-list citation count at which `linkConsensusFloor`'s multiplier reaches 1; counts above it are worth no more than the cap. What stops the factor being an unbounded popularity score. Observed counts on the dev split run 0 to 7 with 95.0% at 3 or below, so the cap sets the steepness of the ramp over the range holding nearly all the mass rather than where it saturates. Not a finite number `>= 1` switches the multiplier off — it is the divisor as well as the ceiling, so its guard is stricter than a floor's. |
| `typeWeightStrength` | `0.35` | how much of stage 1's per-type weight stage 7 applies: `1 + strength × (weight - 1)`. Undamped the type weight spans a median 2.200x within one list — wider than the 1.374x relevance base it multiplies — while separating a relevant claim from an irrelevant one by 1.057x. `1` is PRD 4.1 unmodified, `0` switches stage 1 out of the ranking. The default is the largest strength whose damped range (1.363x) is at or under the base's. **Added 2026-08-12; it is worth 0.6685 -> 0.7027 on the dev split with 13 questions better and 0 worse** — see [section 0](#0-tuning-log). |
| `recencyBiasStrength` | `1` | how hard stage 1's recency bias pulls on the half-life. At `1` a maximally recency-biased query halves every half-life. **On `dec-003` this multiplier makes the answer worse** — see [5.2](#52-four-worked-examples). |
| `verificationWeights` | `{verified: 1, accepted: 0.95, unverified: 0.85, stale: 0.5, superseded: 0.2}` | trust order human-verified > agent-verified > human-unverified > agent-unverified. Constant across the whole eval corpus, therefore unmeasured. **`verified`, `accepted` and `unverified` are all 1 in an org that does not use verification** — see [the config per org](#the-config-per-org). |
| `authorKindWeights` | `{human: 1, agent: 0.9}` | same. Constant across the whole eval corpus. **Both 1 in an org that does not use verification.** |

### Dedup (stage 8)

| Knob | Default | What it does and why |
| --- | --- | --- |
| `dedupCosine` | `0.92` | similarity at or above which two chunks are one claim |

### Filtering

| Knob | Default | What it does and why |
| --- | --- | --- |
| `excludedStates` | `["superseded"]` | never returned, unless the caller asks for `include_superseded` |

**Verification ranks and never withholds.** [ADR 0027](./ADR/0027-verification-is-tracking.md)
removed the retrieval quarantine. Until 2026-09-17 two more knobs, `unreviewedStates` and
`unreviewedAuthorKinds`, dropped an assistant's unverified text from every pack unless the caller
set `includeUnverified`. Both knobs and the flag are gone. `buildRetrievalFilter` now builds the
caller's own clauses (`types`, `since`) and the `superseded` exclusion, and nothing else. A trust
rung and an author kind reach stage 7 as multipliers and nowhere else, so an assistant's unverified
paragraph is in the pack, ranked below the same words at `accepted` or `verified`.

The doors still accept `include_unverified` on the wire, so an older client is not refused, and
they drop it before a query exists. A change that waits in Review is not a candidate: its Markdown
is in a `proposals` row, and no chunk exists for it until a person accepts it.
`apps/worker/test/mcp/proposals-never-answer.test.ts` holds that for `recall`, `read`, `browse` and
Ask.

With the quarantine went every mechanism that existed only to describe it: the `withheld` pile the
pipeline handed to the conflict detector, the `withheld_unreviewed` gap sentence, the post-query
author-kind filter and its over-fetch in `apps/worker/src/retrieval/dense.ts`
(`DEFAULT_DENSE_OVER_FETCH` is 1), and the held-back claim count and score floor in the verify
queue.

### The config per org

[ADR 0018](./ADR/0018-verification-is-an-org-choice.md) makes verification an org's choice, so
`recall` no longer runs one config for every org. `retrievalConfigFor(mode)` in
`packages/core/src/retrieval/types.ts` is the one place a mode becomes a config, and
`orgRetrievalConfig(ctx)` in `apps/worker/src/retrieval/org-config.ts` is the one call every door
makes: MCP `recall`, `POST /api/recall`, Quick Connect `recall`, Ask and the trust report. The
usage queue read it too, to count held-back claims, until ADR 0027 removed them.
`apps/worker/test/invariants/org-retrieval-config.test.ts` refuses a door that passes
`DEFAULT_RETRIEVAL_CONFIG` to the pipeline, or that passes a mode written as a literal.
`POST /api/recall` names the mode it ranked in as a top-level `verification` field, and the
`/recall` screen explains a score with that mode's weights.

| Mode | Config | What changes |
| --- | --- | --- |
| `on` | `DEFAULT_RETRIEVAL_CONFIG` itself | nothing: every table above applies as written, so verified ranks above accepted, which ranks above unverified |
| `off` | `VERIFICATION_OFF_RETRIEVAL_CONFIG` | `verified`, `accepted` and `unverified` weigh 1; both author kinds weigh 1. `stale`, `superseded` and `excludedStates` are unchanged. |

Neither mode holds text back (ADR 0027). The modes differ in rank and in the marks a person sees.

A new org starts `off`, every org that existed on 2026-09-16 was set to `off`, and a NULL value or
a deployment with no org reads `on`.

**A person's own write is now stored `verified`.** The editor, the API, the CLI and an upload stamp
the writer as verifier (B-HUMAN-VERIFIED), and migration `0023` did the same for older human rows. In
an `on` org a person's claim therefore weighs 1.0 where it weighed 0.85, and the calibration comment
on `accepted` in `types.ts` (`0.95 × 0.9 = 0.855`, "about what a person's own unchecked writing is
worth") now compares an accepted agent paragraph with a person's text that a sync from a repository
or a restore wrote. **The in-process eval cannot see this**: every eval chunk is a human claim at
`unverified`, so both trust multipliers are still constant across it. The eval runs twice, once per
mode, and the flat `off` weights move nothing in it either: `node eval/runner.mjs --gate` and
`node eval/runner.mjs --gate --verification=off`. The flag selects `retrievalConfigFor("off")`,
which ships, so the second run can pass the gate where a `--config` override cannot.

### Packing (stage 9)

| Knob | Default | What it does and why |
| --- | --- | --- |
| `defaultBudgetTokens` | `4000` | applied when the caller names no budget (PRD 5.1) |
| `maxConflicts` | `3` | past a handful it is noise rather than a warning |
| `maxGaps` | `3` | most gaps reported, **the packer's own `budget_truncated` notice included**. It used to be appended after the cap was applied, so `maxGaps: 3` produced four gaps and `maxGaps: 0` produced one. It now displaces the least specific detected gap instead. |
| `reportReserveTokens` | `512` | budget at or above which the whole serialised pack, report included, is kept inside the budget — by trimming the echoed query if that is what it takes. 512 is measured, not chosen: an empty pack carrying the largest report the shipped caps allow serialises to 298 tokens, so the previous 256 was below some reachable reports. **It was also dead configuration**: declared, defaulted, documented in three places and read by nothing, so `0`, `256` and `1,000,000` produced byte-identical packs. |
| `fallbackGapText` | `"No claims found for this query."` | emitted identically by the packer and by `recall`'s safety net, so the two cannot drift |
| `countTokens` | `boundedTokenCounter(estimateTokens)` | **not a tunable.** The one dependency in the object. An injected counter is used exactly as given — the guard belongs to this value, not to the packer — so a deployment building its own config must wrap its counter the same way. |

### Tuning tables that are not in `RetrievalConfig` yet

Four stages carry their own tables, declared as optional extensions (`RetrievalConfig & Partial<...>`)
so `DEFAULT_RETRIEVAL_CONFIG` stays assignable everywhere. **They belong in `RetrievalConfig` and
should be moved there the next time that interface is opened** — until then, a change to one of them
does not show up in a diff of the config object, which is exactly the failure mode
[8.7](#87-everything-is-worse-after-a-change-and-you-cannot-tell-why) describes.

| Where | Constant | Contents |
| --- | --- | --- |
| `intent.ts` | `DEFAULT_INTENT_TUNING` | the 14-rule table, `baseRecencyBias: 0`, `maxQueryTokens: 512` |
| `expand.ts` | `DEFAULT_GRAPH_EXPANSION_TUNABLES` | `graphRelationWeights` (the verb table), `graphDefaultRelationWeight: 0.6`, `graphMinRelationWeight: 0` |
| `dedup.ts` | `DEDUP_SHINGLE_SIZE` | `3`, the shingle width of the lexical fallback |
| `conflict.ts` | `DEFAULT_CONFLICT_TUNING` | stopwords, negation cues, unit tables, `conceptClasses`, and the per-signal `gates` below |
| `gaps.ts` | `DEFAULT_GAP_TUNABLES` | the 11 levers below |
| `pack.ts` | `WIRE_SIGNIFICANT_DIGITS` | `3`, the precision `score` and `freshness` are published at |

Conflict gates, per signal kind — a pair must clear all five to be reported:

| Signal | `minShared` | `minSharedStatement` | `minOverlap` | `minDice` | `minTerms` |
| --- | --- | --- | --- | --- | --- |
| `choice` | 3 | 2 | 0.45 | 0.40 | 3 |
| `measure` | 2 | 2 | 0.40 | 0.32 | 3 |
| `count` | 3 | 3 | 0.45 | 0.40 | 3 |
| `polarity` | 3 | 3 | 0.50 | 0.40 | 4 |
| base | 2 | 2 | 0.40 | 0.32 | 3 |

`minSharedStatement` is the load-bearing one: the shared terms must come from the statements
themselves, not from a heading path two notes happen to share.

Gap detection levers (`DEFAULT_GAP_TUNABLES`):

| Lever | Default | What it does |
| --- | --- | --- |
| `gapMinTermLength` | `4` | shortest query word that can be reported as absent |
| `gapMaxTerms` | `24` | query terms considered |
| `gapMaxTermGaps` | `2` | most "no claims found about X" sentences per pack |
| `gapRarePinFraction` | `0.2` | fraction of terms treated as rare enough to pin on |
| ~~`gapRelevanceFloorRank`~~ | ~~`60`~~ | **deleted 2026-08-13 with the rule it configured** — [5.3](#53-the-low_confidence-relevance-floor-is-unreachable). |
| `gapOpenQuestionOverlap` | `0.75` | how much a retrieved `[question]` observation and the query must be the same question, in both directions. 0.5 to 0.8 give an identical verdict on dev |
| `gapStaleFreshness` | `0.25` | freshness below which the pack is called stale |
| `gapImpliedTypeWeight` | `1.25` | type weight above which stage 1's intent is treated as a demand |
| `gapCoverageFloor` | `1` | embedding coverage below which `index_incomplete` fires |
| `gapLagSecondsFloor` | `60` | index lag above which the same fires |
| `gapMaxQueryChars` | `120` | longest query, or term taken from it, quoted back in a gap sentence |
| `gapStopwords` | a list | words never reported as absent |

The relevance floor is measured on the **relevance base** (`rerank ?? rrf`), never on the final score.
A decayed claim is a different complaint (`stale`) from an irrelevant one, and folding freshness in
would make the floor fire on every query over an old corpus.

### Constants that must stay out of `RetrievalConfig`

`tokens.ts`'s `MAX_UNBROKEN_RUN_BYTES` (512) and `pack.ts`'s `MAX_ECHOED_QUERY_CHARS` (16,384) /
`MAX_GUARDED_QUERY_CHARS` (1,024). They bound how much CPU one request may spend, and **a
denial-of-service ceiling a request can raise is not a ceiling.** See
[appendix B](#appendix-b-the-tokeniser-is-quadratic-in-one-unbroken-run).

---

## 3. The eval

`eval/` decides whether kmem is any good. It runs in about seven seconds, needs no account, no
network, no build step and no Cloudflare binding, and it imports `packages/core/src` as TypeScript
**source** (`eval/loader.mjs`) — so a change to a scoring function is measurable in the next run rather
than after the next `tsc`. Without that the tuning loop is a lie.

### Running it

```sh
node eval/runner.mjs                                # run, print a summary line
node eval/runner.mjs --output=eval/results/         # …and write results.json + report.md
node eval/runner.mjs --gate                         # CI: exit non-zero on any gate failure
node eval/runner.mjs --only=runbooks,state          # tuning loop: one or more question files
node eval/runner.mjs --config='{"rerankLimit":12}'  # change one lever, re-run, read the failures
node eval/runner.mjs --retriever=worker --base-url=https://…   # score a deployed instance
npx tsx eval/runner.mjs --retriever=hybrid --split=dev         # the real models over the real index
node eval/runner.mjs --split=heldout --final        # SPENT TWICE. Do not run it a third time
node eval/runner.mjs --help                         # every flag
node eval/validate-questions.mjs                    # check the eval set against the corpus
node eval/validate-questions.mjs --intent           # …and compare declared intent to classifyIntent
npx vitest run --project core packages/core/test/retrieval/metrics.test.ts   # metrics vs hand fixtures
```

**Three backends, and they measure three different things.** Quote the right one.

| flag | what runs | dev `precision@5` | what it proves |
| --- | --- | --- | --- |
| `--retriever=reference` (default) | in-process BM25 over `eval/corpus`, signed-hashing embedder, no cross-encoder | **0.7856** | the ranking arithmetic, deterministically and for free. Byte-identical across processes |
| `--retriever=worker` | `POST /api/recall` over HTTP against a running instance | **0.8180** — **do not quote as a pass** | the HTTP boundary, auth, real D1, real FTS5, real packing. Locally the `AI` and `VECTORS` bindings are `not supported`, so dense and rerank **degrade** and `hit@5` drops to 0.982 |
| `--retriever=hybrid` | `packages/core` + the production ports, with real `bge-m3`, real Vectorize and real `bge-reranker-base` driven over the Cloudflare REST API | **0.7748** | the product's actual ranking. Needs `npx tsx`, not `node` |

**Do not use `pnpm test:core -- <path>` to run one file.** `test:core` is
`vitest run --project core` and the trailing path is not applied as a filter, so you get the whole
project. Measured 2026-08-10: `pnpm test:core -- test/retrieval/decay.test.ts` ran **53 files, 3,922
tests in 22.56 s**; `npx vitest run --project core packages/core/test/retrieval/decay.test.ts` ran
**1 file, 68 tests in 0.34 s**. Inside the tuning loop below, which you will run tens of times, that
is the difference between a minute and an hour. Paths are relative to the repository root, not to the
package.

**`runner.mjs` exits 0 on a failing gate unless you pass `--gate`.** Verified 2026-08-10:
`node eval/runner.mjs; echo $?` prints `FAIL precision@5 0.316` and then `0`; with `--gate` it prints
`1`. Any script that checks the exit code without `--gate` will read a failing eval as a pass.

**`--retriever=worker` cannot be run against the deployed instance**, and that has not changed.
`POST /api/recall` on the deployment answers 401 (`SIGNPOST_AUTH_SECRET` is unset), and
with a session it would return nothing, because its Vectorize index holds **0 vectors**. What can be
run, since 2026-08-13, is a **local** eval instance whose index *is* populated:

```sh
npx wrangler dev --config .wrangler/eval/wrangler.eval.json --local \
  --port 8798 --persist-to .wrangler/eval-state
# in another shell:
node eval/runner.mjs --retriever=worker --base-url=http://127.0.0.1:8798 --sign-in-local \
  --wrangler-config=.wrangler/eval/wrangler.eval.json --split=dev
```

`/health` on that instance answers `status: ok`, `embedding_coverage: 1`, `chunks_total: 459`,
`embedder: "@cf/baai/bge-m3"`. `--sign-in-local` mints an owner invitation in the local database and
signs in — the whole invite → accept → cookie → `/api/recall` chain, walked end to end. Without a
credential `env.AI` and `env.VECTORS` are `not supported`, so that run is BM25 only; see
[round 8](#2026-08-13--round-8-final-numbers-and-the-verdict) for why its 0.8180 is worth less than the
hybrid backend's 0.7748.

### Where the eval runs

The corpus goes in with `node scripts/ingest-eval-corpus.mjs`. Four ways to call it:

- `--stats-only` prints the corpus statistics offline, at no cost.
- `--no-embed` commits the corpus locally, with no account.
- The plain command commits, embeds, upserts and proves.
- `--reset` starts from empty.

The script exits `0` only when every proof passed and none was skipped. The corpus is 100 notes, 1,740
blocks and 459 chunks.

**The target is a dedicated eval instance, not the deployment.** The script generates its config at
`.wrangler/eval/wrangler.eval.json` from `wrangler.jsonc` by changing four things: the Vectorize index
becomes `signpost-eval`, `vars.SIGNPOST_BASE_URL` becomes `http://127.0.0.1:8798`, the Worker `name`
becomes `signpost-eval`, and `routes` is removed. D1, R2, the queues and the Durable Objects are local.
Vectorize and Workers AI are remote, because neither has a local implementation and this repository
never mocks a binding.

**One difference between the two targets decides what a run can measure, and conflating them is the
mistake.** On the eval corpus the declared note ages span 552 days, so stage 7's freshness term varies
and is measurable. On a freshly ingested instance every note shares one `updated_at`, so the freshness
term is one constant for every candidate, a constant multiplier cancels out of a ranking, and anything
age-dependent is untestable there.

459 chunks is above `rerankMinCorpus`, which is 200, so the cross-encoder does run during an eval
against this corpus. Below 201 chunks stage 6 stops calling the model, and nothing in the output says
so.

`docs/EVAL-SETUP.md` held this material until the relaunch folded it into this section.

### The corpus

`eval/corpus/` is 100 interlinked notes for a fictional but internally consistent payments team,
generated by `scripts/seed-corpus.mjs` and never hand-edited. It contains **459 chunks**, 325 wikilinks
over 97 targets, 420 observation lines, and dates spread over 18 months so freshness decay is actually
exercised. Two files make it a *test* corpus rather than a pile of text:

- `conflicts.json` — six subjects two notes disagree about, months apart, neither marked superseded.
- `gaps.json` — eight questions a reasonable person would ask that the corpus genuinely does not
  answer, each next to a well-documented neighbourhood. Each carries `mustNotAppear`, strings asserted
  to occur nowhere in the hundred notes, so a gap cannot quietly become answered as the corpus grows.

Dates are offsets from an anchor recorded in `corpus.json` (2026-08-09), and the eval evaluates at that
anchor rather than at the wall clock. A run today and a run in six months therefore differ only where
the code differs.

**The corpus is old, and that matters for section 5.** Median note age at the anchor is 394 days; **9
of 100 notes have freshness above 0.4 and 28 above 0.25**. Freshness is close to a binary "written in
the last few months" flag for 91% of the corpus.

### The gate, and why it cannot be argued with

Sixteen checks in `runner.mjs`'s `gateChecks`, measured on 2026-09-17. Each threshold is a constant
with no flag behind it. **Five of the checks exist to stop the *population* being moved rather than
the threshold:**

| Check | What it stops |
| --- | --- |
| `no-config-overrides` | any `--config` override makes the gate fail whatever the run scores |
| `whole-eval-set` | re-counts the committed questions from disk; catches `--only` and `--questions` |
| `default-run-parameters` | `--now` and `--budget` must be the corpus anchor and 4,000 |
| `gap-premises-hold` | a `gaps.json` `mustNotAppear` string coming back means the corpus grew the answer |
| `adversarial-contracts` | each adversarial question's own `expect` block |

Two recorded near-misses explain why. Twelve questions copied unmodified into their own directory and
run with `--questions=` passed thirteen checks at `precision@5` **0.850** while the full set of the
same tree scored 0.312. And `--now=2025-01-01` makes every age negative, so `freshnessDecay` correctly
returns exactly 1, stage 7 stops discriminating, and `precision@5` reads **0.564** — a 0.25 improvement
from a flag. The second is consistent with everything in section 5: 0.564 is close to the 0.561 that
switching freshness off scores, and for the same reason.

The other eleven checks are four thresholds and seven safety probes. The thresholds are `hit-at-5`,
`precision-at-1`, `precision-at-5-floor` and `cch`. The safety probes are the budget contract on
100% of queries, no pack with empty claims and empty gaps, superseded content absent, soft-deleted
content absent, and three empty-knowledge-base probes. The superseded and soft-deleted probes ran
twice, with `includeUnverified` false and true, until ADR 0027 removed the flag. The gate had
eighteen checks until then.

### The question set

`eval/questions/` holds eight files: `architecture` (23), `conventions` (27), `decisions` (23),
`runbooks` (45), `state` (19), `people` (20), `negative` (17), `adversarial` (22) — **196 in total, 157
of which have a non-empty `relevant` set and are scored for ranking**. A file is either a bare array or
`{ set, description, questions: [...] }`; both are accepted, so two people writing sibling files cannot
deadlock on the wrapper.

```jsonc
{
  "id": "dec-014",                             // ^[a-z][a-z0-9-]*$, unique across every file
  "question": "why postgres not dynamo",       // as somebody would actually type it
  "intent": "decision",                        // note | decision | convention | state | reference
  "difficulty": "easy",                        // easy | medium | hard
  "relevant": ["decisions/postgres-over-dynamo", "architecture/ledger-model"],
  "primary": "decisions/postgres-over-dynamo", // the single best answer; must be in `relevant`
  "nearest": ["architecture/idempotency"],     // graded 1 for nDCG; `relevant` is graded 2
  "relevantBlocks": ["kmem://decisions/postgres-over-dynamo#blk_839d57a1"],  // optional, sharper
  "primaryBlock": "kmem://decisions/postgres-over-dynamo#blk_839d57a1",      // optional
  "mustContain": ["Aurora PostgreSQL 16"],     // must occur on one line of some `relevant` note
  "mustNotContain": ["ClickHouse"],            // the decoy this question exists to rule out
  "notes": "Why this question is in the set, and what the decoy is."
}
```

Three sets carry extra fields. `state.json` carries a `freshness` block (`fresher`, `staler`, `gate`,
`fresherAgeDays`, `stalerAgeDays`, `why`). `negative.json` carries `relevant: []`, `primary: null`,
`expect: { claims, gaps }`, and — on the ten questions drawn from the eight entries in
`eval/corpus/gaps.json` — a `gapId` and the corpus's own `mustNotAppear` strings. `adversarial.json`
carries `kind` and `expect: { noCrash, budgetRespected, claimsOrGaps }`.

### Adding a question

1. Pick the area file. **Write the question from the corpus, before looking at what the retriever
   returns.** A question written from a failure list is a mirror, and an eval set full of mirrors
   reports that everything is fine. This is the one rule that keeps the gate honest: it must not be
   possible to make the eval green by editing an expectation to match the implementation.
2. Add the object above. **Include `relevantBlocks`** — the block reading is the one that stays honest,
   and only 73 of 157 questions have it.
3. Run `node eval/validate-questions.mjs`. Fix errors; read the warnings.
4. Run `node eval/runner.mjs --only=<file>` and confirm the question behaves as a question rather than
   as a restatement of the implementation.
5. If it is a question the corpus should *not* answer, it goes in `negative.json` with the fields
   above, and its `mustNotContain` strings must be **text the corpus actually contains** or the trap can
   never fire.

### What `validate-questions.mjs` checks

**Errors (exit 1):** id shape and uniqueness across all files; `question` non-empty outside
`adversarial.json`; `intent` a real `NoteType`; `difficulty` one of three; every `relevant` permalink
exists in the corpus with no duplicates; `primary` is in `relevant`; every `mustContain` string occurs
**on one line** of some note it names, checked against the **chunker's own output** rather than the raw
markdown; every `relevantBlocks` ref resolves through the **real parser and chunker** to a block that
exists in that note; `primaryBlock` is in `relevantBlocks` and sits in `primary`; `state.json`'s
declared `gate` matches `exp(-age/halfLife)` recomputed from PRD 4.2; `negative.json`'s
`mustNotContain` equals its gap's `mustNotAppear` and every one of those strings is still absent from
all 100 notes; `adversarial.json` covers every required `kind` and its `pastedFrom` payloads byte-match
the notes they claim to be.

**Warnings (never errors):** a `mustNotContain` string that appears in no chunk, so the trap can never
fire; a `relevantBlocks` ref whose block exists but is no longer the first block of a chunk; and, under
`--intent`, any disagreement between the declared intent and `classifyIntent`. **The intent pass is
deliberately a warning** — a disagreement is a finding about the classifier, and making it an error
would let somebody green the gate by editing the expectation to match the code.

Measured 2026-08-10: `node eval/validate-questions.mjs` prints **`OK — 0 errors, 13 warning(s)`**;
with `--intent`, **`OK — 0 errors, 138 warning(s)`**.

### What each metric means, and what it does not

| Metric | Unit | Means | Does not mean |
| --- | --- | --- | --- |
| `precision@5` | claim | of the 5 claims returned, the fraction from a note the question names. **The gate metric.** | that the right *sentence* came back — see block-level, 0.2219 |
| `precision@5` distinct | note | the same over distinct notes | comparable to the claim reading; its ceiling is 0.504, not 0.995 |
| block-level `precision@5` | block | the claim's own `kmem://…#blk_…` is one the question names | available on all questions — only 73 carry judgements |
| `hit@5`, `hit@10` | claim | anything relevant reached the top k at all | how much of the top k was right. **This is what most RAG evals call "precision@5".** |
| `recall@k` | note | of the notes that answer, the fraction that arrived | that they arrived *usefully*; its ceiling is `min(k, R) / R` for `R` relevant notes, not 1 |
| MRR | claim | rank of the first useful thing the caller was handed | anything about the rest of the pack |
| nDCG@10 | note | graded: a note that answers is 2, a `nearest` note is 1 | a claim-level judgement |
| answer coverage | text | every `mustContain` substring came back (0.847); `anyRate` is 0.943 | that it was ranked well |
| empty-recall | pack | a **knowledge** gap was stated on a question the corpus cannot answer | that claims were withheld — `claimsOnNegativeRate` is 1.000 |
| false-confidence | pack | claims returned and no knowledge gap stated (0.2353) | anything about positive questions |
| `used_tokens` median | pack | the number PRODUCT.md 9 publishes | a packer property — it is about 173 tokens per admitted claim, and `rerankLimit` decides how many are admitted |

Two definitions the report restates every run because both have bitten somebody: `median` averages the
two middle values, `p50` is nearest-rank and returns an observed value; they differ on an even count.
And **negative questions throw rather than score zero** in the ranking metrics — a question the system
correctly refused to answer must not drag the aggregate down — so they are scored only by
`negativeSetRates`.

### The three readings of precision, and why the strictest is published

`precisionAtK` projects a claim onto its note and asks "is this note relevant". That counts a chunk
about deployment windows as a hit because a *different* chunk of the same note answers the question.
The eval set carries **366 hand-curated `relevantBlocks` refs across 73 questions**, every one resolving
through the real parser and chunker to a block that exists, and for a while nothing scored them.

Three readings of the same run, from strictest to loosest. The first two are on the same 73 questions,
recomputed directly from `results.json` on 2026-08-10:

| Reading | Value | What it asks |
| --- | --- | --- |
| **block-level `precision@5`** | **0.2219** | of the first five claims, how many are the *passage* the question names |
| note-projected `precision@5`, same 73 questions | **0.2932** | …how many are from a *note* the question names |
| note-projected `precision@5`, all 157 scored (the gate metric) | **0.3159** | the same, over the whole scored set |
| loosest — `hit@5` | **0.6433** | did *anything* relevant reach the top five at all |

Of the 316 claims the note projection counts as hits across whole packs, **233 are endorsed by the
block truth** (`endorsementRate` 0.7373). **A quarter of the "correct" claims are the wrong chunk of the
right note.**

The gate stays on the PRD's note-level `precision@5` because swapping the go/no-go metric mid-flight
would make two runs incomparable. But publishing the loosest of three available readings while a
stricter one sat unused in the same file was the flattering choice, so **both are on the summary line**.
When somebody quotes "0.64" at you, they are quoting `hit@5`, which is the number most RAG papers call
precision and which this eval deliberately does not.

> **`0.313` does not reproduce.** Earlier drafts of this document, `eval/README.md` and several task
> briefs quote "0.222 against 0.313 on the same questions". Recomputed twice from `results.json`, the
> note-projected figure on those 73 questions is **0.2932**. The nearest number to 0.313 in the file is
> the overall 0.3159 across all 157. Treat **0.2219 against 0.2932** as the pair.

### Reading a failure report

`report.md`'s order is fixed: the gate, the slice tables, the answer-coverage table, the negative set,
the traps, conflicts, tokens and latency, the safety probes, then the failure list. The failure list is
the tuning loop — every question where nothing relevant reached the top five, worst first — and each
entry carries the score decomposition:

```
| # | note | score | rrf | rerank | type w | graph | fresh | verif | author |
| 1 | conventions/payout-operations | 0.01750 | 0.02674 | — | 1.00 | 1.00@0 | 0.769 | 0.85 | 1.00 |
```

Read it as the identity in `types.ts`:
`score = (rerank ?? rrf) × typeWeight × graphPenalty × noteSupport × termCoverage × noteCoverage ×
freshness × verification × authorKind`. The columns multiply out to the score, exactly, which is what
makes the scoring tunable by somebody who did not write it. The sample row above predates three
columns: `support`, which `eval/report.mjs` gained on 2026-08-12 between `graph` and `fresh` — a value
below 1 there is a note that contributed a smaller share of its own chunks than another did — and
`cov` and `note cov`, which it gained on 2026-08-13 after `termCoverage` had been shipped for a day
without one, so for that day the columns did not multiply out. `graph` reads `penalty@hops`, so
`1.00@0` is a direct hit and `0.60@1` is a one-hop neighbour. **A dash under `rerank` means stage 6
did not run**, which is every run today.

Three things to read, in this order.

1. **The last line of the entry.** It says whether this is a *retrieval* failure or a *ranking* failure.
   *"Never reached the packer at all: X"* means stages 2 to 5 lost it and no weight in stage 7 can bring
   it back — 24 of the 56 failures are partly this. *"Every expected note was among the candidates the
   packer was handed"* means the ordering is wrong; 32 of 56 are purely this, and it is what section 5 is
   about.
2. **Declared intent against classified intent.** A mismatch, or "(no cue fired)", means stage 1 sent the
   type weights the wrong way. One row in `INTENT_RULES` fixes it — though see the caveat that stage 1 is
   worth 0.008 in aggregate.
3. **The component columns.** Compare the expected note's row against the row above it and find the
   factor with the largest ratio. That factor names the knob. Today the answer is `fresh` in most of
   them, which is the finding rather than the fix.

`results.json` beside it carries the full list including near misses, and keeps per-claim
`ScoreComponents` only on `failures`, deliberately, to stay diffable.

---

## 4. Current results

> ## These are NOT the current numbers. This section is the frozen pre-tuning baseline.
>
> Everything below was measured on **2026-08-10 at `precision@5` 0.3159**, before any tuning. It is
> kept verbatim because it is the diagnosis the eight rounds acted on, and rewriting it would destroy
> the record of why. **The current numbers are in
> [round 8](#2026-08-13--round-8-final-numbers-and-the-verdict)**: `precision@5` 0.7720 whole set /
> 0.7391 held out / 0.7856 dev on the reference retriever, 0.7748 on the real pipeline, `hit@5` 1.0000
> everywhere, false-confidence 0.0588, median `used_tokens` 1,868.5. Do not quote a figure from this
> section as a statement about the system today.

`node eval/runner.mjs --retriever=reference`, shipped configuration, 196 questions, 157 scored for
ranking. **Measured 2026-08-10** (run timestamp `2026-08-09T23:10:13Z`), `now` = the corpus anchor
`2026-08-09T00:00:00Z`, budget 4,000. Wall time 6.7 s. Reproduces the committed
`eval/results/results.json` to every digit except latency.

```
precision@5 0.3159  precision@1 0.2611  precision@3 0.3206  precision@10 0.3293
hit@5 0.6433        hit@10 0.8662       MRR 0.4380          nDCG@10 0.4590
recall@5 0.4227     recall@10 0.7016    precision@5 over distinct notes 0.192
answer text returned in full 0.847  (at least one required string 0.943)
block-level precision@5 0.2219  over the 73 questions carrying block judgements
empty-recall 0.7647  false-confidence 0.2353  silent 0.000  claims-on-negative 1.000
used_tokens median 3,284.5  p50 3,284  p95 3,765  p99 3,905  max 3,911  violations 0 / 196
claims per pack  median 19  max 20      worst pack 97.8% of budget
conflicts   precision 0.972  recall 0.769   all 6 seeded conflicts surfaced somewhere
decoy traps 6 of 14 fireable (0.429)   7 further trap questions carry strings the corpus lacks
oracle precision@5 ceiling 0.995   (precision@10 ceiling 0.921)
```

**Fourteen of the fifteen gate checks pass. `precision@5 >= 0.80` is the only failure.** The budget
contract, both gaps contracts, all four superseded and soft-deleted probes, the three
empty-knowledge-base probes, the adversarial `expect` blocks, the gap premises, the
shipped-configuration check and the two run-parameter checks all hold.

### By difficulty, with the ceiling

| Slice | n | p@1 | p@3 | **p@5** | p@5 note | hit@5 | hit@10 | r@5 | r@10 | MRR | nDCG@10 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **all scored** | 157 | 0.261 | 0.321 | **0.316** | 0.192 | 0.643 | 0.866 | 0.423 | 0.702 | 0.438 | 0.459 |
| _ceiling: a perfect ranking_ | 157 | 1.000 | 1.000 | **0.995** | 0.504 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| easy | 37 | 0.108 | 0.180 | **0.243** | 0.173 | 0.541 | 0.811 | 0.392 | 0.635 | 0.299 | 0.373 |
| medium | 81 | 0.284 | 0.358 | **0.333** | 0.190 | 0.642 | 0.877 | 0.443 | 0.735 | 0.457 | 0.482 |
| hard | 39 | 0.359 | 0.376 | **0.349** | 0.215 | 0.744 | 0.897 | 0.410 | 0.699 | 0.531 | 0.492 |

Per-difficulty `precision@5` ceilings, computed the same way: **easy 0.989, medium 0.998, hard 0.995**.
The easy-below-hard inversion is therefore **not** a ceiling artefact — see
[5.4](#54-the-difficulty-inversion-resolved).

### By question file

"We are fine except on runbooks" is a different problem from "we are fine except on hard questions",
and only this table tells them apart.

| Slice | n | p@1 | p@3 | **p@5** | hit@5 | hit@10 | r@5 | MRR | nDCG@10 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| architecture | 23 | 0.130 | 0.159 | **0.139** | 0.391 | 0.696 | 0.250 | 0.287 | 0.319 |
| conventions | 27 | 0.481 | 0.531 | **0.504** | 0.963 | 0.963 | 0.642 | 0.671 | 0.612 |
| decisions | 23 | 0.174 | 0.188 | **0.200** | 0.391 | 0.739 | 0.341 | 0.314 | 0.370 |
| people | 20 | 0.300 | 0.317 | **0.340** | 0.700 | 0.800 | 0.362 | 0.463 | 0.475 |
| runbooks | 45 | 0.156 | 0.289 | **0.316** | 0.600 | 0.956 | 0.472 | 0.369 | 0.460 |
| state | 19 | 0.421 | 0.456 | **0.379** | 0.842 | 0.947 | 0.368 | 0.579 | 0.497 |

### By declared intent, by classified intent, and by the note type of the answer

| Declared intent | n | p@1 | **p@5** | hit@5 | MRR |
| --- | --- | --- | --- | --- | --- |
| convention | 29 | 0.483 | **0.517** | 0.966 | 0.670 |
| note | 15 | 0.267 | **0.307** | 0.733 | 0.451 |
| reference | 72 | 0.181 | **0.278** | 0.556 | 0.373 |
| state | 19 | 0.368 | **0.274** | 0.684 | 0.486 |
| decision | 22 | 0.136 | **0.218** | 0.409 | 0.294 |

| Classified intent | n | p@1 | **p@5** | hit@5 | MRR |
| --- | --- | --- | --- | --- | --- |
| convention | 24 | 0.250 | **0.342** | 0.708 | 0.451 |
| decision | 14 | 0.357 | **0.314** | 0.500 | 0.452 |
| note | 95 | 0.274 | **0.314** | 0.653 | 0.449 |
| reference | 21 | 0.143 | **0.286** | 0.571 | 0.342 |
| state | 3 | 0.333 | **0.400** | 1.000 | 0.611 |

The `note` row of 95 is the tell: `classifyIntent` falls back to a flat classification on 60% of the
scored set. Compare its shape to the declared table above — they are different populations, and the
declared and classified intents agree on 46 of 157.

| Note type of the answer | n | p@1 | **p@5** | hit@5 | MRR |
| --- | --- | --- | --- | --- | --- |
| convention | 55 | 0.455 | **0.495** | 0.891 | 0.625 |
| note | 28 | 0.357 | **0.336** | 0.679 | 0.504 |
| reference | 127 | 0.236 | **0.312** | 0.646 | 0.425 |
| decision | 47 | 0.277 | **0.272** | 0.596 | 0.432 |
| state | 49 | 0.286 | **0.265** | 0.633 | 0.443 |

Freshness decay is per type, so a `state` row far below the others is decay doing too much rather than
retrieval doing too little. `convention` has the longest half-life (365 days) and the best score;
`state` has the shortest (14 days) and the worst. That correlation is section 5.

### Answer coverage

| Slice | n | every required string | at least one |
| --- | --- | --- | --- |
| **all scored** | 157 | **0.847** | 0.943 |
| architecture | 23 | 0.652 | 0.957 |
| conventions | 27 | 0.963 | 1.000 |
| decisions | 23 | 0.783 | 0.870 |
| people | 20 | 0.850 | 0.950 |
| runbooks | 45 | 0.889 | 0.956 |
| state | 19 | 0.895 | 0.895 |

### The negative set

Seventeen questions the corpus genuinely does not answer.

| Metric | Value | Target |
| --- | --- | --- |
| empty-recall (a knowledge gap was stated) | 0.7647 | 1.0 |
| **false-confidence (claims returned, no knowledge gap)** | **0.2353** | 0 |
| claims returned at all | 1.000 | context, not a target |
| silent (no claims and no gaps) | 0.000 | must be 0; it is |

**Empty-recall counts knowledge gaps only.** `ContextPack.gaps` also carries the packer's
budget-truncation notice and the index-health notice, which are statements about the *response* rather
than about the knowledge base. Counting those made both rates reachable from a knob with nothing to do
with gap detection: at `--budget=800` they read 1.000 and 0.000 while every negative question still
returned a full pack. The runner splits on `GapReason`; the uncorrected reading is in `results.json` as
`negativeIncludingNotices` (empty-recall 0.824, false-confidence 0.176).

The four false-confidence questions are `neg-warehouse-pii-clickhouse` ("how long do we keep cardholder
names in ClickHouse", 18 claims, no gap), `neg-payout-failure-volume` (15 claims, no gap, **and it
returns the trap** "312 sellers were paid twice, about 1,240 payouts"), `neg-kyc-second-failure` (19
claims, no gap), and `neg-commercial-card-false-positive-rate` (20 claims, whose only gap is the
budget-truncation notice, and which returns the trap "about 3% of cards").

> **A fragility in how that fourth one is counted.** `runner.mjs`'s `knowledgeGaps` filters response
> notices out through *two* mechanisms: by `GapReason` for gaps in the recorded `GapCandidate[]`, and by
> a **text prefix match** (`isResponseNotice`) for everything else. The truncation gap is appended by
> `pack.ts` *after* `detectGaps` has run, so it never appears in the recorded array and **only the
> string comparison catches it**. Reword `BUDGET_TRUNCATED_GAP_TEXT` and the published
> false-confidence rate silently drops from 0.2353 to 0.1765 with no pipeline change. Make the notice
> carry its `GapReason` through to the runner.

### Conflicts and traps

Conflicts: 36 reported, 35 true positives, precision **0.972**, recall on the questions that name a
seeded conflict **0.769**, all 6 seeded conflicts surfaced somewhere in the run. One reported pair is
not in `conflicts.json` (`architecture/vault-service` against `state/pci-audit` on
`pp-who-owns-aws-and-the-vault`) — which is not automatically wrong, since the corpus records the six
that were planted, not the only six that exist.

Traps: 14 questions carry a trap made of text the corpus contains; **6 of those 14 (0.429) returned
it**. Seven further trap questions carry only strings the corpus does not contain — "recovery time
objective", "stipend", "99.9%", "99.95%", "broker outage", "restart the broker" — so they can never
fire and were diluting the rate. The runner now scores over the fireable denominator and names the
unfireable ones; `validate-questions.mjs` warns about each, so the fix belongs in the question files.

### Latency

Median 20.0 ms, p95 27.4 ms, p99 76.1 ms, max 807.9 ms (`adv-single-long-token`). **This is not
comparable to NF1's `recall` p95 < 800 ms.** It is in-process arithmetic over 459 chunks with no
network, no D1 and no model call.

> **Superseded 2026-08-13.** NF1 has now been measured against real Cloudflare services, and it
> **fails**: end to end **p50 ≈ 1,112 ms, p95 ≈ 7,619 ms** against a budget of 800 ms — over by 1.4×
> at the median, before the tail is considered. The cross-encoder is the whole of it (p50 522 ms with
> a session-dependent tail past 13 s); Vectorize is 355 ms and the embedder 120 ms. The full table,
> the three-sample tail measurement and what it implies is in
> [round 8](#2026-08-13--round-8-final-numbers-and-the-verdict). **It has still not been measured on a
> deployed instance** — the deployment has an empty index and cannot search.
>
> **Superseded again 2026-08-14, noted 2026-08-18.** It has now been measured on the deployed
> instance, which searches: `recall` p50 **1.18 s** end to end through the product's own door,
> after the 2026-08-13 change cut the cross-encoder's input from 80 contexts to 30. NF1's 800 ms
> is still not met, and the reason is structural rather than a tuning failure.

---

## 5. Why it fails

### 5.0 Start here: the ceiling is 0.995, so this is a ranking problem

`precisionAtK` is claim-level: of the first five claims, how many came from a note the question names.
A question whose answer lives in a three-chunk note cannot fill five slots, so the metric has an
arithmetic ceiling below 1. `oracleCeiling` in `runner.mjs` computes it: **0.995 at k = 5**, 0.921 at
k = 10. Per difficulty: easy 0.989, medium 0.998, hard 0.995.

That single number does the most important work in this section. **The gate at 0.80 has room, on this
corpus and this question set, with this metric.** Nothing about the eval's construction makes 0.80
unreachable. Whatever is wrong is in the ordering, and the rest of this section says which part.

Two more numbers frame it. A relevant claim reaches the pack at all on `hit@10` **0.866** of questions,
and reaches the top five on `hit@5` **0.6433**. And of the 56 questions with nothing relevant in the
top five, only **2** have nothing relevant anywhere in the pack.

### 5.1 Stage 7 destroys most of the ordering the earlier stages produced

The single most important table in this document. Rank of the best relevant claim on the 155 scored
questions where a relevant claim is present both entering and leaving stage 7, measured 2026-08-10 by
substituting an instrumented `adjustScores` into `DEFAULT_STEPS`:

| Position of the best relevant claim | Entering stage 7 | Leaving stage 7 |
| --- | --- | --- |
| rank 1 | **129** of 155 | **40** of 155 |
| within top 3 | 144 | 81 |
| within top 5 | 150 | 100 |

**Stage 7 moves the best relevant claim in 124 of 155 queries. 112 of those moves are downward**, 12
upward, 31 unchanged. It pushes a relevant claim **out** of the top five in 52 queries and pulls one
**in** in 2. It demotes the right answer roughly nine times for every once it promotes it.

The list entering stage 7 is the RRF order truncated to `rerankLimit`, because `runRerank` returns
`candidates.slice(0, config.rerankLimit)` when the corpus is below `rerankMinCorpus` — which the eval
forces. Ranking on that order alone, with every stage-7 multiplier neutralised:

| Composition | `precision@5` | `precision@1` | `hit@5` | easy / medium / hard |
| --- | --- | --- | --- | --- |
| shipped stage 7 | 0.3159 | 0.2611 | 0.6433 | 0.243 / 0.333 / 0.349 |
| **relevance only** (`base × graphPenalty`) | **0.5898** | 0.8153 | 0.9490 | 0.562 / 0.595 / 0.605 |
| freshness off, trust and type weights on | 0.561 | 0.694 | 0.924 | 0.535 / 0.548 / 0.610 |
| type weights off, everything else on | 0.308 | 0.255 | 0.637 | — |

**Stage 7 costs 0.274 `precision@5` and 0.554 `precision@1`.** The freshness multiplier is responsible
for essentially all of it: turning it off alone recovers 0.245 of the 0.274.

#### The arithmetic: a 1.4x factor multiplied by a 500x factor

Measured over the 191 of 196 queries that produce candidates, taking max/min of each factor within each
query's own list:

| Factor in `score = base × typeWeight × graphPenalty × freshness × verification × authorKind` | Median range in one list | p95 | Max |
| --- | --- | --- | --- |
| relevance base (`components.rerank ?? components.rrf`) | **1.3729x** | 1.4197x | 1.5243x |
| `freshness` | **499.8x** | 4.871e12 | **1.332e19** |
| `typeWeight` | 1.0000x | — | 5.7600x |
| `verification`, `authorKind` | 1.0000x (constant on this corpus) | — | 1.0000x |

The base's narrowness is structural, not incidental. RRF returns `Σ 1/(k + rank)`, and with the shipped
`rrfK: 60`, `lexicalLimit: 50`, `denseLimit: 50` and `rrfAbsentRankOffset: 1`, the entire reachable
range when both lists are non-empty is:

```
best  = 1/(60+1)   + 1/(60+1)   = 0.032787     rank 1 of both
worst = 1/(60+50)  + 1/(60+51)  = 0.018100     rank 50 of both
ratio                           = 1.8114x
```

So the whole relevance signal, from the best document in the corpus to the worst one fusion keeps,
spans less than a factor of two — and *within a single query's top 20* it spans 1.37x. Meanwhile
`freshness = exp(-ageDays / halfLife)` has no lower bound above `MIN_FRESHNESS`, which `decay.ts` sets
to the smallest normal double, 2.225e-308.

**Multiplying the first by the second does not adjust a ranking. It replaces it with a recency ranking
that has been slightly perturbed by relevance.** And because 91 of the corpus's 100 notes sit below
freshness 0.25, that recency ranking is close to a binary "written in the last few months" flag.

This is not a bug in `decay.ts`, which implements PRD 4.2 verbatim, nor in `fuse.ts`, which implements
RRF correctly. **It is a defect in the composition PRD 4.1 stage 7 specifies**: RRF scores are ordinal,
and stage 7 reads them as magnitudes.

### 5.2 Four worked examples

Reproduce with `node eval/runner.mjs --only=architecture,decisions,people --output=/tmp/x/` and read
the failure entries. The `rank` column below is position in the list **leaving stage 7**, which is one
to six places worse than the `bestRelevantRank` the report prints, because dedup collapses
near-duplicates between stage 7 and the pack.

**`arch-018` — "how does a merchant verify a webhook came from us"**

| | note | type | age | BM25 / dense rank | `rrf` | `freshness` | `score` | rank |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| answer | `decisions/webhook-signature-scheme` | decision | 460d | **1 / 1** | 0.03279 | 7.765e-2 | 2.164e-3 | **18** |
| winner | `runbooks/webhook-replay` | reference | 38d | 5 / 5 | 0.03077 | 8.097e-1 | 2.118e-2 | 1 |

The answer is rank 1 in **both** retrievers. It is 1.066x more relevant than the note that beats it and
0.0959x as fresh, so it loses by 9.8x and falls from rank 1 to rank 18.

**`dec-003` — "which acquirer is primary for eea right now"** is the one that should end the argument.
The answer is `state/psp-routing`, a **state** note that is **61 days old** — the freshest thing the
corpus has on the subject, and exactly the kind of note the question asks for. BM25 rank 1.
`classifyIntent` correctly fires `recency.current` and sets `recencyBias: 0.5`, which shortens the
`state` half-life from 14 days to 9.33. The note's freshness is therefore **1.451e-3**, its score
3.979e-5, and it lands at rank 18 behind `runbooks/webhook-replay` at BM25 rank 21. **Recency bias,
whose purpose is to favour current notes, multiplied the penalty on the only current-status note in the
answer set.**

**`arch-004`** (easy): answer `architecture/api-gateway-edge`, reference, 500 days old, BM25 rank 2,
`rrf` 0.03226 against the winner's 0.02797 — 1.153x more relevant, 0.158x as fresh, rank 1 → 16.

**`pp-who-owns-the-ledger`** (easy): answer `people/priya-raghavan`, BM25 rank 1 and dense rank 1,
1.189x more relevant, 0.0075x as fresh, rank 1 → 14.

In all four the relevance signal points at the right answer and stage 7 overrules it.

### 5.3 The `low_confidence` relevance floor is unreachable

> **Closed on 2026-08-13 by round 7 ([section 0](#0-tuning-log)).** `relevanceFloor`,
> `lowConfidenceGaps` and `gapRelevanceFloorRank` are deleted. The diagnosis below is correct and is
> left as measured; what it recommends in its last paragraph — and what [6.3](#63-replace-the-low_confidence-floor-with-a-quantity-that-carries-information)
> spelled out — was measured and does not work, because a term-coverage quantity reads **1.000** on
> every question that fails. Two content rules replaced it: `recordedQuestionGaps` and
> `unmetDurationGaps`. `false-confidence` 0.2353 → 0.0588.

`gaps.ts`'s `relevanceFloor` is `1 / (rrfK + gapRelevanceFloorRank)` = `1/120` = **0.008333**. The rule
fires when the **best** candidate's base is at or below it. The arithmetic, verified 2026-08-10:

| Quantity | Value | Against the floor |
| --- | --- | --- |
| floor | 0.008333 | — |
| **lowest base observed anywhere in 196 queries** | **0.016129** (= `1/62`) | **1.935x above** |
| lowest base *reachable at all* — one list non-empty, its worst rank | `1/(60+50)` = 0.009091 | 1.091x above |
| emissions of "No confident match found for …" in 196 questions | **0** | — |

**The rule names a rank no candidate can ever occupy.** It is unreachable by construction, not
mis-tuned, and retuning `gapRelevanceFloorRank` cannot rescue it: the largest usable value is 50, at
which the rule requires the single best candidate in the whole pack to sit at the very bottom of exactly
one retriever's list and be absent from the other — which happened zero times in 191 non-empty queries.

**And it will be just as dead in the opposite direction the day stage 6 lands.** A real cross-encoder
emits scores near 1.0, so every base will be ~1.0 and the floor at 0.008333 will still never fire. This
is the one number in the file that is wrong in every configuration the product will ever be in.

The problem is the quantity, not the constant. **A reciprocal-rank score says where a document sits in a
list; it never says whether the list is any good.** The consequence is the false-confidence rate of
**0.2353** — 4 of 17 negative questions return a full pack and state no knowledge gap, and two of them
return the decoy.

### 5.4 The difficulty inversion, resolved

`precision@5` is 0.243 on `easy`, 0.333 on `medium`, 0.349 on `hard`. This is the most suspicious number
in the set. Two hypotheses are dead — including the one earlier versions of this document asserted — and
two causes survive.

**It is not the ceiling.** Per-difficulty ceilings: easy 0.989, medium 0.998, hard 0.995.

**It is not that easy questions have a flat relevance signal.** Earlier versions of this document said
"an easy question is short and uses the corpus's own words, so dozens of chunks match it weakly and the
freshness multiplier decides". The first half is measurably false. Median relevance base of the best
relevant candidate against the base of the claim that actually took rank 1:

| Difficulty | base, best relevant | base, rank-1 winner | `freshness`, best relevant | `freshness`, rank-1 winner |
| --- | --- | --- | --- | --- |
| easy | **0.03202** | 0.02683 | **0.1555** | 0.7165 |
| medium | 0.03200 | 0.02778 | 0.2019 | 0.7097 |
| hard | 0.03132 | 0.02737 | 0.3234 | 0.7687 |

**On every difficulty the relevance base of the right answer is *higher* than that of the claim that
beat it.** Query length is also flat: 8.7 words on easy against 9.4 on hard. And BM25-only ranking
scores 0.605 on easy against 0.656 on hard — a gap of 0.05 rather than 0.106.

**Cause 1, worth roughly 0.06 to 0.07: stage 7 punishes easy questions about twice as hard.** Mean
displacement of the best relevant claim through stage 7: **easy +4.70 ranks, medium +3.30, hard +2.36**.
The reason is in the answers, not the questions. Easy questions are answered by notes that are 58%
`reference` type with a median age of 428 days; hard questions by notes that are 36% `reference` and 23%
`state`, median 389 days. Median freshness of the right answer is 0.1555 on easy and 0.3234 on hard — a
2.1x handicap that has nothing to do with how hard the question is.

**Cause 2, worth roughly 0.04: the relevant sets are systematically broader on hard questions.**

| Difficulty | n | mean relevant notes | mean chunks in the relevant set | of 459 chunks |
| --- | --- | --- | --- | --- |
| easy | 37 | 2.41 | 10.95 | 2.38% |
| medium | 81 | 2.36 | 10.57 | 2.30% |
| hard | 39 | **2.97** | **13.44** | **2.93%** |

A hard question labels 23% more of the corpus as a correct answer, so 23% more of any ranking is right
by construction. **This is a labelling asymmetry in the eval set, not a property of the retriever.**

**Confirmation.** Switch freshness off and the inversion nearly vanishes:

| Composition | easy | medium | hard | easy − hard |
| --- | --- | --- | --- | --- |
| shipped | 0.243 | 0.333 | 0.349 | −0.106 |
| freshness off (clamp 0.9) | 0.535 | 0.548 | 0.610 | −0.075 |
| relevance only | 0.562 | 0.595 | 0.605 | −0.043 |
| weighted composition | 0.568 | 0.588 | 0.605 | −0.037 |

The residual −0.04 under a correct ranking is smaller than the 23% breadth advantage predicts, which
means that once stage 7 is fixed, easy questions perform better per unit of labelled breadth than hard
ones — which is what "easy" is supposed to mean. **The difficulty labels are sound. The inversion is
stage 7 plus a labelling asymmetry, and neither is a retrieval finding.**

### 5.5 How much of the 0.3159 is the two missing stages

Simulated by injecting an oracle cross-encoder at stage 6: every chunk of a note the question names
scores `s`, everything else scores `s / separation`, with a 1e-9 rank tiebreak. `separation` is a proxy
for how confidently a real cross-encoder separates relevant from irrelevant text. Measured 2026-08-10 by
supplying `reranker: { score }` to `ReferenceRetrievalPorts` with `rerankMinCorpus: 0`.

| Stage 6 | Stage 7 composition | `precision@5` | `precision@1` |
| --- | --- | --- | --- |
| skipped (today) | shipped | 0.3159 | 0.2611 |
| skipped | relevance only | 0.5898 | 0.8153 |
| oracle, separation 1.5x | shipped | 0.4854 | 0.3949 |
| oracle, separation 3x | shipped | 0.6675 | 0.6178 |
| oracle, separation 9x | shipped | **0.8357** | 0.8917 |
| oracle, separation 30x | shipped | 0.8904 | 0.9809 |
| oracle, separation 1.01x | relevance only / weighted | **0.9427** | 1.000 |
| oracle, separation 9x | relevance only / weighted | **0.9554** | 1.000 |

Read this as four statements.

1. **The gate is reachable, but only through stage 6.** Fixing stage 7 alone tops out at 0.5898. A
   perfect stage 6 with today's broken stage 7 reaches 0.836; a perfect stage 6 with a fixed stage 7
   reaches 0.955 against the 0.995 ceiling.
2. **Stage 7 taxes stage 6 in proportion to how badly stage 6 is calibrated.** With a correct
   composition, a cross-encoder that merely *orders* relevant above irrelevant — separation 1.01x —
   already scores 0.943. With the shipped composition the same reranker scores barely above today's
   number, because a 1.01x separation is annihilated by a 500x freshness range. **Landing stage 6
   without fixing stage 7 makes the outcome depend on a property of the model nobody is measuring.**
3. **These are upper bounds, not predictions.** The oracle knows the answer; `bge-reranker-base` does
   not. What the table establishes is the *shape* of the dependency, not the number to expect.
4. **Attribution of today's 0.3159, on this corpus: 0.274 to stage 7's composition, and 0.365 of the
   remaining 0.405 recoverable by stage 6** (0.590 → 0.955). They are close to independent — fixing
   stage 7 does not reduce what stage 6 is worth, and vice versa.

The missing embedder is separately worth something the eval cannot bound, because the hashing stand-in
is worse than no dense retriever at all. **Do not read "BM25 only scores 0.645" as an argument against
Vectorize.** It is an argument that the current stand-in is not a proxy for `bge-m3`.

### 5.6 Where the candidate set genuinely does fail

"It is purely a ranking failure" is right at the question level and wrong at the note level, and the
difference matters when reading the failure list. Measured from `results.json`:

- 56 scored questions have nothing relevant in the top five.
- Of those, **2** have no relevant claim anywhere in the pack.
- **24** of the 56 have at least one expected note that never reached the packer at all.
- Across all 157 scored questions, **66 of 396 expected notes (16.7%)** never reach the packer, and only
  **106 of 157 questions (67.5%)** see every expected note arrive.

So a relevant note almost always arrives, but a third of questions lose at least one of the several
notes they name. **That residue is a stage 2/3 problem** — which chunk of which note ranks — and it is
where `bge-m3` earns its place. It is roughly uniform across difficulty (easy 19.1% of expected notes
missing, medium 15.2%, hard 17.2%), so it does not explain the inversion.

### 5.7 What the eval cannot see, and therefore what section 5 does not cover

Four coverage gaps. Each one means a number in this document is silent rather than reassuring.

1. **The trust multipliers are constant.** Every chunk is `unverified`/`human`, so `verification` is 0.85
   and `authorKind` is 1.00 everywhere. `decay.ts`'s entire trust half is unmeasured end to end.
2. **Graph expansion is inert.** 27 of 3,820 candidates, surviving in 2 of 196 packs;
   `graphExpandLimit: 0` gives identical ranking metrics — precision@5, hit@5, MRR and nDCG all to
   four decimal places — though median `used_tokens` moves 3,284.5 → 3,283, so the packs are not
   identical. The `seedCeiling` mechanism, which cost two critical defects to get right, is not
   exercised by any scored number.
3. **Intent classification barely fires.** Declared intent matches `classifyIntent` on 46 of 157;
   removing the type weights moves `precision@5` by 0.008.
4. **Reranking is not measured, in either direction.** Section 5.5 is a simulation, not a measurement of
   a model.

And one uncomfortable finding that belongs here rather than in the fix plan.

> **Across every composition measured, freshness decay never improves `precision@5`.** The best it
> manages is to break even: weighted composition 0.5873 against relevance-only 0.5898. Twenty-odd
> variants were measured — clamps from 0.1 to 0.9, powers from 0.1 to 0.75, five weight sets, with and
> without an oracle reranker — and every one in which freshness affects the ranking scores at or below
> the one in which it does not.
>
> Three readings, all probably partly true. **The corpus is too old** (median 394 days, 91 of 100 notes
> below freshness 0.25), so there is almost no fresh content for decay to prefer and it can only
> subtract. **The question set does not ask recency questions** — the one that plainly does, `dec-003`,
> is *hurt* by decay. And **decay may be correct and the eval blind to its value**: freshness protects
> against *acting on a stale claim*, which `precision@5` does not measure at all, since a stale claim
> from a relevant note counts as a hit.
>
> The honest position: **the eval can tell you how much damage a decay configuration does, and cannot
> tell you what it is worth.** Any decision to keep decay at strength rests on PRD 4.2's product
> reasoning, not on evidence from `eval/`. Someone should either build the metric that measures decay's
> value — a "was the returned claim still true" judgement over the seeded conflicts and the superseded
> pairs would do it — or write down that decay is a product commitment the gate metric is expected to
> charge for.

---

## 6. The fix plan

Ranked by measured gain per unit of risk. "Risk" is the chance the change makes something else worse or
encodes a decision the PRD reserves for a human. Every entry names the file, the function, the change,
the metric it should move, and the command that checks it.

**Nothing here should be applied silently.** PRD phase 2 makes the go/no-go a human decision, and
`runner.mjs`'s `no-config-overrides` gate check exists so a retuned default cannot be mistaken for a
passing build. Change `DEFAULT_RETRIEVAL_CONFIG` or `decay.ts` only as part of a decision that is
written down, and re-run the gate afterwards so the new number is on the record. See
[section 7](#7-the-decision-reserved-for-a-human).

### 6.1 Replace the multiplicative composition in stage 7

**File:** `packages/core/src/retrieval/decay.ts`, `adjustOne`. Today one line decides the ranking:

```ts
const score = base * typeWeight * graphPenalty * freshness * verification * authorKind;
```

**Change:** min-max normalise `base` across the candidate list handed to `adjustScores`, then combine
additively. `adjustScores` already receives the whole list, so the normalisation costs one pass:

```
rel   = (base - min(base over the list)) / (max - min)      // 1 when the list is degenerate
trust = verification × authorKind
score = graphPenalty × (wRel × rel + wFresh × freshness + wTrust × trust)
```

**Measured effect** (weights 0.7 / 0.2 / 0.1), 2026-08-10:

| | `precision@5` | `precision@1` | `hit@5` | easy / medium / hard |
| --- | --- | --- | --- | --- |
| shipped | 0.3159 | 0.2611 | 0.6433 | 0.243 / 0.333 / 0.349 |
| **weighted sum** | **0.5873** | 0.7898 | 0.9490 | 0.568 / 0.588 / 0.605 |

The three ranking metrics that are not in that table, measured in the same run: `recall@5` **0.423 →
0.693**, MRR **0.438 → 0.861**, nDCG@10 **0.459 → 0.741**. Every one moves the same way, which is why
this is read as a fix rather than as overfitting to `precision@5`.

`+0.2714 precision@5`, `+0.5287 precision@1`, `+0.270 recall@5`. Per question file:

| file | n | shipped | weighted |
| --- | --- | --- | --- |
| architecture | 23 | 0.139 | **0.409** |
| conventions | 27 | 0.504 | 0.526 |
| decisions | 23 | 0.200 | **0.504** |
| people | 20 | 0.340 | **0.630** |
| runbooks | 45 | 0.316 | **0.693** |
| state | 19 | 0.379 | **0.695** |

**Nothing gets worse.** The two weakest files in the shipped run, architecture and decisions, gain the
most — which is what you would expect if the defect were a systematic demotion of older `reference` and
`decision` notes rather than a per-subject retrieval problem.

**The weights barely matter, which is the sign of a fix rather than a tuning.** Measured: `wRel` 0.9 →
0.5873, 0.8 → 0.5873, 0.7 → 0.5873, 0.6 → 0.5847. Anything in that range is equivalent. Pick 0.7/0.2/0.1
because it is the one that keeps freshness able to reorder a near-tie: with `wRel` 0.7 and `wFresh` 0.2,
freshness can flip a pair only when their normalised relevance differs by less than 0.286 of the list's
relevance span. **That is exactly the intent PRD 4.2 states** — "staleness reorders near-ties but cannot
outrank a substantially more relevant claim" — expressed as an invariant rather than as a hope.

**It also survives the arrival of stage 6.** With an oracle cross-encoder at any separation from 1.01x
upward it scores 0.9427 to 0.9554, identical to a pure relevance ranking, because normalising erases the
scale difference between an RRF score and a cross-encoder score.

**Risks, and how to control them.**

- *This is a change to PRD 4.1 stage 7, not to 4.2.* The half-lives are untouched and `freshnessDecay` is
  untouched. **Write the composition change into the PRD before the code.**
- *It reintroduces a per-query calibration, which is what RRF was chosen to avoid.* True, and bounded:
  the normalisation is over one result list of at most `rerankLimit` items, not across corpora or over
  time, and it is recomputed per query so it cannot drift.
- **Do not use a fixed normalisation range.** Measured, not predicted: calibrating `lo`/`hi` from `rrfK`
  and the list limits scores **0.5669** today — close enough to the per-query version to look
  interchangeable — and then **collapses to 0.3478** the moment an oracle cross-encoder lands, because
  every base clamps to `rel = 1` and freshness decides again. The per-query version scores 0.9427 in the
  same run. This is the same class of bug as the `gapRelevanceFloorRank` miscalibration and it would
  arrive on the same day.
- *`ScoreComponents`' documented identity breaks.*
  `score = base × typeWeight × graphPenalty × freshness × verification × authorKind` is asserted in
  `types.ts`, in the failure report's column layout, and in tests. The decomposition must be redefined
  and the assertions rewritten **in the same change**, or the score becomes un-explainable — which
  `types.ts` rule 3 exists to prevent.
- *`typeWeight` needs a home.* It is currently multiplicative and legitimately exceeds 1. Folding it in
  as `min(1, typeWeight/2)` at weight 0.1 scores 0.585 — no worse, no better. Since stage 1 agrees with
  the declared intent on only 29% of questions and costs 0.008 when removed entirely, **decide what
  stage 1 is for before deciding where its weight goes.**
- *Conflict, trap and dedup metrics were not re-measured under the new composition.* They move with the
  candidate set. Check them before adopting it.

**Verify:**

```sh
node eval/runner.mjs --output=/tmp/x/     # compare precision@5, precision@1 and the per-file table
npx vitest run --project core packages/core/test/retrieval/decay.test.ts   # 1 file, 68 tests, 0.34 s
```

**The four sites that assert the score identity**, and therefore the four that a change to stage 7's
composition must be reconciled with. Named here because §6.1 calls them a category without listing
them:

| Site | What it asserts |
| --- | --- |
| `packages/core/test/retrieval/decay.test.ts:116` | `(rerank ?? rrf) * typeWeight * graphPenalty * freshness * verification * authorKind` — the identity, recomputed by hand and compared to `score` |
| `packages/core/test/retrieval/expand.test.ts:33-38` | the same product, after graph expansion |
| `packages/core/test/retrieval/pack.corpus.test.ts:77` | `rrf * freshness * verification * authorKind` over the seed corpus |
| `eval/report.mjs:494-509` | the failure report's ten-column layout: `#, note, score, rrf, rerank, type w, graph, fresh, verif, author`. A weighted sum has no such columns |

A weighted sum replaces a product, so all four have to change in the same commit as
`packages/core/src/retrieval/decay.ts` and the `ScoreComponents` type, or the suite goes red for a
reason that looks like a regression and is not.

The PRD phase 7 test at `decay.test.ts:518` — "ranks a 400-day-old state note below a 5-day-old one on
an equal-relevance query" — holds trivially, because freshness stays strictly monotone in age.

### 6.2 Cut `rerankLimit` from 20 to 12

**File:** `packages/core/src/retrieval/types.ts`, `DEFAULT_RETRIEVAL_CONFIG.rerankLimit`.

The pack is always full — median 19 claims of about 173 tokens each — so **median `used_tokens` is a
linear function of `rerankLimit` and nothing else.** PRODUCT.md 9 intends to publish a sub-2,500 median,
which the shipped 20 does not meet. Measured 2026-08-10 with `node eval/runner.mjs --config=…`:

| `rerankLimit` | median `used_tokens` | p95 | max | `precision@5`, shipped stage 7 | `precision@5`, fixed composition |
| --- | --- | --- | --- | --- | --- |
| 20 (shipped) | 3,284.5 | 3,765 | 3,911 | 0.3159 | 0.5873 |
| **14** | **2,350** | 2,711 | 2,946 | 0.4089 | 0.5898 |
| **12** | **2,010.5** | 2,319 | 2,464 | 0.4306 | 0.5885 |
| 8 | 1,354.5 | — | — | 0.5363 | 0.5860 |

`rerankLimit ≤ 14` puts the **median** under 2,500. The **p95** needs `≤ 12`: at 13 it is 2,501, one
token over, which is the kind of margin that flips between runs of a reworded corpus.

**The precision column under the shipped stage 7 is a symptom, not a fix.** Shortening the list removes
the fresh-but-irrelevant candidates stage 7 would otherwise promote, so its effectiveness is a direct
measure of how broken stage 7 is. Under the fixed composition the precision effect disappears and only
the token saving remains — 0.5873 at 20, 0.5885 at 12, 0.5860 at 8 — which is what a healthy pipeline
should look like. **Do not report the 0.316 → 0.431 jump as an improvement to retrieval.**

**Risk:** low, and it is a product decision rather than a correctness one. The cost is a small loss of
depth: under the fixed composition `recall@10` goes 0.787 → 0.756 and nDCG@10 0.741 → 0.729, both far
above the shipped 0.702 and 0.459. A caller who wants twenty claims can raise `budget_tokens` but cannot
raise `rerankLimit`, so if twenty claims is a real use case this needs to become a parameter rather than
a constant. It also cuts the cross-encoder's cost per query by 40% when stage 6 lands — 12 contexts
instead of 20 — which is a latency and money argument for the same change.

**Verify:** `node eval/runner.mjs --config='{"rerankLimit":12}'`. The run cannot pass the gate by
construction, which is the point.

### 6.3 Replace the `low_confidence` floor with a quantity that carries information

> **DONE on 2026-08-13, and not the way this section says. Do not implement what is written below.**
> The diagnosis in the first paragraph is right and the prescription in the second is wrong: the
> quantity it names — the share of the query's terms the best claim contains — is **exactly 1.000 on
> every one of the four false-confidence questions**, which is its maximum, so no threshold separates
> them; set low enough to fire, it fires on 61 of 111 questions the corpus answers. The
> cross-encoder's calibrated score, proposed later as the obvious alternative once stage 6 ran, is
> **0.995 on the worst of the four** — the highest score anywhere in the negative set. Both rank the
> retrieval, and on these questions the retrieval is correct; what is missing is the answer, not the
> match. What shipped instead reads the retrieved *content*: a `[question]` observation that **is**
> the query, and a `how long` frame no claim about the query's subject answers. The measurements, the
> nine other features that were tried and are flat, and the one question that remains unfixable are
> in the round 7 entry of [section 0](#0-tuning-log).

**File:** `packages/core/src/retrieval/gaps.ts`, `lowConfidenceGaps` and `relevanceFloor`.

The current rule tests a rank-derived score against a rank-derived threshold, which is a tautology with
no information in it. It is unreachable today at 0.016129 against a floor of 0.008333, and it will be
unreachable in the opposite direction the day a cross-encoder makes every base ~1.0. See
[5.3](#53-the-low_confidence-relevance-floor-is-unreachable).

**Change:** score confidence from the *content* of the best claim rather than from its position. The
cheapest quantity that works, and which `gaps.ts` already computes half of in `buildClaimIndex`: the
fraction of the query's non-stopword terms that the best claim actually contains. Emit the gap when that
coverage is below a threshold. It degrades gracefully across retriever changes because it never reads a
score.

**Expected effect:** the metric to watch is `falseConfidenceRate`, today **0.2353** (4 of 17), target 0.
Three of the four false-confidence questions return 15 to 19 claims and state nothing. This is the number
PRODUCT.md 9 publishes for trust, and **it is the only one of the three headline retrieval numbers that a
user experiences as a lie rather than as a poor result.**

**Risk:** medium, and it is a precision-of-gaps risk rather than a ranking risk. `gaps.ts`'s own module
comment is right that a detector which cries wolf gets turned off. Whatever threshold is chosen must be
measured against the **157 positive questions** as well: a gap stated on a question the corpus does
answer is worse than a missing gap.

**Verify:** `node eval/runner.mjs --output=/tmp/x/` and read the negative-set table in `report.md`, plus
a new check that no scored question gains a spurious "No confident match" gap. While you are in the
file, make the budget-truncation notice carry its `GapReason` through to the runner instead of being
recognised by a string prefix — see [section 4](#the-negative-set).

### 6.4 Land stage 6, and recalibrate everything that reads a relevance score in the same change

**Files:** the seam is already named. `apps/worker/src/retrieval/ports.ts` wires `recall` to
`notIndexedPorts`; its header specifies the replacement, a new `apps/worker/src/retrieval/indexed.ts`
exporting `indexedPorts(ctx)`. Also in scope: `gaps.ts`'s `relevanceFloor` and whatever normalisation
6.1 introduces.

**This is the only change that can carry the gate; nothing else gets past 0.590.** It is also the change
most likely to break two things at once, because two consumers currently assume the base is an RRF-scale
quantity: the gap floor, and — if 6.1 is implemented with a fixed scale instead of a per-query one — the
composition itself, measured at 0.3478.

**Expected effect:** bounded by the oracle table in [5.5](#55-how-much-of-the-03159-is-the-two-missing-stages).
With 6.1 done, a reranker that orders correctly is worth 0.943 to 0.955. Without it, the outcome depends
on the model's score separation and lies between 0.485 and 0.890.

**Risk:** high, and mostly operational — latency, cost, and a model whose behaviour on this corpus
nobody has observed. **Do 6.1 first**, so the outcome does not depend on an unmeasured property of the
model.

**Verify:** `node eval/runner.mjs --retriever=worker --base-url=https://…`, which is the run that makes
the gate mean what it says. **Until that run exists, every number in this document is a floor.** Note
that it also needs `SIGNPOST_AUTH_SECRET` set, which is a human action:
`npx wrangler secret put SIGNPOST_AUTH_SECRET` then
`node scripts/signpost-bootstrap.mjs --email you@example.com`.

### 6.5 Stopgap only: clamp the freshness multiplier

The proposal on the table before this analysis was to clamp `freshness` to roughly `[0.4, 1.0]` inside
`decay.ts`'s `adjustOne`, leaving the composition multiplicative. It works, it is a two-line change, and
it is strictly worse than 6.1 on three counts. The sweep, measured 2026-08-10:

| clamp floor | `precision@5` | `precision@1` | `hit@5` | easy / medium / hard |
| --- | --- | --- | --- | --- |
| none (shipped) | 0.3159 | 0.2611 | 0.6433 | 0.243 / 0.333 / 0.349 |
| 0.2 | 0.4331 | 0.2930 | 0.8408 | 0.405 / 0.422 / 0.482 |
| **0.4** | **0.5223** | 0.4586 | 0.9236 | 0.481 / 0.511 / 0.585 |
| 0.9 | 0.5605 | 0.6943 | 0.9236 | 0.535 / 0.548 / 0.610 |

**Does the eval support a floor of 0.4?** As a compromise, not as an optimum. The curve is **monotone
increasing all the way to 0.9**, where it equals "freshness off". There is no interior maximum, so the
eval's honest answer to "what floor?" is "the highest one you are willing to defend". 0.4 captures 84%
of the recoverable range. **Anyone who reports 0.4 as the eval-supported value is choosing it on product
grounds. That is legitimate; saying the eval chose it is not.**

**It preserves the phase 7 test and destroys the discrimination underneath it.** A 400-day `state` note
clamps to 0.4 and a 5-day one is 0.70, so `decay.test.ts:518` passes. But the clamp deletes freshness
ordering *within* the flat region, and that region is most of the corpus. The age at which a type
saturates is `−halfLife × ln(floor)`:

| type | half-life | ties with everything older than | notes above the floor, of the corpus's own |
| --- | --- | --- | --- |
| convention | 365 d | 334 days | 2 of 16 |
| decision | 180 d | 165 days | 1 of 21 |
| reference | 180 d | 165 days | 2 of 39 |
| state | 14 d | **12.8 days** | 3 of 16 |
| note | 90 d | 82 days | 1 of 8 |

**Nine of the hundred notes are above the floor.** For the other 91 the clamp does not bound freshness's
influence — it deletes freshness entirely and replaces it with a constant. A `state` note is
indistinguishable from any other `state` note after a fortnight, which is precisely the discrimination
the 14-day half-life exists to provide.

**And it does not survive stage 6.** With an oracle cross-encoder that orders correctly but has a 1.01x
score separation, the clamp scores **0.7490** where the weighted sum scores **0.9427**, because a
clamped-but-still-multiplicative freshness of 0.4 can still overturn a 1.01x relevance difference. A
clamp at floor `f` is safe only when the relevance base's own range exceeds `1/f`; today's range is
1.81x and `1/0.4` is 2.5x, so **it is not even safe today** — it is merely less unsafe.

**The other obvious bounded form, `freshness ** α`, is worse than the clamp everywhere**, because it
compresses the range without bounding it: α 0.75 → 0.327, 0.5 → 0.341, 0.25 → 0.405, 0.1 → 0.475. At a
tenth root — barely a decay function any more — it still only reaches 0.475 against the clamp's 0.522 and
the weighted sum's 0.587. **Anything that leaves the composition multiplicative and the range unbounded
inherits the same defect in smaller print.**

Use the clamp only as a one-line stopgap if 6.1 cannot be scheduled. If you do: floor 0.4, and record
that it is a floor chosen for product reasons.

> **Superseded on 2026-08-12 by something that is not this.** `RetrievalConfig.freshnessFloor`
> ([section 0](#0-tuning-log)) bounds freshness, but it is **not** the `Math.max(floor, decay)` clamp
> analysed above and it does not have this section's defect. It maps the decay affinely,
> `floor + (1 - floor) × decay`, which is strictly increasing everywhere, so no ordering is deleted —
> the flat region this section rejects the clamp for does not exist. The saturation ages, computed at
> the same floor, are the whole argument:
>
> | type | half-life | `max(0.9, decay)` ties past | `0.9 + 0.1 × decay` ties past |
> | --- | --- | --- | --- |
> | convention | 365 d | 38.5 d | 12,821 d |
> | decision / reference | 180 d | 19.0 d | 6,323 d |
> | state | 14 d | **1.5 d** | **492 d** |
> | note | 90 d | 9.5 d | 3,161 d |
>
> (The affine column is where the double loses the difference, not where the formula does.) A hard
> clamp at 0.9 would tie every `state` note older than a day and a half, which is why this section is
> right to reject it and why its author had to settle for 0.4. The affine form is what makes a floor
> of 0.9 — and the 0.3279 -> 0.5640 that comes with it — available at all. 6.1 is still worth doing;
> this is no longer a stopgap standing in for it.

### 6.6 Rejected: lower `rrfK`

`eval/README.md` recommends widening the relevance signal via `rrfK: 60 → 1` as fix number one. The
measurement does not support it.

| `rrfK` | with shipped stage 7 | with relevance-only ranking |
| --- | --- | --- |
| 60 (shipped) | 0.3159 | **0.5898** |
| 10 | 0.3758 | 0.5834 |
| 1 | 0.4471 | 0.5580 |

Lowering `rrfK` helps **only** because it inflates the relevance base enough to partially survive the
freshness multiplier. Once the composition is fixed, `rrfK: 1` is **worse** than `rrfK: 60` (0.5580
against 0.5898). It is a workaround for stage 7 wearing the costume of a fusion improvement, and adopting
it would spend RRF's calibration-free property — the whole reason PRD 4.1 chose it — to buy back damage
caused somewhere else.

### 6.7 Not a fix: the eval set's own gaps, in the order they cost credibility

1. **The corpus cannot exercise the trust multipliers.** Every chunk is `unverified`/`human`. Seed a
   verified minority and an agent-authored minority into `scripts/seed-corpus.mjs` and the eval will, for
   the first time, be able to say whether PRODUCT.md bet 4 works end to end.
2. **The relevant sets are broader on hard questions than on easy ones** (13.44 chunks against 10.95).
   Either normalise them or report `precision@5` beside its per-slice ceiling so the comparison is honest.
3. **Graph expansion is untested end to end** — 27 candidates across 196 queries, surviving in 2 packs.
   Add questions whose answer is reachable only through a relation.
4. **Seven of the 21 trap questions carry strings the corpus does not contain**, so they can never fire.
   `node eval/validate-questions.mjs` already warns about each by name; the fix belongs in the question
   files.
5. **There is no metric for what decay buys.** See the box at the end of
   [5.7](#57-what-the-eval-cannot-see-and-therefore-what-section-5-does-not-cover).

---

## 7. The decision reserved for a human

> **A second one arrived on 2026-08-13, and it is now the only thing standing between this project and
> a phase 2 exit: is `precision@5` the right gate?** Round 8 measured the real pipeline and found it
> scores **0.7748 against the reference's 0.7856 while being better on every other measure** —
> `precision@1` 0.955 against 0.892, MRR 0.976 against 0.933, answer-text-in-full 0.955 against 0.901,
> `hit@5` 1.000 on both. The degraded BM25-only configuration, which **fails to find the answer on two
> of 111 dev questions**, scores **0.8180** — the highest `precision@5` measured anywhere in this
> project. A metric that ranks *cannot find it* above *finds it and ranks it first* is measuring
> something other than what PRD §8 phase 2 wants it to measure, and no amount of further tuning fixes
> that. **This is a specification amendment, not a code change**, and it is the project owner's to
> take. The evidence is [round 8](#2026-08-13--round-8-final-numbers-and-the-verdict); the revision box
> at the top of this document is the record.

> **Taken on 2026-08-13, and the stop condition is no longer met.** The operator authorised the
> stage-7 change; it was made, in eight shipped changes over five rounds, and `precision@5` moved
> **0.3159 → 0.7720** on the whole set, **0.7391** on the held-out split, **0.7856** on dev. All three
> are above the 0.6 stop threshold and below the 0.80 gate. The composition shipped is **not** the
> weighted sum [6.1](#61-replace-the-multiplicative-composition-in-stage-7) recommended — it is the
> original multiplicative identity with five bounded multipliers added and the freshness term floored,
> which is a smaller amendment to the stage-7 specification and scores higher than the weighted sum's
> measured 0.587. [Section 0](#0-tuning-log) records the deviation and what replaced
> it. **What follows below is the pre-tuning argument, left as written**, because it is the
> record of why the change was made and statements 2 and 3 are both still true. Statement 1 is not:
> read it as history.

PRD phase 2 sets a stop condition, and it *had* been reached.

> If `precision@5` sits below **0.6** after a genuine tuning effort, the retrieval design gets
> reconsidered before any interface is built on top of it.

Three plain statements, in the order they matter.

**1. The current number is 0.3159, which is below 0.6.** It is below it on every slice: easy 0.243,
medium 0.333, hard 0.349, and the best question file (`conventions`) is 0.504. There is no reading of the
run in which the condition has not been met. *(Superseded 2026-08-13: 0.7720 on the whole set, and no
slice is below 0.60 — the lowest is `architecture` at 0.6609. The condition is no longer met.)*

**2. The pipeline is missing its two most important stages.** Stages 2, 3 and 6 are not implemented in
the deployed Worker; `notIndexedPorts` answers six of the seven ports with an empty list. Every number in
this document was produced by in-memory stand-ins, one of which — the hashing embedder — is measurably
*worse than not having a dense retriever at all*. Reconsidering a retrieval design on the strength of a
run in which the cross-encoder never executed and the embedding model was a hashing trick would be
reconsidering something other than the design.

**3. The interfaces were built anyway.** PRD phase 3 is complete and deployed: seven MCP tools, Quick
Connect, session links. That sequencing decision has already been taken and is not revisited here — but
it means the stop condition is being read *after* the thing it was meant to gate, and whoever picks this
up should know that.

**The honest sequence is: build phase 2B, apply the ranked fixes, re-measure, then decide.** Concretely:

1. ~~Apply [6.1](#61-replace-the-multiplicative-composition-in-stage-7)~~ — **done, 2026-08-13, and
   worth +0.456 rather than the +0.271 estimated here.** Eight shipped changes over five rounds, all
   in `packages/core/src/retrieval/`, all pure: `freshnessFloor`, `noteSupport`, `typeWeightStrength`,
   `termCoverage`, `noteCoverage`, the `rerankLimit` cut moved to the end of stage 7, and
   `linkConsensus`. `precision@5` 0.3159 → 0.7720 on the whole set. See [section 0](#0-tuning-log).
   The multiplicative identity was kept and bounded rather than replaced by 6.1's weighted sum, which
   measured 0.587; PRD §4.1 stage 7 still owes the amendment.
2. Build [6.4](#64-land-stage-6-and-recalibrate-everything-that-reads-a-relevance-score-in-the-same-change):
   `indexed.ts`, `bge-m3` at stage 3, `bge-reranker-base` at stage 6, and recalibrate
   [6.3](#63-replace-the-low_confidence-floor-with-a-quantity-that-carries-information) in the same
   change — after reading round 5's measurement that 6.3's prescribed quantity cannot fire. **This is
   now the whole of the remaining work, and it needs a Cloudflare credential.**
3. Re-run the gate with `--retriever=worker` against an indexed instance. That is the first run in the
   project's history that measures the system the PRD describes. **It has still never happened.**
4. **Then** apply the stop condition to that number. The stop condition is no longer met by the
   reference pipeline (0.7720 > 0.6); it would be met again if a real stage 6 landed and the number
   failed to move past 0.590.

**This document does not make that decision and neither should the agent that reads it.** What it
establishes is the evidence the decision needs: the ceiling is 0.995, so the metric is not the obstacle;
0.274 of the shortfall is a composition defect fixable in pure code today; and the remaining 0.405 is
bounded by an oracle at 0.955, which is what the two unbuilt stages are worth if they work.

Two things that would make the decision wrong if they were assumed rather than checked:

- **That 0.590 is what a fixed stage 7 is worth in production.** It is what a fixed stage 7 is worth
  *with a hashing embedder and no reranker*. BM25 alone scores 0.645 in the same configuration.
- **That the oracle rows predict `bge-reranker-base`.** They do not. The score separation a real
  cross-encoder produces between a relevant and an irrelevant chunk of the same knowledge base is the
  single number this plan is most sensitive to, and **nobody has measured it.**

---

## 8. Tuning guide: symptom to knob

Start every one of these by reading the `ScoreComponents` decomposition for the offending claim. The
eval failure report prints it. **Turning a knob before reading the decomposition is guessing.**

### Before you turn anything: five things this system was caught doing

A tuning guide written before the system was attacked is a list of guesses. These five were found by
attacking it, and each would have absorbed a week of tuning that could not have worked, because in every
case the knob a symptom points at is not the knob that is wrong.

1. **A default can be dead.** `reportReserveTokens` was declared, defaulted, documented in three places
   and read by nothing; `0`, `256` and `1,000,000` produced byte-identical packs. Before attributing a
   behaviour to a knob, change it to something absurd and check that the output moves. Thirty seconds,
   and it is the cheapest experiment in this document.
2. **A constant can be right in the paper it came from and wrong here.** `rrfK = 60` is Cormack's, tuned
   on TREC runs a thousand documents deep. PRD 4.1 imported the 60 and also capped both lists at 50, and
   at that depth it turned fusion into an AND. **Check the reachable range of a score, not just its
   formula:** `1/(k+1)` against `2/(k+listLength)` is one line of arithmetic and it decided the ranking of
   every query.
3. **A cap applied before an append is not a cap.** `maxGaps` was applied by `buildGaps` and the
   truncation notice was added afterwards, so `maxGaps: 3` produced four gaps and `maxGaps: 0` produced
   one.
4. **A multiplier that is recomputed downstream is not a discount.** Stage 5's `graphPenalty` of 0.6 was
   applied to a base that stage 7 then re-derived from the neighbour's own metadata, so the "discount"
   ran as high as 204x. Any factor set in one stage and recomputed in another needs one of them to be the
   authority, in writing.
5. **A metric can be moved by something that is not the system.** `--budget=800` took the published
   `empty-recall` from 0.824 to a perfect 1.000 without changing gap detection at all, because the
   packer's budget notice was being counted as a knowledge gap. If a number improves, check what else
   changed.

### 8.1 Too much noise: irrelevant claims fill the pack

1. **Check the `fresh` column first.** This *was* the answer in most cases: a fresh, irrelevant claim
   beat a stale, relevant one, in 39 of 111 dev questions. See
   [5.1](#51-stage-7-destroys-most-of-the-ordering-the-earlier-stages-produced) for the diagnosis and
   [section 0](#0-tuning-log) for the fix — `freshnessFloor: 0.9`, which took that to 7 of 111. If you
   are still seeing it, the remaining cases are ones where freshness is inside its 1.111x budget and
   the relevance base is genuinely too flat to separate the two claims, which is a
   [6.1](#61-replace-the-multiplicative-composition-in-stage-7) problem and not a decay one.
2. **Do not raise `gapRelevanceFloorRank` expecting anything.** At the shipped numbers the rule cannot
   fire at all — the floor is 0.008333 and the worst base any candidate can hold is 0.009091. The largest
   usable value is 50, and even there it fired 0 times in 191 queries. **Retuning the constant is not the
   fix; the quantity is wrong.** See [6.3](#63-replace-the-low_confidence-floor-with-a-quantity-that-carries-information).
3. **Check whether the noise is link lists.** If the offending claims' text is `verb [[target]]` lines,
   something has bypassed the stage-9 filter — they were 6.1% of all claims and 50 top-five slots before
   it existed. `isRelationOnlyChunk` is the predicate.
4. **Lower `fusedLimit` and `rerankLimit`.** Fewer candidates, less to be wrong about. `rerankLimit` is
   also the direct lever on `used_tokens` — about 173 tokens per claim. Under a fixed stage 7 this stops
   affecting precision, which is how you tell the two effects apart.
5. **Check whether the noise is graph neighbours.** If the offending claims have `graphHops: 1`, lower
   `graphPenalty` (0.6 → 0.4) or `graphExpandLimit` (20 → 10). On the current eval corpus this will change
   nothing, because graph expansion is inert. If a neighbour appears to outrank the seed it borrowed from,
   read its `graphSeed`: it means the cross-encoder promoted it on its own merits, not that the discount
   leaked.
6. **Check whether the noise is one enormous chunk.** A 900-token chunk that matches on one word costs
   five claims' worth of budget. That is a chunker problem, not a retrieval one.

### 8.2 The right note ranks second

Read its components against the winner's. **The factor with the largest ratio between them is the
answer.**

- **`freshness` is the culprit** (the right note is older). This is the common case today and the ratio
  will be large — 9.8x on `arch-018`, 129x in the worked example in section 1. If you are looking at one
  query, `halfLives` for that note's type is the local knob. If you are looking at the aggregate, it is
  the composition, not the half-life. Do **not** raise `halfLives.state` past about 30 without deciding
  that "auth is 70% done" is worth reading a month later.
- **`support` is the culprit** (the right note showed two of its eleven chunks, the winner showed both
  of its two). It is bounded — at the shipped `supportFloor: 0.5` the median gap within one list is
  1.548x and the absolute bound is 2x — so it can only be the culprit when the relevance gap is
  narrower than that. This is the price [section 0](#0-tuning-log) records for the note-support
  multiplier and it now costs 6 of 111 dev questions, all of them large relevant notes that showed a
  small fraction of themselves. The only knob is `supportFloor`, upward, and it gives back
  `precision@5` across the board to fix a handful of queries: 0.65 costs 0.0162. **Do not reach for
  the denominator.** Normalising by the list's maximum was tried and is worth −0.0144; by the
  pre-rerank list and by the fused top 60, −0.1856 each, because both measure retention through the
  rerank cut rather than coverage of a note.
- **`termCoverage` is the culprit** (the winner uses the asker's words and the right note answers in
  the corpus's). Bounded at `1 / coverageFloor` = 2.857x, so it can only decide a comparison narrower
  than that. This is the price [section 0](#0-tuning-log) records for the coverage multiplier and it
  costs 7 of 111 dev questions. The knob is `coverageFloor`, upward, and it gives back precision
  across the board: 0.5 costs 0.0181, 0.7 costs 0.0199. **Do not reach for a term weighting first** —
  inverse document frequency, IDF squared and counting proper nouns double were all measured at three
  floors and every one of them scores at or below the unweighted share. If the right note is a
  paraphrase with no shared vocabulary at all, this factor is not the fix; a real embedder and a real
  cross-encoder are.
- **`graphPenalty` is the culprit** (the right note is `graphHops: 1`, the winner is a direct hit).
  Either raise `graphPenalty`, or — better — look at whether the right note *should* have matched
  directly. A note only reachable through a link often has a title problem.
- **`typeWeight` is the culprit.** The intent classifier misread the question. Check `signals` in the
  report: no cue fired means the phrasing is not in `INTENT_RULES` and the fix is one line in that table;
  the wrong cue fired means a cue is too greedy. It is bounded — at the shipped
  `typeWeightStrength: 0.35` the widest gap between two types in one list is about 1.36x — so it can
  only be the culprit when the relevance gap is narrower than that, and `typeWeightStrength` is the
  knob for the aggregate. Section 5.7's "stage 1 is worth 0.008 in aggregate" was measured through the
  un-floored freshness multiplier and is stale: bounding this factor is worth 0.0342, which is
  [section 0](#0-tuning-log)'s third entry.
- **`rrf` is the culprit and the right note is in only one list.** Check `lexicalRank` and `denseRank`.
  Absent `denseRank` on a paraphrased question is an embedding problem, and today it is *expected*,
  because the embedder is a hashing trick that cannot match a synonym. Absent `lexicalRank` on a question
  that shares words with the note is a tokenisation or chunking problem and is a real finding.
- **Everything is close and the order looks arbitrary.** It is not arbitrary — it is the `chunkId`
  tiebreak — but it does mean the scores are genuinely indistinguishable and the signal you need is not
  in the pipeline. That is the cross-encoder's job.
- **One wrong note holds three of the top five slots.** The obvious remedy is not the remedy. Damping
  the *j*-th chunk of one note is monotonically worse at every setting measured — geometric 0.95
  scores 0.6811 and 0.8 scores 0.5477, a hard cap of four per note scores 0.6919 and of two 0.5586,
  all against 0.7171 — because on this corpus the note holding one relevant chunk usually holds the
  next one too. Reach for a *per-chunk* signal, which is what `termCoverage` is, rather than a
  per-note penalty. The whole diversity family is retired in [section 0](#0-tuning-log).

### 8.3 The right note is not in the pack at all

The report says this explicitly: *"Never reached the packer at all"*. **No weight in stage 7 can promote
a candidate that was never a candidate**, so nothing in 8.1 or 8.2 will help. This is 24 of the 56
current failures, and 66 of 396 expected notes (16.7%) overall.

Look at stages 2 to 5: raise `lexicalLimit` and `denseLimit`, check that the note is embedded
(`index_health.embedding_coverage`), and check that a filter did not remove it — `filteredOut` in the gap
detector's input counts exactly that. On the current eval, the honest answer for most of these is that
the hashing embedder cannot see the paraphrase and `bge-m3` is the fix.

### 8.4 Conflicts are missed

1. **Confirm both sides reached the packer.** Most missed conflicts are missed *retrieval*, not missed
   detection: the detector finds 6 of the corpus's 6 seeded conflicts when both claim-bearing chunks are
   in front of it, and 5 of 6 end to end.
2. If both sides are present and nothing fired, lower the relevant gate in `DEFAULT_CONFLICT_TUNING.gates`.
   `minShared` and `minSharedStatement` are the two that bite: they demand that the two statements share
   vocabulary, and two notes that use different words for the same subject will not clear them.
3. If the two sides use different words for the same concept, the fix is `conceptClasses` — one table
   that folds `store/hold/held/keep/kept/retain/retention/persist/ttl/lifetime/expire` to a single stem.
   Two of the six seeded conflicts are invisible without it.
4. Raise `maxConflicts` only after checking that the missing one is being found and trimmed.

**Conflicts are wrong (false positives):** raise `minOverlap` and `minDice` on the offending signal kind.
A false positive is expensive in a way a missed conflict is not — it teaches the reader to ignore the
section.

### 8.5 Gaps are not firing

- **"No confident match found" never fires, on any query.** That is [5.3](#53-the-low_confidence-relevance-floor-is-unreachable),
  it is unreachable by construction, and no setting of `gapRelevanceFloorRank` rescues it.
- **The rules that do fire** are per-term absence (`No claims found about X`, 70 emissions), whole-query
  absence (5), never-joined co-occurrence (`No single claim covers X and Y together`, 50), staleness (27)
  and implied type (3). If a gap you expect is missing, work out which of those five it should have been
  and read that rule.
- **`gapMaxTermGaps` is 2**, so at most two "no claims found about X" sentences appear per pack, and
  `maxGaps` is 3 overall. A gap can be detected and then trimmed.
- **Before believing a false-confidence number, check what counts as a gap.** The budget-truncation notice
  is filtered out of `knowledgeGaps` by a **string prefix match**, so rewording it changes the published
  rate from 0.2353 to 0.1765 with no pipeline change.

### 8.6 The pack is too big or too small

`used_tokens` is a function of how many candidates survive stage 8, not of the packer. The pack is always
full. Median 3,284.5 tokens at `rerankLimit: 20`, 2,350 at 14, 2,010.5 at 12, 1,354.5 at 8. **PRODUCT.md's
"median tokens per recall under 2,500" is met at 14 and its p95 needs 12** — so either `rerankLimit` comes
down or that figure is republished. It is not a packer bug and no amount of packer work moves it. See
[6.2](#62-cut-reranklimit-from-20-to-12).

### 8.7 Everything is worse after a change and you cannot tell why

`git diff` the config object. **If the diff is empty, the change was to a table that is not in
`RetrievalConfig` yet** — `INTENT_RULES`, `DEFAULT_GRAPH_EXPANSION_TUNABLES`, `DEFAULT_CONFLICT_TUNING`,
`DEFAULT_GAP_TUNABLES`, `DEDUP_SHINGLE_SIZE`, `WIRE_SIGNIFICANT_DIGITS` — which is exactly why they should
be moved into it.

Then re-run with `--only=<the file that regressed>`: a change that helps runbooks and hurts decisions is
not an improvement, and the aggregate hides that. **Never report the aggregate on its own.** The report is
laid out to make that impossible to miss — the aggregate appears inside the first slice table as one row
among the others, and any slice below 0.6 is called out in the header.

### 8.8 How to run an experiment without touching the repository

Every measurement in [section 5](#5-why-it-fails) and [section 6](#6-the-fix-plan) came from a scratch
harness that swaps one step and reuses the eval's own machinery. It changes nothing in the working tree.
The essential shape:

```js
// /tmp/lab.mjs — run with: node /tmp/lab.mjs
import { readdirSync, readFileSync } from "node:fs";
import { register } from "node:module";
import { join } from "node:path";
import { pathToFileURL } from "node:url";

const ROOT = "/Users/ankur/work/signpost";
register(pathToFileURL(join(ROOT, "eval/loader.mjs")));           // TS source imports

const retriever = await import(pathToFileURL(join(ROOT, "eval/retriever.mjs")).href);
const pipeline  = await retriever.loadPipeline();
const corpus    = await retriever.loadCorpus();
const ports     = await retriever.createReferencePorts(corpus, { reranker: "none" });
const config    = { ...pipeline.config, rerankMinCorpus: Number.POSITIVE_INFINITY };

// A stage 7 variant: run the SHIPPED adjustScores to get the components, then
// recompose the score from them. Nothing about how a component is computed
// changes, so a difference in the result is a difference in the composition
// and nothing else.
const weighted = (input, cfg) => {
  const adjusted = pipeline.steps.adjustScores(input, cfg);
  const bases = adjusted.map((c) => c.components.rerank ?? c.components.rrf);
  const lo = Math.min(...bases), hi = Math.max(...bases), span = hi - lo;
  return adjusted
    .map((c) => {
      const b = c.components.rerank ?? c.components.rrf;
      const rel = span > 0 ? (b - lo) / span : 1;
      const k = c.components;
      const score = k.graphPenalty * (0.7 * rel + 0.2 * k.freshness + 0.1 * (k.verification * k.authorKind));
      return { ...c, score };
    })
    .sort(pipeline.compareScoredCandidates);
};

// --- the eval set, loaded exactly as runner.mjs loads it -------------------
// eval/questions/ is eight JSON files. Each is { set?, questions: [...] }; the
// set name falls back to the filename. 196 questions load; the 157 with a
// non-empty `relevant` are the scored ones and are the denominator of every
// number in this document. `nearest` becomes the partial-credit list.
const dir = join(ROOT, "eval/questions");
const questions = [];
for (const name of readdirSync(dir).filter((n) => n.endsWith(".json")).sort()) {
  const file = JSON.parse(readFileSync(join(dir, name), "utf8"));
  for (const q of file.questions) {
    questions.push({ ...q, file: file.set ?? name.replace(/\.json$/, ""),
                     relevant: q.relevant ?? [], partial: q.nearest ?? [] });
  }
}
const scored = questions.filter((q) => q.relevant.length > 0);   // 157

// `kmem://permalink#blk_x` -> `permalink`. A legacy `sp://` ref gives the same
// permalink. This is the runner's own rule: runner.mjs and this script both
// call `noteOfRef` in eval/refs.mjs, which reads the scheme with the core ref grammar.
const { noteOfRef: noteOf } =
  await import(pathToFileURL(join(ROOT, "eval/refs.mjs")).href);

// --- score a step set -------------------------------------------------------
const { rankingMetrics, distinct } =
  await import(pathToFileURL(join(ROOT, "eval/metrics.mjs")).href);

async function score(steps) {
  const cases = [];
  for (const q of scored) {
    const pack = await pipeline.recall(
      { text: q.question, budgetTokens: 4000, now: corpus.anchor },
      ports, config, steps,
    );
    const retrieved = pack.claims.map((c) => noteOf(c.ref));
    const grades = {};
    for (const p of q.partial) grades[p] = 1;      // GRADE_PARTIAL
    for (const p of q.relevant) grades[p] = 3;     // GRADE_RELEVANT
    cases.push({ id: q.id, retrieved, retrievedDistinct: distinct(retrieved),
                 relevant: q.relevant, grades });
  }
  return rankingMetrics(cases);   // precisionAtK, recallAtK, hitRateAtK, mrr, ndcg
}

const shipped  = await score(pipeline.steps);
const variant  = await score({ ...pipeline.steps, adjustScores: weighted });
console.log(shipped.precisionAtK["5"], variant.precisionAtK["5"]);
```

**This harness was run end to end on 2026-08-10 and reproduces every published figure**, which is
what makes it worth copying rather than reading. Its output, to four places:

```
loaded 196 scored 157
shipped  p@5 0.3159 p@1 0.2611 hit@5 0.6433 recall@5 0.4230 mrr 0.4380 ndcg 0.4587
weighted p@5 0.5873 p@1 0.7898 hit@5 0.9490 recall@5 0.6934 mrr 0.8609 ndcg 0.7408
```

Add `readdirSync, readFileSync` from `node:fs` to the imports at the top. A full 157-question pass
takes about six seconds and needs no network and no Cloudflare account, so the edit-measure loop is
seconds rather than minutes. If your `shipped` line is not exactly `0.3159`, stop: something in your
setup differs from the run every number here came from, and comparing a variant against it will
mislead you.

Four details that will otherwise waste an hour.

- **`now` must be `corpus.anchor`.** Anything else changes every freshness in the run.
- **`rerankMinCorpus` decides whether stage 6 runs.** `eval/retriever.mjs` sets it to `Infinity` for
  `--reranker=none`, which is why the reference reranker never fires. Set it to `0` and supply
  `reranker: { score }` to `ReferenceRetrievalPorts` to inject one — that is how the oracle experiments
  were run.
- **Candidates in `adjustScores`'s *input* have `freshness: 1`.** Every multiplier is seeded to 1 and
  filled in by the step that owns it, so reading freshness off the pre-stage-7 list gives 1 for
  everything. Read the components off the *output*.
- **A claim's rank in the pack is better than its rank leaving stage 7**, because dedup collapses
  near-duplicates in between. Be explicit about which you are reporting.

Total runtime for every experiment in this document: about five minutes.

---

## 9. Context pack v2 — PRD_PART2 F2

**Written 2026-08-17.** Five fields on every claim, a budget-accounting object and a `partial` flag,
all added and none renamed. Every number below was measured with `node eval/runner.mjs` over the whole
committed 196-question set at the corpus anchor and the shipped 4,000-token budget.

### 9.1 What F2 asked for, and what shipped

| F2 asks for | Shipped as | Why not literally |
| --- | --- | --- |
| `claims[].note` — "note id" | `note`, the **permalink** (`brand/identity`) | F2's own example is a permalink, and the opaque `note_…` id appears on no other wire field. Publishing it would put a second, unusable identifier on the wire |
| `claims[].heading` | `heading`, the **innermost** heading, `null` above the first one | The full path is already the opening line of `text` — the chunker prefixes every chunk with `{title} > {headingPath}` and that prefix is what was embedded. Publishing the path again charges the caller twice for the same bytes |
| `claims[].version` — an ordinal (`1`) | `version`, the **first 12 hex characters of `content_sha`** | ADR 0011: the ordinal is derived by paging the whole `blobs/` prefix per note and "is the wrong thing to put on a hot path". `recall` is the hot path. Twelve rather than sixty-four is measured, not aesthetic — see 9.3 |
| `claims[].status` | `status`, F3's `current \| superseded \| deprecated` | As specified. Deliberately **not** `provenance.state`: they share the word `superseded` and mean different things everywhere else |
| `claims[].updated` | `updated`, an ISO-8601 UTC **date** | As specified, and a date rather than a timestamp because it answers "how stale is this", which nobody answers to the second |
| `budget: {requested, used}` | `budget: {requested, used, report, dropped}` **beside** `budget_tokens`/`used_tokens` | Additive. `report` is the number the flat pair cannot express — see 9.2 |
| `partial: false` | `partial`, and the `budget_truncated` gap sentence stays | A boolean is what a caller branches on; matching prose is what an agent quotes |
| `conflicts[]: {notes, subject}` | unchanged: `{topic, refs, note}` | `subject` **is** `topic` renamed, and `refs` is strictly more precise than `notes` — block-level rather than note-level, and `distinct(refs.map(noteOf))` recovers `notes` exactly. A pure rename of a published field buys nothing and breaks every client |

`truncated?: boolean` was also folded into `Claim` itself. It existed as `PackedClaim` in `pack.ts`
with a comment asking for exactly this reconciliation; `PackedClaim` is now an alias.

### 9.2 Why `budget` carries four fields and not two

`requested` and `used` restate `budget_tokens` and `used_tokens` on purpose — those names are
published in six places and are not being renamed — so on their own the object would be pure
duplication. `report` is what makes it worth its tokens.

Conflicts and gaps are measured exactly as claims are and then deliberately **not** charged to
`used_tokens`; the argument is on `PackerResult` and it is that charging them puts the budget
contract and the never-silent contract in direct conflict at small budgets. The consequence is that
a caller told `used_tokens: 2245` receives a response that costs more than 2,245 tokens. A
budget-accounting object that omitted the difference would be accounting for the convenient half.
`dropped` is the count behind `partial`.

### 9.3 What F2 costs, measured

Isolated from every other change in flight by re-serialising each emitted claim with and without the
five new fields, using the same `estimateTokens` the budget is counted with, over all 196 questions:

| | tokens per claim | delta |
| --- | --- | --- |
| PRD 4.3 shape | 182.20 | — |
| F2 shape, `version` as the full 64-hex sha | 248.75 | **+66.49 (+36.5%)** |
| **F2 shape, `version` as 12 hex** | **219.33** | **+37.13 (+20.4%)** |

The full sha alone cost **39.30 tokens on every claim** — 59% of everything F2 added, and 9.8% of a
4,000-token budget — for a value that is byte-identical on every claim from the same note. Twelve
characters cost 9.94.

What the short form gives up is precisely what a pack does not need. `version` exists so a caller can
notice that a note it cited has moved, and a prefix does that. It is **not** a concurrency token:
`expected_sha` is, it is all sixty-four characters, `read` returns it, and
`common.ts`'s `^[0-9a-f]{64}$` refuses a prefix with a message naming the constraint rather than
comparing loosely and letting a stale write through. ADR 0011's objection to the demo's twelve-hex
`shortHash` was that it was a hand-rolled FNV used *as* a concurrency token; neither is true here, and
`index-queue.ts` and `ingest.ts` already print `contentSha.slice(0, 12)` when they name a sha.

At the shipped 4,000-token budget the whole of F2 costs **three claims across 196 packs** (1,981 →
1,978 with the long sha, 1,982 with the short one) and moves no quality metric: `precision@1` 0.8981,
`hit@5` 1.000, `MRR` 0.9367 and answer-text coverage 0.8917 are identical before and after. `used_tokens`
median goes 1,862 → 2,245.5 and the worst budget fraction 0.6485 → 0.7588.

`precision@5` also moved in the same wave, 0.7720 → 0.6268, and **none of that is F2**. It is F1's
diversity cap, which the gap study forecast at 0.14–0.16 before either was written; F2 changes how many
tokens a claim costs and not which claims are ranked. The isolation above is per claim, computed by
re-serialising the same emitted claims with and without the five fields, so it is independent of every
other change in flight.

**The number the operator needs for PRD_PART2's proposed default budget of 2,000:** a median pack now
costs 2,245.5 tokens. At a 2,000-token default it would drop claims on more than half of all queries.
The shipped default is 4,000 and the maximum is 32,000, against F2's 2,000 and 6,000, and F2 makes
that conflict materially worse rather than neutral.

### 9.4 The PRD amendment

> **PRD §4.3 amendment, 2026-08-17.** `Claim` gains `note`, `heading`, `version`, `status`, `updated`
> and `truncated`; `ContextPack` gains `budget` and `partial`. Every one is **additive**: no existing
> field is renamed, removed or retyped, so the worked example in §4.3, `docs/API.md`, `docs/MCP.md`,
> `packages/agent-kit/skills/kmem/SKILL.md` and the recall screen all stay true and every client
> already parsing the shape keeps working. Two deviations from PRD_PART2 F2's literal spec are
> recorded above with their reasons: `version` is a twelve-character `content_sha` prefix rather than
> an ordinal (ADR 0011), and `conflicts[]` keeps `{topic, refs, note}` rather than becoming
> `{notes, subject}`. `contextPackSchema` in `routes/api/recall.ts` is `z.strictObject`, so it was
> extended in the same commit; a client validating against the old strict schema will reject the new
> keys, which is the one compatibility cost and it applies to no shipped client.

---

## 10. CCH — the recall eval's one metric (PRD_PART2 F10)

**Written 2026-08-17.**

### 10.1 Is CCH computable from what the harness records? No, and here is what is

F10 defines CCH over an **answer**: a fixed model reads the pack, writes a reply, and a separate pass
judges the reply into `correct-cited`, `honest-gap`, `wrong` or `uncited`. There is no model anywhere
in this harness — `--retriever=reference` is arithmetic and `--retriever=worker` calls a deployment's
`recall` over HTTP — so that metric is **not computable from what the harness records**, and
publishing something else under its name would be the self-reported number PRD_PART2's own rationale
says the category discounts.

What is computable, exactly and for free, is the retrieval half: whether the pack a reader was handed
contained the answer, attributed to the note that says it, or honestly reported the absence. That is
`CCH_r`, and it **bounds F10's CCH from above** — a reader cannot cite an answer it was not handed,
and cannot be honest about a gap the pack did not state. A perfect reader scores exactly `CCH_r`;
every real one scores at or below it. That bound is why the number is worth publishing without a
model, and why it must never be published as if it were the whole metric.

`eval/metrics.mjs`, `citedCorrectOrHonest`.

### 10.2 The five verdicts

A question is judged in one of two ways, and a question that declares neither an answer nor an
absence is not judged at all.

| Verdict | Fires when | In the numerator |
| --- | --- | --- |
| `correct-cited` | every `mustContain` string is present in the claims **from a relevant note** | yes |
| `honest-gap` | the corpus does not answer it, a knowledge gap was stated, no trap string came back | yes |
| `uncited` | every required string is present somewhere in the pack, but at least one only under a note the judgement does not call relevant | no |
| `unsupported` | a required string is absent from the pack altogether | no |
| `wrong` | the pack **stated something false**: it answered a question the corpus cannot answer without saying so, or claimed an absence a returned string contradicts | no |

F10 names four verdicts. `unsupported` is split out of `wrong` because F10 defines `wrong` as
"incorrect or invented" and makes it the number that must reach zero — and inventing is a property of
a reader, which this harness does not have. A retrieval miss states nothing false; a reader that
refuses there is behaving correctly. Both readings are published: `wrong` is 1 and `wrongOrUnsupported`
is 18. **CCH is identical either way**, because neither verdict is in the numerator.

Matching is per needle against the *join* of the relevant-note claims, not per claim. `mustContain`
routinely holds strings that live in several notes, and a sentence the chunker split across two chunks
of one note would otherwise be reported absent for a reason that has nothing to do with citation. Both
sides go through the same `normaliseForMatching` that `substringCoverage` uses — backticks stripped,
the corpus's column-100 wrap flattened, case folded.

### 10.3 The number

`node eval/runner.mjs --gate`, reference retriever, no cross-encoder, `DEFAULT_RETRIEVAL_CONFIG`
unmodified, corpus anchor 2026-08-09, budget 4,000, 100 notes / 459 chunks, split `all` (v1), run
2026-08-17, 4.3 s wall clock, no billable calls:

| | whole set | dev split |
| --- | --- | --- |
| **`CCH_r`** | **0.8966** | **0.9024** |
| judged / unjudged | 174 / 22 | 123 / 16 |
| `correct-cited` | 140 | 100 |
| `honest-gap` | 16 | 11 |
| `uncited` | **0** | **0** |
| `unsupported` | 17 | 11 |
| **`wrong`** | **1** | **1** |
| `wrongOrUnsupported` (F10's four-category reading) | 18 | 12 |

**`uncited` is zero, and that resolves a bound.** The gap study could only bracket CCH at
`[0.810, 0.902]` because `coverage.allRate` (0.8917) and `precision@1` (0.8981) count two sets the
harness never intersected. Judging per needle against the relevant-note claims intersects them: every
time the answer text came back, it came back under a note the judgement calls relevant. So the true
figure is the top of that bracket, 0.8966, and the `uncited` arm of the metric is inert on this corpus
— it would fire only if retrieval returned a near-duplicate of the answer from the wrong note.

**The one `wrong`** is `neg-payout-failure-volume` ("How many payouts failed last month?"), which
returns claims and states no knowledge gap. It is the same question the gap study named and the one
PRD §4.3's amendment records as measured and rejected. F10 requires zero before launch; this is the
whole of the distance.

`CCH_r >= 0.85` is a gate check as of this commit — PRD_PART2's own success-metric threshold,
unmodified, with 0.047 of headroom. `wrong == 0` is **not** gated, because it does not hold today and
adding a gate nobody can pass is not a ratchet.

### 10.4 It cannot be gamed the way `precision@5` was, and here is the measurement

`precision@5` was retired as a gate on a validity proof: a degraded BM25-only configuration that fails
to find the answer at all scored **higher** than the working pipeline. Before adopting CCH as a gate
the same attack was run against it. Six degradations, dev split, one command each:

| configuration | `precision@5` | `CCH_r` | `wrong` |
| --- | --- | --- | --- |
| **shipped** | 0.640 | **0.902** | 1 |
| `--config='{"denseLimit":0}'` | **0.641** ↑ | 0.886 ↓ | 2 |
| `--config='{"denseLimit":0,"titleLimit":0}'` — BM25 alone | **0.641** ↑ | 0.886 ↓ | 2 |
| `--config='{"denseLimit":0,"titleLimit":0,"lexicalLimit":5}'` | 0.573 | 0.772 | 1 |
| `--config='{"lexicalLimit":0,"titleLimit":0}'` — dense alone | 0.589 | 0.854 | 1 |
| `--budget=800` | 0.450 | 0.520 | 1 |
| `--budget=128` | 0.000 | 0.089 | 1 |

Two of the six rank **above** the shipped pipeline on `precision@5` and every one of the six ranks
below it on `CCH_r`. The mechanism is structural rather than lucky: `precision@5` rewards filling five
slots with chunks of one note, which duplication achieves and answering does not, while `CCH_r`
requires the answering sentence to be present and attributed — duplication adds neither.

The last two rows close a second, independent attack. `--budget=800` took `empty-recall` to 1.000 and
`false-confidence` to 0.000 in an earlier round, because every pack acquired a budget-truncation
notice; `CCH_r` is immune, because `knowledgeGaps` excludes response notices before it is judged and
because dropping claims turns `correct-cited` into `unsupported`. At `--budget=128` the negative-set
rates still read 0.917 / 0.000 while `CCH_r` reads **0.089**.

A third one had to be closed rather than measured. `withReadinessGaps` in `routes/api/recall.ts`
prepends a sentence when a channel could not run — "Vector search was unavailable for this query…" and
three others — and those are **not** `GapCandidate`s, so they carry no `GapReason` and only their text
can identify them. A locally run `--retriever=worker` has no `AI` and no `VECTORS` binding, so every
pack acquires one; counting it as a statement about the corpus would have scored every negative
question `honest-gap`. That is exactly the artefact this document already records for that run's
`false-confidence 0.000`, and it would have flattered `CCH_r` on the one configuration F10's
acceptance criterion 3 is measured against. `isResponseNotice` in `runner.mjs` now recognises all
four.

### 10.5 Reproducing it

From a clean checkout, with no credentials, no network and no manual steps:

```sh
pnpm eval                    # node eval/runner.mjs --gate --output=eval/results
```

That writes `eval/results/results.json` and `eval/results/report.md`, prints the summary line, and
exits non-zero if any gate check fails. It takes about 4 s and costs nothing: the reference retriever
has no network, no binding and no model. `pnpm eval:questions` validates the question set;
`pnpm eval:live` runs the same harness against a deployment through `POST /api/recall`
(`SIGNPOST_URL` and `SIGNPOST_TOKEN`, or `--base-url` and `--token`), which is the only mode that
exercises real Vectorize, real bge-m3 and the real cross-encoder.

To reproduce one row of the anti-gaming table, add its flag. A run carrying `--config`, `--budget`,
`--split`, `--only`, `--ids`, `--questions` or `--now` reports its number and **cannot pass the gate**;
`whole-eval-set`, `default-run-parameters` and `no-config-overrides` are three of the gate checks and
each one names what was changed.

### 10.6 What is still missing for F10 as specified

1. **The answer pass.** A fixed model fed only `recall` output and whatever `read` it chooses, with no
   access to `expected_note_ids` or `expected_claim`. The per-question rows CCH is judged from are now
   emitted in `results.json` under `cch.verdicts`, which is the input such a pass would consume.
2. **The judge pass**, separate, with the expectations. F10's acceptance criterion 2 — "the runner
   never sees expected answers at answer time" — makes it two processes, not one.
3. **Determinism and cost.** 174 questions × 2 model calls per run, on every retrieval change. Record
   and replay, the way `eval/hybrid/` already records dense and rerank calls, is the only shape of this
   that belongs in CI.
4. **`type` on every question**: `direct | paraphrase | multi-note | temporal | gap`. Questions carry
   `intent` and `difficulty` today; four of the five map.
5. **Temporal questions. There are zero**, and nothing could answer one until F8 ships. F10 asks for
   five in phase 1 for a feature phase 3 delivers.
6. **Publication as a note in the `engineering` space.** `eval/results/report.md` is regenerated by the
   runner and names the retriever, reranker, corpus size, date, split, every config override and every
   caveat — it is a file in the repository, and nothing publishes it.

### 10.7 The held-out split

Every decision in this section was taken on `--split=dev`. The whole-set number in 10.3 is the run CI
performs on every commit to `main`; it is the reportable figure and it was read once, after the dev
work was finished. The held-out split was **not** run: it has been informed twice already (rounds 6 and
8) and a third spend would buy a number no better than the dev one.

---

## 11. Hybrid recall — PRD_PART2 F1, measured

**Written 2026-08-17.** Every number below is `node eval/runner.mjs --config='…'`, 196 questions,
`--retriever=reference`, everything not named at its shipped default. The runner refuses `--gate`
alongside `--config`, so the rows with an override are measurements and not gate passes; the shipped
row is a gate pass and its exit code was 0.

### 11.1 What was added

Three things, and only the middle one moved a number.

| | Change | `precision@5` | Δ |
| --- | --- | --- | --- |
| baseline | before F1 (`diversityCapPerNote: Infinity`, `titleLimit: 0`) | 0.7726 | — |
| F1.2 | the title-path channel alone | 0.7746 | **+0.0020** |
| F1.4 | the diversity cap alone, at 2 | 0.6280 | **−0.1446** |
| shipped | both | 0.6273 | −0.1453 |

`precision@1` is **0.8981** and `hit@5` is **1.000** in every one of those four rows, to four
decimal places. `MRR` is 0.9370 in all four. Nothing the phase 2 gate enforces moved.

### 11.2 The title-path channel is free, and that is a designed property rather than luck

`+0.0020` on `precision@5`, nothing else measurably changed, and the reason is that the channel
**abstains on almost every question in the set**. `fuse.ts`'s `addAbsentContributions` skips a list
that came back empty — *"a retriever that came back empty has either failed or found no term to
match, and neither is evidence about a document some other retriever found"* — so on a query the
channel has no opinion about, the fusion is bit-for-bit the two-list fusion it was before.

That bounds the risk of a third list to the queries it does answer, which is what made it safe to
add without re-sweeping the five floors. It also means this eval **cannot** show what the channel is
worth: no question in `eval/questions/` names a note by its address, because the set was written
before the channel existed. The measured `+0.0020` is a floor and the honest statement is *"it costs
nothing"*, not *"it is worth 0.002"*.

One precision bug was found by its own test and fixed before it shipped: scoring only *"how much of
the query does this name cover"* let `"a of we"` match `handbook/how-we-work` at exactly the default
floor, on the strength of one function word. The channel now scores the **smaller of two coverages**,
the second being how much of the name the query accounts for — the same two-directional rule, for
the same reason, as `gapOpenQuestionOverlap`. `packages/core/src/retrieval/title-path.ts`'s
`scoreNote` carries the arithmetic.

### 11.3 The diversity cap costs 0.145, and the backfill did not buy back the oracle

`docs/RETRIEVAL.md` already carried three measurements of this knob, and PRD_PART2's own wording
leaves the important half unspecified: the spec line says *"at most 2 chunks per note before
backfill"* and the acceptance criterion says *"no more than 2 chunks from a single note appear
**before other notes are represented**"*. The second is a reordering rule and it is what shipped —
nothing is dropped, the multiset of candidates is unchanged, and only the order moves.

| `diversityCapPerNote` | `precision@5` | `@3` | block-level `@5` | `precision@1` | `hit@5` |
| --- | --- | --- | --- | --- | --- |
| ∞ (before F1) | 0.7726 | 0.8492 | 0.5616 | 0.8981 | 1.000 |
| 4 | 0.7669 | 0.8492 | 0.5644 | 0.8981 | 1.000 |
| 3 | 0.7275 | 0.8492 | 0.5452 | 0.8981 | 1.000 |
| **2**, shipped | **0.6280** | 0.7813 | 0.4548 | 0.8981 | 1.000 |

**The backfill did not reach the oracle's 0.7529, and the reason is now known rather than
suspected.** The oracle knows which chunk answers the question, so it fills a freed slot with a
*relevant* chunk of another note. A real reorder fills it with whatever the ranker scored next,
which on this corpus is usually a chunk of a note that is not relevant. No non-oracle rule closes
that gap: the score is already the best available estimate of relevance and it is already what
decides the backfill order. So the 0.145 is the cost of the criterion on this metric, not the cost
of this implementation, and a cleverer backfill cannot recover it.

The gap study of 2026-08-17 recommended gating this change at `precision@5 >= 0.72` and taking it
back to the operator otherwise. **It does not reach 0.72 at a cap of 2. It reaches it at a cap of
3.** What shipped is 2, because 2 is the number PRD_PART2's acceptance criterion states and shipping
3 while claiming the criterion is met would be the wrong kind of quiet. The consequences, stated
plainly:

- Every enforced gate passes. `hit@5` 1.000, `precision@1` 0.8981, `precision@5` 0.6280 against a
  floor of 0.60.
- **The floor now has 0.0280 of headroom rather than 0.1726.** A corpus edit that used to be
  invisible can now turn CI red.
- Moving the default to 3 buys back 0.0995 and gives up the literal acceptance criterion. Moving it
  to ∞ restores the previous behaviour exactly. Both are one number in
  `DEFAULT_RETRIEVAL_CONFIG`.

This is the collision the gap study's §8.2 predicted between PRD_PART2 F1 and PRD.md §8's phase 2
amendment, which retired `precision@5` as a gate *because* "the metric charges for the diversity the
product exists to provide". It is now charging for it. **The choice of default is an operator
decision and the table above is what it should be made on.**

### 11.4 The soft timeout

`RetrievalQuery.signal`, set by each door from `RetrievalConfig.softTimeoutMs` (1,500). The pipeline
reads no clock — that is what keeps this eval reproducible — so the deadline enters as an
`AbortSignal` the door creates and the pipeline only reacts to. With no signal every line behaves
exactly as it did before F1, which is what the eval and every pure test run with.

Five of the six awaited port calls are raced against it: the three channels, the corpus count, the
health read, the graph walk and the cross-encoder. **`candidates.load` is not**, and that is the one
decision in the feature worth arguing about. Fusion produces chunk *ids*, and an id is not a claim,
so abandoning that step turns every soft timeout into a pack with no claims at all — slower to
nothing than the pipeline was to something. It is the same exemption, for the same stated reason,
that `apps/worker/src/retrieval/ports.ts` already makes on the degradation path. A unit test caught
this by asserting the pack was non-empty when only the dense channel hung; the design changed, not
the test.

What the deadline still cuts is the query embedding (164 ms warm, ~1,000 ms cold), the graph walk
(562–602 ms) and the cross-encoder (640–774 ms), which is the larger half of the 1,499–4,216 ms
in-Worker total measured on 2026-08-17. It does **not** close D13 and it does not make F1's
acceptance criterion 5 true: it makes the failure survivable rather than fatal, by returning a
partial pack instead of a timeout.

---

## Appendix A: the reference implementations

`packages/core/src/retrieval/reference-retriever.ts` implements all seven ports in memory:

| Class | Stands in for | Faithful |
| --- | --- | --- |
| `ReferenceLexicalRetriever` | BM25 over D1's FTS5 | **yes** |
| `ReferenceDenseRetriever` | Vectorize | yes, given the vectors it is handed |
| `ReferenceEmbedder` | `@cf/baai/bge-m3` | **no — signed hashing over stemmed tokens** |
| `ReferenceReranker` | `@cf/baai/bge-reranker-base` | **no — rank-preserving, and not run at all** |
| `ReferenceGraphExpander` | the `relations` join in D1 | yes |
| `ReferenceCandidateStore` | the `chunks`/`notes`/`provenance` join | yes |
| `ReferenceIndexHealth` | the query `/health` serves | yes |

**They are test-only, and that is enforced structurally**, not promised in a comment. PRD 3.7 names two
code paths doing the same job with different behaviour as the bug class that plagues comparable products;
this exception is bounded three ways, each with a test in
`packages/core/test/retrieval/reference-retriever.test.ts`:

1. no file under `apps/` may so much as name the module;
2. nothing in `packages/core/src/` imports it, so it cannot reach production transitively;
3. it is not re-exported from either barrel or from the package's `exports`.

It lives in `src/` rather than `test/` for one reason: `eval/` is not a test directory and cannot import
from `test/`.

**What is faithful.** The BM25 is the real algorithm with FTS5's own parameters — `k1 = 1.2`, `b = 0.75`,
`idf = log((N - n + 0.5) / (n + 0.5))` clamped to `1e-6` when non-positive, three columns with `bm25()`'s
weighting, and `porter unicode61` tokenisation. Two divergences from SQLite are documented in the source:
diacritic removal is NFD-plus-strip-marks rather than SQLite's Latin-only table, and case folding is full
Unicode. Both fold *more* aggressively than SQLite, never less, so a term that matches in production
matches here.

**What is not.** `ReferenceEmbedder` is a signed hashing trick over stemmed tokens: it approximates the
cosine between two term-frequency vectors, which makes it a bag-of-words model that **cannot match a
synonym**. `ReferenceReranker` preserves the incoming order by default and emits scores on the RRF scale,
and the eval switches it off entirely with `rerankMinCorpus: Infinity`. Both choices are deliberate — a
cheap reordering would be a *worse* signal than the RRF order it replaced, so every eval number would
measure the stand-in — but they mean **any figure produced through these ports is a floor, not a
prediction**. Section 3 quantifies the first: fusing the hashing embedder into BM25 costs 0.055
`precision@5`.

---

## Appendix B: the tokeniser is quadratic in one unbroken run

A performance defect that is really a security one, written up here because the shape of it recurs: a
library's cost model differs from the one the calling code assumes, and the difference is only visible on
input a user controls.

js-tiktoken's byte-pair merge is **quadratic in the bytes of a single pre-token piece**. cl100k's
pre-tokenising regex normally cuts text into pieces of a few characters, but two of its alternatives do
not: `[^\r\n\p{L}\p{N}]?\p{L}+` matches a letter run of any length, and `[^\s\p{L}\p{N}]+` does the same
for punctuation and emoji. One word with no space in it is one piece.

```text
estimateTokens("1".repeat(50_000))  ->     15 ms   digits are capped at 3 per piece
estimateTokens("a".repeat( 4_000))  ->  1_213 ms
estimateTokens("a".repeat( 8_000))  ->  5_697 ms
estimateTokens("字".repeat(2_000))  ->  3_335 ms   CJK, no punctuation
```

The packer counts `JSON.stringify` of the pack, the pack echoes the query verbatim, and the packer
measures the shell several times. **A 16 KB query with no whitespace in it cost 57 seconds of Worker
CPU** — 70x NF1's `recall` p95 budget of 800 ms, from one request body, with no write access and nothing
to rate-limit against but the request itself.

**The eval harness could not see it.** `runner.mjs` disclaims latency as "not comparable to NF1", there is
no latency gate, and the one adversarial question that names this case used `"1".repeat(50_000)`, which
the tokeniser splits into three-character pieces and finishes in 15 ms. **The test named the case and did
not exercise it.** Changing that fixture from `"1"` to `"a"` used to hang the suite.

Three things fix it, each a different kind of bound.

1. **`boundedTokenCounter` (`retrieval/tokens.ts`)** makes counting linear. On text whose longest unbroken
   run is at or under `MAX_UNBROKEN_RUN_BYTES` it delegates unchanged, so the count is bit-identical for
   every realistic input; beyond that it counts in bounded pieces and sums, which can only over-count.
   Over-counting is the direction the budget needs.
2. **The limit is 512 UTF-8 bytes, and both halves of that were measured.** *Bytes* because that is the
   unit the quadratic is in — 512 CJK characters are 1,536 bytes and nine times the work, and a character
   limit left a 4,000-character Japanese question at 4.4 seconds. *512* because the constraint is
   exactness on JSON, which is nearly whitespace-free: the longest run in a serialised claim is 195 bytes
   at the shortest, 270 at the median and **456 at the longest**, against 58 in the prose those claims are
   made of. A limit of 256 would have taxed every request about 1% for ever to make a deliberate attack
   twice as cheap to absorb.
3. **The packer clamps the echoed query**, because linear cost still needs a bounded input.
   `MAX_ECHOED_QUERY_CHARS` is 16,384 — about 2,700 English words, comfortably above the two-thousand-word
   query PRD 8 names as adversarial — and `MAX_GUARDED_QUERY_CHARS` is 1,024 when the question contains an
   over-long run, because a character is not a unit of work.

Measured through the whole packer at the default budget afterwards: an ordinary question 5 ms, a
2,000-word question 38 ms, 16,000 unbroken letters 343 ms, 4,000 unpunctuated CJK characters 517 ms,
50,000 unbroken letters 354 ms. In the eval harness the slowest question, `adv-single-long-token`, went
from **11,732 ms to 1,066 ms**; it is now the slowest at 807.9 ms.

**The write path had the same hazard, and `chunk/tokens.ts` now closes it at the source.**
`chunk/chunk.ts` counts a whole block before `hardSplit` gets a chance to cut it, so a note whose body
was one long letter run could not be saved: a 267 KB pasted line never finished committing, and
`"a".repeat(20_000)` alone cost 31 seconds. `estimateTokens` now counts every pre-tokeniser piece at or
under `MAX_EXACT_PIECE_BYTES` (256) exactly and charges a longer one its UTF-8 byte length. That is an
upper bound, because byte-pair merging starts from one part per byte and only joins parts, so the budget
contract holds. It is linear in the input, so the same 300 KB line counts in about a millisecond. And on
text with no piece over the limit it is still one call to js-tiktoken on the whole text, so every real
count is unchanged. The recall-path guard above still stands in front of it.

---

## Appendix C: honest limitations

Everything in section 5 is a limitation and is not repeated here. These are the rest.

**Dense retrieval is not measured, and its stand-in is harmful.** Every number in this document was
produced with a hashing embedder standing in for `bge-m3`. It handles morphology (`payouts` and `payout`
are one term) and nothing else. Fusing it into BM25 costs 0.055 `precision@5`. Lexical-only figures
understate the deployed system on paraphrased questions and say nothing at all about it on multilingual
ones.

**Reranking is not measured at all**, in either direction. Stage 6 does not run. When
`bge-reranker-base` lands, expect the top-5 ordering to change substantially and expect two calibration
bugs: the gap floor stops firing for the opposite reason it does not fire today, and any fixed-scale
normalisation collapses. Both are measured in [6.1](#61-replace-the-multiplicative-composition-in-stage-7)
and [6.3](#63-replace-the-low_confidence-floor-with-a-quantity-that-carries-information).

**End-to-end conflict recall is 5 of 6, and the sixth is a retrieval miss, not a detection one.** The
detector finds 6 of 6 when both claim-bearing chunks are in front of it, with zero false positives across
114 query-scoped candidate sets. The one that does not surface end to end is `pan-storage` ("does Meridian
store card numbers?"), and the reason is pinned by a test: **both notes reach the pack, but only one of
the two incompatible sentences is in a chunk that ranked**. No conflict gate can fix that — it is a stage
2/3 problem about *which chunk* of the right note ranks, and it is the clearest single argument for a real
embedding model. A whole-corpus adversarial sweep over all 459 chunks at once finds 6 of 6 plus 2 unseeded
pairs, one of which is a genuine false positive between `conventions/testing` and
`architecture/three-d-secure`.

**Conflict detection does not see the graph.** "Neither side is superseded" is implemented as "neither
side's verification state is in `excludedStates`", plus an age-gap confidence bonus. `ConflictInput`
carries no relation edges, so a supersession recorded as a `supersedes` *edge* rather than as a *state* is
invisible to it. Adding `edges: readonly GraphEdge[]` to `ConflictInput` makes this exact.

**Conflict notes can be uninformative even when the pair is right.** The ledger conflict renders
`JPY (Feb 2025) vs FX (Mar 2026)`, because the only lexical link between the two notes is the currency
example each happens to lead with. The pair is correct; the values quoted are not the disagreement.

**One hand-written vocabulary table exists.** `conflict.ts`'s `CONCEPT_CLASSES` folds
`store/hold/held/keep/kept/retain/…` to one stem. It is general English with no payments word in it, and
two of the six seeded conflicts are invisible without it — but hand-written vocabulary is how a retriever
quietly becomes tuned to one corpus. Review it when the corpus changes.

**Intent classification is English-only and cue-based.** Five non-English queries and six contentless
queries all produce flat weights, which is the correct fallback but is not classification. A query phrased
in a way no cue covers gets no help at all — which is 60% of the scored set.

**`Claim.related` is capped by nothing.** A note with forty outbound links puts forty refs on every one of
its claims, and each ref is charged to `used_tokens`. The corpus's most-linked note has six, so it has not
bitten yet.

**Graph expansion is one hop and cannot explain itself.** A claim admitted at `graphHops: 1` does not
record *which* relation brought it in, only the resulting `graphPenalty` — the verb weight is folded into
that number. Adding `relationWeight` to `ScoreComponents` would make the decomposition complete.

**Dedup is O(n²) in the candidate count.** At 20 candidates that is 190 comparisons and immaterial; at 200
it is 19,900 and takes about 5 ms with exact Cauchy–Schwarz pruning. It would need rethinking if
`rerankLimit` ever grew by an order of magnitude.

**Scores are emitted at three significant figures**, which keeps a thoroughly decayed `3.02e-13`
distinguishable from zero but means a consumer cannot reconstruct the exact product of the components from
the wire format.

**NF1 (`recall` p95 < 800 ms) is measured on the deployment, per stage, and still fails.** On
2026-08-14, n=100 sequential calls against the live instance with `rerankCandidateLimit: 30` in place:
pipeline **p50 983 ms, p95 1,649 ms**; client, including network, **p50 1,180 ms, p95 1,867 ms**. 4 of
100 requests met 800 ms at the pipeline and 0 met it at the client. It is **not** dominated entirely
by the cross-encoder, which is what the 2026-08-13 REST measurement concluded: reranking is 46% of the
median, the dense leg 30%, and three sequential D1 round trips 21%. The floor with the cross-encoder
removed altogether is 580 ms at p50. The eval's own latency figures remain in-process arithmetic with
no network and no model call, and are not the NF1 number; see
[the deployed per-stage measurement](#nf1-measured-per-stage-on-the-deployment-2026-08-14--and-it-is-not-the-cross-encoder-alone).

---

## Appendix D: documents that disagree with the code

Recorded rather than fixed, because a document that disagrees with the code should report reality and name
the disagreement. Each was checked against a run on 2026-08-10. **None of these changes a conclusion; they
are here so the next person does not lose an afternoon deciding whether they broke something.**

| Document | Says | Measured |
| --- | --- | --- |
| `eval/README.md`, gate table | precision@5 "fails at 0.296" | **0.3159** |
| `eval/README.md`, "Where retrieval actually stands" | 0.312, `used_tokens` median 3,542, answer coverage 0.764, hit@5 0.611 | 0.3159, 3,284.5, 0.847, 0.6433 — that whole block predates the committed `results.json` |
| `eval/README.md`, lever table | `rrfK` 1 → 0.436; decay off → 0.566 | 0.4471; 0.5898 |
| `eval/README.md`, fix 1 | "widen the relevance signal, `rrfK` 60 → 1" | with stage 7 fixed, `rrfK` 1 scores 0.558 against `rrfK` 60's 0.590 — a workaround, not a fix. See [6.6](#66-rejected-lower-rrfk) |
| `eval/README.md`, traps | "Five of twenty-one traps are currently returned" | 6 of 14 fireable (0.4286); `results.json` reports over the fireable denominator |
| earlier drafts of this file, and several task briefs | block-level "0.222 against **0.313** on the same questions" | 0.2219 against **0.2932**, recomputed twice from `results.json`. 0.313 does not reproduce |
| earlier drafts of this file | hit@5 0.611 in prose | 0.6433, which the same draft's own table gave correctly |
| earlier drafts of this file | easy 0.227, hard 0.344 | 0.243, 0.349 |
| earlier drafts of this file | the inversion is because easy questions match many chunks weakly and the relevance base is flat | the base of the right answer is *higher* than the winner's on all three difficulties — [5.4](#54-the-difficulty-inversion-resolved) |
| earlier drafts of this file | freshness "spans about 15x over an 18-month corpus" | median 499.8x within one result list, max 1.332e19 |
| earlier drafts of this file, `rerankLimit` table | 20 → 0.312 / 3,542; 12 → 0.427 / 2,090; 8 → 0.536 / 1,379 | 0.3159 / 3,284.5; 0.4306 / 2,010.5; 0.5363 / 1,354.5 |
| earlier drafts of this file, negative set | "three negative-set questions come back with twenty confident claims and no gap" | **4 of 17 (0.2353)** by the runner's own rule; 3 if the budget-truncation notice is counted as a knowledge gap. Both readings exist, and the difference is a string comparison — see [section 4](#the-negative-set) |
| `types.ts`, `countTokens` doc comment | the guard is "exact on every input whose longest unbroken run is **64 characters** or fewer" | `tokens.ts` sets `MAX_UNBROKEN_RUN_BYTES = 512`, and it is **bytes**, not characters. The comment is stale in both units |
| `eval/README.md` vs `report.md` | "eight questions" pinned to `gaps.json` vs "10 questions pinned" | both right: `gaps.json` has 8 entries and `negative.json` has 10 questions carrying a `gapId` |

---

## Appendix E: where the code is

| Concern | File |
| --- | --- |
| The vocabulary, the config, the step contracts | `packages/core/src/retrieval/types.ts` |
| The seams to Cloudflare (seven port interfaces) | `packages/core/src/retrieval/ports.ts` |
| The pipeline itself | `packages/core/src/retrieval/pipeline.ts` |
| Stage 1 intent | `packages/core/src/retrieval/intent.ts` |
| Stage 4 fusion | `fuse.ts` |
| Stage 5 graph expansion | `expand.ts` |
| **Stage 7 score adjustment — the defect** | **`decay.ts`, `adjustOne`** |
| Stage 8 dedup | `dedup.ts` |
| Stage 9 packing | `pack.ts` |
| Conflicts | `conflict.ts` |
| **Gaps — the unreachable floor** | **`gaps.ts`, `lowConfidenceGaps` and `relevanceFloor`** |
| The token counter and its guard | `tokens.ts` |
| The eight pure stages, bound | `steps.ts` (`DEFAULT_STEPS`) |
| Test-only port implementations | `reference-retriever.ts` |
| End-to-end tests over the real corpus | `packages/core/test/retrieval/pipeline.test.ts` |
| **Where stages 2, 3 and 6 are not implemented** | **`apps/worker/src/retrieval/ports.ts` (`notIndexedPorts`)** |
| **Where they should be implemented** | **`apps/worker/src/retrieval/indexed.ts` — does not exist yet** |
| The eval harness | `eval/runner.mjs`, `eval/metrics.mjs`, `eval/report.mjs`, `eval/retriever.mjs` |
| **CCH, the one metric of PRD_PART2 F10** | **`eval/metrics.mjs`, `citedCorrectOrHonest` — see §10** |
| The eval corpus and question set | `eval/corpus/`, `eval/questions/` |
| The committed baseline run | `eval/results/results.json`, `eval/results/report.md` |