--- title: "ADR 0025 — Meter the cost of each organization at the binding, and roll it up each night" type: decision tags: [signpost, adr, decision] proposed_by: tok_bc09b340db294c709763789154d8113d model: kmem-docs-publisher extracted_at: "2026-10-08T22:42:14.602Z" --- # ADR 0025 — Meter the cost of each organization at the binding, and roll it up each night **Status** Accepted · **Date** 2026-09-16 · **Supersedes** nothing **Relates to** ADR 0010 (a compensating control needs a way to see that it stopped), ADR 0013 (one shared D1 for every tenant), `docs/PRICING_ANALYSIS.md` (the unit costs and the tier recommendation), `packages/core/src/metering/`, `apps/worker/src/metering/`, `packages/db/migrations/0026_cost_metering.sql`. ## Context The owner asked what one organization costs to run, and what kmem must charge for it. Before this change, kmem recorded per-org cost for Workers AI only, and only on some doors: - `spend_ledger` recorded Workers AI calls and Vectorize queries on the Ask and recall doors. - MCP `recall` spent inference and wrote no ledger row (finding F2). - Nothing attributed D1 rows, R2 operations, Worker requests, CPU time, Durable Object requests, Queue operations, email or log events to an organization. - Nothing recorded how large an organization was over time. - `services/spend/vectorize-estimate.ts` priced a stored vector at about one twentieth of the published rate, and it billed upserts, which Cloudflare does not bill (finding F6). So nobody could say whether a price covered its cost, or which organization caused a bill. ## Decision ### 1. One meter per invocation, found through `AsyncLocalStorage` Each entry point opens a meter and runs its handler inside it: - `fetch`, `queue` and `scheduled` in `apps/worker/src/index.ts`; - every event of the three Durable Object classes (`NoteDoc`, `IngestAgent`, `GitAgent`). `apps/worker/src/metering/entry.ts` is the only place that does this. `resolveProjectContext` and `deriveProjectContext` tell the meter its organization. Some work serves more than one organization in one invocation: a queue batch, or the nightly sweep. That work opens one scope per organization with `runMetered`. A request that never resolves a project counts under `platform`. A binding call with no meter in scope counts under `unattributed`, and the rollup warns when that share is high. No call site passes a meter, so no call site can forget one. ### 2. The environment is wrapped once, at the entry point `apps/worker/src/metering/bindings.ts` wraps each billed binding in a proxy that counts Cloudflare's billed units: | Binding | Counted | | --- | --- | | D1 | statements, `meta.rows_read`, `meta.rows_written`, and statements with no `meta` | | R2 | Class A and Class B operations | | Vectorize | queries and queried dimensions (upserts are free) | | Workers AI | calls, neurons as reported, or neurons priced from token counts | | Queues | operations, in Cloudflare's 64 KB units | | Durable Objects | requests, and SQLite rows written | | Email | messages sent | `METERED_BINDINGS` names every binding in `wrangler.jsonc` and the wrapper that meters it. `apps/worker/test/invariants/metered-bindings.test.ts` is a source scan. It refuses a binding with no row, and it refuses an entry point that reaches the raw `env`. A Worker cannot read its own CPU time. The meter records **wall time** per invocation as a proxy, and names it `wall_ms`. Wall time is an upper bound on CPU time. A Tail Worker could read the real CPU time, and it is not in this change (see "Rejected"). ### 3. Points go to Workers Analytics Engine At the end of each invocation, the meter writes one data point per organization to the dataset `kmem_cost_events` (binding `COST_EVENTS`): - the index is the organization id; - the twenty doubles are the units in `COST_UNITS` order; - the blobs are the schema version, the invocation kind, the matched route, the status class and the deployed version. The meter never writes more than 250 points per invocation. Above that, it folds the rest into one `platform` point and logs `cost_points_capped`. `SIGNPOST_COST_METER` is `on` in production. `off` removes every wrapper at once. `echo` also prints each point, for the local load test. ### 4. A nightly rollup writes three D1 tables `apps/worker/src/metering/rollup.ts` runs as the second step of the 03:00 UTC cron. It writes: - `org_cost_daily`: the day's units per organization, from the Analytics Engine SQL API; - `org_size_daily`: what each organization holds (docs, blocks, chunks, vectors, D1 rows and bytes, R2 objects and bytes) and what D1 recorded that day; - `meter_rollup_runs`: one row per day per run, with the status and the cause of a skip. The SQL API needs the secrets `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_ANALYTICS_TOKEN`. Without them, the rollup still writes the size facts. It logs `cost_rollup_skipped` and records the skip on the run row. The owner creates the token later. A second run for the same day replaces the first run's rows. It never adds to them. `apps/worker/test/metering/rollup.test.ts` runs the rollup twice and compares. The tables are control-plane tables with no `project_id`, like `spend_ledger`. `apps/worker/src/metering/store.ts` is their only writer, and `apps/worker/test/invariants/write-path-allowlist.ts` names it with the reason: a cost row is not knowledge-base content, so it does not go through `CommitService`. ### 5. One dated price table `packages/core/src/metering/prices.ts` holds every price that Cloudflare can bill kmem. Each row has its source URL and the date somebody read it (2026-09-16). The report gives two views: - **list**: each unit at the overage rate, with no included amount taken off; - **pro rata**: the account's real overage and base fee for a month, split across organizations by their share of each product (`allocate.ts`, largest-remainder rounding). `vectorize-estimate.ts` now uses Cloudflare's published formula, and this change fixes its callers' arithmetic with it. ### 6. The owner reads it in two places - `GET /api/admin/costs` (JSON, or CSV with `?format=csv`), for the instance operator only, behind the same fences as the rest of `/api/admin`. Each read writes an `admin_org_cost_report` audit row. The response says whether the meter is stale. - `scripts/cost/report.mjs`, which reads the same tables through `wrangler d1 execute` and builds the same rows with the same function. `scripts/cost/reconcile.mjs` compares one day of the meter with Cloudflare's GraphQL totals for the account. It prints the difference and writes nothing. ### 7. The load test uses the product's own doors and its own meter `scripts/loadtest/run.mjs` seeds a synthetic organization and replays a mix of operations through the Quick Connect doors. A local run costs nothing: it leaves Workers AI and Vectorize unbound, reads every other unit off the Worker's `echo` output, and computes inference units from token counts. A remote run refuses to start unless its target is not production and `KMEM_LOADTEST_APPROVED_USD` is at least the printed estimate. ### 8. Two small fixes that the numbers asked for - MCP `recall` now writes its `spend_ledger` rows (finding F2). - Ask sends at most about 4,000 tokens of history to the model (`MAX_ASK_HISTORY_TOKENS`). Before this cap, a client that sent eight turns of 8,000 characters made one answer cost about four times a typical one. The per-request `token_used` audit row stays, because it is an accountability record. The meter counts its cost. ## Rejected - **A D1 row per request.** It would double the D1 writes of every request, and D1 rows written are already the largest cost in normal use. It would also put a write on the read path. - **Cloudflare's GraphQL Analytics API only.** It gives account totals and totals per Worker, but not per organization, because every tenant runs in one Worker. It is the reconciliation, not the meter. - **A Tail Worker for CPU time, now.** It is a second deployed Worker with its own cost and its own failure modes. Wall time is enough to find the expensive paths. Add the Tail Worker when a price depends on CPU time. - **A wrapper at each call site.** There are hundreds of call sites. A wrapper at the entry point covers all of them, and the scan refuses the one that escapes. ## Consequences - Every request pays for some proxy calls and one Analytics Engine write. Cloudflare does not bill Analytics Engine writes today, and the plan will include 10 million points a month when it does. - The `CLOUDFLARE_ANALYTICS_TOKEN` secret can read the analytics of the whole account, including other Workers on it. It cannot write. This is a residual risk: keep the token scoped to "Account Analytics: Read", and rotate it with the other secrets. - Wall time over-states CPU time, most on requests that wait for I/O. A price that depends on CPU time needs the Tail Worker first. - The load test computes its inference units. It does not measure them. The first approved remote run replaces them. ## How to see that it stopped - `meter_rollup_runs` has a row for each night. The admin report sets `meter.stale` when the newest run is more than 36 hours old, and the rollup logs `cost_rollup_stale`. - `cost_rollup_skipped` says each night that the token is missing. - `cost_rollup_unattributed_high` says that too much work belongs to no organization. - `scripts/cost/reconcile.mjs` shows how much of the account's usage the meter did not see. - `apps/worker/test/metering/accountability.test.ts` holds every metering log line at `warn`, because a line below `warn` does not reach production.