--- title: "Git sync and folder ingest" type: reference tags: [signpost, docs, reference] proposed_by: tok_bc09b340db294c709763789154d8113d model: kmem-docs-publisher extracted_at: "2026-10-08T22:28:55.980Z" --- # Git sync and folder ingest **Written 2026-08-15.** What kmem does with a connected repository, what a person must create before it works, and what it refuses to do on its own. Two features share this document because they share a model: a repository and a folder are both *somebody else's files*, and the rule for both is that kmem never silently replaces what somebody wrote. - **Git sync** is two-way and continuous. `docs/**/*.md` in the repository become notes; notes become files; `AGENTS.md` is a projection out of kmem and never a source. - **Folder ingest** is one-way and on demand. `npx kmem ingest ./docs` walks a directory and files each changed file as a proposal, or commits it with `--force`. --- ## 1. State on this instance **Git sync is built, proven against github.com, and unconfigured.** Every operation in §8's table has now been run against a real private repository — real trees, real blobs, real commit parents, real 409s, real rate-limit headers — and §10 is what that found. What is still missing is a GitHub App: creating one needs a signed-in github.com session, which no API and no token substitutes for, so `GET /api/git/status` reports `not-configured` and names the five environment values that are unset. Nothing degrades. An unconnected instance is the normal state and the permanent one for most deployments. **Folder ingest works, on the deployment, against a real folder.** `POST /api/ingest` is live behind the ordinary session cookie, `npx kmem ingest` drives it, and §10.4 has the numbers from running it over this repository's own `docs/`. **Git sync is per-deployment, not per-organization.** The five environment values in §7 configure one repository and one GitHub App for the whole deployment: the operator's. Per-organization git configuration is not built yet. On a hosted deployment, a tenant organization is therefore refused: `POST /api/git/sync` answers 503 with code `git-not-available`, a webhook delivery to a tenant hostname answers 503 before the signature is checked, and `GET /api/git/status` answers `connected: false` with no repository named. The reason is a disclosure, not a policy preference — a tenant sync served with the deployment-wide connection would pull the operator's repository into the tenant's project. `gitTenancyRefusal` in `apps/worker/src/services/git/config.ts` is the one gate, and `test/routes/api/git-tenant.test.ts` and `test/routes/hooks/git-tenant.test.ts` hold it. The deployment's own default project is unaffected. §7 is the operator's checklist for connecting a repository. §10 is the evidence and the six defects it produced; §7.4 is now down to one outstanding assertion rather than four. --- ## 2. The model ### 2.1 What syncs, and in which direction | Path | Direction | What happens | | ------------------------ | ----------------- | ---------------------------------------------------------------- | | `{path}/**/*.md` | both ways | Notes. Pulled on webhook delivery or `POST /api/git/sync`, pushed on commit. | | `AGENTS.md` | kmem → file | The managed region only. Everything outside the markers is never touched. | | `CLAUDE.md` | kmem → file | Reduced to the single line `@AGENTS.md`, and only when it is absent or already that line. | | `README.md` | neither | Written for humans browsing GitHub. kmem leaves it alone. | | `.signpost/config.json` | file → kmem | The repository's own sync settings. | | `.signpostignore` | file → kmem | What not to import, in `.gitignore` syntax. | `{path}` is `SIGNPOST_GIT_PATH`, or `path` in `.signpost/config.json`, or the repository root. A repository path maps to a permalink by stripping the prefix and the `.md`: `docs/backend/auth.md` ⇄ `backend/auth`. There is nothing else to learn, and `apps/worker/test/services/git/paths.test.ts` asserts the two directions are exact inverses — because a mapping that is not invertible produces a pull and a push that fight over the same file for ever. ### 2.2 Everything lands through `CommitService` A git pull is a writer like any other. Every note it lands goes through `CommitService.commit(ctx, change)` with `source: "git"`, which is what gives a pull the content-addressed blob, the reconciled block ids, the provenance rows, the `chunks_fts` sync, the audit row and the index job — invariant I1, and the reason "reindexed" needs no code in the sync at all. Nothing in `apps/worker/src/services/git/**` writes to R2 or D1. ### 2.3 Conflicts A pull is a three-way merge over (the text the last sync agreed on, kmem's text, the repository's text). - Edits in different places merge, and nobody is asked anything. - The same lines edited on both sides becomes a **proposal** carrying the whole incoming change. kmem's version stays live, the repository's version is on the proposal, and both are readable: the losing side is in `blobs/`, which nothing ever deletes. - No common ancestor — a first sync, or a note created independently on both sides — is treated as a whole-file conflict. Loud, and never lossy: with no ancestor there is no basis on which to prefer a side, and guessing is the failure this design exists to prevent. Accepting a proposal re-runs `commit` with a compare-and-swap, so accepting a stale one raises a conflict rather than quietly reverting the note. ### 2.4 Visibility survives the round trip A note's level lives in its frontmatter precisely so a clone carries it (ADR 0009). Two rules follow: - **A push writes the canonical bytes**, `visibility:` included. Not a re-render. - **A pull cannot widen access.** A repository file that would move a note from `private` to `open` becomes a proposal, and the note stays private until a person accepts it. Narrowing applies immediately: a mistake in that direction hides a note rather than exposing one, and it is visible to whoever made it. A change to the names on an `access:` list is not a level change, and a pull carries it ([ADR 0023](./ADR/0023-sharing-people-and-groups.md)). The sync reads every note, private ones included, under the `ingest` system read scope. That is deliberate: the repository is a mirror, and a mirror that omitted the private notes would be a backup nobody could restore from. What protects them is that the repository is private and the level travels with the file. ### 2.5 `AGENTS.md` The file is a projection, never a source. kmem owns the bytes between ``` ``` and nothing else in the file, ever. The projection is a slice-and-concatenate, so every byte outside the markers is identical afterwards — not "equivalent", identical. It is asserted against *this repository's own* `AGENTS.md`, which has prose on both sides of the region that somebody would be annoyed to lose. **A hand-edited region raises a proposal rather than being overwritten.** The proposal keeps the text somebody typed, verbatim, as a note at `_signpost/agents-md-hand-edit` — so accepting it loses nothing. It does **not** publish those lines into the region: the region is capped at 200 lines and every line in it needs human approval, and a bulk paste out of a file is not that. To publish any of it, add it to a note of type `convention` with `briefing: true`. Accepting and rejecting are both answers, and both authorise the next sync to write the file. That is what stops the question being asked every five minutes, which is the review-fatigue failure. Review fatigue is worse than capturing nothing. Telling "somebody edited the file" from "kmem's own conventions changed" needs one extra fact: the digest of the region kmem last wrote. `GitAgent` records it. Without it — a first sync, or an operator who reset the agent — a differing region is treated as a hand edit, which is the conservative direction and costs one proposal. --- ## 3. `GitAgent` One Durable Object per repository, addressed by the lowercased `owner/repo` slug. It is the only door to a sync, and that is what makes two syncs racing unrepresentable rather than unlikely — the platform serialises every request to one object id. It remembers three things in its own storage: the cursor (the commit the last successful pull agreed on), a base `content_sha` per note (the merge ancestor, read back out of `blobs/`), and which `AGENTS.md` hand edits have already been asked about. None of it is knowledge-base content; losing it costs one full sync and not one byte of anybody's writing. ``` POST (to the object) {"op":"pull"} pull, then project AGENTS.md {"op":"push"} push named notes, or every note {"op":"sync"} both, pull first — the only safe order {"op":"status"} state, without touching the repository ``` **The cursor advances only when nothing failed.** A pull that could not apply four notes must re-read those four next time, and moving the cursor past them would mean it never did. **There is no poll.** The only two things that start a sync are a webhook delivery and `POST /api/git/sync`; `index.ts`'s cron sweeps the index and nothing else. That is why a missed delivery is a repository kmem stays behind on until somebody pushes again, and why the stale-tip guard in `sync.ts` matters as much as it does — see §10.3 item 2. --- ## 4. Folder ingest ```bash npx kmem ingest ./docs # → review queue npx kmem ingest ./docs --force # → commit directly npx kmem ingest ./docs --dry-run # → report only npx kmem ingest ./docs --into handbook # → land under handbook/ ``` `SIGNPOST_URL` and `SIGNPOST_COOKIE` name the instance and the credential. Both can be flags; prefer the environment for the cookie, so it does not end up in a shell history. ### 4.1 Three requests, not one upload 1. The CLI walks the folder and hashes every candidate file. Nothing is uploaded. 2. It posts a **manifest** — paths and hashes, no content — and gets a plan back: which files kmem already has byte for byte, and which it wants. 3. It uploads only the wanted files, twenty-five at a time. That shape is what makes the three interesting properties true rather than hoped for. Idempotence is decided by the server from `content_sha`, so a second run of an unchanged folder uploads nothing. `--dry-run` is step 2 and stops — there is no code path from it to a write. And a thousand-file import is forty separate Worker invocations, each with its own CPU budget, resumable through `IngestAgent`. ### 4.2 The report ``` Ingest /Users/ankur/handbook Filing proposals for review. queued backend/auth.md -> backend/auth — waiting for review: … skipped backend/index.md -> backend/index — already in kmem, byte for byte. 312 created, 4 updated, 891 skipped, 0 queued, 2 failed ``` Every line carries a reason, because `891 skipped` is indistinguishable from `891 silently broken` without one. Failures print last and always, `--quiet` included: a run that failed four files and said nothing about them is a run that lost four files. ### 4.3 What ingest will not do - It never deletes. A file missing from the folder is a file missing from the folder. - It refuses two files that derive the same address, rather than letting the second win silently. - It refuses content whose hash does not match the manifest, because the plan for it is then stale. - Without `--force` it queues, on every instance, whatever `SIGNPOST_WRITE_POLICY` says. "Agents may write here" is not the same sentence as "replace four hundred of my notes with this folder". ### 4.4 A limit worth knowing before a large import **An instance cannot hold more than 64 notes that share any byte-identical block at the same heading path.** `blocks.id` is unique across the instance and is derived from `heading path + text + occurrence` with a 64-attempt collision escape (`packages/core/src/blocks/identity.ts`, `MAX_ID_DERIVATION_ATTEMPTS`). The 65th note carrying that block is refused with *"one of its blocks says exactly what 64 other notes already say"*. Frontmatter is the most common way to hit it, because it is parsed as a block like any other and a folder of files that all begin ```yaml --- type: reference --- ``` commits 64 and then refuses the 65th. Measured 2026-08-15 on a thousand-file import: 64 created, 936 failed. **It is not only frontmatter, and an earlier version of this section said it was.** A shared heading does it too. A 120-note import whose frontmatter was already unique — every file carried its own `title:` — still failed on the 65th, because every note began with the same `# Heading` line. A repeated `## See also`, or one boilerplate sentence copied into every runbook, reaches the same ceiling. So the advice is stronger than "give the files a `title:`": before a large import, make sure nothing is byte-identical across more than 64 files — the header, the first heading, and any boilerplate line. It is not folder ingest's to fix and ingest does not work around it, because a workaround would be ingest rewriting somebody's Markdown. The fix belongs in `packages/core/src/blocks/identity.ts`. --- ## 5. Webhooks `POST /hooks/git` takes a GitHub App delivery. It is the one route group with no kmem credential on it, so the HMAC is the entire security of the surface: - the signature is checked over the **raw body**, before anything parses it; - the comparison is constant time; - **a missing secret refuses.** It does not skip verification when unconfigured, which is the most common way this endpoint is backdoored; - `/hooks` is mounted with no per-group middleware, so nothing can consume or transform the bytes the signature was computed over. The handler verifies, decides whether the event matters, hands the work to `GitAgent`, and answers `202`. It does not sync inline: GitHub gives a webhook ten seconds, a pull of a large push is not reliably inside ten seconds, and a synchronous handler would paint an operator's GitHub dashboard red for syncs that in fact succeeded. Five kinds of delivery are answered `200` and acted on in no way: a `ping`, an event that is not `push`, a push to another branch or repository, **a branch deletion**, and a push kmem itself made. The last two matter most. A branch deletion reads as an empty tree, and with `deleteOnPull` on that is a knowledge base deleted by one `git push --delete`; a push kmem made would otherwise loop for ever. --- ## 6. `.signpost/` ``` .signpost/ config.json the repository's own sync settings ``` ```json { "path": "docs", "branch": "main", "pushMode": "direct", "createOnPull": true, "deleteOnPull": false, "authorName": "kmem", "authorEmail": "signpost@users.noreply.github.com" } ``` Every key is optional. Credentials are never in here — a credential in a repository is a leaked credential — and the environment wins for `path` and `branch`, so an operator can repoint an instance during an incident without a commit. **`deleteOnPull` is off by default and that is a data-safety decision.** A `git rm` of a directory, a botched rebase and a checkout of the wrong branch all look identical to a sync: many files vanished. With it off, the notes stay and the pull reports what it did not do. A `config.json` that will not parse is reported and ignored, and the defaults apply. One bad JSON commit must not silently stop every note syncing. --- ## 7. Connecting a repository Nothing below can be done by code. It needs somebody with admin on the repository. **§7.1 is the only step that genuinely cannot be scripted**, and it is worth knowing why before you try: GitHub has no REST endpoint that creates an App. The documented alternative, the app manifest flow, is an HTML form posted to `github.com/settings/apps/new` and needs a signed-in browser session. A personal access token — of any scope, classic or fine-grained — cannot do it. §7.2 onward is ordinary CLI work and has been run end to end on this deployment (§10). ### 7.1 Create a GitHub App , or the organisation equivalent. | Setting | Value | | ---------------------- | ----------------------------------------------------------- | | Name | Anything. It appears as the commit author. | | Homepage URL | Your instance's URL. | | Webhook URL | `https://your-instance/hooks/git` | | Webhook secret | Generate one: `openssl rand -hex 32`. Keep it. | | Repository permissions | **Contents: read and write** | | | **Pull requests: read and write** — only if `pushMode` is `branch` | | | **Metadata: read** (GitHub adds this automatically) | | Subscribe to events | **Push** | | Where can it be installed | Only on this account, unless you have a reason otherwise. | Then **Generate a private key** and keep the `.pem` file. It is shown once. Then **Install App** on the repository, and note the installation id — it is the last path segment of the URL you land on: `.../installations/12345678`. ### 7.2 Set the values ```sh npx wrangler secret put SIGNPOST_GITHUB_APP_ID # the numeric App ID npx wrangler secret put SIGNPOST_GITHUB_PRIVATE_KEY < app.pem npx wrangler secret put SIGNPOST_GITHUB_INSTALLATION_ID npx wrangler secret put SIGNPOST_GITHUB_WEBHOOK_SECRET ``` and in `wrangler.jsonc` under `vars`: ```jsonc "SIGNPOST_GIT_REPO": "acme/handbook", "SIGNPOST_GIT_BRANCH": "main", // optional, defaults to main "SIGNPOST_GIT_PATH": "docs", // optional, defaults to the repository root "SIGNPOST_GIT_PUSH_MODE": "direct" // optional: direct | branch ``` The private key is the one that catches people out. Paste the whole file, `BEGIN` and `END` lines included; `<` does that correctly. kmem accepts both the PKCS#1 form GitHub hands out (`BEGIN RSA PRIVATE KEY`) and PKCS#8, converting the first rather than telling you to run `openssl`. That path was exercised on the deployment on 2026-08-15 against a key from `openssl genrsa -traditional`, and it imports. `SIGNPOST_GIT_REPO` works as `npx wrangler secret put SIGNPOST_GIT_REPO` too, and that is the better choice while you are still trying it: a secret is one command to remove, where a `vars` entry needs an edit and a deploy. Move it into `wrangler.jsonc` once you have decided, because a repository name is not a secret and belongs where somebody reading the config can see it. **Nothing works until all five are set.** `resolveGitConnection` treats the set as all-or-nothing, so an instance with only `SIGNPOST_GITHUB_WEBHOOK_SECRET` refuses every delivery with 401 — which is correct, and is not the same failure as a wrong secret even though the status code is. ### 7.3 Check it ```sh curl -s https://your-instance/api/git/status # cookie-authenticated ``` `connected: false` names exactly which values are unset. `connected: true` with a `cursor` of `null` means it is configured and has never synced. ### 7.4 What is still unproven Three of the four items that used to be here were closed on 2026-08-15 by §10. What is left: - **That the app JWT is accepted by GitHub's own `/access_tokens`.** The signing is proven — a real RSA key, a real RS256 signature, and the PKCS#1 wrapper exercised against a key generated by `openssl genrsa -traditional` — and the *failure* path is proven against the real endpoint, which is how §10 found the token-exchange 404. What is not proven is a success, because that needs a GitHub App and creating one needs a person signed in to github.com. It is the last thing between the code and end-to-end evidence, and it is a five-minute job for whoever has that session (§7.1). - **That a repository large enough to truncate `/git/trees?recursive=1` behaves as §2.1 says.** The refusal is written and unit-tested; GitHub truncates somewhere above 100,000 entries and the probe repository is nowhere near it. Low priority: `SIGNPOST_GIT_PATH` exists for exactly this. Closed by §10: the permission set (a token with `repo` reached all eight operations and nothing more), the webhook signature (a real delivery verified on the deployment), and the whole of the two-way sync. --- ## 8. Where the code is | Path | What | | ----------------------------------------------- | ------------------------------------------------- | | `apps/worker/src/services/git/paths.ts` | The mapping, and `.gitignore` semantics | | `apps/worker/src/services/git/merge.ts` | Three-way merge, pure | | `apps/worker/src/services/git/repository.ts` | The port: eight operations, and why there is one | | `apps/worker/src/services/git/github.ts` | GitHub over REST, the App JWT, the PKCS#1 wrapper | | `apps/worker/src/services/git/webhook.ts` | Signature verification and event reading | | `apps/worker/src/services/git/sync.ts` | Pull and push, through `CommitService` | | `apps/worker/src/services/git/projection.ts` | The `AGENTS.md` decision | | `apps/worker/src/services/git/config.ts` | What an operator must set, and what is missing | | `apps/worker/src/services/git/ports.ts` | The one seam a suite replaces | | `apps/worker/src/durable/git-agent.ts` | `GitAgent` | | `apps/worker/src/durable/ingest-agent.ts` | `IngestAgent` | | `apps/worker/src/services/ingest.ts` | The manifest, the plan, the batch | | `apps/worker/src/routes/hooks/git.ts` | `POST /hooks/git` | | `apps/worker/src/routes/api/ingest.ts` | The CLI's door | | `packages/cli/src/**` | `npx kmem ingest` | `apps/worker/test/support/git-host.ts` is the in-memory host, and its header says why a fake of GitHub is not the mocked binding this project forbids. ## 9. What the reconciliation pass proved, and what it did not — 2026-08-15 `GET /api/git/status` on an unconnected instance returns the whole of what such an instance should say — the cause, then the fix, and no pretence that anything is syncing: ```json {"connected": false, "missing": ["SIGNPOST_GITHUB_APP_ID", "SIGNPOST_GITHUB_PRIVATE_KEY", "SIGNPOST_GITHUB_INSTALLATION_ID", "SIGNPOST_GITHUB_WEBHOOK_SECRET", "SIGNPOST_GIT_REPO"], "because": "Git sync is not connected: … are unset.", "fix": "Create a GitHub App, install it on …"} ``` `GitAgent` and `IngestAgent` are bound and deployed, and the read-scope audit confirmed that neither adds a leak: 65 checks across every surface, 0 findings, with `services/git/sync.ts`, `durable/git-agent.ts` and `services/ingest.ts` each carrying a named, dated entry in `read-scope-allowlist.ts` rather than a silent bypass. That pass ended by saying the honest status was **code, no evidence**, because the eight checks passed against a fake host and a fake host is no test of the assumption that GitHub behaves as the port says. §10 is that sentence being acted on. --- ## 10. Run against github.com — 2026-08-15 **Repository:** `rush-skills/signpost-git-probe`, private, created empty for this and safe to delete. **Harness:** `apps/worker/test/services/git/github-live.test.ts`, running inside workerd against real D1, real R2, real `CommitService` and real `api.github.com`. It skips itself unless two environment values are exported, so `pnpm test` still needs no account, no credential and no network: ```sh SIGNPOST_LIVE_GITHUB_TOKEN=$(gh auth token) \ SIGNPOST_LIVE_GIT_REPO=owner/throwaway-repo \ npx vitest run --project worker apps/worker/test/services/git/github-live.test.ts ``` **13 tests, all passing, ~3 minutes**, twice in a row after the fixes in §10.3 — which matters, because before them three of the thirteen failed intermittently and the intermittency *was* the finding. Each run works on its own `gate/` branch, so the evidence accumulates rather than being overwritten, and the branches on the probe repository are the audit trail. The default suite is unchanged by this: `pnpm test` is **248 files, 10,454 tests, 1 file and 13 tests skipped** — the skipped ones are exactly this suite, with no credential exported. ### 10.1 The one substitution, stated plainly Creating a GitHub App needs a person signed in to github.com. There is no API for it — the manifest flow is a browser form and the REST API has no equivalent — so **no App exists for this instance**, and one request in the harness is answered locally: `POST /app/installations/{id}/access_tokens` returns a token the operator already has. Everything else goes to `api.github.com` untouched, and the app JWT is still signed with a real RSA key on every request. What that buys and what it does not: | | Proven against github.com | Not proven | | --- | --- | --- | | `headOf`, `listFiles`, `readFile`, `changedBetween`, `commit`, `propose`, `check`, the ref compare-and-swap | yes | | | RS256 signing, PKCS#1 → PKCS#8 wrapping | yes | | | GitHub *accepting* the app JWT | | needs an App (§7.4) | **No gate below is marked passing on a fake host.** Where the substitution matters, it is named. ### 10.2 The eight gates | Gate | Result | | ---- | ------ | | A commit in kmem appears in the repository | **Pass.** Pushed, then read back over the `/contents` API rather than through the code that wrote it: byte-identical, one parent, author `kmem `. | | A commit in the repository appears in kmem, reindexed | **Pass.** Committed by hand in git, pulled with a cursor set so the real `/compare` endpoint decided the range; the note exists and has chunks, because the pull went through `CommitService`. | | Conflicting edits produce a conflict record and never silent data loss | **Pass.** Same line edited on both sides; one proposal, the repository's text on it whole, kmem's text still live in `blobs/`. | | A hand-edited `AGENTS.md` managed region proposes rather than overwrites | **Pass.** The plan is `propose` and the bytes on github.com afterwards are the ones the person typed, unchanged. | | The projection writes only inside the markers | **Pass.** Through a real blob and a real commit, against a file with hand-aligned table columns, a fenced block containing the marker syntax, and a trailing space — every byte outside the markers identical. | | Ingest is idempotent | **Pass.** Second `--force` over the same folder: 0 created, 0 updated, 41 skipped *"already in kmem, byte for byte"*, in 3.4 s against 118.9 s for the first. Nothing was uploaded, because the manifest decided it. | | `--force` commits, plain ingest queues, `--dry-run` writes nothing | **Pass**, all three on the deployment. See §10.4. | | A large ingest completes without exceeding Worker limits | **Pass at the size measured, and the general claim is false.** 120 notes pushed in 246 requests and 98.4 s. `push` has no bound and no resumption, so it stops working at about 497 notes. | **Visibility survives the round trip.** A `sealed` note was pushed and read back off github.com — not through the code that wrote it, through the `/contents` API, decoded the way a clone would: ``` $ gh api repos/…/contents/docs/secrets/postmortem.md?ref=… --jq .content | base64 -d --- type: reference visibility: sealed --- # Heading The incident postmortem nobody outside the team reads. ``` Pulled back unchanged, it is still sealed. Then the file was edited in git to `visibility: open` and pulled: the note **stayed sealed** and the change became a proposal saying *"…which lets more people read it. A pull never widens access on its own."* That is the assertion the permission model rests on, and it now holds against a real clone rather than against a `Map`. This measurement predates [ADR 0026](./ADR/0026-two-visibility-levels.md). `sealed` now reads as `open`, so the same file would widen nothing today. The rule is the same for `private`, and the sync suite in `apps/worker/test/services/` asserts it. ### 10.3 What GitHub does that the fake host did not Six things, and each produced a defect. 1. **An empty repository answers `409`, not `404`** — `{"message":"Git Repository is empty."}` on every git-data read. This is the first thing a new operator meets and the error told them to wait for a retry that can never help (fixed). 2. **A ref read from a Worker is eventually consistent.** 9 of 27 reads issued immediately after a write GitHub had already acknowledged returned the previous commit; 0 of 12 from Node on the same machine. `cache: "no-store"` and `cf.cacheTtl: 0` change nothing, so it is replica lag rather than the Workers cache. Every stale read settled on the next one, inside ~700 ms. A stale read turned a webhook-driven pull into a silent no-op that never revisited the commit (fixed). 3. **A repository webhook is delivered in about a second, and the signature is exactly as documented.** A real `ping` answered 200 in 0.79 s; a real `push` answered 202 in 1.16 s with the range on it. Forged deliveries were refused on the deployment: no signature 401, wrong signature 401, correct signature over a body naming a branch deletion answered 200 and *"main was deleted, and kmem never acts on that."* 4. **A 404 on the token exchange is not about the repository.** GitHub says *"Integration not found"*, and the message sent the operator to check the two settings that are not in that request (fixed). 5. **Rate limit is 5,000 requests per hour** on the `core` resource, reported on every response in `x-ratelimit-limit` and `x-ratelimit-remaining`. It is not what constrains a sync — a push costs 2.05 requests per note, so a Worker runs out of its 1,000 subrequests long before an instance runs out of hourly quota. Two syncs of a 500-note corpus in an hour would reach both at once. Worth knowing rather than worth designing around today. 6. **Force-push handling is not the problem it looks like.** `blobs/` is content-addressed and nothing deletes from it, so a merge base survives a rebase or a force-push regardless of what GitHub still has. The one that bites is the ordinary case in (2). There is a seventh, found by connecting rather than by GitHub: **a failed sync leaves no trace on `/api/git/status`**. `lastRunAt` and `lastOutcome` stay `null`, byte for byte what an instance that has never synced reports (unresolved; the fix is three lines). ### 10.4 Folder ingest, against this repository's own `docs/` 41 files, run against the deployment with `--into _gate6` so the probe corpus could be removed afterwards. It was: the instance is back to 143 notes and 1,927 chunks. | Run | Result | Time | | --- | ------ | ---- | | `--dry-run` | 0 created, 0 updated, 0 skipped, 41 queued, 3 failed. Note count unchanged, 143 → 143. *"Nothing was written."* | 3.8 s | | `--force` | 41 created | 118.9 s | | `--force` again | 0 created, 0 updated, **41 skipped**, 0 queued | 3.4 s | | plain, folder unchanged | 41 skipped, 0 queued — there is nothing to review | 3.5 s | | plain, one file edited | 40 skipped, **1 queued** as `prop_9e8ce…`, origin `ingest`, and the note was **not** updated | 3.6 s | The 3 failures are the three `README.md` files, refused by design with the reason on each line (§2.1). The CLI exits non-zero when anything failed, which is right: a run that lost three files and exited 0 is a run nobody checks. ### 10.5 What a person still has to do One thing, and it is §7.1: **create a GitHub App and install it on the repository.** It cannot be scripted — GitHub has no API for creating an App, and the manifest flow is a browser form that needs a signed-in session. Everything after it is `wrangler secret put` (§7.2) and `curl /api/git/status` (§7.3), both of which have now been run end to end on the deployment. Until then `/api/git/status` says `not-configured` and names the five values, which is the truth. --- ## 11. Browser upload: the durable import job **Written 2026-09-11.** A third way somebody else's files arrive: a person drags a zip into the Import screen (called Upload until 2026-09-17). The rule of this document holds here too — kmem never silently replaces what somebody wrote — and the outcomes are the ones §2.1 names for ingest. What changed on 2026-09-11 is *who waits*. `POST /api/import` used to unzip the archive, classify every file and commit every one of them before it answered. The page had to stay open. A large archive sat on a request for minutes, and closing the tab lost the run with no id to resume from. ### 11.1 The three steps 1. **The request validates and stages.** It decodes the archive, reads it, and maps every entry to a note address. That is the same work it always did, so a broken archive is still refused in the request, with the cause and the fix. It then writes one row in `import_jobs`, stores the bytes, and answers `202 { "job_id", "status_url" }`. 2. **The queue imports.** A message on `signpost-ingest` names the job. The consumer imports at most 20 files per invocation, one after another, through the same `applyImportFile` the synchronous path uses. It updates the row after every file. It enqueues itself again until every file has an outcome. 3. **The screen polls.** `GET /api/import/jobs/{id}` reports the state, the counts and one row per file. A refresh resumes from the URL, because the URL is the whole of the client's state. ### 11.2 Where the archive waits In the `IngestAgent` Durable Object's own storage, keyed by the job id, in 64 KiB chunks. The job deletes them when it ends. An alarm deletes them 24 hours later if it never does. Not in R2. The bucket is the ground truth: "if D1, Vectorize and the app all vanish, the R2 bucket plus a git clone still contain everything, readable in any editor". An opaque zip there would be the first thing in that bucket that is not readable Markdown. It would also need the first `r2-put` and `r2-delete` exceptions invariant I1 has ever granted. The Durable Object is neither R2 nor D1. It already holds a sync's own bookkeeping for the same reason (§3), so the invariant's scan has nothing to excuse. ### 11.3 How a job cannot stall or import a file twice - **The cursor is on the row, never on the message.** A queue delivers at least once. A message that carried its own cursor would re-run the files the first delivery finished. - **A consumer claims a slice with a lease.** The claim is one `UPDATE … WHERE` nothing else holds it, so two deliveries that arrive together cannot import the same twenty files. - **A job always ends, one of three ways.** Every file has an outcome, so the row is `done`. Or a step no retry can fix failed, so the row is `failed` with a cause and a fix. The archive is gone, or it no longer maps to what it mapped to. Or a step a retry might fix failed three times, so the row is `failed` with the last cause. No job stays `running` while its message dead-letters out of sight. - **One file failing is one row.** `applyImportFile` never throws. A file the commit service refuses is `failed` in the report, and the other 299 still land. - **A read repairs a job nothing is driving.** Two gaps sit outside the consumer. A request can die between writing the row and sending the message. A message can spend its attempts. Both leave a live job with no message naming it. So `GET /api/import/jobs/{id}` puts a job back on the queue when it has not changed for longer than a lease and nothing holds one. The screen polls, so the repair happens while somebody is looking at it. The claim is a compare-and-swap, so a message sent twice still imports each file once. ### 11.4 What the docs the job writes still obey Every one of them goes through `CommitService.commit` under the person who ran the import, with `source: ingest`, exactly as the synchronous path always did. Invariant I1 holds because the job writes no note itself; invariant I4 holds because those commits enqueue their index jobs like any other. The job row is a record of the work, not the work. ### 11.5 The compatibility path `POST /api/import?mode=sync` runs the whole import inside the request. It answers the old `200 { report }`. It exists for the clients that have not moved: the live e2e harness and this repository's own route tests. It is not a second copy of the importer. Both modes plan the archive with `planImport` and apply each file with `applyImportFile`. It goes when the last caller does. ### 11.6 Who may read a job The person who ran the import, and an owner of the organization. A job of another organization answers `404`, not `403`: the difference between "not yours" and "does not exist" is itself a disclosure. The rule is narrower than "anybody signed in", for the reason invariant I7 gives for notes. A job's rows quote the paths in somebody's archive, and the addresses those files became.