Topic resolution & incorporation — Design

Problem

The collaborative-annotations initiative needs a way for a Topic to reach its terminal state. Sub-projects #1 and #2 fixed the data shape — proposals are an append-only log, Topics carry commit_sha / discarded_at outcomes, anchors are data-orcha-anchor markers stamped into Source — and #3 shipped the subprocess runtime that can invoke the Agent. Three holes remain:

1. The wb-incorporate skill exists only as a scaffold with a <REWRITE CONTRACT OWNED BY #4> placeholder. No actual prompt body tells the Agent what to do.
2. There is no HTTP surface to start a proposal job, fetch its diff, approve it, or discard a Topic. The only Topic-mutating endpoints today are message append and Topic creation.
3. The diff review gate from #1's "commit-immediately with UI review" decision has no implementation — Tier 1 (unified) and Tier 2 (rendered side-by-side) are unbuilt.

Without these, a Topic is a one-way thread that can never close.

Goals & non-goals

Goals

• Fill in the wb-incorporate rewrite contract: how the Agent produces a proposed Source given a Topic + thread + every other open Topic on the same Source.
• Ship the seven HTTP endpoints needed for the Resolve flow, all behind collaborator auth from #7.
• Ship Tier 1 (unified) and Tier 2 (side-by-side rendered) diff views as decided in #1.
• Add a post-job invariant check that catches silent anchor drops without imposing arbitrary caps on prompt context.
• Have the Agent attach a short natural-language explanation to each proposal so humans can verify it understood the discussion before reading the diff.
• Simplify wb-agent's surface to match its single consumer (the skill): no redundant or defensive flags.
• Reshape the prompt the Agent receives so it reads as if a human asked the Agent to help, not as if a system is feeding it parameters. The bootstrap minimum is three values (Job ID, Config path, wb-agent path); everything else the Agent discovers on its own.

Non-goals

• Perspective refresh on Incorporation. Owned by #5. #4 lands the Source commit and stops there.
• Resolve UI chrome. Button styling, modal layouts, the affordance for adding rework feedback, the diff toggle widget — all #8. #4 ships functional endpoints + minimal client wiring so #8 has something to skin.
• Push notifications. v1 polls agent_jobs and the proposals list. SSE is #8.
• Per-Document personas / Perspective switcher in the Resolve view. #5 + #8.
• New topic_messages.kind values. #4 introduces no new kinds. Rework feedback rides on existing 'human' messages (see the Decision on "no rework concept in the data model" below); discard reasons likewise (see the Decision on "discard reason rides on kind='human'" below). No annotation columns on proposals either.

Approach

Three changes, layered on the existing #1–#3 substrate:

1. Slim and shape the wire contract for one consumer. wb-agent's only caller is the skill. Each subcommand keys on whatever it naturally operates on: get-topic and insert-proposal on --job-id; list-open-topics on --source-path + --exclude-topic. Every subcommand still takes --config=<path>. The prompt the skill receives drops everything derivable: only Job ID, Config path, and wb-agent path remain, framed as a human asking the Agent for help (not as a harness shipping parameters). Proposal rows keep the producing job ID for audit and precise UI joins.
2. Fill in the rewrite contract. Replace the <REWRITE CONTRACT OWNED BY #4> marker in wb-incorporate/SKILL.md with a body that (a) applies the Topic's discussion to the Source, (b) drops the incorporated Topic's marker, (c) places at least one data-orcha-anchor marker for every other open non-global Topic on the Source — naturally where each maps, otherwise in an "Other ideas (potentially to discard)" parking-lot section at the bottom of the Source, (d) writes a 1–3 paragraph explanation of what it understood the Topic to be asking for and how its rewrite addresses it, persisted as the body of the agent-proposal message that already accompanies every proposal row.
3. Add the Resolve flow. Seven HTTP endpoints exposing the operations the Resolve UI needs: enqueue a proposal job (idempotent per Topic), list proposals for a Topic, render the unified diff, serve the proposed Source through the existing render pipeline (Tier 2), approve a proposal (calls existing collab.Incorporate), and discard a Topic. All behind collaborator auth from #7.

Design

User-visible state machine

The Resolve UI surfaces seven derived states on an open Topic, all read from DB rows plus current Source/proposal bytes. No new state column.

"Latest" everywhere in the table means the row with the highest revision_number for proposals, and the row with the highest started_at (then created_at as tiebreaker) for agent_jobs filtered to this Topic. A linked job gates only whether an Agent proposal is approvable at all; it is not used for freshness. Agent proposals whose linked job is failed/timed_out remain in history but are not approvable.

The proposal-fresh / proposal-stale distinction is detected at read time, not stored. The server computes fresh = (agent_job_id IS NULL OR linked job status == 'succeeded') and (SourceSHA(repo_root, source_path) == proposal.base_source_sha) and (all current open non-global Topic IDs on this Source, excluding the incorporated Topic, occur as data-orcha-anchor markers in proposed_source) when serving the proposals list. The same checks happen server-side at approval time before collab.Incorporate writes the file; the UI's display is purely advisory. The list response includes stale_reasons (["source_sha"], ["missing_topic_markers"], both, or empty) plus missing_topic_ids so the UI's banner can name the cause. Proposal rows from failed jobs stay in the DB for audit history but never surface as actionable, and Approve is gated server-side.

HTTP surface

Seven endpoints, all behind auth.RequireCollaborator (the helper from #7). The API endpoints return application/json with the same error envelope as #2 ({ "error": "<code>" }); the preview route returns rendered HTML.

Approve, propose, and discard attribute to the request principal via users.id (the normalised email from #7). The bootstrap-operator path is already gone.

wb-incorporate rewrite contract

The skill replaces the <REWRITE CONTRACT OWNED BY #4> placeholder. The body is written in the voice of someone asking the Agent for help, not as system instructions. Below is the spec text; the actual SKILL.md mirrors it.

SKILL.md (body)# What you're helping with

You're helping a small group of writers iterate on a shared document. They
hold conversations about specific parts of the document — call those
conversations Topics — and when they reach agreement, they ask you to
translate the outcome into a concrete rewrite. The rewritten document is
then reviewed and (usually) committed as the new version.

Vocabulary you'll see:
  • Source — the canonical Markdown/HTML file being edited.
  • Topic — a discussion thread attached to a region (or the whole) of the
    Source. Topics are open until they are incorporated or discarded.
  • Anchor — an inline marker stamped into the Source as an HTML element
    with a data-orcha-anchor="<topic-id>" attribute. Tells the UI
    which region each open Topic is about. You maintain these.
  • Proposal — the rewritten Source you produce, plus a short explanation
    of what you understood the Topic to be asking for.

# Inputs you have

The message that invoked this skill carries three values:

    Job ID:        <uuid>
    Config path:   <absolute path to wiki-browser.yaml>
    wb-agent path: <absolute path to the wb-agent binary>

Always invoke wb-agent via the absolute path you were given.
Always pass --config=<Config path>.

# Steps

1. Load the working set

   Run:
     <wb-agent path> get-topic --config=… --job-id=…

   It returns the Topic you're working on with its anchor and the full
   message thread, the absolute path to the Source file, and the
   base_source_sha of the Source at this moment. Prior proposals you've
   made for this Topic appear inside the message thread as kind
   "agent-proposal", with their full proposed_source bodies inlined — read
   them as your own earlier thinking.

   Then list the other open discussions on the same Source so you can
   keep them anchored:
     <wb-agent path> list-open-topics --config=… --source-path=<abs> --exclude-topic=<current-id>

   Pass the absolute source_path get-topic returned, and exclude the
   Topic you're working on. The response is every other open non-global
   Topic on that Source, each with its anchor and message thread.

   Read the current Source via the Read tool at the absolute source_path.

2. Produce a rewrite

   Apply the Topic's discussion to the Source. The discussion is
   authoritative — don't invent changes the humans didn't agree to. If
   the discussion is ambiguous or unresolved, prefer the most recent
   human messages and explicit decision markers ("yes", "approved",
   "let's go with X"); if still ambiguous, prefer the smallest change
   that reflects the conversation.

3. Re-anchor every other open non-global Topic

   For every Topic returned by list-open-topics, the rewritten Source
   MUST contain at least one occurrence of:
     data-orcha-anchor="<that-topic-id>"

   How to place markers:
     • If the Topic's intent still maps to a region of the new Source,
       wrap the relevant content. Inline anchors use
       <span data-orcha-anchor="<id>">…</span>; block-level anchors use
       <div data-orcha-anchor="<id>"></div> immediately preceding the
       block, separated by a blank line.
     • If a Topic's intent naturally spans multiple regions, use multiple
       markers — one per region is fine.
     • If a Topic's intent no longer maps cleanly to anything in the new
       Source, append it to a section titled exactly:
         ## Other ideas (potentially to discard)
       at the very bottom of the Source. Each parked Topic gets at least
       one sub-bullet referencing the discussion, wrapped in a marker.
       If a parked discussion has multiple distinct sub-ideas to preserve,
       use multiple sub-bullets — each with its own marker is fine.
     • Do NOT include any marker for the Topic you're incorporating. Its
       discussion outcome lives in the prose now; the marker would be a
       dead UUID in the committed Source.

4. Write a short explanation

   In 1–3 paragraphs, summarise what you understood the Topic to be asking
   for and how your rewrite addresses it. This goes in the UI alongside
   the diff so the humans reviewing know what your thinking was. Plain
   prose; no Markdown headings.

5. Persist

   Pipe the rewritten Source bytes on stdin to wb-agent, with the
   explanation inline as a flag:

     <wb-agent path> insert-proposal --config=… --job-id=… \
       --explanation="$EXPLANATION" < rewritten-source

   Build $EXPLANATION as a single shell variable so multi-line text and
   special characters pass safely as one argv value. No temp files.

   Exit 0 on success. Exit non-zero on any unrecoverable error; the
   stderr you produce becomes the error message humans see in the UI.

Four invariants the prompt encodes:

• At least one marker per other open non-global Topic. Missing markers mean a Topic was silently dropped (the post-job invariant catches this).
• No marker for the incorporated Topic. Its outcome is in the prose now; the marker would be dead UUID in committed Source.
• The parking-lot section is the safety valve. The Agent never has to drop a Topic to fit; it can always park unmappable ideas under ## Other ideas (potentially to discard).
• An explanation accompanies every proposal. Empty explanations fail the job; the human-facing review depends on this.

The harness enforces all four post-exit (see "Job-kind invariants" below).

Diff viewer — Tier 1 (unified)

GET /api/proposals/{id}/diff computes the unified diff server-side using an in-process Go diff library (github.com/sergi/go-diff/diffmatchpatch and github.com/hexops/gotextdiff are both viable; the implementation plan picks one). No git diff --no-index shell-out. The response carries the diff text plus base_sha, proposed_sha, and fresh. The client renders it as a <pre> block with line-level +/− classes. The chrome (toolbar, tier toggle button) belongs to #8.

Diff viewer — Tier 2 (side-by-side rendered)

Two iframes side by side. The left iframe loads /content/{source_path} — the current Source rendered through the existing internal/render pipeline. The right iframe loads a new route:

internal/server/handler_preview.goGET /content/preview/proposals/{id}
  → look up proposal by id (auth check)
  → load proposed_source bytes from the row
  → render through internal/render using source_path for relative-link
    resolution (so images, links, mermaid all resolve identically to the
    current Source)
  → Content-Type: text/html; Cache-Control: no-store

Same render pipeline as live Source means rich content (images, tables, SVG, mermaid, JS-driven knobs) works for free. Base-URL handling matches today's /content/ route because the path under the namespace is the same.

Anchor resolution on the preview. The preview route runs ResolveAnchors (the same post-render pass used by /content/{source_path}) in a render-only post-commit mode. It reads current open non-global Topics on the Source, excludes the incorporated Topic (its marker is gone from the bytes by contract), and keeps only Topics whose data-orcha-anchor="<id>" marker appears in proposed_source. For those Topics, the route passes a temporary {"kind":"marker"} anchor to ResolveAnchors even if the stored DB anchor is still pre-marker. The DB is not mutated by preview rendering. The result looks like the post-commit page would: re-anchored Topics highlight in place, the parking-lot section highlights its parked entries, and the incorporated Topic has no highlight because its outcome is now in the prose.

The preview iframe deliberately does not emit the <meta name="wb-source-sha"> tag the live /content/ route uses for Topic creation, so client-side Topic-composer code on the preview cannot mistake the preview for a live Source and create a Topic against bytes that aren't yet committed.

Default tier on first open: Tier 2 (rendered side-by-side). Tier 1 is a toggle for precision. Users approve content, not patches; the rendered view shows the actual user-facing change.

Mobile fallback for Tier 2. Below ~700px wide, side-by-side iframes are unusable. The recommended pattern is tabs (Current | Proposed) — same two URLs, one visible at a time. The API surface doesn't change; the breakpoint logic, tab chrome, and toggle widget belong to #8.

Discard flow

Discard is a pure SQLite transition. No Agent job, no Source write. The endpoint:

1. Validates the Topic exists and is non-terminal.
2. If reason is present and non-empty, appends a kind='human' message with that body, author_user_id = the discarding user — through the existing message-append path so the topic_messages.sequence allocation stays consistent. If reason is absent or empty, no message is appended; the bare outcome columns suffice as the audit trail.
3. Sets topics.discarded_at and topics.discarded_by.
4. Returns { discarded_at }.

Steps 2 and 3 run through the existing single-writer goroutine in one SQLite transaction.

Job-kind invariants for incorporate

#3 ships a single post-exit check ("a new incorporation_proposals row exists for the expected topic_id with created_at >= job.started_at"). #4 replaces the timestamp inference for Agent proposals with the explicit link from migration 004: a proposal row must exist with agent_job_id = job.id, then pass the three anchor/explanation invariants below. All checks run after wb-agent insert-proposal has committed the proposal row and its accompanying agent-proposal message to the DB but before the harness reports status='succeeded' on the job. Failure on any invariant flips the job to status='failed' with a descriptive error_tail; the proposal row stays (the proposal log is append-only) but the UI does not surface a failed proposal as fresh.

• Every other open non-global Topic on the Source has at least one marker. For each Topic returned by list-open-topics at job start, the proposed Source must contain at least one occurrence of the literal substring data-orcha-anchor="<topic-id>". Missing occurrences fail the job with error_tail = "anchor invariant: topic <id> not stamped in proposal". Multiple occurrences are allowed (a Topic may span multiple regions).
• The incorporated Topic's marker is absent. The proposed Source must not contain the literal substring data-orcha-anchor="<incorporated-topic-id>". Failure: error_tail = "anchor invariant: incorporated topic's marker leaked into proposal".
• The accompanying agent-proposal message has a non-empty body. The Agent must attach an explanation. Empty body fails the job with error_tail = "explanation invariant: agent-proposal body is empty".

The checks run in internal/agent/service.go's post-exit step alongside the existing "did a row appear?" verification. Anchor checks read the proposal's proposed_source column and do plain substring matching on data-orcha-anchor="…" — counting literal substring occurrences, not parsing HTML. The marker syntax is fixed (Topic IDs are UUIDs, attribute uses double quotes), so substring matching is unambiguous. Explanation check is a simple LENGTH(TRIM(body)) > 0 against the agent-proposal message row.

No explicit cap on open Topics per Source. Earlier drafts considered capping the number of open Topics whose context the skill loads, to bound prompt size. The first invariant above replaces that — it catches the actual failure mode (silent drop) regardless of prompt size, so we don't need an arbitrary cap rejecting work that might fit. If runs start timing out on real Sources, #5 or a future iteration can revisit per-Topic message truncation.

Explanation length, unified contract. Three numbers appear in this spec and have caused confusion: "1–3 paragraphs" in the SKILL.md body is content guidance only — what the Agent should aim for so humans can use the explanation. The only hard constraints are #2's validateMessage rule (non-empty body, body length ≤ 64 KiB — applied to all topic_messages rows including kind='agent-proposal') and the Linux argv ceiling (~128 KiB on the wire from skill to wb-agent, well above 64 KiB). wb-agent insert-proposal validates the explanation against the same validateMessage the server uses, so the cap is enforced server-side regardless of whether the Agent's prompt followed the guidance. Failure: non-zero exit from wb-agent with the validator's error message; the harness reports status='failed' and the error_tail surfaces in the UI.

Auth wiring

All seven endpoints sit behind auth.RequireCollaborator. The request principal (users.id = normalised email) is used for:

• incorporation_proposals.proposed_by stays NULL for Agent-produced rows (per #3's migration 003).
• topics.incorporated_by = principal at approval time.
• topics.discarded_by = principal at discard time.
• topic_messages.author_user_id = principal for rework feedback messages.
• incorporation_attempts.approved_by = principal at approval time (already wired by #1's IncorporateInput.ApproverID).

Anonymous users get 401; authenticated non-collaborators get 403, per #7's envelope decision. API endpoints never redirect to Google; the preview content route returns HTML only after collaborator auth passes.

All five mutating endpoints (POST /api/topics/{id}/proposals, POST /api/proposals/{id}/incorporate, POST /api/topics/{id}/discard, plus the message-append and topic-create endpoints from #2 that they coexist with) additionally require X-CSRF-Token per #7. The two GET endpoints (/api/topics/{id}/proposals, /api/proposals/{id}/diff) and the preview content route do not.

Schema migration 004

#4 adds one provenance column to the proposal log:

sqlALTER TABLE incorporation_proposals
  ADD COLUMN agent_job_id TEXT
  REFERENCES agent_jobs(id);

CREATE INDEX incorporation_proposals_agent_job
  ON incorporation_proposals(agent_job_id);

The column is nullable. Existing rows and any future manual/user-authored proposals can keep NULL; Agent-authored rows inserted by wb-agent insert-proposal --job-id=… must set it. The migration does not rebuild the table because SQLite allows adding a nullable FK column. Tests cover insertion with a valid job ID, rejection of a missing job ID when foreign keys are enabled, and legacy NULL rows.

Anchor JSON updates on Incorporation

The Agent's rewrite contract requires every other open non-global Topic on the Source to receive at least one data-orcha-anchor marker in the proposed Source. That stamps the new Source bytes correctly, but doesn't touch the corresponding topics.anchor JSON in SQLite — and #2's invariant is that any Topic whose anchor.kind == "pre-marker" resolves through byte offsets in the Source pinned by source_sha. Once Incorporation lands and that source_sha no longer matches the committed Source, those offsets refer to nothing useful; the locator must flip to marker (UUID-based) for resolution to keep working.

So Incorporation extends the SQLite transaction inside collab.CompleteIncorporation to also re-anchor the other open Topics. The harness:

1. Reads the open-Topic set on the Source at approval time (same query list-open-topics uses, minus the incorporated Topic), then filters it to Topic IDs whose marker appears in proposed_source. This is the post-stale-check set, so if any current open Topic is missing a marker, approval has already returned 409 stale_proposal.
2. Passes those Topic IDs to collab.CompleteIncorporation as a new ReanchorTopicIDs []string field on CompleteIncorporationInput.
3. Inside the existing transaction (so the commit-sha write, attempt completion, and anchor updates all succeed together or all roll back), runs:

   UPDATE topics
      SET anchor = '{"kind":"marker"}',
          updated_at = unixepoch()
    WHERE id IN (<reanchor list>)
      AND discarded_at IS NULL
      AND commit_sha IS NULL
      AND json_extract(anchor, '$.kind') = 'pre-marker'

   The WHERE filters guard against Topics that transitioned terminal in the time between read and write (concurrent Discard, concurrent Incorporation on a sibling Topic that somehow made it past the per-Source queue), and skip Topics already at marker kind (idempotent retry safety). Topics in the global-anchor kind aren't in the input set to begin with (list-open-topics excludes them).

The post-job invariant (every other open Topic has data-orcha-anchor="<id>" in the proposed Source) is the partner check: it guarantees that when the anchor JSON flips to marker, the marker that locator refers to actually exists in the committed bytes. Without both halves, the system silently corrupts anchor resolution the first time two Topics share a Source.

The Discard-flow callout's claim "the renderer treats markers for non-open Topics as inert" still holds: markers for discarded or already-incorporated Topics stay in committed Source until the next Incorporation rewrites the file. This update only converts open Topics' anchor JSON.

Slim prompt body (replaces #3's parameter block)

The harness's prompt builder in internal/agent/claude_cli_runner.go writes the prompt as a brief human-style request, not a parameter dump. Concretely:

prompt bodyPlease help me incorporate a Topic discussion into a shared document.
Use the wb-incorporate skill to load the conversation, produce a rewrite,
and persist it as a proposal for review.

Job ID:        ab12cd34-…
Config path:   /etc/wiki-browser/wiki-browser.yaml
wb-agent path: /opt/wiki-browser/dist/wb-agent

The skill discovers everything else through wb-agent get-topic --job-id=…. Removed from the prompt: Repo root, Source path, Base source SHA, Topic ID. The #3 decision "job parameters travel in the prompt body, not env vars" still holds for the three remaining values; only the derivable ones drop.

The skill's description field stays accurate (#3 already wrote it); the body change is the one that swaps from infrastructure tone to user-task tone.

wb-agent surface (the only consumer is the skill)

Every subcommand takes --config=<path>. The identifier flag matches what each call naturally operates on.

No temp files. The explanation rides as a single argv value (Linux argv limit is 128 KiB; 1–3 paragraphs is well under that). The skill's SKILL.md documents the Bash pattern (single-quoted variable assignment) for safely passing multi-line text with arbitrary characters.

Concurrency & recovery

The per-Source agent queue from #3 (agent.max_concurrent_jobs=1, keyed by source_path) serialises proposal jobs on the same Source. Approval is serialised by two staleness checks before the file write: (1) the existing SHA guard — the first commit on a Source wins; any in-flight proposal for another Topic on that Source becomes stale (base_source_sha mismatch, stale_reasons includes "source_sha"); (2) the marker-set guard — every current open non-global Topic on the Source, excluding the Topic being incorporated, must have at least one marker in the proposal bytes (stale_reasons includes "missing_topic_markers" when not). The UI surfaces both via the fresh, stale_reasons, and missing_topic_ids fields in the proposals list and the 409 stale_proposal response from approval. No new locking primitives — Topic creation, proposal generation, and message append all stay unblocked.

Startup recovery is unchanged: #3's agent_jobs sweep marks any orphaned running/queued rows as failed; #1's collab.Recover reconciles any partial Incorporation attempt against the working tree. #4 introduces proposal/job provenance but no new durable in-flight state.

Open questions

• Per-Source approval race. Two collaborators clicking Approve on different fresh proposals in the same second (each for a different Topic on the same Source) will both pass the read-side fresh check; the second one's collab.Incorporate will fail the SHA check after the first one commits. UX is acceptable (the loser sees a stale error, the propose endpoint dedups so they re-queue the same Topic without ceremony), but worth observing once two-collaborator usage starts in earnest.
• Bash-quoting robustness for very long or unusual explanations. The skill builds the --explanation argv value as a shell-quoted variable. Standard 1–3 paragraph natural-language explanations pass through cleanly; pathological inputs (literal null bytes, etc.) would fail at the shell level before reaching wb-agent. If observed, we can switch to a stdin framing (length-prefixed) without changing the spec — the SKILL.md is the only thing that knows the wire shape.

References

• Collaborative annotations — Domain model
• Decisions & parking lot
• #1 — Document model & persistence
• #2 — Topic core: data model + anchoring
• #3 — Agent runtime & harness invocation
• #7 — Identity & permissions