Topic core — data model & anchoring — Design

Problem

Sub-project #2 of the collaborative-annotations initiative fills the gap left by sub-project #1: it defines what goes inside topics.anchor (the JSON column #1 left empty), settles the full topic_messages.kind enum, and specifies the semantic-anchor mechanism — how a Topic stays bound to a meaningful region of its Document as the Source changes.

This is the technically hardest piece of the initiative because "stays bound across rewrites" sounds like an algorithm problem but is really a contract problem: the Agent owns Source writes, so the right design hands the Agent enough information and convention to keep anchors valid by construction. The cross-cutting decisions that produced this design live in the decisions log; this spec is the implementable contract that downstream sub-projects (#3 Agent runtime, #4 Incorporation, #8 UI integration) build against.

Goals

• A concrete JSON shape for topics.anchor, with a small enum of kinds and a clear lifecycle between them.
• A marker convention the Agent stamps into the Source, robust enough to survive arbitrary rewrites because the Agent maintains it deliberately on every Incorporation.
• A renderer-side substrate (Source-position attributes on rendered HTML) that downstream features — selection capture, the diff viewer (#4), the UI overlay (#8) — can all consume.
• A backend API for Topic creation, listing, and message threads.
• A minimal UI surface that demonstrates the loop end-to-end: select text → open Topic → see highlight → post replies.
• A precise re-anchor contract for the Agent so sub-project #4 can wire Incorporation against it without renegotiation.

Non-goals

• Incorporation mechanics. The state machine, rework loop, diff UI, and the Agent's prompt are owned by #4. This spec defines the inputs/outputs of the re-anchor step #4 will invoke.
• Agent invocation runtime. How the harness spawns Claude Code, ships context, and receives results is owned by #3. This spec describes what the Agent must do during re-anchor; #3 owns how.
• Per-Perspective anchors. Anchors on Perspectives (independent of the Source anchor) are owned by #5. This spec covers the Source anchor only.
• Polished UI chrome. The Resolve button, sidebar styling, persona editor, and Perspective switcher belong to #8. This spec ships the bare functional UI needed to drive Topic creation and reading.
• Real-time / multi-user editing of Topic threads. Owned by #6.

Approach

Three moves work together:

1. Anchors are inline markers the Agent stamps directly into the Source for open, non-global Topics — HTML elements carrying data-orcha-anchor="<topic-id>". Because the Agent is the sole writer of Source (per #1's invariants), it can place markers wherever the idea now lives during every rewrite. The set of "what's anchorable" collapses to "anything the Agent can wrap": text span, image, heading, figure, code block, mermaid fence.
2. The renderer exposes Source byte positions on every block-level element of the rendered HTML (data-source-start / data-source-end) and returns a server-side rendered-text-to-Source map for those blocks. Selection capture maps a rendered selection back to exact Source offsets through that map; the diff viewer (#4) and the inline annotator (#8) reuse the same substrate.
3. Three anchor kinds tracked in topics.anchor: pre-marker (just created, awaiting the next Source rewrite while still open), marker (Agent has stamped one), global (deliberately document-level). Three states, two transitions, no orphan state.

Design

Anchor JSON — three kinds

All anchors live in topics.anchor (TEXT, JSON). The kind is a discriminator; extra fields depend on the kind. The column is non-null for any Topic; an "empty" anchor is represented explicitly.

json// kind: "pre-marker"
{
  "kind":        "pre-marker",
  "source_sha":  "<40-char git blob sha>",
  "start":       1234,                // byte offset, inclusive
  "end":         1289,                // byte offset, exclusive
  "quote":       "the exact selected rendered text"
}

// kind: "marker"
{ "kind": "marker" }

// kind: "global"
{ "kind": "global" }

Field meanings:

• source_sha on pre-marker pins which Source version the offsets were captured against. It must equal the current Source's git blob SHA for the offsets to be valid. If it ever doesn't match (mainly an invariant check; eager re-anchor on Incorporation should prevent this), the resolver treats the Topic as awaiting re-anchor.
• start, end are byte offsets into the Source file (UTF-8 bytes, half-open range).
• quote is informational — the original selected rendered text. Used for sidebar previews and as input to the Agent's re-anchor decision; it is not required for resolution while source_sha still matches.

Why three kinds and not two

A user can open a Topic two ways: with a text selection ("comment on this") or without ("comment on the doc"). These are distinct intents that the model should preserve. Collapsing them would force one of two unattractive options: lose the "global" intent (the Agent eventually has to put the Topic somewhere in the Source, but there's no reason to do so for genuinely document-level comments) or pretend every Topic has a region (the Agent has to invent one, surfacing as a poorly-placed marker).

Marker conventions in the Source

The Agent stamps anchors into Source files (both .md and .html) as HTML elements carrying data-orcha-anchor="<topic-id>". Two placement forms:

• Inline anchor wraps the relevant text: <span data-orcha-anchor="<topic-id>">the relevant text</span>. Used when the idea is about a phrase or sentence.
• Block-level anchor uses an empty <div data-orcha-anchor="<topic-id>"></div> immediately preceding the block, with a blank line between them in Markdown. div is one of CommonMark's block-level HTML tags, so the marker becomes its own HTML block rather than being absorbed into the following paragraph. The resolver detects the pattern and projects the highlight onto the next sibling block. Used when the idea is about a heading, paragraph, figure, code block, or mermaid fence.

Both forms are CommonMark-legal (inline HTML is passed through) and don't break Markdown read in any other tool. The Agent picks the form that best represents the idea's scope.

The Agent's anchoring obligation

Every Incorporation involves exactly one Topic's resolution, but the Agent's rewrite touches the whole Source and must preserve anchors for every other open Topic on the same Source as well. The contract from #2 to #4 is:

1. Before invoking the Agent, the harness loads the Topic being incorporated plus every other open Topic on this Source path.
2. The Agent's prompt includes the full list of open Topics, their current anchors, and their resolved content (the text or region each is currently pinned to).
3. In the proposed Source, the Agent must emit a data-orcha-anchor for each other open non-global Topic — placed naturally if the idea fits a region of the rewrite, or in an "Other ideas (potentially to discard)" section appended to the document if the idea no longer maps cleanly.
4. The Agent never drops markers silently. The parking-lot section serves as visible evidence of unmappable ideas, suitable for a human's eventual discard decision.
5. On commit, every re-anchored other open Topic transitions to {kind: "marker"} as part of the same SQLite transaction that records the commit SHA on the Topic being incorporated. The incorporated Topic becomes closed; its marker is not retained in the committed Source.
6. global Topics are untouched — neither markers placed nor anchor JSON changed.

The wording of the Agent's prompt (how to phrase the parking-lot rule, how to describe the open Topics, how to format the "Other ideas" section) is owned by sub-project #4.

Renderer changes — Source-position attributes

Every block-level element in the rendered HTML carries:

html<p data-source-start="234" data-source-end="389">…</p>
<h2 data-source-start="412" data-source-end="428">…</h2>
<pre data-source-start="450" data-source-end="612">…</pre>

Offsets reference UTF-8 byte positions in the Source file. The half-open range [start, end) covers the bytes that produced this block.

Markdown (internal/render/markdown.go): a new goldmark NodeRenderer wraps the default html.NewRenderer. For each block-level ast.Node, the wrapper writes data-source-start and data-source-end attributes derived from the node's Lines() segments and contributes entries to the render map (defined below). goldmark already tracks segment positions on every text node; the renderer records those segments verbatim instead of trying to reverse-engineer Markdown from DOM text later.

HTML (internal/render/html.go): a tokenizer pass over the Source HTML, using golang.org/x/net/html. As tokens are emitted to the output, block-level opening tags receive data-source-start and data-source-end attributes recording the byte range of the original token stream. Text tokens inside those blocks contribute render-map entries using decoded text for rendered offsets and token byte ranges for Source offsets; entity references collapse into a single entry (e.g., rendered & ↔ five Source bytes &). Self-closing and inline tags are passed through unmodified.

Block-level scope for both formats is the standard set: p, h1–h6, ul, ol, li, blockquote, pre, table, tr, figure, div, and the wiki-browser-specific pre.mermaid. Inline elements (span, a, strong, em, code) do not carry DOM attributes; selection capture interpolates within a block via the render map.

The server content path changes for .html files as part of #2: handleContent must render HTML files through internal/render and the resolver pipeline, not stream raw bytes directly. Raw Source remains available through ?raw=1.

Render map structure

The render map is a sibling output of the renderer, returned alongside the rendered HTML in the *render.Document struct. It is a per-block list of linear segments, where each segment translates between rendered-text offsets and Source byte offsets. The map is the contract between selection capture, the resolver, and the diff viewer (#4).

gotype RenderMap struct {
    Blocks []BlockMap // indexed lookup by source range
}

type BlockMap struct {
    SourceStart, SourceEnd int      // matches the block's data-source-* attrs
    Segments               []Segment
}

// Segment is a maximal linear stretch where rendered-text offsets
// map proportionally to Source byte offsets (1:1 per character when
// the renderer copies bytes through; many-to-one for entities and
// Markdown escapes). Renderer-injected content (syntax-highlighting
// tokens, Mermaid SVG labels, list-marker glyphs, "View more" etc.)
// is represented by a Segment with Source == empty, marked NonSource.
type Segment struct {
    RenderedStart, RenderedEnd int
    SourceStart,   SourceEnd   int
    NonSource                  bool
}

Translation rules used by selection capture and the resolver:

• Rendered → Source. A rendered offset r inside a linear segment S maps to Source offset S.SourceStart + (r − S.RenderedStart). An offset inside an entity / escape segment maps to the whole Source range of that segment — selections that begin or end mid-entity expand outward to the entity's boundaries.
• Selection rejection. A selection whose rendered range overlaps any NonSource segment is rejected (the user is asked to reselect). Selections that span entity boundaries are accepted with the boundary-expansion rule above.
• Source → Rendered (used by the resolver to project a stored Source range onto the rendered output): take the union of every segment whose Source range intersects the target. Edges that fall mid-segment use the same linear formula in reverse.

Cross-block selections are out of scope for v1: a selection that crosses block boundaries — two paragraphs, two list items, two table cells — is rejected client-side (the composer asks the user to narrow to one block). The render map carries one block per BlockMap, and the API shape carries one block range per request. Multi-block anchors can be added later by extending the API to accept an array of BlockMap references; the storage shape (single start/end) would need a follow-up decision before that lands.

Selection capture

Every rendered page carries the Source's git blob SHA in a <meta name="wb-source-sha" content="…"> tag inside the iframe document. The client reads it once on load and includes it in the Topic-creation request so the server can detect a stale iframe before translating offsets.

The flow when a user selects text and creates a Topic:

1. Client JS listens for selectionchange in the iframe. On a non-empty selection, a floating composer (described under Topic creation chrome) appears anchored to the selection. If the selection spans more than one block element (different data-source-start ancestors), the composer disables Save and explains "please select inside a single block."
2. On Save, the client computes: the selected quote (DOM toString()), the rendered-text start/end offsets of the selection within its block (measured by walking text nodes), and that block's data-source-start / data-source-end values.
3. It POSTs {source_path, source_sha, first_message_body, selection: {quote, block_source_start, block_source_end, rendered_start, rendered_end}} to /api/topics.
4. Staleness check. The server computes the Source's current source_sha (via the collab.SourceSHA helper from #1) and compares to the request's source_sha. If they differ, return 409 Conflict with code stale_source — the iframe was rendered against an older Source, and the captured offsets are no longer trustworthy. The client reloads the iframe and asks the user to reselect.
5. Translation. The server loads the render map for the named path (re-rendering if not cached) and locates the block whose SourceStart/SourceEnd match the request. It applies the render-map translation rules above to convert [rendered_start, rendered_end) into a Source byte range. Selections that overlap a NonSource segment return 409 Conflict with code non_source_selection.
6. Write. The server writes {kind: "pre-marker", source_sha, start, end, quote} into the new Topic row, alongside the first message. Translation and write happen inside the single SQLite write funnel from #1, so a concurrent Incorporation cannot interleave; if the Incorporation lands first, the funnel still serialises this write after it — but the staleness check in step 4 will have already rejected the request because the request's source_sha referenced the pre-Incorporation Source.

The resolver

When the harness renders a Document with active Topics, a resolver pass overlays anchor highlights onto the rendered HTML. The resolver runs server-side after the renderer produces its *render.Document (HTML body + render map) and before it's served. Conceptually:

gofunc ResolveAnchors(doc *render.Document, topics []OpenTopic) *render.Document {
    // doc.HTML and doc.RenderMap are produced together by the renderer.
    // 1. Walk the topics; for each non-global anchor, compute its
    //    target range in the Source bytes, then project it through
    //    doc.RenderMap into rendered-text intervals (Source → Rendered
    //    direction documented under "Render map structure").
    // 2. Walk the rendered HTML once with a tokenizer.
    //    - On any block opening tag: read data-source-start / -end and
    //      look up the matching BlockMap.
    //    - When inside a block whose rendered intervals overlap one
    //      or more target intervals, split text nodes at interval
    //      boundaries and wrap each segment in <mark class="wb-anchor"
    //      data-topic-id="..." [data-topic-ids="..."]>.
    //    - On a span/element with data-orcha-anchor=<topic-id> that
    //      matches a marker-kind Topic, wrap its contents the same way.
    // 3. Return the rewritten Document (HTML updated; map unchanged).
}

Behaviour by kind:

• pre-marker: locate [start, end) in the Source, project that Source range through the render map into one or more rendered-text intervals, and wrap those intervals in <mark class="wb-anchor">. If source_sha doesn't match the current Source's sha, skip the inline highlight and surface the Topic in the sidebar's "Awaiting re-anchor" group — a state that should not normally occur.
• marker: find the element carrying data-orcha-anchor="<topic-id>" in the rendered HTML. If the element has no content (block-level marker, conventionally an empty <div>), wrap its next sibling; if it has content (inline marker, conventionally a <span>), wrap its contents. The data-orcha-anchor attribute is preserved on the wrapped element so the client can correlate marker → mark.
• global: no inline rendering. The sidebar lists global Topics in a dedicated group.

Overlapping anchors (two Topics whose target ranges intersect) are handled by interval splitting, not by assuming simple containment. For every text segment, the resolver computes the set of covering Topic IDs. Segments with one Topic emit <mark class="wb-anchor" data-topic-id="...">; segments with multiple Topics emit one <mark class="wb-anchor wb-anchor-overlap" data-topic-ids="id-a id-b">. Styling distinguishes overlap visually (#8's call); the resolver only needs to sort Topic IDs deterministically before emitting attributes.

Topic creation chrome (minimal UI)

Two affordances are introduced in #2:

1. Selection composer. Click-and-drag text in the iframe → a small floating popover appears near the selection: textarea + "Save" + "Cancel". Save creates the Topic with a pre-marker anchor.
2. Topics sidebar. A panel adjacent to the iframe lists open Topics on the current Document, grouped: Anchored (those with pre-marker or marker anchors) and Global. Each entry shows the originator, the quote preview (for anchored Topics), and the first message snippet. Clicking an entry opens the thread view — message list plus a textarea to post a new 'human' message. A "New global Topic" button in the sidebar header creates a Topic with {kind: "global"}.

Clicking a wb-anchor highlight in the iframe focuses the sidebar on the corresponding Topic (via data-topic-id). No keyboard shortcuts, no rich formatting, no notifications, no presence indicators, no Resolve button. Visual polish — sidebar position and width, highlight colour, popover styling — uses the existing wiki-browser design tokens and is intentionally minimal; #8 owns the production-quality treatment.

HTTP API

All endpoints are JSON in/out. Authentication is not in scope for v1; every request attributes to the bootstrap operator from wiki-browser.yaml (see #1).

Topic resolution endpoints (Discard, Incorporate) are owned by #4, which adds them alongside the Resolve flow. Existing internal/collab mutators (InsertTopic, InsertMessage) underpin the write paths; this sub-project adds an internal/collab reader API for sidebar queries and an HTTP layer in internal/server.

Message kinds

For v1 the topic_messages.kind column carries one of two values:

• 'human' — a human-authored message. author_user_id non-null, proposal_id null.
• 'agent-proposal' — written by #4 when the Agent produces a proposal. author_user_id null, proposal_id non-null.

Topic resolution (incorporated / discarded) is fully derivable from topics outcome columns; the UI can synthesise "Topic discarded by Alice on 2026-05-11" without persisting it as a message. Additional kinds (system events, rework requests) can be added when a concrete UI need surfaces — the column accepts any string at the schema level, so adding a kind is a non-migration change.

Validation rules (server-side)

• A POST /api/topics body must contain exactly one of selection or global: true.
• source_path must resolve through the existing ValidateSourcePath helper from #1 (no traversal, must be within root, must have an allowed extension).
• For selection-based creation, source_sha in the body must equal the current Source's git blob SHA. If it doesn't, return 409 Conflict with code stale_source.
• selection.quote must be non-empty, and [block_source_start, block_source_end) must match exactly one block in the current render map. If it doesn't (e.g., block boundaries shifted, which should already have been caught by the staleness check), return 409 Conflict with code unknown_block.
• [rendered_start, rendered_end) must lie within that block's segments and must not overlap any NonSource segment. Otherwise return 409 Conflict with code non_source_selection.
• POST /api/topics/{id}/messages rejects messages whose Topic is incorporated or discarded (outcome columns non-null) with 410 Gone.
• first_message_body and message body are non-empty strings, max 64 KiB.

Re-anchor contract (handoff to #4)

Sub-project #4 implements the actual re-anchor step as part of Incorporation. This spec defines its input/output shape so #4 doesn't have to renegotiate with #2:

re-anchor task — invoked once per IncorporationInput  (from harness, prepared by #4):
  - source_path
  - current Source bytes (pre-rewrite)
  - the Topic being incorporated:
      { id, anchor, messages[] }
  - all other open non-global Topics on this source_path:
      [{ id, anchor, messages[] }, ...]   // excludes 'global' Topics

Output (from the Agent, consumed by #4):
  - proposed Source bytes (post-rewrite)
  - INVARIANT: for every "other open" input Topic id,
    proposed Source bytes contain a data-orcha-anchor="<id>"
    element exactly once.
  - INVARIANT: any Topic whose idea did not map naturally is
    represented by a marker inside an "Other ideas (potentially
    to discard)" section appended to the proposed Source.

Side effects on commit (in the same SQLite transaction as the
commit-sha write owned by #1):
  - For every "other open" Topic id in the input list, set
      topics.anchor = '{"kind":"marker"}'

Two role-based distinctions to keep straight:

• The Topic being incorporated is always handed to the Agent as rewrite context, regardless of its anchor kind — a global Topic can itself be the one incorporated, and its conversation tells the Agent what to change. It is not part of the re-anchor output set: once approved, it closes and its marker (if any) must not appear in the committed Source.
• Other open Topics are handed to the Agent so it can preserve their anchors. Global Topics in this group are excluded from the re-anchor obligations — the Agent neither sees them nor places markers for them, and their anchor JSON is unchanged.

Storage — no schema change

Sub-project #1 already shipped topics.anchor TEXT as nullable. This sub-project tightens the contract: every row created via the API has a non-null anchor JSON, and the JSON validates against one of the three kinds defined above. The NOT NULL constraint is enforced in Go (in InsertTopic), not as a schema-level constraint — keeping the schema unchanged from #1 means no second migration.

topic_messages.kind remains an unconstrained TEXT column at the schema level. The Go layer constrains it to {'human', 'agent-proposal'} on insert.

The existing index topics_by_source_path covers the sidebar's primary query ("list open Topics for this source_path"). No new indexes for v1.

Code organisation

New files and the packages they live in:

The Topic core sits between internal/collab (storage primitives owned by #1) and internal/server (the HTTP+UI surface). Re-anchoring logic that the Agent will eventually drive lives in internal/collab/anchor.go as the type definitions plus pure helpers; #4 will add the runtime that invokes the Agent against these types.

Testing

• Anchor JSON: round-trip tests for each kind; validation tests for missing fields and impossible combinations (e.g., marker with start).
• Source-position renderer + render map: golden-file tests covering one Markdown file with every block element class and one HTML file with nested blocks. Verifies data-source-start/-end values match the corresponding BlockMap entries and that segments cover entity references (e.g., &), Markdown escapes, syntax-highlighting tokens (flagged NonSource), list-marker glyphs (NonSource), and Mermaid SVG labels (NonSource).
• Render map translation helpers: table tests for RenderedToSource and SourceToRendered, including selections that straddle entity boundaries (expanded outward) and selections that overlap NonSource segments (rejected).
• Selection capture (server side): table tests — given a Source, a request with source_sha + block range + rendered offsets, the handler returns expected Source offsets, stale_source, unknown_block, or non_source_selection.
• Resolver: golden tests covering each kind, partially overlapping anchors (interval splitting with data-topic-ids), and a Topic with a missing marker (verifies the bug-path: skip the inline highlight, emit a structured log line).
• HTTP API: httptest-driven tests for each endpoint covering happy paths, validation failures, the staleness check (request source_sha mismatches), and the resolved/discarded → 410 path.
• End-to-end: a Playwright test (using the project's existing playwright-cli skill) that drives selection → Save → highlight visible → reply posted → reply visible.

Open questions

Small items, none blocking implementation. Resolve during writing-plans or first PR.

• Sidebar collapse state persistence. Should the sidebar's open/closed state persist per user across reloads (localStorage), or reset each time? Cosmetic; default to localStorage.
• Quote truncation in sidebar previews. Long quotes need a max length and ellipsis. Pick a number (80 chars?) during implementation.
• Block-level anchor "next sibling" edge cases. An empty marker div at the end of the document has no next sibling; treat as a placement bug (Agent should not emit this) and log a structured error if encountered.
• Tokenizer choice for HTML source-position pass. golang.org/x/net/html is the obvious pick; if its byte-position tracking is incomplete, fall back to a small handwritten tokenizer over a fixed list of block-level tag names. Decide once implementation starts.

References

• Domain model — vocabulary, invariants, sub-project decomposition.
• Document model & persistence (sub-project #1) — schema this design fills in, write funnel, hash utilities.
• Decisions & parking lot — full trail of choices and items deferred to other sub-projects.
• wiki-browser/internal/render/markdown.go — existing goldmark setup; the Source-position attributes hook into the renderer here.
• wiki-browser/internal/collab/mutators.go — existing InsertTopic, InsertMessage; the API in this spec calls into these directly.