Agent runtime & harness invocation — Design

Problem

The collaborative-annotations initiative introduces an Agent that owns Source rewrites, anchor maintenance, and Perspective generation. Sub-projects #1 (document model & persistence) and #2 (topic core: data model + anchoring) are implemented and assume the Agent exists; both punt the question of how the harness actually invokes it.

Today, internal/collab stores proposals and applies approved ones, but nothing produces them. The wiki-browser Go server has no way to spawn a Claude Code instance, ship it the context for a job, or learn whether the job succeeded. This sub-project defines that runtime.

The scope here is narrow on purpose. #4 (Topic resolution & incorporation) and #5 (Perspectives) own the prompt wording and the per-job invariants their skills must respect. #3 owns the substrate that lets those skills run, persist results, and surface state.

Goals & non-goals

Goals

• Define the process model for invoking Claude Code from the Go server, and the wire contract between them.
• Pin the skill-storage layout in the repo so #4 and #5 can drop in their prompts without renegotiating paths or input/output conventions.
• Make agent invocation testable without spawning subprocesses — a substitutable Runner behind a small interface.
• Track agent activity in a new agent_jobs table so the UI can surface state (queued, running, succeeded, failed, timed_out) without inventing new mechanisms.
• Plumb the Agent's git identity (Orcha Agent <agent@orcha.local>) into the existing Incorporate apply step.
• Keep the v2 path to claude --channels open: skill code never assumes a fresh process; only the runtime mode changes.

Non-goals

• Prompt wording. The text inside each skill file is owned by #4 (incorporate) and #5 (perspective). #3 ships skill scaffolds with a "TODO: prompt body owned by #4/#5" marker so the runtime can be tested end-to-end with a stub.
• Incorporation state machine, rework loop, diff UI. Owned by #4.
• Persona format, refresh policy, cache invalidation specifics. Owned by #5. #3 only defines the wb-agent CLI surface those subprojects will use.
• Push-to-UI (SSE / WebSocket). The UI polls agent_jobs for v1. SSE deferred to #8.
• Authentication on agent-triggering HTTP endpoints. Owned by #7 (identity & permissions); #3 uses the principal helper #7 provides.
• Distributed or multi-process agent execution. Single-machine, single-server-process v1.
• Job persistence across server restarts. In-flight jobs are lost on restart; the user retries from the UI.

Approach

Use a headless claude -p subprocess per job. The Go server spawns Claude Code, hands it a short prompt that names a skill and includes the job parameters as body text, and waits for exit. The agent does its work using its standard tools — filesystem reads/writes for Source, and a new wb-agent CLI for DB access. The server learns "succeeded" or "failed" purely from exit code and a stderr tail; everything else is read from the collab DB after exit.

The agent owns the work surface. The Go server is a launcher and a status mirror: it spawns the process, surfaces success/failure to humans, and reads the rows the agent created. It does not parse agent stdout, does not enforce job-specific invariants (anchor placement, persona shape), and does not retry. Validation that lives inside the skill belongs to #4/#5; validation that lives in wb-agent reuses the existing internal/collab code paths.

Design

Process model

One subprocess per job, spawned via Go's exec.Command:

gocmd := exec.CommandContext(ctx,
    cfg.Agent.ClaudeBin,                       // "claude" by default
    "-p", promptBody,
    "--dangerously-skip-permissions",
)
cmd.Dir = wikiBrowserRoot                      // for .claude/skills/ discovery only
cmd.Stdout = &stdoutBuf                        // captured for debug logs
cmd.Stderr = &stderrBuf                        // last 4 KiB → agent_jobs.error_tail

// Process-group + graceful shutdown. cmd.Cancel + WaitDelay are Go 1.20+:
// on ctx cancel (shutdown or timeout) we SIGTERM the whole process group, then
// SIGKILL after 5 s if claude or any child still hasn't exited.
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error {
    return syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM)
}
cmd.WaitDelay = 5 * time.Second                // then default kill (SIGKILL)

• No env vars on the subprocess. All job parameters live in promptBody. The subprocess inherits the server's environment as-is.
• --dangerously-skip-permissions for two reasons: (1) the agent is a trusted system actor invoked by the server, not a user-driven assistant — interactive permission prompts are incompatible with unattended invocation; and (2) Claude Code's default Read/Write tools are restricted to cwd and its descendants. Source files live under cfg.Root, which is the parent of wikiBrowserRoot in the typical orcha-monorepo layout, so the agent must reach above its own cwd. Skipping permissions lifts that restriction. --add-dir plumbing is not needed.
• Working directory is wikiBrowserRoot (the absolute path to the wiki-browser/ subproject). This is purely for skill discovery — Claude Code auto-loads .claude/skills/wb-incorporate/ and .claude/skills/wb-perspective/ as project-local skills from this directory. Source paths live one level up (under cfg.Root, the orcha monorepo root) and are conveyed in the prompt as absolute paths the agent reconstructs from the Repo root field — never via working-directory relative resolution.
• No --output-format flag. The server doesn't parse stdout; default human-readable output is fine.
• No MCP servers, no extra tool grants. The skill uses Claude Code's built-in tools (Read, Write, Edit, Bash, Glob, Grep) and nothing else. Bash is what runs wb-agent.
• Process-group teardown is explicit. Claude Code spawns child processes for Bash tool calls and may keep them alive past its own exit if signalled naively. Setpgid: true creates a new process group, and cmd.Cancel signals the whole group on context cancellation. cmd.WaitDelay = 5s gives the group a graceful window before Go's default behavior (SIGKILL on the leader) takes over. The bare exec.CommandContext default — single-process SIGKILL — is insufficient here.

Skill layout

Project-local skills, versioned in git as part of wiki-browser/:

treewiki-browser/.claude/skills/
├── playwright-cli/           # existing
├── wb-incorporate/
│   └── SKILL.md
└── wb-perspective/
    └── SKILL.md

Each SKILL.md follows the standard frontmatter format:

markdown---
name: wb-incorporate
description: Produce a proposed Source rewrite for an open Topic, re-anchoring
  every other open non-global Topic on the same Source. Used by wiki-browser
  during Topic incorporation.
---

# wb-incorporate

Parse the job parameters from the prompt body. They will look like:

    Job ID:          <uuid>
    Topic ID:        <topic-id>
    Source path:     <repo-relative>
    Base source SHA: <git blob SHA>
    Repo root:       <absolute path to the orcha monorepo root>
    wb-agent path:   <absolute path to the wb-agent binary>

The Repo root + Source path concatenation gives you the absolute Source
file path. Always use the absolute path the harness gave you — never
rely on the current working directory for resolving Source files.
Always invoke wb-agent via the absolute path the harness gave you —
never rely on PATH lookup.

Then:

1. Read the Source file at <Repo root>/<Source path>.
2. Run `<wb-agent path> get-topic --id=<topic-id>` to load the topic,
   anchor, and full message thread.
3. Run `<wb-agent path> list-open-topics --source-path=<source-path>`
   to load every other open Topic on this Source, with anchors.
4. <REWRITE CONTRACT OWNED BY #4 — re-anchor + rewrite contract goes here.>
5. Pipe the proposed Source to:
     `<wb-agent path> insert-proposal --topic-id=<topic-id> --base-sha=<sha>`
6. Exit.

#3 ships both SKILL.md files with stub prompt bodies (clearly marked <REWRITE CONTRACT OWNED BY #4/#5>) plus a minimal exit-zero pass so the end-to-end runtime is testable. #4 and #5 fill in the substantive prompt content when they land.

Prompt format

The prompt body is a short instruction naming the skill, followed by job parameters as a labelled block:

textUse the wb-incorporate skill.

Job ID:          0d4b9a2f8c7e4a13b6a01e9c2d8f5b34
Topic ID:        t-xyz
Source path:     docs/foo.md
Base source SHA: deadbeefcafef00d...
Repo root:       /home/volrath/code/orcha
wb-agent path:   /home/volrath/code/orcha/wiki-browser/dist/wb-agent

When done, exit 0. On any unrecoverable error, exit non-zero — the wiki-browser
server will surface stderr to the operator.

The parameter block is the same for every wb-incorporate invocation; the skill's first paragraph teaches Claude how to read it. wb-perspective uses an analogous block with Persona name, Source SHA, and Persona SHA in place of Topic ID and Base source SHA. Repo root and wb-agent path appear in both.

The harness computes Repo root from cfg.Root (the orcha monorepo root) and wb-agent path from cfg.Agent.WBAgentBin — which defaults at startup to filepath.Join(filepath.Dir(os.Executable()), "wb-agent"), the path next to the running wiki-browser binary. Both are validated at config-load time.

Capability surface

Two job kinds in v1:

The wb-agent CLI

A new binary, built from the same Go module as wiki-browser, lives at cmd/wb-agent/main.go and ships as dist/wb-agent. It opens the collab DB directly using the existing internal/collab code paths so all validation, FK enforcement, CHECK constraints, and the sequence-allocation logic that Store applies are reused. The agent invokes it via Claude Code's Bash tool.

Because wb-agent is a separate process from the running server, it cannot share the server's in-memory write funnel. SQLite's WAL mode plus the per-DSN busy_timeout=5000 already configured in collab.Open handle cross-process write contention: brief blocking is acceptable for low-volume agent writes. wb-agent opens its own short-lived connection per invocation; the funnel goroutine model remains the discipline within the server process.

v1 subcommands:

wb-agent reads wiki-browser.yaml to locate the collab DB. The path is resolved via the same -config flag the server uses (default wiki-browser.yaml in the working directory). The agent's working directory is wikiBrowserRoot when invoked, so the default works without configuration.

Concurrency and the in-memory queue

A new internal/agent package owns the runtime. The Service wraps an in-memory queue keyed by Source path:

• Per-Source serialization: at most one job per source_path runs at a time. New job for an already-running Source returns 409 Conflict from the HTTP layer; the UI disables triggers while a job is in flight.
• Global cap: agent.max_concurrent_jobs (default 1 for Pi). Beyond the cap, jobs queue.
• Non-persistent: the queue is process-memory. A server crash loses queued and in-flight jobs; users retry. The startup sweep (below) marks orphaned agent_jobs rows.
• Cancellation: each job runs under a per-job context.Context that the service cancels on shutdown and on timeout. Cancellation calls cmd.Cancel, which sends SIGTERM to claude and waits up to 5 s before SIGKILL.

Schema: relaxing incorporation_proposals.proposed_by

The current 001_initial.sql migration declares proposed_by TEXT NOT NULL with a FK to users(id). That contradicts #1's intent — recorded in the decisions doc as "the Agent is not a user; agent-authored content may have null user references." The constraint never fired because nothing has produced an agent proposal yet. #3 drops NOT NULL via SQLite's twelve-step table-rebuild procedure.

The rebuild has constraints that the existing migration runner cannot satisfy:

• Foreign keys must be disabled for the rebuild because topics, topic_messages, and incorporation_attempts reference incorporation_proposals — dropping it under foreign_keys=ON either fails or silently invalidates referencing rows.
• PRAGMA foreign_keys is a no-op inside a transaction. It can only be toggled when no BEGIN is active. The current applyOne wraps every migration in a transaction, so an inline PRAGMA foreign_keys = OFF; at the top of the SQL file would silently do nothing.

The fix is a small enhancement to the migration runner: a per-file directive that opts out of the runner's transaction wrapper and lets the migration manage its own boundaries.

go// internal/collab/migrate.go — runner change
// If the first non-blank line is exactly "-- migrate:no-tx",
// applyOne skips its own BEGIN/COMMIT. The migration file is
// executed as-is and is responsible for its own tx + FK toggling.
// schema_migrations bookkeeping runs in a separate short tx afterward.

sql-- migrations/003_agent_runtime.sql
-- migrate:no-tx

-- FK toggling must happen outside any transaction; the runner honors the
-- no-tx directive above and lets this file own its own BEGIN/COMMIT.
PRAGMA foreign_keys = OFF;

BEGIN;

-- SQLite cannot ALTER a column's NOT NULL; rebuild the table.
CREATE TABLE incorporation_proposals_new (
  id              TEXT PRIMARY KEY,
  topic_id        TEXT NOT NULL,
  revision_number INTEGER NOT NULL,
  proposed_source TEXT NOT NULL,
  base_source_sha TEXT NOT NULL,
  proposed_by     TEXT,                  -- now nullable; NULL = Agent
  created_at      INTEGER NOT NULL,
  FOREIGN KEY (topic_id)    REFERENCES topics(id),
  FOREIGN KEY (proposed_by) REFERENCES users(id)
);
INSERT INTO incorporation_proposals_new SELECT * FROM incorporation_proposals;
DROP TABLE incorporation_proposals;
ALTER TABLE incorporation_proposals_new RENAME TO incorporation_proposals;
CREATE UNIQUE INDEX incorporation_proposals_topic_rev
  ON incorporation_proposals(topic_id, revision_number);
CREATE UNIQUE INDEX incorporation_proposals_id_topic
  ON incorporation_proposals(id, topic_id);

-- Composite FK in incorporation_attempts targets the renamed table,
-- but the rename preserves it automatically.

PRAGMA foreign_key_check;            -- raises if any orphan slipped through

COMMIT;

PRAGMA foreign_keys = ON;

The same migration file then creates agent_jobs (see below). Tests cover: (a) the rebuilt table accepts proposed_by IS NULL; (b) existing non-null rows survive the rebuild with values intact; (c) foreign_key_check returns no rows after the rebuild on a populated DB; (d) the runner's no-tx directive correctly leaves schema_migrations recorded even when the migration manages its own transaction.

collab.InsertProposal changes its ProposedBy field from string to *string, with the existing required-fields check dropping that key. Existing rows produced before the migration are unaffected because they all already have non-null proposed_by values.

The agent_jobs table

The single source of truth for "what is the agent doing right now, and what did it do last." Added by the same migration (still inside the no-tx file, after the rebuild commits and FKs are re-enabled):

sqlCREATE TABLE agent_jobs (
  id            TEXT PRIMARY KEY,
  kind          TEXT NOT NULL,   -- 'incorporate' | 'perspective'
  source_path   TEXT NOT NULL,
  topic_id      TEXT,            -- non-null iff kind = 'incorporate'
  persona_name  TEXT,            -- non-null iff kind = 'perspective'
  status        TEXT NOT NULL,   -- queued|running|succeeded|failed|timed_out
  started_at    INTEGER,         -- unix seconds; null until run begins
  completed_at  INTEGER,         -- unix seconds; null until terminal
  exit_code     INTEGER,         -- null until terminal
  error_tail    TEXT,            -- last 4 KiB of stderr; null on success
  created_at    INTEGER NOT NULL,
  CHECK (status IN ('queued','running','succeeded','failed','timed_out')),
  CHECK (
    (kind = 'incorporate' AND topic_id IS NOT NULL AND persona_name IS NULL) OR
    (kind = 'perspective' AND persona_name IS NOT NULL AND topic_id IS NULL)
  ),
  CHECK ((status IN ('queued','running')) =
         (completed_at IS NULL)),
  FOREIGN KEY (topic_id) REFERENCES topics(id)
);
CREATE INDEX agent_jobs_status      ON agent_jobs(status);
CREATE INDEX agent_jobs_source_path ON agent_jobs(source_path, created_at DESC);

The kind/discriminator CHECK requires the opposite field to be NULL — a perspective job with topic_id set, or an incorporate job with persona_name set, is rejected at the schema level. The source_path column is validated through ValidateSourcePath at insert time (same discipline as topics.source_path). Lifecycle transitions are routed through the existing single-writer funnel in collab.Store via new mutators (InsertJob, StartJob, CompleteJob) — the same pattern as topics, messages, and proposals.

Startup sweep

On server startup, before collab.Recover runs:

sqlUPDATE agent_jobs
   SET status       = 'failed',
       completed_at = unixepoch(),
       error_tail   = 'server restarted while job in flight'
 WHERE status IN ('queued','running');

This restores the invariant that no running row outlasts a server process. A more sophisticated recovery (re-queueing) is rejected for v1: the agent's work may have partially landed (e.g. a proposal row exists) and the safest thing is to surface "this job didn't finish — retry if you still want it." The user retries through the UI.

HTTP surface

New endpoints under /api/agent/:

These endpoints are triggers and observers; they are not how the agent itself talks to the server. The agent uses wb-agent for its own writes and has no HTTP access.

Git identity

The Agent does not commit. The harness commits, post-approval, via the existing collab.Incorporate → CommitSourceRewrite path. #3's git work is purely config plumbing: new agent.author_name and agent.author_email fields, threaded into the IncorporateInput.AuthorName/AuthorEmail arguments already in place.

go// in the handler that approves a proposal:
sha, err := collab.Incorporate(store, collab.IncorporateInput{
    RepoRoot:     cfg.Root,
    ProposalID:   proposalID,
    ApproverID:   principal.ID,
    ApproverName: principal.DisplayName,
    Subject:      subject,
    Body:         body,
    AuthorName:   cfg.Agent.AuthorName,   // new
    AuthorEmail:  cfg.Agent.AuthorEmail,  // new
})

This decouples "who wrote the new Source" (the Agent, via the git author trailer) from "who approved it" (the human, via the commit trailer). git blame attributes the rewrite to the Agent; git log reveals the human approver and Topic ID.

Failure modes

No automatic retries. A failed job is surfaced to the user, who decides whether to re-trigger.

Observability

• Structured slog entries at job-start and job-end: job_id, kind, source_path, duration_ms, exit_code, status. Errors logged at WARN.
• Per-job stderr log file at ${agent.log_dir}/${job_id}.log when agent.log_dir is configured. Captures the full stderr stream (not just the 4 KiB tail). Absent config disables file logging.
• Stdout is buffered for debug but not persisted in v1 — Claude Code's default human-readable output is not a stable format and would bloat disk.

Configuration

New agent: block in wiki-browser.yaml:

yamlagent:
  author_name:         "Orcha Agent"
  author_email:        "agent@orcha.local"
  claude_bin:          ""           # optional; default "claude" (resolved against $PATH)
  wb_agent_bin:        ""           # optional; default sibling of the wiki-browser binary
  max_concurrent_jobs: 1
  incorporate_timeout: "5m"
  perspective_timeout: "3m"
  log_dir:             "./agent-logs"   # optional; empty disables file logging

The block is required once #3 lands. If agent: is missing, startup fails with a clear error — there is no implicit default for git authorship, and silent fallback would corrupt the audit trail.

Validation at config load:

• author_name and author_email are required and non-empty.
• claude_bin defaults to "claude"; the harness verifies it resolves via exec.LookPath at startup and fails fast if not.
• wb_agent_bin defaults to filepath.Join(filepath.Dir(os.Executable()), "wb-agent"); the harness os.Stats it at startup and fails fast if missing. Explicit values are checked the same way. The absolute resolved path is what flows into the prompt body.
• max_concurrent_jobs defaults to 1. Must be ≥ 1.
• incorporate_timeout and perspective_timeout parse via time.ParseDuration; defaults 5m / 3m.
• log_dir, if non-empty, is created with 0o755 if absent.

Module layout

New code goes in two packages and one binary directory:

treeinternal/agent/
├── service.go          # queue + lifecycle + agent_jobs writes
├── service_test.go
├── runner.go           # Runner interface + ClaudeCLIRunner
├── runner_test.go
└── fake_runner.go      # test impl, also usable from external tests

internal/collab/
└── agent_jobs.go       # InsertJob / StartJob / CompleteJob mutators

cmd/wb-agent/
└── main.go             # subcommand dispatcher + handlers

internal/agent depends on internal/collab but not on the HTTP layer. internal/server wires agent.Service into its dependency bundle and exposes the HTTP handlers. The existing internal/collab/incorporate.go is unchanged: it remains the post-approval apply step.

Test boundary

The runtime is testable end-to-end without ever invoking claude. The Runner interface:

gotype Job struct {
    ID         string
    Kind       string           // "incorporate" | "perspective"
    SourcePath string
    TopicID    string           // "" when Kind == "perspective"
    Persona    string           // "" when Kind == "incorporate"
    BaseSHA    string           // "" when Kind == "perspective"
    PersonaSHA string           // "" when Kind == "incorporate"
    SourceSHA  string           // "" when Kind == "incorporate"
}

type RunResult struct {
    ExitCode  int
    ErrorTail string         // last 4 KiB of stderr
    Err       error          // non-nil on spawn errors / timeouts
}

type Runner interface {
    Run(ctx context.Context, j Job) RunResult
}

ClaudeCLIRunner spawns the real subprocess. FakeRunner takes a user-supplied func(Job) RunResult and runs it inline, letting tests assert on queue state, simulate the agent's wb-agent writes against the test DB, and exercise every failure-mode branch deterministically.

Open questions

The remaining unknowns belong to other sub-projects, not to #3:

• Prompt content. The wb-incorporate and wb-perspective SKILL bodies stay as scaffolds with <REWRITE CONTRACT OWNED BY #4/#5> markers until #4 and #5 land.
• Push notifications to the UI. Polling agent_jobs is acceptable for v1. A future move to SSE or WebSocket is owned by #8 and does not require schema changes.
• Channels migration. v2 work, gated on operational interest (e.g. wanting to trigger jobs from Telegram). Replaces ClaudeCLIRunner with a channels-backed runner; the rest of the stack (skills, wb-agent, agent_jobs, HTTP surface) is unchanged.
• Resource budgets. Token usage is not tracked in v1 — Claude Code does not surface it in default -p output. If usage becomes interesting, --output-format stream-json exposes per-event token counts and can be opted into without changing the contract.

References

• Domain model — Collaborative annotations (defines the Agent's role and the sub-project boundary)
• Decisions & parking lot (cross-cutting decisions log; #3's decisions append here)
• Sub-project #1 — Document model & persistence
• Sub-project #2 — Topic core: data model + anchoring
• Sub-project #7 — Identity & permissions (provides the principal helper used on /api/agent/ endpoints)
• Claude Code — Channels documentation (reference for the v2 runtime path)